From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DBC9542DC8; Tue, 4 Jul 2023 07:02:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5F91D40F18; Tue, 4 Jul 2023 07:02:14 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 335DC40E03 for ; Tue, 4 Jul 2023 07:02:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688446932; x=1719982932; h=message-id:date:subject:to:references:from:in-reply-to: content-transfer-encoding:mime-version; bh=q/8PEpoMLs9OikKWYT/k6sniat0aucZsYMo1FdkqQs4=; b=jVFMn6g7v8KPF/bx4ZJ5vXfbYRJPkIRaFyKgLvRHp8pI4J2Bx6RTo9MZ D/6DG591n0l2rAoPKMaxM3nHiIZzAg9kYstMXbU/OwR8YCIAbj5aL+liP 3mWghTDEJJZ3OOGqbfLi3PiMTksjy2YKJKv8lnHBk92feIvT5iQkf5xPd 27JZ7yzLxdWmqggIv6F5VHFkKJwJyWP+zHTnscBTw13a5reePQn4RJiUM Jd+BPv5DMWBE9IXj7Mtee40r6wjvRY9dVYKw9/ToF7PrAFmfF7ug2hsQC Bpa5kFebqjpcr/uXpFGtyG/p2Wrt6IBkIRyzapTC88QmJadM+7+kIEIgF w==; X-IronPort-AV: E=McAfee;i="6600,9927,10760"; a="449406314" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="449406314" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 22:02:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10760"; a="721970694" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="721970694" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmsmga007.fm.intel.com with ESMTP; 03 Jul 2023 22:02:10 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 3 Jul 2023 22:02:09 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Mon, 3 Jul 2023 22:02:09 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.100) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Mon, 3 Jul 2023 22:02:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L9F1GpYifooo1eBvBVX+kx9Nh/U62SggyncwdkmPHLxsAzZv+HlfBo49c3SV4IIG21wCJ2T9TxdY/hb7L+QS9/fpc53fg/3bU3Rx4RJA7GSU0DxLiUehg5rJIbyzY3yQt9Hl6aQx4ZjDvdjd6KAHFRvOeBJe3Zlh6AZzzqvEUptmopa5AYW6PXcf5C9Vp6V/iHP+UsEib7fpmNNfkd+UH05pKksPlmLvfg9s0E+jl+6uQDaf1ejGwp0Ptd9udQg04+yVGGGAYZeNQxcOL1DMGTa7cuPa+OmH7rBudzDd4vjkT0K/d89kkIPWX3UpuPUAEbvTQGNOkdmZpIZUCen1Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u5/iBRUZgfLIbFDqRKAC4GClzaWQ7KorfjJIaMUQqss=; b=U500nBcZ9aNUx1UcLgF9TItYOUpX6jDu88e1wdshFnIFPiJGkqVilvPeM/I5Pzylt6qhCu5KMnAnS27JUobYn2wOhGKoWKSmp2McnqjhdLmsqZk0yF3hqGfzo2rWRok4WAVTxA7t2k/jAi1iwYfDf0z/7yjCiqHAFslhrIdr1pEkJnJ1YrODipbKV37pZn1E16yfAbSOsdy4SS7w6rgsCIgrcTHm30ItxfpAPTw1fYksCLgoQdD2RvVlhsXShK1Syd4ZkUiGe9AVo/bB/pCwx8sb6M7Bpl0P3rH3HJIMaHRNy//05a2Py1lTmggzK8lmsmgsuU0cPxDCMOO2+cbG/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from IA0PR11MB7838.namprd11.prod.outlook.com (2603:10b6:208:402::12) by SA0PR11MB4736.namprd11.prod.outlook.com (2603:10b6:806:9f::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 05:02:02 +0000 Received: from IA0PR11MB7838.namprd11.prod.outlook.com ([fe80::6d31:14e5:d786:8329]) by IA0PR11MB7838.namprd11.prod.outlook.com ([fe80::6d31:14e5:d786:8329%5]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 05:02:02 +0000 Message-ID: <1cdebe12-3545-fa4d-dbd4-9e867beca58f@intel.com> Date: Tue, 4 Jul 2023 13:01:52 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.12.0 Subject: Re: [PATCH V2] doc: add iavf live migration guide To: Lingyu Liu , , , References: <20230703051449.191259-1-lingyu.liu@intel.com> <20230704022718.227538-1-lingyu.liu@intel.com> Content-Language: en-US From: "Cao, Yahui" In-Reply-To: <20230704022718.227538-1-lingyu.liu@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: SG3P274CA0001.SGPP274.PROD.OUTLOOK.COM (2603:1096:4:be::13) To IA0PR11MB7838.namprd11.prod.outlook.com (2603:10b6:208:402::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR11MB7838:EE_|SA0PR11MB4736:EE_ X-MS-Office365-Filtering-Correlation-Id: 295557af-40b6-4914-6795-08db7c4bcd87 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Qtp7T2C5eL5/yY/7x9M5xAKfXHw8HVIq5ORxqNCBV7uXaAdX10oJNABZd9OTFlED82CG4CTMVoTyGW5n0waKzQmU+DMqqFpvFuy50Dpjnsi7EQbh7zlmLbUE6lCFGX7YWQypFk+RMTCtjfexeO2QVFrYA2UJs4lD2boSc+UEn+PdWRaxkRtjENAuPoeBghUZLHKTQdU4/hvHquQ3GKRKLdIQUUShldcXhuAzm8BE8brkE0d9rxDdfsJqlDrKqdIPqnx38RY4DSkIEdWHJOlStbyDvzucXQwvUBxLx5LzKJGImFTU3fUNGym4RC85AGRdRzu8cehwAkcm+buT2Nfrs53YhbLXHrWuMRANJRlygAtbovSUoN+NYL9jqt7k8ntXteIBoGMJIGaVi37hDdRWmNnd8OVpqfdmRRn42UoQ6zd+ABdaPuS9/k4zyI4k5VLrIr7eNeHVMjgX0xT/hitJ/UHoZqhC7Mtoawy22uDJswR/FFaqjKSbviJSUS+BQOZNQyALqChIkXGq/RqgiwMxsMWX1pjKb7n4WyV5cPo1fXSkT2yAs0Qc1pgwMyXB/Sp3qSZ1ymHKB5FY2iF883i5722694WvNlkP1tmw5ie+KZd15AksdugIf5Jniqpgyroy+hOEQ/X1tc3BvsLeeZrNeA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA0PR11MB7838.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(366004)(396003)(136003)(39860400002)(346002)(376002)(451199021)(2906002)(41300700001)(8936002)(8676002)(36756003)(5660300002)(31696002)(86362001)(186003)(2616005)(31686004)(26005)(478600001)(6506007)(6512007)(966005)(6666004)(82960400001)(53546011)(6486002)(316002)(66476007)(66556008)(66946007)(38100700002)(6636002)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?amIvd2F4S3BnYVdMQmpld3JkYjRzL29PWHhHMHNnZXNYL0wzZm80dGQrZkVz?= =?utf-8?B?NEFYZUdIc2xNbGdSc2h4YnV6YzA5MlA2YkxrcS9VWm1abFJ1ZE81Q2Y5MENs?= =?utf-8?B?OXB2WFhQN2hJajJNdWYrWjAzRmZMS2E4MUhxWjRQTTJvdEliWEpGSHRmUENy?= =?utf-8?B?cDdBR2kxL1MwNG1hWkkxMTYyUVF2WitiVVBhRU53ZlZSSWVEWDJDWWhMR1NL?= =?utf-8?B?ZkZhc2hpRzFWTzFxYlhtSzJ3MDhPSDRhMTJnNzRYdEtoOTVuaitWMytBanIz?= =?utf-8?B?OFBBTDhHY2I1cDRoYnFoWSt1VzI5K1pDbUdHMzZEUUpHWUl0bGQrOFlKc2VI?= =?utf-8?B?dmltV3ZqWGFzNmttdTBBdUhFNkZNSVREL3JER2hPSHQ3OU9QS1hEbXA5cWFQ?= =?utf-8?B?Y0VqWm1Ed1h2cWFtdE1WODM5TXZDR0hSVm11WTFZQXcydlBRVTJEVVk2TGg3?= =?utf-8?B?VnVZTkxDZXdqOW82QXU0MXdsMWRha0NQdXVaMDR4TWN5UEdqdUxIRXBmejVW?= =?utf-8?B?b09vZFVFWXRpckJqenhOTmRtdDVqNWt6RjNyMWxISjZpY253TDVZY0VRYmRQ?= =?utf-8?B?M2JSck5Pd3BGWGcxQld6S2lLcnczNnZnN1Zsb1N0VEtzaVpLdm1SdTBWZVkx?= =?utf-8?B?NCtkWEhmOHJhSXhXQlEyejNMWFFSOXhDcW1Qc0RmTUZ5cVlteVc3VFphY2ZE?= =?utf-8?B?YUlCNWhKVmFUOEZicE5Tc1VGQTF0aEw5MG1NZ05rQmUyVmdwS0J0ZDIwc3JZ?= =?utf-8?B?VzZPWUR0Rm5ZSGlUc012cktZR3VObjlsOGgyME1qU215T2l3NU1hMVpSOVdz?= =?utf-8?B?TUFKVngxcVJjWHFSQmErTnRqeWNGTG1JUSs2NmdvVmNZUmx4aWRZUmRTVHRp?= =?utf-8?B?VmUrYWdGQVdxUVNTSWc3NEhJM1FlUWZEektMUUxqQkFTM0pCcHk3dTBaeHdK?= =?utf-8?B?blRNREFWeE95SjlnSXE4VkNLbTNscmFQVE9qcGxxUFc0U2VwOTNUdXNTVEN6?= =?utf-8?B?RTR0RVNjSmluYzJuSWJLc1pJUnlpdUVORkxvdHVRMjhFSGNYWGFnaWsxbTJR?= =?utf-8?B?c0dDRVNnUkxvYnRlZm0vK1FhclR1czNpQ1ppM1ZoL1JHb0EwUEJNKytGbHph?= =?utf-8?B?UUs1bVg1Z0ZYRkk3dG82dCtCTElTV1k2Q1FZVkJhR0dvT3dXRU9Cbnkydlo3?= =?utf-8?B?RjNZUDQ4cTlTZEo3S1dCc013cFkwRVIyaVoxSlh1WlU2S2hJMkdwaU5KcGJV?= =?utf-8?B?dXVBbHVzcWlML2Q1UndPL1NXREgwKzdLTm5iS3FwZnFmVjQzKzRVRVNDc3l0?= =?utf-8?B?VEduRUFodzg3U3dyajZKREUzd1duVmcvdjVhTHhYckplMkdPMWpYV2FIajVZ?= =?utf-8?B?R2M1NDlKY0laelpTSUFac0ViNGt3UWs4MmtTcXlrb1VFcE9KMVhDeXlReHpI?= =?utf-8?B?cldSYWdYQ292SmpVbUxraTlKdittSHpkdDF0enVuVnFpcWo0U05maHVlM3NX?= =?utf-8?B?UTJ2UmlXTHU2ZEJScmJCQUtXNys5OXFSSEJYVHFubE5CdmdFV3pPN2w0aHFY?= =?utf-8?B?amtwTmdjQzg4VithaWFHQW8wUnM5aXJpckJ4YUd0U2Q0YmN2eXA4aXEvb0VI?= =?utf-8?B?ZWVJWHFsYTE1M3FIQVBHdFY4QVd3VUFEWTFWNGYvNWYzZ01tQ1RSaVpCTUla?= =?utf-8?B?YURFcXZiVHRadytaeHBrV1hrcitxU3FXUHBWd1o1cUtGeWZvUnJmVHFMSUtx?= =?utf-8?B?N3hWOGU3dHpkY1kyanBpU1dsV0RDRTVPWkVkU1hDa1dwSm5PRjNTWUJweDF0?= =?utf-8?B?K1E1WlVQZFRPNitwaWlVaXRXN1FRdjBVSzJjVmh6ZzdtWUw1ak9ZbzN5bExx?= =?utf-8?B?REtiazhRT01xRVZkQmVlUjAzNDdQeDVrUWNJd3lEU0Uvc0dhNWNWbjIvRHBQ?= =?utf-8?B?djJLampPOUlsNGNENldGWWRwb1puL2xkUzE3bXV0bm1xYnM1cnhzZUNDVThN?= =?utf-8?B?Y2VET3hsOWxnL2d2ajB4R0JNVVZFdVNvU2Zzem9xc2RwSDRNaWRVazNWcklx?= =?utf-8?B?NE52ekRNZk5KZWc1YXJUZHVGdEQrR2Q0YUROYTF4TkhnSDJJQWQwSlBScnZ3?= =?utf-8?Q?ekTvGHT7BgMsPstnPyo8DGl8i?= X-MS-Exchange-CrossTenant-Network-Message-Id: 295557af-40b6-4914-6795-08db7c4bcd87 X-MS-Exchange-CrossTenant-AuthSource: IA0PR11MB7838.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 05:02:01.8701 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yh1TqeR2shBWYW+Ikzl6mPGm7RgFsX0QOX7EEkrNcATF/6d/Oh9RWCMFnvLnIQtQ4o1SgyBWVEx2wJMUrnAIAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4736 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Lingyu, On 7/4/2023 10:27 AM, Lingyu Liu wrote: > Describe iavf live migration steps. > > Signed-off-by: Lingyu Liu > --- > v2: Fixed CI. > Added brief introduction about live migration. > Clarified this is iavf feature. > --- > doc/guides/nics/intel_vf.rst | 101 +++++++++++++++++++++++++ > doc/guides/rel_notes/release_23_07.rst | 3 + > 2 files changed, 104 insertions(+) > > diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst > index d365dbc185..dd75aadf51 100644 > --- a/doc/guides/nics/intel_vf.rst > +++ b/doc/guides/nics/intel_vf.rst > @@ -622,6 +622,107 @@ which belongs to the destination VF on the VM. > Inter-VM Communication > > > +Live Migrating a VM running DPDK > +-------------------------------- > + > +Live migration refers to the process of moving a running virtual machine (VM) or application > +between different physical machines without disconnecting the client or application > +(see https://en.wikipedia.org/wiki/Live_migration for more information). > + Please also add some introduction about the vfio migration and the link for qemu vfio migration doc: https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html > +This part describes how to migrate a Virtual Machine which has a iavf device. > + > +The following describes a target environment: > + > +* Host Operating System: Ubuntu 20.04.5 > + > +* Guest Operating System: Ubuntu 20.04.5 > + > +* Linux Kernel Version: 5.15.0-72-generic > + > +* Target Applications: dpdk-testpmd > + > +* Ice Kernel Driver Version: 1.11.17.1 ``_ > + > +* Qemu Version: 7.2 > + > +The setup procedure is as follows: > + > +#. Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**. > + Can you add some description about kernel boot parameters? > +#. In the Host OS > + > + Install the ice driver and migration driver: > + > + .. code-block:: console > + > + insmod ice.ko > + insmod ice-vfio-pci.ko > + > + Create 2 VFs and bind them to vfio pci driver: > + > + .. code-block:: console > + > + echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs > + echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id > + dpdk-devbind.py -b ice-vfio-pci 0000:ca:11.0 > + dpdk-devbind.py -b ice-vfio-pci 0000:ca:11.1 ice-vfio-pci binding is not supported on dpdk-devbind.py yet. Please use sysfs to bind device and driver > + > + .. note:: > + > + The command above creates two vfs for device 0000:ca:00.1: > + > + .. code-block:: console > + > + 0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf > + 0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf > + > +#. Now, start the migration source Virtual Machine by running the following command: > + > + .. code-block:: console > + > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio > + > + .. note:: > + The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device > + to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed, > + x-enable-migration=true indicates that this VF supports migration. Dirty page tracking > + is not supported, so set x-pre-copy-dirty-page-tracking=off. > + > +#. In VM, install iavf driver and vfio-pci driver > + > + .. code-block:: console > + > + insmod iavf.ko > + modprobe vfio enable_unsafe_noiommu_mode=1 > + moodprobe vfio-pci > + > +#. Bind net device to vfio-pci driver and launch dpdk-testpmd > + > + .. code-block:: console > + We need to ensure dpdk-testpmd should be running independently of ssh console and suggest to put it in a background process like tmux/screen so that migration will not causing ssh console exit and dpdk-testpmd killed. Please add related description. > + dpdk-testpmd -l 0-1 -- -i > + testpmd> set txpkts 64 > + testpmd> start tx_first > + > +#. Start the migration destination Virtual Machine > + > + .. code-block:: console > + > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444 > + > + #. Start migration by issuing the command in qemu console > + > + .. code-block:: console > + > + migrate -d tcp:127.0.0.1:4444 > + > +#. Log in the destination VM, and dpdk-testpmd is not interrupt > + > +.. note:: > + > + Host kernel version above 5.17 is currently not supported due to kAPI change. > + > + > Windows Support > --------------- > > diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst > index 4459144140..a187cb6146 100644 > --- a/doc/guides/rel_notes/release_23_07.rst > +++ b/doc/guides/rel_notes/release_23_07.rst > @@ -200,6 +200,9 @@ New Features > > Enhanced the GRO library to support TCP packets over IPv6 network. > > +* **Updated Intel iavf driver.** > + > + Added the document for iavf driver live migration. Suggest to highlight iavf driver live migration based on KVM vfio migration. > > Removed Items > -------------