From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 85C4C42DF4; Fri, 7 Jul 2023 08:22:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4CA6A41101; Fri, 7 Jul 2023 08:22:21 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 88CF341101 for ; Fri, 7 Jul 2023 08:22:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688710939; x=1720246939; h=message-id:date:subject:to:references:from:in-reply-to: content-transfer-encoding:mime-version; bh=+dEc9jpZ9J5Nej+UTqEGsZsJLAYbNzroeuFK62IKkWk=; b=ieiOxnr/mRCd772dNTorHU2Xx/u6GmsgHAS3CNGhuA/5mUlaebE9iaEy ZbokEaJYwZ6PEn2E69v0HPa+1iyzLKQqy4DMtqdN7vQSw8+XtFTJa3xz5 dCINmkZlHvXtDUzELg5z6jg5Q0DCRUUhFXF2Uz4lcCVE5GUQl8NgQcYku cbjCYmtfnaJX8CRcg64wKTjuzbxOYyHkWZqphl87AOTU7ZtVuF67g5gJq BgFwgNZIJUojvGFIJvWndm0WJZ0zeo19adt8Bu0RcRM7a1N7/SFegPsCt ykfn8uuoR0E6+eHPlG8d51WKz2yWf/X/g5TxgxjXJXqK+EHAdzEno8/J3 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10763"; a="362682402" X-IronPort-AV: E=Sophos;i="6.01,187,1684825200"; d="scan'208";a="362682402" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jul 2023 23:22:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10763"; a="755036939" X-IronPort-AV: E=Sophos;i="6.01,187,1684825200"; d="scan'208";a="755036939" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga001.jf.intel.com with ESMTP; 06 Jul 2023 23:22:09 -0700 Received: from fmsmsx601.amr.corp.intel.com (10.18.126.81) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 6 Jul 2023 23:22:09 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Thu, 6 Jul 2023 23:22:09 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.173) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Thu, 6 Jul 2023 23:22:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bOgbwFTCL9+pRc0yDtBYfiCtMc4fLCiP6Gzo1HVeQirYZ42K0ibNxUNiePntvoyJBHOtxqoruol6eccUPRaPCzNzOEDyo7M5tA8rZH0DHrHSR69YO55SEAifNXZs/jdHDUevHuslcrejmFzwJZA17ca8C3OSofL4po/fJUpjVPRsrk3/0sizMBpaaR+Hh8h+gRYXe8PtfkswYqRe/nB6ERk1z0mKSsw8lcCVCxfYCcaBpTzWUGYYVtDDTjdMf04juAyk8FTYoSV10fzi+qTzRPjdvGd4uPm3mAPdtl5pI/voBwl64tFPLAIst/dzNa+9IMy8pJCUAqoHHU6hWBA6aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iINnBuHHoXzamv6t8A5c+vyo6JGS41kJcFSMlxJXtuc=; b=RzbXeqnK/V75C6XcE1SAmfRvAqAPw93AZeSnSVFmli1Gw1DjvyPqGp4u5XL9m9Lj+em+RtjTL/OosHnXPjqU58AUcUNT5s5/M0lMNgVmHUm/HdRHtlgBj8ujFtJ2GfvPwE4yjlFmUOcKU1TnGpeOx+AFnCRsOdetEa/mgUnhl/i+AkknCIaHhvyFkDnjQCZI/PphG6QI4EuX+HD8ESFkjiCpg9iZ533og3uMar6dKzk/fh9EtkVgc2J9emVsW8BePHx0tHUFmvZAICCUsQiG3kPHx/mtWvRnc9vcQzVtTVpeA9DsqpBoq0owVNdS3LTLC4w28no7XFy0e8D/DMBiTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from IA0PR11MB7838.namprd11.prod.outlook.com (2603:10b6:208:402::12) by SN7PR11MB6852.namprd11.prod.outlook.com (2603:10b6:806:2a4::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.24; Fri, 7 Jul 2023 06:22:07 +0000 Received: from IA0PR11MB7838.namprd11.prod.outlook.com ([fe80::6d31:14e5:d786:8329]) by IA0PR11MB7838.namprd11.prod.outlook.com ([fe80::6d31:14e5:d786:8329%5]) with mapi id 15.20.6565.019; Fri, 7 Jul 2023 06:22:07 +0000 Message-ID: <88200a75-fe01-2d31-1d20-bacaa030468f@intel.com> Date: Fri, 7 Jul 2023 14:21:56 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.12.0 Subject: Re: [PATCH v4] doc: add iavf live migration guide To: Lingyu Liu , , , References: <20230706072448.30063-1-lingyu.liu@intel.com> <20230707060841.70826-1-lingyu.liu@intel.com> Content-Language: en-US From: "Cao, Yahui" In-Reply-To: <20230707060841.70826-1-lingyu.liu@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: SI1PR02CA0021.apcprd02.prod.outlook.com (2603:1096:4:1f4::14) To IA0PR11MB7838.namprd11.prod.outlook.com (2603:10b6:208:402::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA0PR11MB7838:EE_|SN7PR11MB6852:EE_ X-MS-Office365-Filtering-Correlation-Id: ddc93278-618a-4315-4d70-08db7eb27cf1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /2WX5jBi72xokdHL4NrDhBR7Weq8uppLsfTpnGRcSkm+mLONb/NTKS6+bCS8EbcvIfe8a45MPPdI5i2kdoYG14Mt0qMLuU+sztLuS7vhAU9vxS99ErMp+Ppfy7BHh7HyD2cVFngTM1Nht7ZtDI7ECOPOxDa+nN+j+XHOMZYHZyHjhDdwn7lig7AJ7NTPLilv4ECKJgVzTtUwo9Xbw8tcnhPCfmRRZxtgAyMaBDCqRMfOaiZ4Vr3d8glvFfPLCE8CdRTyJdSQeykSd79zoxFcpiirkrH3uy32LFLIqdT8JBXRxz+sFs8bE+vVwDZDNlFE3kphNhk5fOBfgyC/41Buhxvj54tCIjSVAtKUFqGVZTeFO1c+Drzs6vGGSQFGoF44ro+Qi21T2PMmYAFRAb4iDRwV0Jxt1zEZfP/qX7r3WUCbh/m+niYyyN9JiGu1pItcnubInbBi0U8s0LMMkOHdIkJcizmEzjezFaXvh6n9RF2041i+xqHyWcoC4k9K+V6tAc1tKbvs3uSrJe33cU7HLC5l/WUQyO+hphemJi6cuIoMhSPcxoZ/nNgUpgmxp2sseCYj7TOcFsqpcvNcc5cxHJ1XrjZzPrH+fO0DyhDHz6y92NZz+MzTJ/GaOcZu9YXN7CbmNV+El8CATp7VemnLkoEgT9fxHtcR9sjjrR3n1L0= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA0PR11MB7838.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(136003)(376002)(346002)(366004)(396003)(39860400002)(451199021)(8936002)(5660300002)(478600001)(8676002)(66556008)(66946007)(6636002)(316002)(66476007)(2906002)(41300700001)(6486002)(6666004)(6512007)(966005)(2616005)(26005)(186003)(53546011)(31686004)(82960400001)(83380400001)(6506007)(31696002)(86362001)(38100700002)(36756003)(43740500002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cXlDY0hFWk94TXloYW9yN3hNdlIxcjYrRFJSekltN2NGSXN6bktWN2t2L20r?= =?utf-8?B?VlBTM1JHSkdQaXRuQUlkOWs4UmNJSEVBWHZXV3pYWDRBYjVjZHNac01BaXJ0?= =?utf-8?B?V0YyS1hPa0kvK2NoM1d4Rk42M3kxMUQzR25scDlJYkJ0WXp3OFZFU0NhWmtq?= =?utf-8?B?UWVTOStEUnZWV2ppMjRHbHgzNURjcnh1ZGJlOXJ1K3BScXVOTk85Z1hyZzVM?= =?utf-8?B?bnhIUXV3d0JQa0d2cjZBWVNDVnZuQWRueUN2eUhRZkJhRmd4dWdxSFp1bDNw?= =?utf-8?B?OVBHdGVvanJhRzB2dHh0Z3ZxbkZZVW9jUkpZNXIxVzloa253Ync0alU0UmNm?= =?utf-8?B?QThpWnI1RU9KWXlMV0IzM0ZJd2dPT0piU3BGdk5tZ201dzhxcWVHV2hubHpU?= =?utf-8?B?Q3dRMjNDTWtXK3dhcGUrUDFmVVY5b0FiakJmb2tsQlZIaVZKWlNwVXU4YXBY?= =?utf-8?B?U1BQUytyMFdrY3FBa053cXRxYzM1MmF2VTE2OWdNVFJhMHZQNGc1eFZvZHlH?= =?utf-8?B?V1RCOXJTWC9Db2ErNHFoUHpLcVpDbHVYLzRwb3IzM0xuUFUybitTWnY5eFNZ?= =?utf-8?B?NWkxb2cwNVFEaDh5N1NVZzFMcE9jL0F1L3lxZjF3NmVjK0YySk1mWWI0K0Q3?= =?utf-8?B?WVUrSDhhL1hTaUp3TmtCZXR2YWNhcittbDhmREhGM0k3NDc2NDhJaWNmRElt?= =?utf-8?B?Sk5hM1hDbDZpZDI3UWRaZjJwSGFSZEFMTEJQZHpEOHNoNzR4TGZOeGNIS1E3?= =?utf-8?B?UUkzNVN3NkR0ZGcxKzdtWXNvRjVra0xQRmY4RDVoazJOZ09SOExyQlAxTHc2?= =?utf-8?B?Nk9RUmI3UytPL0ZpajROMGlqb0FNSGJVaDNvSzZRSzNLeUVnbFU5T04zUEdN?= =?utf-8?B?WkVyYnZZL0FJdU14Q0pxdDB4QldLeWNxbnlVMWNQR3BBVXYwYWZnSFBaMlg2?= =?utf-8?B?bDJUVm1qUXdNMGpOVFRqUFJEZVE4MDAxVVMxRzErNGJrQkZPVFN3TzlOVU81?= =?utf-8?B?UEpMWDBYVEQ3K2ppUm82S1pxK3JkTWdXaExjeGlDRThhSUY2VVU2OUZqM1I5?= =?utf-8?B?R095amRMV2QxSDRja2pvNUNhTjYrQlFKaFFXVDhjZVI0a0xRWEhBalBNMkpQ?= =?utf-8?B?UUtJRllGWHBOUnZiTmc2RmFUTnZWQlhienduR0M1aWV2VTVUUnMxUVJUQUZQ?= =?utf-8?B?c1BoYzRMbE9UN3IxLzEySnNDZzJMWkVQeHlvbWY5S0RxY2tBMit3Q05rZWNO?= =?utf-8?B?TE5McVRFRE1CQk5uTEpCMVBvcXFlMDFMN0t3b1lJU2xncTMxQkJ4WkVlU1dP?= =?utf-8?B?ZU9HczdiYVROeC9LclNzRE5wM2NGTzZYUVFYY3grSk14UVZYTUdQeUc4Q3p4?= =?utf-8?B?YXhpUWtKUk5oL2xuOGJxM0NCRnlxMVUySlNRcG5xbldQWWR0VnRXbzh3aXN4?= =?utf-8?B?OG4wL2VpL1FFbTc4Nk1lS1JDZEs5WmgzdGNoaVErMkRSUTRteStPQkFhL1p5?= =?utf-8?B?R3hKT01OTkpyekk5Z2czOFFra0o2MFRnaW1SbUNaQjVLR3FFeUhxV2VmZFJi?= =?utf-8?B?Ujc0ZnhhTzF1T3grOU1hbzJuN2d6OWlkNHNuMUFYNngyUmN5L05WQnJJL1Jk?= =?utf-8?B?MEwyR0pmenU2cHpNejlKSjZsaURVRkx3TytsWUxQNVJyOThUQytibU8yaGU1?= =?utf-8?B?bWExdWFwdWlxMUJPMUQ2bUU1RUFEV1gwUEprWUx6RWRrNWFwOUpjV3JVaXBC?= =?utf-8?B?b3Z4dWUzTGN1WmRpaldBaUhza0cwQVJlODk2Ylh0cWwxcTZPdEVvYVM1MDV5?= =?utf-8?B?SVRRZENEdUwxUlRvUnNWMmJ0UmtHSWRGQitjVTZXWERwUVdydlVNbDRBVFVl?= =?utf-8?B?bVBDeGNrNDNGTzRlN0xodnQwblVYQnU2UTZiNUQvTHJMRWY3OEJKakhINGhR?= =?utf-8?B?TTIxU3E0YTdlbkIrbGF0RFAxakxCY2czWVgxcjFDQTRzSGVnb3RvdUQxdFVx?= =?utf-8?B?L0toSEsxaUdRMUt1NWgrb0pmaHRXeUJQVjFGRUhxZUFoUzQyWCtBMTZ6alBE?= =?utf-8?B?RGc0Q3FCQ1dQZnFiQ1NXeTFrNTQrN3kxK01uaGF2YnJIMm9SUjJXT0R5cm9z?= =?utf-8?Q?Sky9d4Nd1/txMnO43UTci4CkB?= X-MS-Exchange-CrossTenant-Network-Message-Id: ddc93278-618a-4315-4d70-08db7eb27cf1 X-MS-Exchange-CrossTenant-AuthSource: IA0PR11MB7838.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jul 2023 06:22:07.2126 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XEvBAc3B6T3aVzfjLSbEbHy6ZB+9dNjJel0nSXo8FC3NE2caNJK7MV+EpVlLNjVqI+HifoG2GfO4FuhRndIR6A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6852 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 7/7/2023 2:08 PM, Lingyu Liu wrote: > Add iavf live migration steps based on KVM VFIO migration. > > Signed-off-by: Lingyu Liu > --- > v2: Fixed CI. > Added brief introduction about live migration. > Clarified this is iavf feature. > > v3: Added intro and link about vfio live migration. > Added description about kernel boot parameters. > Changed to use sysfs to bind device to driver. > Noted for running dpdk-testpmd. > Highlighted KVM vfio migration. > > v4: Removed ice kernel driver version. > --- > doc/guides/nics/intel_vf.rst | 113 +++++++++++++++++++++++++ > doc/guides/rel_notes/release_23_07.rst | 3 + > 2 files changed, 116 insertions(+) > > diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst > index d365dbc185..85b3b4fc24 100644 > --- a/doc/guides/nics/intel_vf.rst > +++ b/doc/guides/nics/intel_vf.rst > @@ -622,6 +622,119 @@ which belongs to the destination VF on the VM. > Inter-VM Communication > > > +Live Migrating a VM running DPDK > +-------------------------------- > + > +Live migration refers to the process of moving a running virtual machine (VM) or application > +between different physical machines without disconnecting the client or application > +(see https://en.wikipedia.org/wiki/Live_migration for more information). > + > +VFIO device migration refers to migrating a VM which have VFIO device pass-through > +(see https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html for more information). > + > +This part describes stpes to migrate a VM which has a iavf device pass through. > + > +The following describes a target environment: > + > +* Host Operating System: Ubuntu 20.04.5 > + > +* Guest Operating System: Ubuntu 20.04.5 > + > +* Linux Kernel Version: 5.15.0-72-generic > + > +* Target Applications: dpdk-testpmd > + > +* Ice Kernel Driver: Download from ``_ > + > +* Qemu Version: 7.2 > + > +The setup procedure is as follows: > + > +#. Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**. > + > +#. While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB. > + > +#. In the Host OS > + > + Install the ice driver and migration driver: > + > + .. code-block:: console > + > + insmod ice.ko > + insmod ice-vfio-pci.ko > + > + Create 2 VFs and bind them to vfio pci driver: > + > + .. code-block:: console > + > + echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs > + echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id > + echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind > + echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind > + echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind > + echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind > + > + .. note:: > + > + The command above creates two vfs for device 0000:ca:00.1: > + > + .. code-block:: console > + > + 0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf > + 0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf > + > +#. Now, start the migration source Virtual Machine by running the following command: > + > + .. code-block:: console > + > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio > + > + .. note:: > + The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device > + to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed, > + x-enable-migration=true indicates that this VF supports migration. Dirty page tracking > + is not supported, so set x-pre-copy-dirty-page-tracking=off. > + > +#. In VM, install iavf driver and vfio-pci driver > + > + .. code-block:: console > + > + insmod iavf.ko > + modprobe vfio enable_unsafe_noiommu_mode=1 > + moodprobe vfio-pci > + > +#. Bind net device to vfio-pci driver and launch dpdk-testpmd > + > + .. code-block:: console > + > + dpdk-testpmd -l 0-1 -- -i > + testpmd> set txpkts 64 > + testpmd> start tx_first > + > + .. note:: > + Please ensure dpdk-testpmd to run independently of ssh console. > + Suggest to put it in a background process like tmux/screen so that > + migration will not causing ssh console exit and dpdk-testpmd killed. > + > +#. Start the migration destination Virtual Machine > + > + .. code-block:: console > + > + qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444 > + > + #. Start migration by issuing the command in qemu console > + > + .. code-block:: console > + > + migrate -d tcp:127.0.0.1:4444 > + > +#. Log in the destination VM, and dpdk-testpmd is not interrupt > + > +.. note:: > + > + Host kernel version above 5.17 is currently not supported due to kAPI change. > + > + > Windows Support > --------------- > > diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst > index 4459144140..794a86eba3 100644 > --- a/doc/guides/rel_notes/release_23_07.rst > +++ b/doc/guides/rel_notes/release_23_07.rst > @@ -200,6 +200,9 @@ New Features > > Enhanced the GRO library to support TCP packets over IPv6 network. > > +* **Updated Intel iavf driver.** > + > + Added the document for iavf driver live migration based on KVM vfio migration. > > Removed Items > ------------- Reviewed-by: Yahui Cao