From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8741E4281B; Thu, 23 Mar 2023 04:26:27 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EFB614161A; Thu, 23 Mar 2023 04:26:26 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 540ED40156; Thu, 23 Mar 2023 04:26:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679541985; x=1711077985; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=pdHTBM5sC4v6/sTc7mO43BOpg99mcNf/+QC8jFJghvg=; b=OqKdi04yq5dj/foUXKyXs52AUxjuj/sh6WHJNyW/LCUbcapIZhOf5au8 UwC7NqkJWiWAvlKvLO8UyOWKRJXdC+7BeCZE8v54naya5ZEMFBjRrXDb+ o8OzZszJ1SVokb03K4oHergHEn4KJam4/KRnBDJodwHFHc5FDDaISGvJ8 79k49X1v8ScwvzfVMOQKQtJL1o+oKMIlM4el+I8i4L9xlZcFaZykB+y/p 4lqXy7h+TchAZDUxf+ou0kQDlTeatGB1jIqIYXKY1imssq7Tb8YJWV82X O22CEN7TUCjS+eQLY5k76dbv64VJ7phLzot92AZ9Zg0+TwhmKKVmKXOU0 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10657"; a="367123810" X-IronPort-AV: E=Sophos;i="5.98,283,1673942400"; d="scan'208";a="367123810" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Mar 2023 20:26:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10657"; a="682157872" X-IronPort-AV: E=Sophos;i="5.98,283,1673942400"; d="scan'208";a="682157872" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga002.jf.intel.com with ESMTP; 22 Mar 2023 20:26:16 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 22 Mar 2023 20:26:16 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 22 Mar 2023 20:26:16 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21 via Frontend Transport; Wed, 22 Mar 2023 20:26:16 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.103) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.21; Wed, 22 Mar 2023 20:26:15 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bZ95dvot1g21hcdSaKV7RKjewbM9RDKjaWuzyiDJomugbBD0vsM2kKGqjgaTN6pnb0evm3TpEhx79zbgWrhw+ICy7VmYGJFIYqaAzotN/oq/3c8TzEyuSVz9lEBE/EtGj5alEBx4MsBSccP2EvTKdfYjb3C+JmSwNWzgDU73jgNHvig3Kke3M/mhREYmpQHTbyirrEZrrE5NdglYI/vLw3bP4uzmjQXyWocTauGe0glhjG0xGuXzHyyu9hKJ2eaRODOq5vnXmQ99EKulent2G1KPugH88pHHimNX5wD1idYx7FwUhYk1lzNJLwI7zZbhY9RqhKDneWVnUKW5y9JyvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uEiy6HVS4z8HPSC8AF0j2R4sfnBXAWMpl0fdCEe20ho=; b=Vvy42WCW6ynXkXXVnvGSgZDsl9Y6thaqGF6Nd1rUuB2GBK+/RdZcX7zsHBfV4RG3A42TcGnRKGKqZR/ETskTXA+F45l8W0kkfB5A+5IP7ZYXwtosBQeIBvBQrf222HEx5Yg9Gmcgj7AWQdZ83detMwTLEkI9q/YnW6Tpp/KCb+ux7wsuKJ9DHi7cvByOaYhuL5I2c3auh7q89VIzt5XUnVa1XRDN5RV8mchIDKFd3DPUG6ogQHN5KW/fmj+RRAJSZk3+C7lqKItE2blrhtOM2BG2SFpqU8lRstFIFwgirr4cx3oGnLeCU85ZaywEVxPRjVZJVAzERPupS7PCHA2x2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DM6PR11MB3530.namprd11.prod.outlook.com (2603:10b6:5:72::19) by BL1PR11MB5351.namprd11.prod.outlook.com (2603:10b6:208:318::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37; Thu, 23 Mar 2023 03:26:06 +0000 Received: from DM6PR11MB3530.namprd11.prod.outlook.com ([fe80::2a1b:9bf9:445f:567f]) by DM6PR11MB3530.namprd11.prod.outlook.com ([fe80::2a1b:9bf9:445f:567f%3]) with mapi id 15.20.6178.037; Thu, 23 Mar 2023 03:26:06 +0000 From: "Huang, Wei" To: Thomas Monjalon CC: "dev@dpdk.org" , "david.marchand@redhat.com" , "stable@dpdk.org" , "Xu, Rosen" , "Zhang, Tianfei" , "Zhang, Qi Z" Subject: RE: [PATCH v1] raw/ifpga: remove virtual device unplug operation Thread-Topic: [PATCH v1] raw/ifpga: remove virtual device unplug operation Thread-Index: AQHZWHHvun1srISqFUW9KK/CSBP1OK8D2uEAgACG2SCAAIedgIAAAFNwgAAltoCAAPfrAIAAsegAgADg8PA= Date: Thu, 23 Mar 2023 03:26:06 +0000 Message-ID: References: <20230316204445.360330-1-wei.huang@intel.com> <2637984.KRxA6XjA2N@thomas> <5367342.29KlJPOoH8@thomas> In-Reply-To: <5367342.29KlJPOoH8@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR11MB3530:EE_|BL1PR11MB5351:EE_ x-ms-office365-filtering-correlation-id: 994d8c20-00db-44d6-5296-08db2b4e56a8 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JW5oB0goHvPxQwppvGcKyC7m6y53IU6/uX1FFZWquk9vsBVMIy4y9VXxak7NCkVp37ayLW8z/dTw3yDZjEOzlJQRS2T+ytVSdwAwte0a8T/Os5axWc/0y5phqjw+vOBUhw6QygTlr8wigPeeKQg2o7dSZdT1r/tODV1eX/TaJZ/uEEoquryw+Tz4gBe0Mbsafft/TShPCM/q5qV8IHo50Nzo2J2wF5+aYKIBicxY6Qfp2/T886zBaBgxPtGV4dDWpjIifqv80s8J6hjqbxSP4+2EX84uY6iRJxryNc9puKxaXUZ1wqEZqv3y6quLEnSuEjIUbJp1vsBW0PwUDKeqjHYiYEQ9r4tpkXh7jy1NE6gySjDIE/sTc35P6xzuVyL7jYOxkRJ0b66dN7VY6PIH8lFWY8hYO8xZm5xmWQJbBaHX4C4QVbCMzcLnXYO22fl3CqTqBzCo/RGun3dapc55awscZhqwMVwWi/mnLI/9BkVWS854fu98xzDdiIzHPdfZ1WmoNe3af/eDP4HeqpoaTREkQp9hskB6vx0sO5VfbDrhDdAWNmsHSx4ZFpsT4d3hWWjQjXjcExiswXD3w2fL+0XeT/JHppJZvSzmBJLP97YnhaLoY+hFEGSZQS6iJFjKY1bYryfwFNEd/o0zksGvLObd9X6vyMTgCGRshBXbnFY1k4zT2VOw/jeg02Kuwb6yk3cp8M1J1H5a3OoPqo2dvw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB3530.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(366004)(346002)(39860400002)(136003)(396003)(376002)(451199018)(53546011)(186003)(38070700005)(9686003)(107886003)(82960400001)(6506007)(26005)(38100700002)(7696005)(316002)(478600001)(2906002)(86362001)(71200400001)(122000001)(83380400001)(33656002)(55016003)(4326008)(54906003)(66446008)(41300700001)(66476007)(5660300002)(64756008)(66556008)(76116006)(52536014)(8676002)(66946007)(8936002)(6916009); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?dIM/UzSVrxhYiS/eGo8D13zzgpJJcebfF3OTiXb6trXJVco7HeMsOQ0WktIx?= =?us-ascii?Q?AWkpcWagcCi8XCH/fFh9I6n4fK/R38f18mDcxZ4mxny0VaQZ0XLpwWZ+sH59?= =?us-ascii?Q?J+9/rStuz4Bqz9hdEts4yPyJa51awdNy2E3itLVSQBr/xSNJjdyPrfHcVQA5?= =?us-ascii?Q?I1mFIzW7BDQN9GaLp82eY1mK7G87G52JsuZidvb36xDjx0CxwY2HpIJB3S1j?= =?us-ascii?Q?ty7Gjys7+5sBbF34IMfOGGly30tpnSxxd21zDlkLNGRQEoTsDF6nrPHaBOkq?= =?us-ascii?Q?9uQgxJP3P0YPOHVKJXRKNlHYa5tqtfrIfueYYq28WDxrN4Z4XrQjqnm5Bdse?= =?us-ascii?Q?D9sNovb8LQgu2p4OHcYgCpeiGJgyuBajts3Cm4lPk22OH0tLXMt6uMSVIaRN?= =?us-ascii?Q?U8z9m9YGIhXTNsd/37QUPfpzGDrHxW2GPBrNHfM3BN6Amw/j2yH0XXqCjtEv?= =?us-ascii?Q?VVT5BdeGDS7PKukXzfmQ++2ImD2KUS6PPvSy/B0+coJfhQ0bbqpSnphv4vy3?= =?us-ascii?Q?gK6uMN4KKPDlCQY3YjSs9KhxQm4WXlU/eLWJOI/p376IRPb2jWu0vJiNeVek?= =?us-ascii?Q?4+ttgnpimti0uRQrizzUGMfMGWcJpc6l03InH6219ReHCzhp/L6qgMxVySbK?= =?us-ascii?Q?qoZUBQPXxyZ33SqbP7G5kKibe8x2XOLFvLIAIpdk6ofAWwU7jiAyP7iSQ6vE?= =?us-ascii?Q?TJiu5rZA83kcaLvIJHL/1RKYxZJ/1B8em93Sm8cxw94c3IcZLE3la8zpiPqF?= =?us-ascii?Q?Bqaom50RV0rj0R54JBCWHh81ILMX7+Wji5LdTsv1T2F02FVLiqJsjfGl/gaY?= =?us-ascii?Q?AK8je0r27SbX6abJyscSKJmqDdyj3th8S7tl+fFd0KTBF6RzVzTSQvZt9kGM?= =?us-ascii?Q?c2TJyKLl0j39ctMc7IlFUGtuh9dAIxwxjiXqWLFn7kREq1YpKaA/6E7QgnYK?= =?us-ascii?Q?PERxnMzaMHgg3CNXdlg1ZCRglTz3I97KvPIogSUiZo5YVhyZ18TCkWxUYqHv?= =?us-ascii?Q?2DMra8nqOWzxU5gjSB6Vpza+zF51ZHhwt4l/hlVSoGQk/7/ZVMMJ6QtKy8se?= =?us-ascii?Q?d1Q6daIU51Psi5HME7QHGWkyRYbkFkh4bAoiZZ2lpsdlygF99MkrzZUn5drM?= =?us-ascii?Q?AIHAQ6FmfTtznTchW8KXI7fk2V+ziGV5ODGhcLXdkl2gRJWQkXiIqDDiyVVb?= =?us-ascii?Q?QJzFTU66gUpYRspEjsBA3Zc7OXfq7deG9bmxYQnqnfG2z7iFfgePlfDGtpWc?= =?us-ascii?Q?xTzC/xMjGnibnJfP2DQIrxQUMTEb0kdE0hzkeJm0yboOusKq1ul+OJwKckAq?= =?us-ascii?Q?kqFC3LgxXgLzmwAY/B8uv1XTwYzeac5no1QMfg8MRi3IKVgGvxi6cPJfnT0A?= =?us-ascii?Q?TbsMLUpaJjZNMLPDLbt9+1Wy3mSZwqMKjovKR94YK1eC7beL6bH/MCHAXwzc?= =?us-ascii?Q?M3BE055quW4JytrTv7lKpdbrdKZEa/f74ZQOpMkHSdN2eK5/c8kO7MTTLhgM?= =?us-ascii?Q?zkgshmYlYWLZyOz7BvkR1Q8LrD/izuyR8IWLzCE8kt1HK/eYsJT+dYN3p3VE?= =?us-ascii?Q?qQJnEEgDGK9zT0rhVK69rtsIQTJORLmyAcwoImPt?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3530.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 994d8c20-00db-44d6-5296-08db2b4e56a8 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Mar 2023 03:26:06.4216 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: AOYr3rlgPM9itg4q4GKXLoedzmDyaPwxcPbsAP6WwOmtY1VrRutBIp445mSMMcYxZIfiGpYSdZ8viUsX2I6Z0g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB5351 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Thomas Monjalon > Sent: Wednesday, March 22, 2023 19:54 > To: Huang, Wei > Cc: dev@dpdk.org; david.marchand@redhat.com; stable@dpdk.org; Xu, > Rosen ; Zhang, Tianfei ; > Zhang, Qi Z > Subject: Re: [PATCH v1] raw/ifpga: remove virtual device unplug operation >=20 > 22/03/2023 02:26, Huang, Wei: > > From: Thomas Monjalon > > > 21/03/2023 09:41, Huang, Wei: > > > > From: Thomas Monjalon > > > > > 21/03/2023 01:11, Huang, Wei: > > > > > > From: Thomas Monjalon > > > > > > > 16/03/2023 21:44, Wei Huang: > > > > > > > > VDEV bus has implemented cleanup() function to perform > > > > > > > > cleanup for devices on the bus during eal_cleanup(), so > > > > > > > > there is no need for ifpga driver to record virtual devices= and > unplug them. > > > > > > > > > > > > > > Why no need? > > > > > > > If the application wants to explicitly remove a device, what > happens? > > > > > > > > > > > > > > > > > > > > EAL will output an error information "Cannot find plugged devic= e > (%s)". > > > > > > > > > > It does not look what we expect. > > > > > > > > > Let me clear it. > > > > With this patch, no error information will be outputted. > > > > Without this patch, error information will be outputted. > > > > Because bus cleanup action will unplug virtual device, then ifpga > > > > PMD unplug the virtual device which is already be cleanup, > > > > > > Why ipfga unplug the device after the bus cleanup? > > > I'm not following. > > > > > The virtual device is created upon ifpga, if VDEV bus doesn't perform > > cleanup, ifpga has the responsibility to unplug these virtual devices. >=20 > Really I don't understand the flow. > Are you talking about EAL cleanup case? Yes, it's about EAL cleanup. > What happens first? Do you need ifpga to be called first? The cleanup flow is rte_eal_cleanup() -> eal_bus_cleanup() eal_bus_cleanup() will call each bus's cleanup method to complete cleanup w= ork. =20 There are three types of device related to ifpga PMD: PCI device, VDEV devi= ce and AFU device. VDEV device is created on PCI device, it's a mediate device which purpose i= s to plug a AFU device on IFPGA bus. Before eal_bus_cleanup() is implemented, application will call rte_pmd_ifpg= a_cleanup() to remove PCI device, VDEV device will be removed when PCI devi= ce is removed, AFU device will be removed when VDEV device is removed, it w= orks fine. Now eal_bus_cleanup() takes the job, application has no need to call rte_pm= d_ifpga_cleanup(), ifpga PCI device has no need to remove ifpga VDEV device= and ifpga VDEV device has no need to remove ifpga AFU device. > I think you need the correct checks to allow any order of cleanup. When this patch is committed, no order dependent on cleanup. > > > > bus->find_device() returns NULL, > > > > EAL output "Cannot find plugged device (%s)\n" at line 302 in > > > > eal_common_dev.c > > > > > > Anyway, the good answer is not to completely remove the "remove" > > > operation. > > > > > If not to completely remove the "remove", the same virtual device will = be > unplug twice, is it reasonable? >=20 > You need to add a check to not unplug something already unplugged. > But you must allow the user calling "remove" directly. >=20 rte_pmd_ifpga_cleanup() is the only one interface for user to calling "remo= ve" directly, when this patch is committed, VDEV and AFU device will not be= unplugged twice. For PCI device, the implementation of rte_pmd_ifpga_cleanup() is like below for (i =3D 0; i < IFPGA_RAWDEV_NUM; i++) { dev =3D &ifpga_rawdevices[i]; if (dev->rawdev) { rte_rawdev_pmd_release(dev->rawdev); dev->rawdev =3D NULL; } } If ifpga PCI device is already removed, dev->rawdev is NULL, it will not be= unplugged again.