From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C71F6A04FF; Tue, 24 May 2022 17:22:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CE72342BA1; Tue, 24 May 2022 17:22:01 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 67F204281F for ; Tue, 24 May 2022 17:21:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653405719; x=1684941719; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=RwGJBV4O32BhwNC2+cgn1qlw0CJZ7RnsGoNt5JzWGPU=; b=WgdP9eyG0ThqU5/RRpPOtJNdzcOnjiYCL0Y+ZOWF41I/DpSqg1O+9fb5 Ef8wJxLIHVDUiHZhcMBbM5Kcn/l/PbofbuKkO48Itekrfb06+GomvBsni s/DRDBsBZZvekGCX1LhPTorubZBOORCXXBNpuHo6mkPohqjxIcMaecmCN fH9A/qNCbxBs7/GlfYe8DvcGJ23Wcgp/niskEhN5xzB13DJnpHx8FWtgn 6OpVDldQ1MVS/VLNaXAkWFTsU9TkpdR9yJn4wfsMzvRmLYZVrWlcJvouv lRaSgckKj262v3BZf+h58e6sJW5lmqjd7h9EkvRudLboFlzreFdGtQk9A A==; X-IronPort-AV: E=McAfee;i="6400,9594,10357"; a="255624015" X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="255624015" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 May 2022 08:19:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,248,1647327600"; d="scan'208";a="629927052" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga008.fm.intel.com with ESMTP; 24 May 2022 08:19:54 -0700 Received: from orsmsx604.amr.corp.intel.com (10.22.229.17) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Tue, 24 May 2022 08:19:53 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Tue, 24 May 2022 08:19:53 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.42) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Tue, 24 May 2022 08:19:53 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QC3MVVFoXaCCL8KRCmYuzTLxK2oVT+ENZGzcW8WNaHzuqFJ1RY7zAy+ZGjlF5oSdviyic/lbsYYJZ97an1pPh2l7na15EAFONVP4U3Cq3Y1Hd9v46nDf5Ez5GDlcy6NrFixO4uXUQU4gIaOtjAvtiZn63XYLYOnsnazKp8hWNWtHeGWPS7r2iNcJwyyXAr/fsA/YLrnKpqZO1PpqJXkF+Yr0tRc5hjr/IRFBTGbAxeuFiBubjR23deQ8QvqNeiFNRfaDh057bP2iY55q2ikM8AwAOjv8vPTB7rD9cUPy4GhHAfai+y8ETnxL1Jf6XHMK1PXY4FMI9PwETCeUtXzynQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wijZRAGRPUadlbzMViHIWITZApzzaQc2eNG9IbJKwv0=; b=lgyyIJ81DLQ5y2FeCAGZwW8O2gXe56wzXNi844a0GYtE+i/9TuYUhmyqcXd0Yf5zzveWQ4zAHbwSFoS4U1WRAzleC4u6DDG2/hk9A5qmvLuJR/eXhdFLQaBMJmngalOaoIJU38O7ya6Xt3XFwfYJRoHvR6f4IXyYI9OwLQsBAtj5cJEKqkpJUnKduQgzeBSdYpuCAkb8ruCEf7IbjG2PK08SY+C5pcKd1n4hNCWDsHTGLRu133nvudZ22v0RGh94KDu7jagHzyo63MOZXtf2UxF9MHur8uK0Tr0vPwDNuHnw2XSSZe0kktsny8SpymSvBsb5mkHNGAgF9qkJSMy4vQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from MW4PR11MB5872.namprd11.prod.outlook.com (2603:10b6:303:169::14) by DM6PR11MB4105.namprd11.prod.outlook.com (2603:10b6:5:5::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.15; Tue, 24 May 2022 15:19:52 +0000 Received: from MW4PR11MB5872.namprd11.prod.outlook.com ([fe80::d55d:28c1:bfab:3dd]) by MW4PR11MB5872.namprd11.prod.outlook.com ([fe80::d55d:28c1:bfab:3dd%5]) with mapi id 15.20.5273.023; Tue, 24 May 2022 15:19:52 +0000 Message-ID: <3bd7ebff-5abf-9627-70d0-9459a95ea51f@intel.com> Date: Tue, 24 May 2022 16:19:46 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1 Subject: Re: [PATCH v4] eal: add bus cleanup to eal cleanup To: Bruce Richardson CC: , =?UTF-8?Q?Morten_Br=c3=b8rup?= References: <20220419161438.1837860-1-kevin.laatz@intel.com> <20220524092501.419761-1-kevin.laatz@intel.com> From: Kevin Laatz In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0372.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18e::17) To MW4PR11MB5872.namprd11.prod.outlook.com (2603:10b6:303:169::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2acd0ed8-2861-4fa4-9b0d-08da3d98d98d X-MS-TrafficTypeDiagnostic: DM6PR11MB4105:EE_ X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kNRMIfxT+f0yqDrlNEjFhhgF8GtyO6odbySI0puVM1dS6tnwehuGVahLB/PkpUsaUAZC4AABYSPCcLKBsFctQpldcWVbjQ6MJcqETbVI/c0AGI2XjI0obOCwo7w9jUjYwzdcng2dl9HaDrLdzYSIo8rkYZK+6Mz8BTeDdWX2IeULDcAEHb7f0XQ/fTl4pPAzz2tCdvg0ibybHeez/r1RtT363kvzuvnT8k7gu7Dq2+UEQVEnDLtflNiZhF5w+8a+WKabsAfrkzxYrEr3R4xx3ZVjwPLpsaowChtuNDY0g8RqBMi41DMDJhkOQbk3vVNxOIglbN0YRBtrx1V2uEVFEFDWJh9lplDlk0WByv1Y9cL+zp/ev40qYSxg1uxUzn8dglHxeOC5ohyKbf/JC3yS/l2p2/NNkFq7kFyqg6DDIY9qU+R5o2VB6d9dcytT8bbWJ9wsEWAPNOPIlf0THS7ZnVmgGlP/gCoKCKv4eHGREPNQXWXjwpFzZQcVmOvvnaYG7VMS55uM367nHzR7n9F7IHERU0913w3xpy2k7zt6DMScym607w/0Zkd0ZEPLqS+/hscZ1s5rzruxx2VN24QfBAOod5WgD28UncN4mEJ25n96CxN3Iw3lyGyv4RBGS4/rPhdwITV1Z6d0wTKMPOStRB+PItpxsHdGAnozJEo/ACdD3wW7M77wLmYibctqIiMd7WgXzdoc76WfRl5NVnvuHUX0BmavczH3vPgh0FY0bn2psN5NAt5kMa8NP3I9SFGv+DPPLR4ysDHQiGMmyqI2ig== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MW4PR11MB5872.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(508600001)(37006003)(6636002)(53546011)(82960400001)(6486002)(83380400001)(316002)(86362001)(5660300002)(8936002)(31696002)(6512007)(6506007)(186003)(4326008)(44832011)(66476007)(6666004)(31686004)(66556008)(66946007)(36756003)(26005)(2906002)(2616005)(8676002)(66574015)(6862004)(87944003)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TDNNbWhrYUc2azZCa0VpSmVQUDdDOERCb0dBRlpOZDZnWU4xQ29XbjJnSUhF?= =?utf-8?B?Q3BXU0xLTzd4VmViZmlWUFphc09GRGJSRWlZTjhDYkhpLyt5OC8rNmlSOW1J?= =?utf-8?B?SktIcEs0YzlvRmszdG5MRkNtNWdDNnpTRG1tOVlETUI2eXNWby9nbWFjUWNL?= =?utf-8?B?YlZvUEcwMTcxQ3NhYTRlbTMzUno2RUk5VGs2ZkRWczBvSnNCMXk5b3N6dVpo?= =?utf-8?B?SXVMODEyM3hoUDFFeUpnVEZPL01keFpIM1lGVlNickNUaTVZNW9NY1FHWUls?= =?utf-8?B?R2JLQW0vVjNlTDN1dmRJUE5wZjlFb1lYbXhxb01pTFpTaW5FVFBzYUhWMnc5?= =?utf-8?B?OEMzZ1ptdGhlNENQQ2lFZVRJVHR0QlJYODNQTW14cnNCMEdSWWVobEEzVTdE?= =?utf-8?B?Siszb0cxYUg2dTVleVRKVW5aMllrTjNDV2NVSm14eDdmR3gzdHU5Z05lQWY5?= =?utf-8?B?MDA4cUQzcUdrbW5zUjBSUXFiOHh5anB5MU1iVjlFMStNR0ZhaGNGQ2EwZUxQ?= =?utf-8?B?dGRhR2NQSkJJOUlMTTc4U1JQemhmdmpDV3VPc1VyR2VBV2FtcjlvM09wc2dp?= =?utf-8?B?RmtjUm9kTXhCRVZyaHQvVkRhWEdkTE5SeVV0WmdycFBaVGVBYlI4amc0MjFC?= =?utf-8?B?eXpqaDlHWldZT1U2TjZJNVVKQ0lMdVZ2ZzFQNzVnVUtLSkNsbG1oNXNpN2Mx?= =?utf-8?B?amozRmFzTUhkMEx3WHU0enBPYnBBdUZjNXZQNWpST090eVlod3dFRDFNdkNL?= =?utf-8?B?eGZQVmtBRHFJSzM5SHpSdHNnb29vMjZXWXNNbU5zNzdkd3Zvc0pET0d2aDg4?= =?utf-8?B?azRmcWlKZ1pudk9WT0twSi91aTl5WnVNV1hmZlYrcXNGWUlFTHJGM214T0JT?= =?utf-8?B?RlJDYU0vS3JpRCtGUGxJZlNzQmVQblBoV3pTZ0NhSFhINGl6bUtyaXc4M1Za?= =?utf-8?B?OTVtam9vakxleVRCQnpWNkJLQnZ6NlhySDFnSGlubkhuOWhkZHZ0ZVFwbW92?= =?utf-8?B?WWhzNDNJYVk2VWp4cDZGR3JsWFRIV3V2NjBXTEtuSm0xT3hITnc4bnNJbmxa?= =?utf-8?B?QXZXbkFySVoxZHV5TytEQTNjVVRSbTRrRzdHc3dOUlpCZVl1MzVPMkYrbnAr?= =?utf-8?B?RHBLeUczeE54Yi9LV3pueDIra3h3R0wwZXFUT3dQazB5T3ZqWEZtaEZrM3lr?= =?utf-8?B?MTFsN0tsdytnUDBJRGFOdENrdWlpYytkVllNWFJWODAzeGsyVU1QcHRTMFFY?= =?utf-8?B?RFc4Vms5VzBjRm5pUEUvZng2Z1NTNGtDMngzZWd0UnVzOUExa0VDQVpPNnJy?= =?utf-8?B?MkxWNCs4VzVqV1ArS3RieThQcXhpdk5pY1Y0NUhJUy9TNXpqU1dsV3I0RGgv?= =?utf-8?B?dWtBbE1SY0Zjd3BWcDFsdG03blByMFlQaDNNQ3N4U2FUU242UU1xdWVVcTJ3?= =?utf-8?B?aVFEbHhWZVNwSnJSRmdQOHZrL295aXVzaWFqSGU3cHB0MnMyRndLLy9kZEND?= =?utf-8?B?ZEpCRCtNY0x1TDlFa2RRakk0M2JHMnQyK2dpWVE3eXlwWU5LeWVEVDg4Vk54?= =?utf-8?B?OFdDUWgxKzk0T0pmTzRjRG13cnIwTFVMTisyZnFYWUZ2Z2QzelNaYTZTaDFZ?= =?utf-8?B?V1J5TllhUW1TRE1qUFdjdENvYjFQR1J1YlNIL0I1TC9LbjkwTVVCSW11WkVi?= =?utf-8?B?aU9BK1RoSFdSZEI5S213ejVNa1dMYVNnb1M4S01hZEhtKzlmZnJoK3FNaFVv?= =?utf-8?B?ZWVTWEptakZRaHVJcUQwZTd4WFR2MWVacEZvZkZpN0QzVmRhUmpzN2c0M0J3?= =?utf-8?B?NFZtQkhrVVpEb0xLd1VhbjQ5TUZCaDk4dlJHRlIrbnFhWXZWQ1A5bXZaUmZY?= =?utf-8?B?VldwZHhOZzVQM1Vhb3RKS2RzdWdES1B5T3pYMnJKbjhqNENmaDhVWTB5WmJ4?= =?utf-8?B?d2FLeXpLQzVRTzg1NUNsQVJkUWt6Sm5KdW1rNmpZZnZnNlg0SG93UysvM25a?= =?utf-8?B?NW8yTVVhUVJqWmFZQjA4bWlZOGgwUkpBL0lIUVA3M2FnNGNoOTBFZnVvaW5h?= =?utf-8?B?SzNtcW1CZ0xjVFlIK0h2VjJucWNveDF4VVBNZldkNU9HYUttK3BkaExIeGwy?= =?utf-8?B?ZjFmUDE0TFoyV25PMUh3Z01VbFpzNUZNelNua09jUEIxODY0UE00SXNtWWlJ?= =?utf-8?B?YkhkRXdzNC8vRzd6UEx2NGJVcWEvdmJqV0ZzRFgwOHhodWtFYnNnQVM5cTVt?= =?utf-8?B?LzB1U2FwWC8vSWdDdlBGcVE0ZnBUSXNTYVVTSEllbVAveGtVUVYxWTd4SmRa?= =?utf-8?B?MlBabHdWMVFIbGVPOURTalJYamFrcXBvbGpvQVJ0REV0SUpMdnNmZVJjakxB?= =?utf-8?Q?rP5qizd7atEy7j/g=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 2acd0ed8-2861-4fa4-9b0d-08da3d98d98d X-MS-Exchange-CrossTenant-AuthSource: MW4PR11MB5872.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2022 15:19:52.2793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5FaPk7O97G0/qE5gKePP2L8E9mfvDbHaTP8OfzuNT51SDH7SmlaOq7gL+9nlGuIoqCtnjWVqkrveGOTPfnJ3bA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4105 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 24/05/2022 10:38, Bruce Richardson wrote: > On Tue, May 24, 2022 at 10:25:01AM +0100, Kevin Laatz wrote: >> During EAL init, all buses are probed and the devices found are >> initialized. On eal_cleanup(), the inverse does not happen, meaning any >> allocated memory and other configuration will not be cleaned up >> appropriately on exit. >> >> Currently, in order for device cleanup to take place, applications must >> call the driver-relevant functions to ensure proper cleanup is done before >> the application exits. Since initialization occurs for all devices on the >> bus, not just the devices used by an application, it requires a) >> application awareness of all bus devices that could have been probed on the >> system, and b) code duplication across applications to ensure cleanup is >> performed. An example of this is rte_eth_dev_close() which is commonly used >> across the example applications. >> >> This patch proposes adding bus cleanup to the eal_cleanup() to make EAL's >> init/exit more symmetrical, ensuring all bus devices are cleaned up >> appropriately without the application needing to be aware of all bus types >> that may have been probed during initialization. >> >> Contained in this patch are the changes required to perform cleanup for >> devices on the PCI bus and VDEV bus during eal_cleanup(). There would be an >> ask for bus maintainers to add the relevant cleanup for their buses since >> they have the domain expertise. >> >> Signed-off-by: Kevin Laatz >> Acked-by: Morten Brørup >> > Thanks for the non-RFC versions. Some comments inline. > > /Bruce > >> --- >> v2: >> * change log level from INFO to DEBUG for PCI cleanup >> * add abignore entries for rte_bus related false positives >> >> v3: >> * add vdev bus cleanup > Missing v4 update note. > Please reverse the order here, so it goes from newest to oldest, so those > of us tracking the patch can just look at top entry. Ack >> --- >> devtools/libabigail.abignore | 9 +++++++++ >> drivers/bus/pci/pci_common.c | 32 ++++++++++++++++++++++++++++++++ >> drivers/bus/vdev/vdev.c | 23 +++++++++++++++++++++++ >> lib/eal/common/eal_common_bus.c | 18 ++++++++++++++++++ >> lib/eal/include/rte_bus.h | 23 +++++++++++++++++++++++ >> lib/eal/linux/eal.c | 1 + >> 6 files changed, 106 insertions(+) >> >> diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore >> index 79ff15dc4e..3e519ee42a 100644 >> --- a/devtools/libabigail.abignore >> +++ b/devtools/libabigail.abignore >> @@ -56,3 +56,12 @@ >> ; Ignore libabigail false-positive in clang builds, after moving code. >> [suppress_function] >> name = rte_eal_remote_launch >> + >> +; Ignore field inserted to rte_bus, adding cleanup function >> +[suppress_type] >> + name = rte_bus >> + has_data_member_inserted_at = end >> + >> +; Ignore changes to internally used structs containing rte_bus >> +[suppress_type] >> + name = rte_pci_bus, rte_vmbus_bus, rte_vdev_bus >> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c >> index 37ab879779..7d0c49f073 100644 >> --- a/drivers/bus/pci/pci_common.c >> +++ b/drivers/bus/pci/pci_common.c >> @@ -394,6 +394,37 @@ pci_probe(void) >> return (probed && probed == failed) ? -1 : 0; >> } >> >> +static int >> +pci_cleanup(void) >> +{ >> + struct rte_pci_device *dev = NULL; >> + int ret = 0; >> + >> + FOREACH_DEVICE_ON_PCIBUS(dev) { >> + struct rte_pci_addr *loc = &dev->addr; >> + struct rte_pci_driver *drv = dev->driver; >> + >> + if (loc == NULL || drv == NULL) >> + continue; >> + >> + RTE_LOG(DEBUG, EAL, >> + "Clean up PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n", >> + drv->driver.name, dev->id.vendor_id, dev->id.device_id, >> + loc->domain, loc->bus, loc->devid, loc->function, >> + dev->device.numa_node); >> + >> + ret = drv->remove(dev); >> + if (ret < 0) { >> + RTE_LOG(ERR, EAL, "Cleanup for device "PCI_PRI_FMT" failed\n", >> + dev->addr.domain, dev->addr.bus, dev->addr.devid, >> + dev->addr.function); >> + rte_errno = errno; >> + } >> + } >> + >> + return ret; >> +} > This function returns the status of the last remove call only, which is > probably not what we want. Better to make "ret" a local variable inside the > loop and have a new variable in function scope called "error", initialized > to zero. Then where you assign rte_errno you can also assign error to -1 > and return error from the function at end. Will fix for v5. >> + >> /* dump one device */ >> static int >> pci_dump_one_device(FILE *f, struct rte_pci_device *dev) >> @@ -813,6 +844,7 @@ struct rte_pci_bus rte_pci_bus = { >> .bus = { >> .scan = rte_pci_scan, >> .probe = pci_probe, >> + .cleanup = pci_cleanup, >> .find_device = pci_find_device, >> .plug = pci_plug, >> .unplug = pci_unplug, >> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c >> index a8d8b2327e..eca1a3d536 100644 >> --- a/drivers/bus/vdev/vdev.c >> +++ b/drivers/bus/vdev/vdev.c >> @@ -569,6 +569,28 @@ vdev_probe(void) >> return ret; >> } >> >> +static int >> +vdev_cleanup(void) >> +{ >> + struct rte_vdev_device *dev; >> + int ret = 0; >> + >> + TAILQ_FOREACH(dev, &vdev_device_list, next) { >> + const char *name; >> + struct rte_vdev_driver *drv; >> + >> + name = rte_vdev_device_name(dev); >> + if (vdev_parse(name, &drv)) >> + continue; >> + >> + ret = drv->remove(dev); >> + if (ret < 0) >> + VDEV_LOG(ERR, "Cleanup for device %s failed\n", rte_vdev_device_name(dev)); >> + } >> + >> + return ret; >> +} > Same comment for ret as above. > >> + >> struct rte_device * >> rte_vdev_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, >> const void *data) >> @@ -627,6 +649,7 @@ vdev_get_iommu_class(void) >> static struct rte_bus rte_vdev_bus = { >> .scan = vdev_scan, >> .probe = vdev_probe, >> + .cleanup = vdev_cleanup, >> .find_device = rte_vdev_find_device, >> .plug = vdev_plug, >> .unplug = vdev_unplug, >> diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c >> index baa5b532af..046a06a2bf 100644 >> --- a/lib/eal/common/eal_common_bus.c >> +++ b/lib/eal/common/eal_common_bus.c >> @@ -85,6 +85,24 @@ rte_bus_probe(void) >> return 0; >> } >> >> +/* Clean up all devices of all buses */ >> +int >> +rte_bus_cleanup(void) >> +{ >> + int ret; >> + struct rte_bus *bus; >> + >> + TAILQ_FOREACH(bus, &rte_bus_list, next) { >> + if (bus->cleanup == NULL) >> + continue; >> + ret = bus->cleanup(); >> + if (ret) >> + RTE_LOG(ERR, EAL, "Bus (%s) cleanup failed.\n", bus->name); > Is this error message needed, if the individual buses all print out their > own failure logs? Probably harmless enough. > >> + } >> + >> + return 0; > Do you want to pass back up the fact that a bus cleanup failed rather than > always returning 0? Will change this and review the need for the logging. > >> +} >> + >> /* Dump information of a single bus */ >> static int >> bus_dump_one(FILE *f, struct rte_bus *bus) >> diff --git a/lib/eal/include/rte_bus.h b/lib/eal/include/rte_bus.h >> index bbbb6efd28..42da38730f 100644 >> --- a/lib/eal/include/rte_bus.h >> +++ b/lib/eal/include/rte_bus.h >> @@ -66,6 +66,18 @@ typedef int (*rte_bus_scan_t)(void); >> */ >> typedef int (*rte_bus_probe_t)(void); >> >> +/** >> + * Implementation specific cleanup function which is responsible for cleaning up >> + * devices on that bus with applicable drivers. >> + * >> + * This is called while iterating over each registered bus. >> + * >> + * @return >> + * 0 for successful cleanup >> + * !0 for any error during cleanup >> + */ >> +typedef int (*rte_bus_cleanup_t)(void); >> + >> /** >> * Device iterator to find a device on a bus. >> * >> @@ -277,6 +289,7 @@ struct rte_bus { >> /**< handle hot-unplug failure on the bus */ >> rte_bus_sigbus_handler_t sigbus_handler; >> /**< handle sigbus error on the bus */ >> + rte_bus_cleanup_t cleanup; /**< Cleanup devices on bus */ >> >> }; >> >> @@ -317,6 +330,16 @@ int rte_bus_scan(void); >> */ >> int rte_bus_probe(void); >> >> +/** >> + * For each device on the buses, perform a driver 'match' and call the >> + * driver-specific function for device cleanup. >> + * >> + * @return >> + * 0 for successful match/cleanup >> + * !0 otherwise >> + */ >> +int rte_bus_cleanup(void); >> + > This is in a public header file, so it's visible to users, but it's not in > the version.map file, so not actually usable by anything other than EAL. We > need to decide whether to make this function explicitly public (in which > case it probably needs to be marked as experimental and added to > version.map), or to decide its for EAL use only, in which case move the > defintion to a private header file e.g. eal_private.h. The intention is to only call this in eal_cleanup() since all applications should call it when closing, so I think its reasonable to move it to a private header. Will move in v5. Thanks for reviewing. > >> /** >> * Dump information of all the buses registered with EAL. >> * >> diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c >> index 1ef263434a..27014fdc27 100644 >> --- a/lib/eal/linux/eal.c >> +++ b/lib/eal/linux/eal.c >> @@ -1266,6 +1266,7 @@ rte_eal_cleanup(void) >> vfio_mp_sync_cleanup(); >> #endif >> rte_mp_channel_cleanup(); >> + rte_bus_cleanup(); >> /* after this point, any DPDK pointers will become dangling */ >> rte_eal_memory_detach(); >> eal_mp_dev_hotplug_cleanup(); >> -- >> 2.31.1 >>