From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE552A0542; Mon, 6 Jun 2022 13:47:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 52312427F9; Mon, 6 Jun 2022 13:47:26 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2081.outbound.protection.outlook.com [40.107.94.81]) by mails.dpdk.org (Postfix) with ESMTP id 434B8427EE; Mon, 6 Jun 2022 13:47:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PLMBbBTBraG0TI1C+LSze5PlAY4W3H6VWROBa5ymZ0XnfB3ifoJJqZzUMlsy3N+FQFnltUKu05XEEwf8bvYiU19chI+vVocQhb2p2RgKRGSQAAW5BAIH+Ug7vogBneYC6XNLbYe8Ky3H4dNqFI84vBi9i2PEA1qNhW3jdcH1GjTKKL7BTZUbG6BBMoaI9Vc2wcCKaH6r37B2V/KmBewSPWDPxFB5SJqjtPiZtQBbnKyWYNkIPQ/yq8ea2RQ95UsI6MIJgDJjoooxUpkUbXlT3ytj7VnoeOdoYszHgf1jZTXbIxAIsjmG+LyxV23wwldIojFSMMOKHWOpP+Oxw54Vrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GxYjiKxrccCuUfqLT+hVMguof9KBb2t6cO9cFphAuQI=; b=RpK5AUBq6fzKS3p20xZ++Hgm1H/wVCjYBYqgMdifqt7uxESNu05lrFOi/gEEgkvEdsZxzTb0jiMJ+PknWiODSdYRRNZbQPALveO8wRaQ0YoHhFX7sUUMT9LkBlP+8ZRSfGu3QTu2TkOcfS1oFmqkR7ox0IfZ6eRdW2/YGmeYiTxgVkF1og2UuWAoH7kNGb/YrNlo+lw7wo3SKNenc7PnKW0tgks/fx7cOyXcbaQ8xwPcz88DlJO047EaE5IZaghgyIdkOOLK1s6nvZICBDCbjdT3REI1VSjymEOFMLD/zYcIt9MHpWyQWITaosZoUeqWchjDH6BzwJveNFQvbauwUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GxYjiKxrccCuUfqLT+hVMguof9KBb2t6cO9cFphAuQI=; b=o/WsuRgF4waiz4WZQuxP2gor1cM6Naj3XjhUr4dNgiucUUagk4OTlltlMkNwF+BxVJZJckdgnlFq7EbMRgOmT2xqsSLi027Zu78WCDJxAkSKE1b/lrWFKdIOXVvgrpSKR3sYMttlChgj3mP5p2rO8G/R6FAqBrl55xQEQobqypFkEb1IS6rdoNRDBN0JWnSYQIKuFnQoZWnEnos1cYTt2Leet9m5Ql6pQbOnuEODrmz+oCRhLT2n3wI54E0rirrsdvTJx7HPY4rf56aDCaoypwhsUK8mwApkvWH2dMDXLk/b+9nCQ8emRgp2g6s6GpG9qEDZE3BymNyiFV0ZA3RzOA== Received: from DM6PR11CA0020.namprd11.prod.outlook.com (2603:10b6:5:190::33) by MW4PR12MB5603.namprd12.prod.outlook.com (2603:10b6:303:16a::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.19; Mon, 6 Jun 2022 11:47:22 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:190:cafe::a1) by DM6PR11CA0020.outlook.office365.com (2603:10b6:5:190::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.17 via Frontend Transport; Mon, 6 Jun 2022 11:47:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5314.12 via Frontend Transport; Mon, 6 Jun 2022 11:47:21 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 6 Jun 2022 11:47:21 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 6 Jun 2022 04:47:16 -0700 From: Li Zhang To: , , , , Bruce Richardson , "Dmitry Kozlyuk" , Narcisa Ana Maria Vasile , Dmitry Malloy , Pallavi Kadam CC: , , , , Yajun Wu , Subject: [PATCH v1 02/17] eal: add device removal in rte cleanup Date: Mon, 6 Jun 2022 14:46:35 +0300 Message-ID: <20220606114650.209612-3-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606114650.209612-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> <20220606114650.209612-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5f5b50d4-be90-45ab-80c6-08da47b25128 X-MS-TrafficTypeDiagnostic: MW4PR12MB5603:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EM3/HfdpClQ1fdiQhtZbeo/CJ4XRKw1QKQAaLm3ktCC6zRhVoAzAEYW377ZtkN3QHTpvBgXZfMvSq0O1s4PwqUjw/4QLf+wbjGIFiXBKc24zuy1XgJjw8CTSh+2id3zxkE+7kld67iXTAMn1OV7h8ykVwdOW+5fQfis8CfrlmY4TG9f2nb69unh82LfUZfwbgZsqC1AIJY+BAX90LfMWvUQbH0YLns8BBzINSyauNdWdixrGZKVnOHKE74qtjQ5X30Rt5UvyjIbuUuRerE5AQ2A1qDUuQ58mUcRkEj8q/9iyvTI7VVtiXmLVZ/7apzsdjO4pYFZM5JHTX7/yR3IByNBYLrD9UcOLFv5dbBSBNDSXV1shzOLZG8/GF/H9yndufq5X+wo/b8EE+JIDtJXlW4itHKlqWFYSK6AO9wuomWO7m8pTEyEc/ssRJ1TLpm+qWOQiCByLevuADEEI+uE1m9HgG6bR1Hi9w9IQP7DwK2Ja+7sSiGC6UnpbMxPazQu3ucfe5V/HACRZX9Wy/+5GW3D+jusTLJzjQhLmTZGLkcq1pxHoiM/OyuLXOvf2lg3lo64gTus1UG7pqfaHt+MbBJfxAGkT03iUzUMFg25xTBkI8cdYpw1Lko/dRjZBAxz3vPKt9fXdM65OCpeFcSpRgRKOrowD41oylE0+rDcvCckxY1FIUKTxu6MrSnqfQFoWMsTkPgTSMOjNrf+GTSkG4A== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(16526019)(1076003)(6286002)(36860700001)(426003)(186003)(2906002)(40460700003)(47076005)(508600001)(54906003)(336012)(70206006)(2616005)(70586007)(55016003)(110136005)(316002)(8936002)(6666004)(36756003)(356005)(81166007)(4326008)(8676002)(7696005)(26005)(5660300002)(86362001)(82310400005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 11:47:21.6456 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5f5b50d4-be90-45ab-80c6-08da47b25128 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB5603 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Yajun Wu Add device removal in function rte_eal_cleanup. This is the last chance device remove get called for sanity. Loop vdev bus first and then all bus for all device, calling rte_dev_remove. Cc: stable@dpdk.org Signed-off-by: Yajun Wu --- lib/eal/freebsd/eal.c | 33 +++++++++++++++++++++++++++++++++ lib/eal/include/rte_dev.h | 6 ++++++ lib/eal/linux/eal.c | 33 +++++++++++++++++++++++++++++++++ lib/eal/windows/eal.c | 33 +++++++++++++++++++++++++++++++++ 4 files changed, 105 insertions(+) diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c index a6b20960f2..5ffd9146b6 100644 --- a/lib/eal/freebsd/eal.c +++ b/lib/eal/freebsd/eal.c @@ -886,11 +886,44 @@ rte_eal_init(int argc, char **argv) return fctret; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); rte_service_finalize(); rte_mp_channel_cleanup(); /* after this point, any DPDK pointers will become dangling */ diff --git a/lib/eal/include/rte_dev.h b/lib/eal/include/rte_dev.h index e6ff1218f9..382d548ea3 100644 --- a/lib/eal/include/rte_dev.h +++ b/lib/eal/include/rte_dev.h @@ -492,6 +492,12 @@ int rte_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len); +#define RTE_DEV_FOREACH_SAFE(dev, devstr, it, tdev) \ + for (rte_dev_iterator_init(it, devstr), \ + (dev) = rte_dev_iterator_next(it); \ + (dev) && ((tdev) = rte_dev_iterator_next(it), 1); \ + (dev) = (tdev)) + #ifdef __cplusplus } #endif diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index 1ef263434a..30b295916e 100644 --- a/lib/eal/linux/eal.c +++ b/lib/eal/linux/eal.c @@ -1248,6 +1248,38 @@ mark_freeable(const struct rte_memseg_list *msl, const struct rte_memseg *ms, return 0; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { @@ -1257,6 +1289,7 @@ rte_eal_cleanup(void) struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); if (rte_eal_process_type() == RTE_PROC_PRIMARY && internal_conf->hugepage_file.unlink_existing) rte_memseg_walk(mark_freeable, NULL); diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c index 122de2a319..3d7d411293 100644 --- a/lib/eal/windows/eal.c +++ b/lib/eal/windows/eal.c @@ -254,12 +254,45 @@ __rte_trace_point_register(rte_trace_point_t *trace, const char *name, return -ENOTSUP; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); eal_intr_thread_cancel(); eal_mem_virt2iova_cleanup(); /* after this point, any DPDK pointers will become dangling */ -- 2.31.1