From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F02E9A0543; Mon, 6 Jun 2022 13:21:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 334E94280B; Mon, 6 Jun 2022 13:21:44 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2075.outbound.protection.outlook.com [40.107.212.75]) by mails.dpdk.org (Postfix) with ESMTP id 73977415D7; Mon, 6 Jun 2022 13:21:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EeZgGzPyo9oLRGLxElaWul5mgFD4evZbWx3Y2SFGVQyFOhcvRJlOmscJxRpFBvevLevSHfQYJ1mgSUsEUq/gE4sD/qflPbCE7gweNr1AGJNYNq0gzz8A1It6xYy4wKY8vtapaPuk2rxe+SwNLBII/r/MpgBLz6dQE+kjSS6x7Z7w4S2hvqkuDkMyjcymJTSoT7HIn6YRsSKTfgieX3H+/uytHc3GvpDTG79uQr0YxvqOV7WCJOL+QKHv0mhdBfN5wl1HyoJ4fWGtNE9TBS8Jo+E69YOK+hpuo/ORaXVhx/JyCgyZvEuEs/m0xeSIIRkmPPU/xNIurcjpy5cnhQfwnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GxYjiKxrccCuUfqLT+hVMguof9KBb2t6cO9cFphAuQI=; b=YUirClGwHplGVGgbGDfT1srJvU6i6MYzVRjvhN04MgJXWHZ39TXfYHPk9rPBNfJ9ZJIYV0TgEbLVWzNhYH56cIdSsU+awYSAPRxbixfFtWew94yNXm3gCNVnzWWokMepbi1mIMDqxqbsB20NA1O1dRf/dCjUcyVpf4iepR2633QKbGCj5O7s4otK8h+CyaxTwp3jObDJxDKQg14cELMs3aj/hYhgM8KUArIfcHpdL0zmRJaDIyun6jIsaW53kBqoowM8dUBSaQT77F360IzEm2KCdFd5+/ltOdU9ssvZN2ai/5Bv9V6oWtXV0dRnoc/ch6PpleObMsDpMOj1qEPhNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GxYjiKxrccCuUfqLT+hVMguof9KBb2t6cO9cFphAuQI=; b=P7uy81Q9eSJkK6T49/+MNlrT/pjPx692tEz4wZUwOM9DZ9ke5JpzJCFdLGPpz648VvLBeEEdbDG0fJE1vsWcv3ihc0vtN0QzB+oMb44X/slXZhI+dNcIIgtl5FMtt5IDGsSLV/terBYV7beZmBCwAzjbpobVIBeVRHTvDUOoqBYRksASMSUWGBNbBCtEbXS64O/n3RQwqhG3ENCFDTDMeTWe7YznvtmyG+UEkD1s8lINFBitsU5qqhjn2NLHpHjl62LryCsHIBhlOwoBDvwQfqIShXPFbNYxkqTZ/+B/3PkdEiIcGwoWUeZs0R1hrtPYz1IDtoFku5hNAKAygHa/Jw== Received: from DM6PR02CA0056.namprd02.prod.outlook.com (2603:10b6:5:177::33) by BN6PR1201MB0161.namprd12.prod.outlook.com (2603:10b6:405:55::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.12; Mon, 6 Jun 2022 11:21:41 +0000 Received: from DM6NAM11FT053.eop-nam11.prod.protection.outlook.com (2603:10b6:5:177:cafe::d3) by DM6PR02CA0056.outlook.office365.com (2603:10b6:5:177::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13 via Frontend Transport; Mon, 6 Jun 2022 11:21:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT053.mail.protection.outlook.com (10.13.173.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5314.12 via Frontend Transport; Mon, 6 Jun 2022 11:21:41 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 6 Jun 2022 11:21:40 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 6 Jun 2022 04:21:35 -0700 From: Li Zhang To: , , , , Bruce Richardson , "Dmitry Kozlyuk" , Narcisa Ana Maria Vasile , Dmitry Malloy , Pallavi Kadam CC: , , , , Yajun Wu , Subject: [PATCH v1 02/17] eal: add device removal in rte cleanup Date: Mon, 6 Jun 2022 14:20:39 +0300 Message-ID: <20220606112109.208873-3-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220606112109.208873-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> <20220606112109.208873-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c0f73efc-82af-4fb8-fb56-08da47aebaf7 X-MS-TrafficTypeDiagnostic: BN6PR1201MB0161:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oTdCxXd4OnBkbE/fOQ+qOI5DaXN8fXpLhzeiTEEtmRwqLUj6vPywgZ4YO5NUaxxo3t/QvX/WmLAck96k7uSdewSdVeDq7sA3ZrLlckyARceWXy/ewa3aeGcssRoChKrHm7tWraWpiL+H6p59js0srvQgbNT9/0DSfQByJnrlgwcts/r5K6yRyE5odw22YunNiHk7danR0c+FxGexSmtwDyf2uK6446tceTFqq27+8Am4aad6LGKRBalK7UWsA4uRIlcrzF+4F2xhHYU5MgUAJsATAFrF3hN28u1wm1Awn11AAJo/qka7pjb0/6WgMeW2AN9BAiAeKs6Tmyj9bTU4c3m/I/nevv6MUBERxEt5Dq5PqMd9V/t44bhl60eYR1h3uFK09bD9JYZKS6UOJWVvEulg8S92+KfJdnABXTx1BU6efGKLtWmcMpzsh52RdnPDFg4MfRAUktn+OBeWXoc6VHg9mVyU1zCh78DiI4jsMZXUCN81wnQ6FIYgBlr9HjUNICAKwkypLxzK4QEjER7YifmQMKfAHRBghdqGX2O8gf4kgYyZF+wINWw84xL9ApkarQgV28Rr8fNl/LkIbWI04cQClT15JjUuj+1V0XvkPPZu0/pfDF8j5lk2Z+lLc6DMV1+uFp5JsR3GLMp65k6ROx0k4t/cTQjUKF762cD2s5cS/ZzwOWU0mDhJPBHmJCuwriGTX4DGimPc1Ue/qIxrLA== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(508600001)(47076005)(7696005)(36860700001)(4326008)(86362001)(70586007)(82310400005)(8676002)(186003)(2616005)(8936002)(40460700003)(70206006)(5660300002)(16526019)(6286002)(55016003)(36756003)(2906002)(1076003)(426003)(54906003)(110136005)(356005)(26005)(81166007)(316002)(336012)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 11:21:41.1167 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c0f73efc-82af-4fb8-fb56-08da47aebaf7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT053.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1201MB0161 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Yajun Wu Add device removal in function rte_eal_cleanup. This is the last chance device remove get called for sanity. Loop vdev bus first and then all bus for all device, calling rte_dev_remove. Cc: stable@dpdk.org Signed-off-by: Yajun Wu --- lib/eal/freebsd/eal.c | 33 +++++++++++++++++++++++++++++++++ lib/eal/include/rte_dev.h | 6 ++++++ lib/eal/linux/eal.c | 33 +++++++++++++++++++++++++++++++++ lib/eal/windows/eal.c | 33 +++++++++++++++++++++++++++++++++ 4 files changed, 105 insertions(+) diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c index a6b20960f2..5ffd9146b6 100644 --- a/lib/eal/freebsd/eal.c +++ b/lib/eal/freebsd/eal.c @@ -886,11 +886,44 @@ rte_eal_init(int argc, char **argv) return fctret; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); rte_service_finalize(); rte_mp_channel_cleanup(); /* after this point, any DPDK pointers will become dangling */ diff --git a/lib/eal/include/rte_dev.h b/lib/eal/include/rte_dev.h index e6ff1218f9..382d548ea3 100644 --- a/lib/eal/include/rte_dev.h +++ b/lib/eal/include/rte_dev.h @@ -492,6 +492,12 @@ int rte_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len); +#define RTE_DEV_FOREACH_SAFE(dev, devstr, it, tdev) \ + for (rte_dev_iterator_init(it, devstr), \ + (dev) = rte_dev_iterator_next(it); \ + (dev) && ((tdev) = rte_dev_iterator_next(it), 1); \ + (dev) = (tdev)) + #ifdef __cplusplus } #endif diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index 1ef263434a..30b295916e 100644 --- a/lib/eal/linux/eal.c +++ b/lib/eal/linux/eal.c @@ -1248,6 +1248,38 @@ mark_freeable(const struct rte_memseg_list *msl, const struct rte_memseg *ms, return 0; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { @@ -1257,6 +1289,7 @@ rte_eal_cleanup(void) struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); if (rte_eal_process_type() == RTE_PROC_PRIMARY && internal_conf->hugepage_file.unlink_existing) rte_memseg_walk(mark_freeable, NULL); diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c index 122de2a319..3d7d411293 100644 --- a/lib/eal/windows/eal.c +++ b/lib/eal/windows/eal.c @@ -254,12 +254,45 @@ __rte_trace_point_register(rte_trace_point_t *trace, const char *name, return -ENOTSUP; } +static int +bus_match_all(const struct rte_bus *bus, const void *data) +{ + RTE_SET_USED(bus); + RTE_SET_USED(data); + return 0; +} + +static void +remove_all_device(void) +{ + struct rte_bus *start = NULL, *next; + struct rte_dev_iterator dev_iter = {0}; + struct rte_device *dev = NULL; + struct rte_device *tdev = NULL; + char devstr[128]; + + RTE_DEV_FOREACH_SAFE(dev, "bus=vdev", &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + while ((next = rte_bus_find(start, bus_match_all, NULL)) != NULL) { + start = next; + /* Skip buses that don't have iterate method */ + if (!next->dev_iterate || !next->name) + continue; + snprintf(devstr, sizeof(devstr), "bus=%s", next->name); + RTE_DEV_FOREACH_SAFE(dev, devstr, &dev_iter, tdev) { + (void)rte_dev_remove(dev); + } + }; +} + int rte_eal_cleanup(void) { struct internal_config *internal_conf = eal_get_internal_configuration(); + remove_all_device(); eal_intr_thread_cancel(); eal_mem_virt2iova_cleanup(); /* after this point, any DPDK pointers will become dangling */ -- 2.31.1