From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C1ABA0093; Fri, 22 Apr 2022 18:27:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD0E3410D5; Fri, 22 Apr 2022 18:27:31 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 27A2F40042 for ; Fri, 22 Apr 2022 18:27:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1650644850; x=1682180850; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=asMzRMSmN+/BuRqm1TW7E9OM6eHN4dxisOXgjMn8dcc=; b=ddFe46Zo3oFqyl/yt5Lvu7cXdNhtZkyluD3LIhijYLNffJzd818UcCl6 oI34AlBjV2AgCqMcO4zgWdVE1jJGXfFL918FWTkugDlGtj4jCKNQ1ybA+ E8AHi/nuZD3d4s8I/z+1rmCJsnxQp7lGznx3LKdAHX1CbS1ISDwYbl3NP nK1BpOlnJ3ZqHp5OnEzjTC9k3CGV0mT+YbriGv/G7RDKm7JZdu4us5QdE /8mB3zvfZQuwKLUg2ZHmnJ1tYqd/CKDpZFw7F1O+pg5TBxtSUpOIlCd2m Slyi19+H2bnNyGJvflfQpZh/Sw9PDeIEERWg05CVj4aJWyQZI6/V4UJwv A==; X-IronPort-AV: E=McAfee;i="6400,9594,10324"; a="252051316" X-IronPort-AV: E=Sophos;i="5.90,282,1643702400"; d="scan'208";a="252051316" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Apr 2022 09:27:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,282,1643702400"; d="scan'208";a="511648313" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga003.jf.intel.com with ESMTP; 22 Apr 2022 09:27:27 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: Kevin Laatz , =?UTF-8?q?Morten=20Br=C3=B8rup?= Subject: [RFC v2] eal: add bus cleanup to eal cleanup Date: Fri, 22 Apr 2022 17:27:17 +0100 Message-Id: <20220422162717.3249369-1-kevin.laatz@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220419161438.1837860-1-kevin.laatz@intel.com> References: <20220419161438.1837860-1-kevin.laatz@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During EAL init, all buses are probed and the devices found are initialized. On eal_cleanup(), the inverse does not happen, meaning any allocated memory and other configuration will not be cleaned up appropriately on exit. Currently, in order for device cleanup to take place, applications must call the driver-relevant functions to ensure proper cleanup is done before the application exits. Since initialization occurs for all devices on the bus, not just the devices used by an application, it requires a) application awareness of all bus devices that could have been probed on the system, and b) code duplication across applications to ensure cleanup is performed. An example of this is rte_eth_dev_close() which is commonly used across the example applications. This RFC proposes adding bus cleanup to the eal_cleanup() to make EAL's init/exit more symmetrical, ensuring all bus devices are cleaned up appropriately without the application needing to be aware of all bus types that may have been probed during initialization. Contained in this RFC are the changes required to perform cleanup for devices on the PCI bus during eal_cleanup(). This can be expanded in subsequent versions if these changes are desired. There would be an ask for bus maintainers to add the relevant cleanup for their buses since they have the domain expertise. Signed-off-by: Kevin Laatz Acked-by: Morten Brørup --- v2: * change log level from INFO to DEBUG for PCI cleanup * add abignore entries for rte_bus related false positives --- devtools/libabigail.abignore | 9 +++++++++ drivers/bus/pci/pci_common.c | 29 +++++++++++++++++++++++++++++ lib/eal/common/eal_common_bus.c | 18 ++++++++++++++++++ lib/eal/include/rte_bus.h | 23 +++++++++++++++++++++++ lib/eal/linux/eal.c | 1 + 5 files changed, 80 insertions(+) diff --git a/devtools/libabigail.abignore b/devtools/libabigail.abignore index c618f20032..3ff5d4db7c 100644 --- a/devtools/libabigail.abignore +++ b/devtools/libabigail.abignore @@ -40,3 +40,12 @@ ; Ignore visibility fix of local functions in experimental gpudev library [suppress_file] soname_regexp = ^librte_gpudev\. + +; Ignore field inserted to rte_bus, adding cleanup function +[suppress_type] + name = rte_bus + has_data_member_inserted_at = end + +; Ignore changes to internally used structs containing rte_bus +[suppress_type] + name = rte_pci_bus, rte_vmbus_bus diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index 37ab879779..1bee8e8201 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -394,6 +394,34 @@ pci_probe(void) return (probed && probed == failed) ? -1 : 0; } +static int +pci_cleanup(void) +{ + struct rte_pci_device *dev = NULL; + int ret = 0; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + struct rte_pci_addr *loc = &dev->addr; + struct rte_pci_driver *drv = dev->driver; + + RTE_LOG(DEBUG, EAL, + "Clean up PCI driver: %s (%x:%x) device: "PCI_PRI_FMT" (socket %i)\n", + drv->driver.name, dev->id.vendor_id, dev->id.device_id, + loc->domain, loc->bus, loc->devid, loc->function, + dev->device.numa_node); + + ret = drv->remove(dev); + if (ret < 0) { + RTE_LOG(ERR, EAL, "Cleanup for device "PCI_PRI_FMT" failed\n", + dev->addr.domain, dev->addr.bus, dev->addr.devid, + dev->addr.function); + rte_errno = errno; + } + } + + return ret; +} + /* dump one device */ static int pci_dump_one_device(FILE *f, struct rte_pci_device *dev) @@ -813,6 +841,7 @@ struct rte_pci_bus rte_pci_bus = { .bus = { .scan = rte_pci_scan, .probe = pci_probe, + .cleanup = pci_cleanup, .find_device = pci_find_device, .plug = pci_plug, .unplug = pci_unplug, diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c index baa5b532af..046a06a2bf 100644 --- a/lib/eal/common/eal_common_bus.c +++ b/lib/eal/common/eal_common_bus.c @@ -85,6 +85,24 @@ rte_bus_probe(void) return 0; } +/* Clean up all devices of all buses */ +int +rte_bus_cleanup(void) +{ + int ret; + struct rte_bus *bus; + + TAILQ_FOREACH(bus, &rte_bus_list, next) { + if (bus->cleanup == NULL) + continue; + ret = bus->cleanup(); + if (ret) + RTE_LOG(ERR, EAL, "Bus (%s) cleanup failed.\n", bus->name); + } + + return 0; +} + /* Dump information of a single bus */ static int bus_dump_one(FILE *f, struct rte_bus *bus) diff --git a/lib/eal/include/rte_bus.h b/lib/eal/include/rte_bus.h index bbbb6efd28..42da38730f 100644 --- a/lib/eal/include/rte_bus.h +++ b/lib/eal/include/rte_bus.h @@ -66,6 +66,18 @@ typedef int (*rte_bus_scan_t)(void); */ typedef int (*rte_bus_probe_t)(void); +/** + * Implementation specific cleanup function which is responsible for cleaning up + * devices on that bus with applicable drivers. + * + * This is called while iterating over each registered bus. + * + * @return + * 0 for successful cleanup + * !0 for any error during cleanup + */ +typedef int (*rte_bus_cleanup_t)(void); + /** * Device iterator to find a device on a bus. * @@ -277,6 +289,7 @@ struct rte_bus { /**< handle hot-unplug failure on the bus */ rte_bus_sigbus_handler_t sigbus_handler; /**< handle sigbus error on the bus */ + rte_bus_cleanup_t cleanup; /**< Cleanup devices on bus */ }; @@ -317,6 +330,16 @@ int rte_bus_scan(void); */ int rte_bus_probe(void); +/** + * For each device on the buses, perform a driver 'match' and call the + * driver-specific function for device cleanup. + * + * @return + * 0 for successful match/cleanup + * !0 otherwise + */ +int rte_bus_cleanup(void); + /** * Dump information of all the buses registered with EAL. * diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index 025e5cc10d..37983b98c0 100644 --- a/lib/eal/linux/eal.c +++ b/lib/eal/linux/eal.c @@ -1268,6 +1268,7 @@ rte_eal_cleanup(void) vfio_mp_sync_cleanup(); #endif rte_mp_channel_cleanup(); + rte_bus_cleanup(); /* after this point, any DPDK pointers will become dangling */ rte_eal_memory_detach(); eal_mp_dev_hotplug_cleanup(); -- 2.31.1