From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10358A0C43; Fri, 8 Oct 2021 19:43:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C9180410EA; Fri, 8 Oct 2021 19:43:41 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by mails.dpdk.org (Postfix) with ESMTP id 79FFF410DB for ; Fri, 8 Oct 2021 19:43:40 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mlu3zD0Ka33eielUZsyJsdwtzPAapmx1dhTTwpAo2ThubjBuBli2H/jy58wACrz1NZUWFeAOdE27hc3lA2U1SwAGxsSqCCC+TFRP5ku6rT5v9ODXxe4JhqlQ6fFpiI6tUJe3UZLNGJvuvwe2R46H3jU48i1hFvJhvIwAmbA0KPep0GzpXrLX5KONlqOKIB2RTSz1SNPX1RhQHlVfDeZBReVGLkQfAvivHK+TL09oyztgblgbCJ3Ngtpv0Gu/ahRnanN+nt77WshMbUEEEquyTsuli84cyrjIjhWifjK3P3o9NBTD3W969WrhPytx1UYYYeA3RGsQl88NjZQs0SxItg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=; b=Ko1XKyvHU7/R/kDxPAz2YKUgoO+HXI+orwV93/BRxKWfjlDnjk/VVq03yApHnlb27yO9IdCB+OkK1NIfOkDgZJVPjfAv3BsGYdkg0j1zk5p2Jar/C5IGFyyIycCZY5uOxFVdFLbpTACrgHDwuUcU95n1Pz00CvYbC3u7AW3CAkOSR0BxdTxqEGQEJeNbnnT81kPhDfnBqv519EN2KKPy99JGw7ydq5B9nnj4P2wf0YupwuyTi8DdO17KershUPZ3ye5mA8najAYN1mM00tiEyJkiiWSwyANATKCJCXo3Hld79zet27sgdKLWFOvmA/zNvV5RAk8BUmn90bTVNQC1PQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=; b=iziVPftoJ8CqWnNaDIKoYv6OqExRPgnojdSQcXQYNqkpr5tk5eXa8yziUG9Bnd6nKzt8NivFpi0HFczpdjtcJBb3nigiYdKRkzW7RqG3JBhybw70iy4zqn73TXcdgU8atnQYnSQ+Mx6WcJ60Mcy53jCPz9+zDOJIH9rwQXr2wO3P0rU0+FVFzrrtKTui/x7iIhADbFJGR5Ac7DNzl2fCPT9r4X1/1Ju2uBsbOp9qIvMSPCMnPvtuktKAE7GR/PyFY1cAQdr+FsaJAYdp4WJjVo7exXDlRHmwOouxKcqBgUHB4UBLqju2klICrhiDqqDZeC0652nhsS4OlOLeHwvZ0Q== Received: from MW4PR03CA0131.namprd03.prod.outlook.com (2603:10b6:303:8c::16) by DM4PR12MB5182.namprd12.prod.outlook.com (2603:10b6:5:395::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Fri, 8 Oct 2021 17:43:39 +0000 Received: from CO1NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8c:cafe::fc) by MW4PR03CA0131.outlook.office365.com (2603:10b6:303:8c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18 via Frontend Transport; Fri, 8 Oct 2021 17:43:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT047.mail.protection.outlook.com (10.13.174.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4587.18 via Frontend Transport; Fri, 8 Oct 2021 17:43:38 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 8 Oct 2021 17:43:34 +0000 From: To: CC: Thomas Monjalon Date: Sat, 9 Oct 2021 01:53:43 +0000 Message-ID: <20211009015349.9694-4-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211009015349.9694-1-eagostini@nvidia.com> References: <20210602203531.2288645-1-thomas@monjalon.net> <20211009015349.9694-1-eagostini@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9cfdf352-57f4-489c-cde3-08d98a83295c X-MS-TrafficTypeDiagnostic: DM4PR12MB5182: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6FJa0P/CqhJMV+68EX22Z4zx6UkhywFpIBEvTTk69HiTQ7MZ0QM8+j40Dl+eJ/6ta9qWF0OYb2t7fn3bTdDA9dKYi2bjBBYoXEFZsdVIwfnB91GQ2N9n1n4WeXmWjb7WirgkjXPwwytXmzSCROPbfDxjomSv5P4njmxcPaayHAPIOqnPfkEVi16IPZp780bZKggdbWliNc5loBs/mdaYKQbSnUgnehMN0HPW2kxtBpnnYzccJbXT1wuok8sxOVtyfKJaSQANZHaWkbnHDiJCyYgNa/aPTG2b+8C2EGdJYxwKJvgdpRe4K93jQm+fAiF7OF1BLAB6ptdz6uWZxiv9bw10u2vV4YVnBcjxyEMoXfM3SCyYU1ofwGaO3Z9MaiAhfZZTDHY3tg/urnO8grVrqYXsaTQQk5Aoiat+ijBkKwCpWzyvJyquZ839Jq3cSo63/Fj2c0YFwKW7ZMeTc7gKs1fxLFPtFhbBSP/GE2IC/cliEVZwrp7hgeP5dpaIcc82UjUS7GvwGfyxXOFPGanIY7PGGIli4O/tOL25qacnbDGW1jhA6vb1YT2kBKxhXN4S6JgmwGWXJZlA436CXbFUxLHbAG22iQhmPilKCAEQoeicKPTBJB5nzfDpkX5QQZZN2C8x8qMzHnxg/8e58oC8Y0ytdtKw/pKndeINCGZvCWaKlN6I/kj9XdtkhrLz2AmCj29dgTyUgncpn39Cx+CmYQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(6916009)(1076003)(2616005)(4326008)(5660300002)(70586007)(70206006)(316002)(356005)(7696005)(6286002)(8936002)(8676002)(86362001)(426003)(186003)(16526019)(6666004)(36756003)(7636003)(508600001)(55016002)(26005)(2876002)(36860700001)(83380400001)(2906002)(336012)(82310400003)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Oct 2021 17:43:38.7853 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9cfdf352-57f4-489c-cde3-08d98a83295c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5182 Subject: [dpdk-dev] [PATCH v3 3/9] gpudev: add child device representing a device context X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon The computing device may operate in some isolated contexts. Memory and processing are isolated in a silo represented by a child device. The context is provided as an opaque by the caller of rte_gpu_add_child(). Signed-off-by: Thomas Monjalon --- doc/guides/prog_guide/gpudev.rst | 12 ++++++ lib/gpudev/gpudev.c | 45 +++++++++++++++++++- lib/gpudev/gpudev_driver.h | 2 +- lib/gpudev/rte_gpudev.h | 71 +++++++++++++++++++++++++++++--- lib/gpudev/version.map | 1 + 5 files changed, 123 insertions(+), 8 deletions(-) diff --git a/doc/guides/prog_guide/gpudev.rst b/doc/guides/prog_guide/gpudev.rst index 6ea7239159..7694639489 100644 --- a/doc/guides/prog_guide/gpudev.rst +++ b/doc/guides/prog_guide/gpudev.rst @@ -34,3 +34,15 @@ This library provides a number of features: API Overview ------------ + +Child Device +~~~~~~~~~~~~ + +By default, DPDK PCIe module detects and registers physical GPU devices +in the system. +With the gpudev library is also possible to add additional non-physical devices +through an ``uint64_t`` generic handler (e.g. CUDA Driver context) +that will be registered internally by the driver as an additional device (child) +connected to a physical device (parent). +Each device (parent or child) is represented through a ID +required to indicate which device a given operation should be executed on. diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c index d57e23df7c..74cdd7f20b 100644 --- a/lib/gpudev/gpudev.c +++ b/lib/gpudev/gpudev.c @@ -80,13 +80,22 @@ rte_gpu_is_valid(int16_t dev_id) return false; } +static bool +gpu_match_parent(int16_t dev_id, int16_t parent) +{ + if (parent == RTE_GPU_ID_ANY) + return true; + return gpus[dev_id].info.parent == parent; +} + int16_t -rte_gpu_find_next(int16_t dev_id) +rte_gpu_find_next(int16_t dev_id, int16_t parent) { if (dev_id < 0) dev_id = 0; while (dev_id < gpu_max && - gpus[dev_id].state == RTE_GPU_STATE_UNUSED) + (gpus[dev_id].state == RTE_GPU_STATE_UNUSED || + !gpu_match_parent(dev_id, parent))) dev_id++; if (dev_id >= gpu_max) @@ -177,6 +186,7 @@ rte_gpu_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node = -1; + dev->info.parent = RTE_GPU_ID_NONE; TAILQ_INIT(&dev->callbacks); gpu_count++; @@ -185,6 +195,28 @@ rte_gpu_allocate(const char *name) return dev; } +int16_t +rte_gpu_add_child(const char *name, int16_t parent, uint64_t child_context) +{ + struct rte_gpu *dev; + + if (!rte_gpu_is_valid(parent)) { + GPU_LOG(ERR, "add child to invalid parent ID %d", parent); + rte_errno = ENODEV; + return -rte_errno; + } + + dev = rte_gpu_allocate(name); + if (dev == NULL) + return -rte_errno; + + dev->info.parent = parent; + dev->info.context = child_context; + + rte_gpu_complete_new(dev); + return dev->info.dev_id; +} + void rte_gpu_complete_new(struct rte_gpu *dev) { @@ -199,10 +231,19 @@ rte_gpu_complete_new(struct rte_gpu *dev) int rte_gpu_release(struct rte_gpu *dev) { + int16_t dev_id, child; + if (dev == NULL) { rte_errno = ENODEV; return -rte_errno; } + dev_id = dev->info.dev_id; + RTE_GPU_FOREACH_CHILD(child, dev_id) { + GPU_LOG(ERR, "cannot release device %d with child %d", + dev_id, child); + rte_errno = EBUSY; + return -rte_errno; + } GPU_LOG(DEBUG, "free device %s (id %d)", dev->info.name, dev->info.dev_id); diff --git a/lib/gpudev/gpudev_driver.h b/lib/gpudev/gpudev_driver.h index 2a7089aa52..4d0077161c 100644 --- a/lib/gpudev/gpudev_driver.h +++ b/lib/gpudev/gpudev_driver.h @@ -31,7 +31,7 @@ typedef int (rte_gpu_info_get_t)(struct rte_gpu *dev, struct rte_gpu_info *info) struct rte_gpu_ops { /* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get; - /* Close device. */ + /* Close device or child context. */ rte_gpu_close_t *dev_close; }; diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h index e1702fbfe4..df75dbdbab 100644 --- a/lib/gpudev/rte_gpudev.h +++ b/lib/gpudev/rte_gpudev.h @@ -41,8 +41,12 @@ extern "C" { struct rte_gpu_info { /** Unique identifier name. */ const char *name; + /** Opaque handler of the device context. */ + uint64_t context; /** Device ID. */ int16_t dev_id; + /** ID of the parent device, RTE_GPU_ID_NONE if no parent */ + int16_t parent; /** Total processors available on device. */ uint32_t processor_count; /** Total memory available on device. */ @@ -110,6 +114,33 @@ uint16_t rte_gpu_count_avail(void); __rte_experimental bool rte_gpu_is_valid(int16_t dev_id); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Create a virtual device representing a context in the parent device. + * + * @param name + * Unique string to identify the device. + * @param parent + * Device ID of the parent. + * @param child_context + * Opaque context handler. + * + * @return + * Device ID of the new created child, -rte_errno otherwise: + * - EINVAL if empty name + * - ENAMETOOLONG if long name + * - EEXIST if existing device name + * - ENODEV if invalid parent + * - EPERM if secondary process + * - ENOENT if too many devices + * - ENOMEM if out of space + */ +__rte_experimental +int16_t rte_gpu_add_child(const char *name, + int16_t parent, uint64_t child_context); + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. @@ -118,13 +149,17 @@ bool rte_gpu_is_valid(int16_t dev_id); * * @param dev_id * The initial device ID to start the research. + * @param parent + * The device ID of the parent. + * RTE_GPU_ID_NONE means no parent. + * RTE_GPU_ID_ANY means no or any parent. * * @return * Next device ID corresponding to a valid and initialized computing device, * RTE_GPU_ID_NONE if there is none. */ __rte_experimental -int16_t rte_gpu_find_next(int16_t dev_id); +int16_t rte_gpu_find_next(int16_t dev_id, int16_t parent); /** * @warning @@ -136,15 +171,41 @@ int16_t rte_gpu_find_next(int16_t dev_id); * The ID of the next possible valid device, usually 0 to iterate all. */ #define RTE_GPU_FOREACH(dev_id) \ - for (dev_id = rte_gpu_find_next(0); \ - dev_id > 0; \ - dev_id = rte_gpu_find_next(dev_id + 1)) + RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_ANY) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Macro to iterate over all valid computing devices having no parent. + * + * @param dev_id + * The ID of the next possible valid device, usually 0 to iterate all. + */ +#define RTE_GPU_FOREACH_PARENT(dev_id) \ + RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_NONE) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Macro to iterate over all valid children of a computing device parent. + * + * @param dev_id + * The ID of the next possible valid device, usually 0 to iterate all. + * @param parent + * The device ID of the parent. + */ +#define RTE_GPU_FOREACH_CHILD(dev_id, parent) \ + for (dev_id = rte_gpu_find_next(0, parent); \ + dev_id >= 0; \ + dev_id = rte_gpu_find_next(dev_id + 1, parent)) /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. * - * Close device. + * Close device or child context. * All resources are released. * * @param dev_id diff --git a/lib/gpudev/version.map b/lib/gpudev/version.map index b3b6b76c1c..4a934ed933 100644 --- a/lib/gpudev/version.map +++ b/lib/gpudev/version.map @@ -2,6 +2,7 @@ EXPERIMENTAL { global: # added in 21.11 + rte_gpu_add_child; rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close; -- 2.17.1