From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EC284A0C53; Wed, 3 Nov 2021 12:05:16 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A306F41233; Wed, 3 Nov 2021 12:05:02 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2085.outbound.protection.outlook.com [40.107.220.85]) by mails.dpdk.org (Postfix) with ESMTP id AE945411CE for ; Wed, 3 Nov 2021 12:04:58 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ViM6t+Cro1KjzIDWDbIkOwxboh+ooFv35QSLFF4QAfpYZdNDkBom41FNgbnQp0oQkGjWzA+spSUM4dqON4tH/KNWv0LQg6G3blxnY/GGWubNTs2HXy0UfdYv2fqMblRlwBK/GdyMchePs+Huz4o9zCVH5Py1z5mkvQjAz3JyBTX03pvrVVTOsSHb26zrZSUijf4fTBGsbbxSCq9pEB1+cXZQOHN4Ny1Gix09x6dml48FQgwDCKO2nBxw2tTEcRxhVoOboDodw774dMSenqMYYdhmIumqqTM8ujVcPQbjxaQeq6HCKmayPZSZpDsypHIedk+nUbhArTvr+p1fGEvnzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=; b=kO87SDqqndsL9lvRZWj91HTYzaxKbTtsuum3JNacwFmeC7NrVVsvtoh3/ToQn32XP3bamyahwGFSKTF5A7hmIdwfFTIOcAicMGBEtfzeUNZMLeyF5kYBru0ifOKQS+Y/6v/Nd0cU+m3Q1eBRnRrcmhxaGAG1x8j09NFFWOs9HGmYcylJ1NPWletuRZfbxScEkMWYlPyjLeo1FQEnsiwHWT86BbFzb6dBbSs4TJLjs/U6dktwDHtExvTBSh8RONQjVE66CWI8QyT6iXKRwMf033iEH9fuiZ04SRXBHwrucXwak18ZiQWcl8qJf3hvz+Ld+cWHAmjaY0BXNpBn7Wgpeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=; b=trDaw6XqTf408BuOIUEcvHeRENSL1jupiDkooLayLkRei62CaSnagZ+tZUtciWPfGKRYGlPO78IAs/eZigklSulDz7bBetajmTQiJleHt6v5zRp3gmM0Owmlrdz/OCaljDgUqdYsJ3jnJd3haTD9TKgEuAzKd/p9rrHAEhttWnhbiEQGL4vuL+xn2Ia73jLxhIeGX3/4OeNZ26kSb9dTqEUlkONzBNKOXbExvVrFplvqKoPOgG/9eD+CvtrklEM4A5rVN8fLnDeK+ySCRsVnUe4e4NkysbV1VGTp9r0mcBXKMhsw9tcqiZtNDt9T2kcCWjBT5JBBVHBcd0+dxbcSIA== Received: from DM6PR03CA0080.namprd03.prod.outlook.com (2603:10b6:5:333::13) by BN9PR12MB5337.namprd12.prod.outlook.com (2603:10b6:408:102::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Wed, 3 Nov 2021 11:04:56 +0000 Received: from DM6NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:5:333:cafe::42) by DM6PR03CA0080.outlook.office365.com (2603:10b6:5:333::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 11:04:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT064.mail.protection.outlook.com (10.13.172.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 11:04:56 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 3 Nov 2021 11:04:54 +0000 From: To: CC: Thomas Monjalon Date: Wed, 3 Nov 2021 19:15:48 +0000 Message-ID: <20211103191554.16449-4-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211103191554.16449-1-eagostini@nvidia.com> References: <20210602203531.2288645-1-thomas@monjalon.net> <20211103191554.16449-1-eagostini@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 12e359f8-2aa3-4122-a6f3-08d99eb9c537 X-MS-TrafficTypeDiagnostic: BN9PR12MB5337: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qShQm7bxVsy1DE3qzgtLYx5o7jNry6VRYl5maPSXR8hMrkb2En8Czp3rFyomzgPCSL4q3me+FvYh9cZFtlzkXMm+Dl/fmQsn+YeKD+gFgiTIytZQDCtThJUrfUnWArLTbQCP6e2stuawIFea3MTlZSZLsXXt9bdU7z+Vc69M2aFHL2hAndpiRb3f0U4WhT0Cy5FKchQYQIWZS9Imx7WIlsrGpm/HDYF40G9mgS8fdrNF5gGbnbBWnXRFgpZ+LB8KVWuS7U6gXHYm3kycXh6J62T9jXx6PWIUfdPbcrjQKxBwKmzSr/wOovQ3VakZyBE5TL3rMl9Rcf0qKe7g0cdBkHQKxSs0fugoYzVkIdCUV0Jjsa8nJtgLfuX94+GHnjnVWxYR3U6ezbgLATKzNe2a8yghnbdnQyp2iReGRuHTiJY1pw+r10+wlw5wqc50W8RwhOEcigCVob6+KM6vna2DAgLBV2B/lHyDFdkR/pM/OtOxLktXv1XOxYaVVJ/GjYZP4n6t9Gaeu65AOXl2s8223pyB7+zEIjCrcNrszbXqLkbh2ZELf4Ve5vrtwWUBmnO+n5YD4p0PYR9hV8VyRgr/TlpjS71IQ6JUtiBcBW4SQuhpp4J+gwWaUTQQX4TRrNrKUKlGtXkRGaSmqubZ2EIImP6ObXeuWCL4rVRwbPUyT0eEafeUWPP4MKwKIVcoqmlUEkAYtxWtPl8fR8P4rQ9Dow== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(86362001)(6666004)(36756003)(82310400003)(2616005)(36906005)(7696005)(316002)(55016002)(7636003)(26005)(36860700001)(186003)(356005)(8936002)(16526019)(2876002)(6916009)(70586007)(47076005)(336012)(8676002)(83380400001)(6286002)(70206006)(426003)(508600001)(2906002)(4326008)(5660300002)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 11:04:56.3363 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 12e359f8-2aa3-4122-a6f3-08d99eb9c537 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5337 Subject: [dpdk-dev] [PATCH v4 3/9] gpudev: add child device representing a device context X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon The computing device may operate in some isolated contexts. Memory and processing are isolated in a silo represented by a child device. The context is provided as an opaque by the caller of rte_gpu_add_child(). Signed-off-by: Thomas Monjalon --- doc/guides/prog_guide/gpudev.rst | 12 ++++++ lib/gpudev/gpudev.c | 45 +++++++++++++++++++- lib/gpudev/gpudev_driver.h | 2 +- lib/gpudev/rte_gpudev.h | 71 +++++++++++++++++++++++++++++--- lib/gpudev/version.map | 1 + 5 files changed, 123 insertions(+), 8 deletions(-) diff --git a/doc/guides/prog_guide/gpudev.rst b/doc/guides/prog_guide/gpudev.rst index 6ea7239159..7694639489 100644 --- a/doc/guides/prog_guide/gpudev.rst +++ b/doc/guides/prog_guide/gpudev.rst @@ -34,3 +34,15 @@ This library provides a number of features: API Overview ------------ + +Child Device +~~~~~~~~~~~~ + +By default, DPDK PCIe module detects and registers physical GPU devices +in the system. +With the gpudev library is also possible to add additional non-physical devices +through an ``uint64_t`` generic handler (e.g. CUDA Driver context) +that will be registered internally by the driver as an additional device (child) +connected to a physical device (parent). +Each device (parent or child) is represented through a ID +required to indicate which device a given operation should be executed on. diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c index d57e23df7c..74cdd7f20b 100644 --- a/lib/gpudev/gpudev.c +++ b/lib/gpudev/gpudev.c @@ -80,13 +80,22 @@ rte_gpu_is_valid(int16_t dev_id) return false; } +static bool +gpu_match_parent(int16_t dev_id, int16_t parent) +{ + if (parent == RTE_GPU_ID_ANY) + return true; + return gpus[dev_id].info.parent == parent; +} + int16_t -rte_gpu_find_next(int16_t dev_id) +rte_gpu_find_next(int16_t dev_id, int16_t parent) { if (dev_id < 0) dev_id = 0; while (dev_id < gpu_max && - gpus[dev_id].state == RTE_GPU_STATE_UNUSED) + (gpus[dev_id].state == RTE_GPU_STATE_UNUSED || + !gpu_match_parent(dev_id, parent))) dev_id++; if (dev_id >= gpu_max) @@ -177,6 +186,7 @@ rte_gpu_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node = -1; + dev->info.parent = RTE_GPU_ID_NONE; TAILQ_INIT(&dev->callbacks); gpu_count++; @@ -185,6 +195,28 @@ rte_gpu_allocate(const char *name) return dev; } +int16_t +rte_gpu_add_child(const char *name, int16_t parent, uint64_t child_context) +{ + struct rte_gpu *dev; + + if (!rte_gpu_is_valid(parent)) { + GPU_LOG(ERR, "add child to invalid parent ID %d", parent); + rte_errno = ENODEV; + return -rte_errno; + } + + dev = rte_gpu_allocate(name); + if (dev == NULL) + return -rte_errno; + + dev->info.parent = parent; + dev->info.context = child_context; + + rte_gpu_complete_new(dev); + return dev->info.dev_id; +} + void rte_gpu_complete_new(struct rte_gpu *dev) { @@ -199,10 +231,19 @@ rte_gpu_complete_new(struct rte_gpu *dev) int rte_gpu_release(struct rte_gpu *dev) { + int16_t dev_id, child; + if (dev == NULL) { rte_errno = ENODEV; return -rte_errno; } + dev_id = dev->info.dev_id; + RTE_GPU_FOREACH_CHILD(child, dev_id) { + GPU_LOG(ERR, "cannot release device %d with child %d", + dev_id, child); + rte_errno = EBUSY; + return -rte_errno; + } GPU_LOG(DEBUG, "free device %s (id %d)", dev->info.name, dev->info.dev_id); diff --git a/lib/gpudev/gpudev_driver.h b/lib/gpudev/gpudev_driver.h index 2a7089aa52..4d0077161c 100644 --- a/lib/gpudev/gpudev_driver.h +++ b/lib/gpudev/gpudev_driver.h @@ -31,7 +31,7 @@ typedef int (rte_gpu_info_get_t)(struct rte_gpu *dev, struct rte_gpu_info *info) struct rte_gpu_ops { /* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get; - /* Close device. */ + /* Close device or child context. */ rte_gpu_close_t *dev_close; }; diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h index e1702fbfe4..df75dbdbab 100644 --- a/lib/gpudev/rte_gpudev.h +++ b/lib/gpudev/rte_gpudev.h @@ -41,8 +41,12 @@ extern "C" { struct rte_gpu_info { /** Unique identifier name. */ const char *name; + /** Opaque handler of the device context. */ + uint64_t context; /** Device ID. */ int16_t dev_id; + /** ID of the parent device, RTE_GPU_ID_NONE if no parent */ + int16_t parent; /** Total processors available on device. */ uint32_t processor_count; /** Total memory available on device. */ @@ -110,6 +114,33 @@ uint16_t rte_gpu_count_avail(void); __rte_experimental bool rte_gpu_is_valid(int16_t dev_id); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Create a virtual device representing a context in the parent device. + * + * @param name + * Unique string to identify the device. + * @param parent + * Device ID of the parent. + * @param child_context + * Opaque context handler. + * + * @return + * Device ID of the new created child, -rte_errno otherwise: + * - EINVAL if empty name + * - ENAMETOOLONG if long name + * - EEXIST if existing device name + * - ENODEV if invalid parent + * - EPERM if secondary process + * - ENOENT if too many devices + * - ENOMEM if out of space + */ +__rte_experimental +int16_t rte_gpu_add_child(const char *name, + int16_t parent, uint64_t child_context); + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. @@ -118,13 +149,17 @@ bool rte_gpu_is_valid(int16_t dev_id); * * @param dev_id * The initial device ID to start the research. + * @param parent + * The device ID of the parent. + * RTE_GPU_ID_NONE means no parent. + * RTE_GPU_ID_ANY means no or any parent. * * @return * Next device ID corresponding to a valid and initialized computing device, * RTE_GPU_ID_NONE if there is none. */ __rte_experimental -int16_t rte_gpu_find_next(int16_t dev_id); +int16_t rte_gpu_find_next(int16_t dev_id, int16_t parent); /** * @warning @@ -136,15 +171,41 @@ int16_t rte_gpu_find_next(int16_t dev_id); * The ID of the next possible valid device, usually 0 to iterate all. */ #define RTE_GPU_FOREACH(dev_id) \ - for (dev_id = rte_gpu_find_next(0); \ - dev_id > 0; \ - dev_id = rte_gpu_find_next(dev_id + 1)) + RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_ANY) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Macro to iterate over all valid computing devices having no parent. + * + * @param dev_id + * The ID of the next possible valid device, usually 0 to iterate all. + */ +#define RTE_GPU_FOREACH_PARENT(dev_id) \ + RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_NONE) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Macro to iterate over all valid children of a computing device parent. + * + * @param dev_id + * The ID of the next possible valid device, usually 0 to iterate all. + * @param parent + * The device ID of the parent. + */ +#define RTE_GPU_FOREACH_CHILD(dev_id, parent) \ + for (dev_id = rte_gpu_find_next(0, parent); \ + dev_id >= 0; \ + dev_id = rte_gpu_find_next(dev_id + 1, parent)) /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. * - * Close device. + * Close device or child context. * All resources are released. * * @param dev_id diff --git a/lib/gpudev/version.map b/lib/gpudev/version.map index b3b6b76c1c..4a934ed933 100644 --- a/lib/gpudev/version.map +++ b/lib/gpudev/version.map @@ -2,6 +2,7 @@ EXPERIMENTAL { global: # added in 21.11 + rte_gpu_add_child; rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close; -- 2.17.1