From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id EC284A0C53;
Wed, 3 Nov 2021 12:05:16 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id A306F41233;
Wed, 3 Nov 2021 12:05:02 +0100 (CET)
Received: from NAM11-CO1-obe.outbound.protection.outlook.com
(mail-co1nam11on2085.outbound.protection.outlook.com [40.107.220.85])
by mails.dpdk.org (Postfix) with ESMTP id AE945411CE
for <dev@dpdk.org>; Wed, 3 Nov 2021 12:04:58 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=ViM6t+Cro1KjzIDWDbIkOwxboh+ooFv35QSLFF4QAfpYZdNDkBom41FNgbnQp0oQkGjWzA+spSUM4dqON4tH/KNWv0LQg6G3blxnY/GGWubNTs2HXy0UfdYv2fqMblRlwBK/GdyMchePs+Huz4o9zCVH5Py1z5mkvQjAz3JyBTX03pvrVVTOsSHb26zrZSUijf4fTBGsbbxSCq9pEB1+cXZQOHN4Ny1Gix09x6dml48FQgwDCKO2nBxw2tTEcRxhVoOboDodw774dMSenqMYYdhmIumqqTM8ujVcPQbjxaQeq6HCKmayPZSZpDsypHIedk+nUbhArTvr+p1fGEvnzw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=;
b=kO87SDqqndsL9lvRZWj91HTYzaxKbTtsuum3JNacwFmeC7NrVVsvtoh3/ToQn32XP3bamyahwGFSKTF5A7hmIdwfFTIOcAicMGBEtfzeUNZMLeyF5kYBru0ifOKQS+Y/6v/Nd0cU+m3Q1eBRnRrcmhxaGAG1x8j09NFFWOs9HGmYcylJ1NPWletuRZfbxScEkMWYlPyjLeo1FQEnsiwHWT86BbFzb6dBbSs4TJLjs/U6dktwDHtExvTBSh8RONQjVE66CWI8QyT6iXKRwMf033iEH9fuiZ04SRXBHwrucXwak18ZiQWcl8qJf3hvz+Ld+cWHAmjaY0BXNpBn7Wgpeg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com;
dmarc=pass (p=quarantine sp=quarantine pct=100) action=none
header.from=nvidia.com; dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
s=selector2;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=jU6qN/2cZBay1ysZuzS9t+JNTNnvUCIuXrt06SvDRt4=;
b=trDaw6XqTf408BuOIUEcvHeRENSL1jupiDkooLayLkRei62CaSnagZ+tZUtciWPfGKRYGlPO78IAs/eZigklSulDz7bBetajmTQiJleHt6v5zRp3gmM0Owmlrdz/OCaljDgUqdYsJ3jnJd3haTD9TKgEuAzKd/p9rrHAEhttWnhbiEQGL4vuL+xn2Ia73jLxhIeGX3/4OeNZ26kSb9dTqEUlkONzBNKOXbExvVrFplvqKoPOgG/9eD+CvtrklEM4A5rVN8fLnDeK+ySCRsVnUe4e4NkysbV1VGTp9r0mcBXKMhsw9tcqiZtNDt9T2kcCWjBT5JBBVHBcd0+dxbcSIA==
Received: from DM6PR03CA0080.namprd03.prod.outlook.com (2603:10b6:5:333::13)
by BN9PR12MB5337.namprd12.prod.outlook.com (2603:10b6:408:102::9) with
Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Wed, 3 Nov
2021 11:04:56 +0000
Received: from DM6NAM11FT064.eop-nam11.prod.protection.outlook.com
(2603:10b6:5:333:cafe::42) by DM6PR03CA0080.outlook.office365.com
(2603:10b6:5:333::13) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend
Transport; Wed, 3 Nov 2021 11:04:56 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34)
smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed)
header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
216.228.112.34 as permitted sender) receiver=protection.outlook.com;
client-ip=216.228.112.34; helo=mail.nvidia.com;
Received: from mail.nvidia.com (216.228.112.34) by
DM6NAM11FT064.mail.protection.outlook.com (10.13.172.234) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 11:04:56 +0000
Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com
(172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 3 Nov
2021 11:04:54 +0000
From: <eagostini@nvidia.com>
To: <dev@dpdk.org>
CC: Thomas Monjalon <thomas@monjalon.net>
Date: Wed, 3 Nov 2021 19:15:48 +0000
Message-ID: <20211103191554.16449-4-eagostini@nvidia.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20211103191554.16449-1-eagostini@nvidia.com>
References: <20210602203531.2288645-1-thomas@monjalon.net>
<20211103191554.16449-1-eagostini@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [172.20.187.5]
X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To
HQMAIL107.nvidia.com (172.20.187.13)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 12e359f8-2aa3-4122-a6f3-08d99eb9c537
X-MS-TrafficTypeDiagnostic: BN9PR12MB5337:
X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr
X-Microsoft-Antispam-PRVS: <BN9PR12MB5337404B948FDFCB5F8E4C77CD8C9@BN9PR12MB5337.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:6790;
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: qShQm7bxVsy1DE3qzgtLYx5o7jNry6VRYl5maPSXR8hMrkb2En8Czp3rFyomzgPCSL4q3me+FvYh9cZFtlzkXMm+Dl/fmQsn+YeKD+gFgiTIytZQDCtThJUrfUnWArLTbQCP6e2stuawIFea3MTlZSZLsXXt9bdU7z+Vc69M2aFHL2hAndpiRb3f0U4WhT0Cy5FKchQYQIWZS9Imx7WIlsrGpm/HDYF40G9mgS8fdrNF5gGbnbBWnXRFgpZ+LB8KVWuS7U6gXHYm3kycXh6J62T9jXx6PWIUfdPbcrjQKxBwKmzSr/wOovQ3VakZyBE5TL3rMl9Rcf0qKe7g0cdBkHQKxSs0fugoYzVkIdCUV0Jjsa8nJtgLfuX94+GHnjnVWxYR3U6ezbgLATKzNe2a8yghnbdnQyp2iReGRuHTiJY1pw+r10+wlw5wqc50W8RwhOEcigCVob6+KM6vna2DAgLBV2B/lHyDFdkR/pM/OtOxLktXv1XOxYaVVJ/GjYZP4n6t9Gaeu65AOXl2s8223pyB7+zEIjCrcNrszbXqLkbh2ZELf4Ve5vrtwWUBmnO+n5YD4p0PYR9hV8VyRgr/TlpjS71IQ6JUtiBcBW4SQuhpp4J+gwWaUTQQX4TRrNrKUKlGtXkRGaSmqubZ2EIImP6ObXeuWCL4rVRwbPUyT0eEafeUWPP4MKwKIVcoqmlUEkAYtxWtPl8fR8P4rQ9Dow==
X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE;
SFS:(4636009)(36840700001)(46966006)(86362001)(6666004)(36756003)(82310400003)(2616005)(36906005)(7696005)(316002)(55016002)(7636003)(26005)(36860700001)(186003)(356005)(8936002)(16526019)(2876002)(6916009)(70586007)(47076005)(336012)(8676002)(83380400001)(6286002)(70206006)(426003)(508600001)(2906002)(4326008)(5660300002)(1076003);
DIR:OUT; SFP:1101;
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 11:04:56.3363 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 12e359f8-2aa3-4122-a6f3-08d99eb9c537
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34];
Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT064.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5337
Subject: [dpdk-dev] [PATCH v4 3/9] gpudev: add child device representing a
device context
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
From: Thomas Monjalon <thomas@monjalon.net>
The computing device may operate in some isolated contexts.
Memory and processing are isolated in a silo represented by
a child device.
The context is provided as an opaque by the caller of
rte_gpu_add_child().
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
doc/guides/prog_guide/gpudev.rst | 12 ++++++
lib/gpudev/gpudev.c | 45 +++++++++++++++++++-
lib/gpudev/gpudev_driver.h | 2 +-
lib/gpudev/rte_gpudev.h | 71 +++++++++++++++++++++++++++++---
lib/gpudev/version.map | 1 +
5 files changed, 123 insertions(+), 8 deletions(-)
diff --git a/doc/guides/prog_guide/gpudev.rst b/doc/guides/prog_guide/gpudev.rst
index 6ea7239159..7694639489 100644
--- a/doc/guides/prog_guide/gpudev.rst
+++ b/doc/guides/prog_guide/gpudev.rst
@@ -34,3 +34,15 @@ This library provides a number of features:
API Overview
------------
+
+Child Device
+~~~~~~~~~~~~
+
+By default, DPDK PCIe module detects and registers physical GPU devices
+in the system.
+With the gpudev library is also possible to add additional non-physical devices
+through an ``uint64_t`` generic handler (e.g. CUDA Driver context)
+that will be registered internally by the driver as an additional device (child)
+connected to a physical device (parent).
+Each device (parent or child) is represented through a ID
+required to indicate which device a given operation should be executed on.
diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c
index d57e23df7c..74cdd7f20b 100644
--- a/lib/gpudev/gpudev.c
+++ b/lib/gpudev/gpudev.c
@@ -80,13 +80,22 @@ rte_gpu_is_valid(int16_t dev_id)
return false;
}
+static bool
+gpu_match_parent(int16_t dev_id, int16_t parent)
+{
+ if (parent == RTE_GPU_ID_ANY)
+ return true;
+ return gpus[dev_id].info.parent == parent;
+}
+
int16_t
-rte_gpu_find_next(int16_t dev_id)
+rte_gpu_find_next(int16_t dev_id, int16_t parent)
{
if (dev_id < 0)
dev_id = 0;
while (dev_id < gpu_max &&
- gpus[dev_id].state == RTE_GPU_STATE_UNUSED)
+ (gpus[dev_id].state == RTE_GPU_STATE_UNUSED ||
+ !gpu_match_parent(dev_id, parent)))
dev_id++;
if (dev_id >= gpu_max)
@@ -177,6 +186,7 @@ rte_gpu_allocate(const char *name)
dev->info.name = dev->name;
dev->info.dev_id = dev_id;
dev->info.numa_node = -1;
+ dev->info.parent = RTE_GPU_ID_NONE;
TAILQ_INIT(&dev->callbacks);
gpu_count++;
@@ -185,6 +195,28 @@ rte_gpu_allocate(const char *name)
return dev;
}
+int16_t
+rte_gpu_add_child(const char *name, int16_t parent, uint64_t child_context)
+{
+ struct rte_gpu *dev;
+
+ if (!rte_gpu_is_valid(parent)) {
+ GPU_LOG(ERR, "add child to invalid parent ID %d", parent);
+ rte_errno = ENODEV;
+ return -rte_errno;
+ }
+
+ dev = rte_gpu_allocate(name);
+ if (dev == NULL)
+ return -rte_errno;
+
+ dev->info.parent = parent;
+ dev->info.context = child_context;
+
+ rte_gpu_complete_new(dev);
+ return dev->info.dev_id;
+}
+
void
rte_gpu_complete_new(struct rte_gpu *dev)
{
@@ -199,10 +231,19 @@ rte_gpu_complete_new(struct rte_gpu *dev)
int
rte_gpu_release(struct rte_gpu *dev)
{
+ int16_t dev_id, child;
+
if (dev == NULL) {
rte_errno = ENODEV;
return -rte_errno;
}
+ dev_id = dev->info.dev_id;
+ RTE_GPU_FOREACH_CHILD(child, dev_id) {
+ GPU_LOG(ERR, "cannot release device %d with child %d",
+ dev_id, child);
+ rte_errno = EBUSY;
+ return -rte_errno;
+ }
GPU_LOG(DEBUG, "free device %s (id %d)",
dev->info.name, dev->info.dev_id);
diff --git a/lib/gpudev/gpudev_driver.h b/lib/gpudev/gpudev_driver.h
index 2a7089aa52..4d0077161c 100644
--- a/lib/gpudev/gpudev_driver.h
+++ b/lib/gpudev/gpudev_driver.h
@@ -31,7 +31,7 @@ typedef int (rte_gpu_info_get_t)(struct rte_gpu *dev, struct rte_gpu_info *info)
struct rte_gpu_ops {
/* Get device info. If NULL, info is just copied. */
rte_gpu_info_get_t *dev_info_get;
- /* Close device. */
+ /* Close device or child context. */
rte_gpu_close_t *dev_close;
};
diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h
index e1702fbfe4..df75dbdbab 100644
--- a/lib/gpudev/rte_gpudev.h
+++ b/lib/gpudev/rte_gpudev.h
@@ -41,8 +41,12 @@ extern "C" {
struct rte_gpu_info {
/** Unique identifier name. */
const char *name;
+ /** Opaque handler of the device context. */
+ uint64_t context;
/** Device ID. */
int16_t dev_id;
+ /** ID of the parent device, RTE_GPU_ID_NONE if no parent */
+ int16_t parent;
/** Total processors available on device. */
uint32_t processor_count;
/** Total memory available on device. */
@@ -110,6 +114,33 @@ uint16_t rte_gpu_count_avail(void);
__rte_experimental
bool rte_gpu_is_valid(int16_t dev_id);
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Create a virtual device representing a context in the parent device.
+ *
+ * @param name
+ * Unique string to identify the device.
+ * @param parent
+ * Device ID of the parent.
+ * @param child_context
+ * Opaque context handler.
+ *
+ * @return
+ * Device ID of the new created child, -rte_errno otherwise:
+ * - EINVAL if empty name
+ * - ENAMETOOLONG if long name
+ * - EEXIST if existing device name
+ * - ENODEV if invalid parent
+ * - EPERM if secondary process
+ * - ENOENT if too many devices
+ * - ENOMEM if out of space
+ */
+__rte_experimental
+int16_t rte_gpu_add_child(const char *name,
+ int16_t parent, uint64_t child_context);
+
/**
* @warning
* @b EXPERIMENTAL: this API may change without prior notice.
@@ -118,13 +149,17 @@ bool rte_gpu_is_valid(int16_t dev_id);
*
* @param dev_id
* The initial device ID to start the research.
+ * @param parent
+ * The device ID of the parent.
+ * RTE_GPU_ID_NONE means no parent.
+ * RTE_GPU_ID_ANY means no or any parent.
*
* @return
* Next device ID corresponding to a valid and initialized computing device,
* RTE_GPU_ID_NONE if there is none.
*/
__rte_experimental
-int16_t rte_gpu_find_next(int16_t dev_id);
+int16_t rte_gpu_find_next(int16_t dev_id, int16_t parent);
/**
* @warning
@@ -136,15 +171,41 @@ int16_t rte_gpu_find_next(int16_t dev_id);
* The ID of the next possible valid device, usually 0 to iterate all.
*/
#define RTE_GPU_FOREACH(dev_id) \
- for (dev_id = rte_gpu_find_next(0); \
- dev_id > 0; \
- dev_id = rte_gpu_find_next(dev_id + 1))
+ RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_ANY)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Macro to iterate over all valid computing devices having no parent.
+ *
+ * @param dev_id
+ * The ID of the next possible valid device, usually 0 to iterate all.
+ */
+#define RTE_GPU_FOREACH_PARENT(dev_id) \
+ RTE_GPU_FOREACH_CHILD(dev_id, RTE_GPU_ID_NONE)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Macro to iterate over all valid children of a computing device parent.
+ *
+ * @param dev_id
+ * The ID of the next possible valid device, usually 0 to iterate all.
+ * @param parent
+ * The device ID of the parent.
+ */
+#define RTE_GPU_FOREACH_CHILD(dev_id, parent) \
+ for (dev_id = rte_gpu_find_next(0, parent); \
+ dev_id >= 0; \
+ dev_id = rte_gpu_find_next(dev_id + 1, parent))
/**
* @warning
* @b EXPERIMENTAL: this API may change without prior notice.
*
- * Close device.
+ * Close device or child context.
* All resources are released.
*
* @param dev_id
diff --git a/lib/gpudev/version.map b/lib/gpudev/version.map
index b3b6b76c1c..4a934ed933 100644
--- a/lib/gpudev/version.map
+++ b/lib/gpudev/version.map
@@ -2,6 +2,7 @@ EXPERIMENTAL {
global:
# added in 21.11
+ rte_gpu_add_child;
rte_gpu_callback_register;
rte_gpu_callback_unregister;
rte_gpu_close;
--
2.17.1