From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 86B8AA0C43; Fri, 8 Oct 2021 19:44:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2E7CF41121; Fri, 8 Oct 2021 19:43:46 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2054.outbound.protection.outlook.com [40.107.237.54]) by mails.dpdk.org (Postfix) with ESMTP id 4FD42407FF for ; Fri, 8 Oct 2021 19:43:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Sln2hejquHLCf4AG1kzj+sqYgMOzmqMJDrw8zE24e82+qJq6KVBOSOg1nciJj5OGSy1vZOZ2EfxOiTzPtJ7H66n/TlDqyR/lV3fmAF2Y2f2m3waoeegguw9uDRyuRpSz2nY6sBa7Yrn3+jZop9AuxWOV/hInv7gW0GyQcsSCyLM0cE4glfjyRJ4PpUgi4AvTBb5oKKSA2Q1tudupJfBMxP8QM/JA073JlOS0IRzKD/ojMNh24uXbO5zhQEwkaQUySauEiiEXECa0ZOowEMvZzN60WAFSKzi7uruzxtKgQOmNUyfNQHIF9Oq2p7wxURxDIF0cNq5lFgL5fubCv+Unfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2TJJfWuag8aNaZVVL61bL7JJLJN39ujkKZD8whqRu/U=; b=EF4OZ0t9lJkNVVpRv3bS4EvkXKNOniIGgSUexlHV+icd0fhFEyUzQ0layBmRXICFdCz8YqRsh5hmWQ6EkZJk7U5YArwNiFlx8Xj9AwviPQNWCzV2wekfO+7PjSfqSX5QvfX3NYkpL5YfJ1pvQeMlto6xc87/6IKpXXL+r4eKQ644mkS+YEU9eKp/sbaOwC8lBciAsXEo0/JqtyrznYq198txaEhk6D8b7c6q+BVCV2WLflGtJFX9qmjiT3L+HeHMCklxnd+5HlUgrMEVUEG/pTWCC2li0o6KFKcnkrJkw5UxpPjqiWakZziiwJcyFQx+8FyIjN8cJ/VobOEvnhK00g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2TJJfWuag8aNaZVVL61bL7JJLJN39ujkKZD8whqRu/U=; b=X2XLty+PQe1/BQDZDKnYjnNOK0nD2xXhYDRDkqjr0Qrnl4lOUQZnpAZ5h9LS3Os6vjhxZePKY139DGxJhd9iJKUUHBvLz6YBcIKMMlc4ffsLu0hdY9jLNsXrbbnunLs8siMrkfv9yYhpfUkJMdeM1dSr0/ETs0pRoWRJdp+bA9HoKbBg6HZqUAextIFsqQ2sLaFpV9xjDmqzqpoPKmwqMWfD7qWLQ+d5K8vB6gPLW0v+iuT82UJKVsjLKdcuDaW0AWvXcarvxGO1/TNjTsxNOO/BzFpjvN51IaYhfw4uHVW2UOGz0egxYb6A+KVzxq2OBVDIQPgq7QZr7Vr9ngjpbg== Received: from MW4PR03CA0126.namprd03.prod.outlook.com (2603:10b6:303:8c::11) by SN1PR12MB2399.namprd12.prod.outlook.com (2603:10b6:802:2b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.22; Fri, 8 Oct 2021 17:43:37 +0000 Received: from CO1NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8c:cafe::21) by MW4PR03CA0126.outlook.office365.com (2603:10b6:303:8c::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.19 via Frontend Transport; Fri, 8 Oct 2021 17:43:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT047.mail.protection.outlook.com (10.13.174.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4587.18 via Frontend Transport; Fri, 8 Oct 2021 17:43:37 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 8 Oct 2021 17:43:32 +0000 From: To: CC: Thomas Monjalon Date: Sat, 9 Oct 2021 01:53:42 +0000 Message-ID: <20211009015349.9694-3-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211009015349.9694-1-eagostini@nvidia.com> References: <20210602203531.2288645-1-thomas@monjalon.net> <20211009015349.9694-1-eagostini@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 25ee5b00-58f6-4689-2a63-08d98a832873 X-MS-TrafficTypeDiagnostic: SN1PR12MB2399: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: p0JBoTP2vJ9LjLHM8gHqoQ02ryEeoI5ffFf+3NN5wu1Yy67rnPzm0y+pkO9T9FF3I0wCTdhmWk9shtAe0RxEhwQASED7obuevSPkS/MqA1NcsJ6Ck3cjNxxzMvd3L7asLycl9A0XhMs/lRvbArO5TjxVkC/8ft7CgnJJ356QvbJUSFtJW59KyCyZRwIV5gbwBf9aedWxi0qiIp/BYDDiuvKIHZNO6H6VIBDJOMCYH63UwcMpiqvsTwxGrBPRWW8qtzpEz+idMB9raPxNFv+0dBg/tMlwvdcUo/Ga7mAh5fjk8yYGTuCk3g9t08QJkgShsR8o9V9GsRORG5ATPqCNlui7Y+2zNJefSlqIvEl2wIQhjsCkdbZoW/BtE1wJZ5aipALHsUIGSVcJhiPsuUxffZkNtgMdehQVFJ1/lgpimXYeq15EW3erBnnnH+npLRZVGV49V+yGakWc5YD8ukUrhUSlf2ejeCQEkVZLu2mZ2eSACv8Y5yu8jJNe5HBkdKilwXTKy+RD5pcAxpIAQF+RAhK6OuVQZk1VkNhnbKyYIvTRqJUzCAIeAzcU0wWpC2UXdFF0SSdOsaTlY1KEcu+Ny1U1150/pxyUw8LMSDLOgEXRmEKEzfS7HnBtKbMzL0LP0fJ2iDzuuc2qZe/n36oFnv0Sf0kmVqrt24hjn8mFH1zu+thNE6qu7ppucekOOBnZz/Y0ZpPK76kc+KnoyHEXqg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(16526019)(26005)(83380400001)(8676002)(4326008)(7636003)(186003)(55016002)(70206006)(70586007)(508600001)(6286002)(356005)(47076005)(2616005)(2876002)(1076003)(86362001)(7696005)(336012)(82310400003)(8936002)(36860700001)(36756003)(316002)(5660300002)(6666004)(426003)(15650500001)(2906002)(6916009); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Oct 2021 17:43:37.2461 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 25ee5b00-58f6-4689-2a63-08d98a832873 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2399 Subject: [dpdk-dev] [PATCH v3 2/9] gpudev: add event notification X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon Callback functions may be registered for a device event. Callback management is per-process and not thread-safe. The events RTE_GPU_EVENT_NEW and RTE_GPU_EVENT_DEL are notified respectively after creation and before removal of a device, as part of the library functions. Some future events may be emitted from drivers. Signed-off-by: Thomas Monjalon --- lib/gpudev/gpudev.c | 148 +++++++++++++++++++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 7 ++ lib/gpudev/rte_gpudev.h | 70 ++++++++++++++++++ lib/gpudev/version.map | 3 + 4 files changed, 228 insertions(+) diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c index c839c530c8..d57e23df7c 100644 --- a/lib/gpudev/gpudev.c +++ b/lib/gpudev/gpudev.c @@ -3,6 +3,7 @@ */ #include +#include #include #include #include @@ -27,6 +28,16 @@ static int16_t gpu_max; /* Number of currently valid devices */ static int16_t gpu_count; +/* Event callback object */ +struct rte_gpu_callback { + TAILQ_ENTRY(rte_gpu_callback) next; + rte_gpu_callback_t *function; + void *user_data; + enum rte_gpu_event event; +}; +static rte_rwlock_t gpu_callback_lock = RTE_RWLOCK_INITIALIZER; +static void gpu_free_callbacks(struct rte_gpu *dev); + int rte_gpu_init(size_t dev_max) { @@ -166,6 +177,7 @@ rte_gpu_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node = -1; + TAILQ_INIT(&dev->callbacks); gpu_count++; GPU_LOG(DEBUG, "new device %s (id %d) of total %d", @@ -180,6 +192,8 @@ rte_gpu_complete_new(struct rte_gpu *dev) return; dev->state = RTE_GPU_STATE_INITIALIZED; + dev->state = RTE_GPU_STATE_INITIALIZED; + rte_gpu_notify(dev, RTE_GPU_EVENT_NEW); } int @@ -192,6 +206,9 @@ rte_gpu_release(struct rte_gpu *dev) GPU_LOG(DEBUG, "free device %s (id %d)", dev->info.name, dev->info.dev_id); + rte_gpu_notify(dev, RTE_GPU_EVENT_DEL); + + gpu_free_callbacks(dev); dev->state = RTE_GPU_STATE_UNUSED; gpu_count--; @@ -224,6 +241,137 @@ rte_gpu_close(int16_t dev_id) return firsterr; } +int +rte_gpu_callback_register(int16_t dev_id, enum rte_gpu_event event, + rte_gpu_callback_t *function, void *user_data) +{ + int16_t next_dev, last_dev; + struct rte_gpu_callback_list *callbacks; + struct rte_gpu_callback *callback; + + if (!rte_gpu_is_valid(dev_id) && dev_id != RTE_GPU_ID_ANY) { + GPU_LOG(ERR, "register callback of invalid ID %d", dev_id); + rte_errno = ENODEV; + return -rte_errno; + } + if (function == NULL) { + GPU_LOG(ERR, "cannot register callback without function"); + rte_errno = EINVAL; + return -rte_errno; + } + + if (dev_id == RTE_GPU_ID_ANY) { + next_dev = 0; + last_dev = gpu_max - 1; + } else { + next_dev = last_dev = dev_id; + } + + rte_rwlock_write_lock(&gpu_callback_lock); + do { + callbacks = &gpus[next_dev].callbacks; + + /* check if not already registered */ + TAILQ_FOREACH(callback, callbacks, next) { + if (callback->event == event && + callback->function == function && + callback->user_data == user_data) { + GPU_LOG(INFO, "callback already registered"); + return 0; + } + } + + callback = malloc(sizeof(*callback)); + if (callback == NULL) { + GPU_LOG(ERR, "cannot allocate callback"); + return -ENOMEM; + } + callback->function = function; + callback->user_data = user_data; + callback->event = event; + TAILQ_INSERT_TAIL(callbacks, callback, next); + + } while (++next_dev <= last_dev); + rte_rwlock_write_unlock(&gpu_callback_lock); + + return 0; +} + +int +rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event, + rte_gpu_callback_t *function, void *user_data) +{ + int16_t next_dev, last_dev; + struct rte_gpu_callback_list *callbacks; + struct rte_gpu_callback *callback, *nextcb; + + if (!rte_gpu_is_valid(dev_id) && dev_id != RTE_GPU_ID_ANY) { + GPU_LOG(ERR, "unregister callback of invalid ID %d", dev_id); + rte_errno = ENODEV; + return -rte_errno; + } + if (function == NULL) { + GPU_LOG(ERR, "cannot unregister callback without function"); + rte_errno = EINVAL; + return -rte_errno; + } + + if (dev_id == RTE_GPU_ID_ANY) { + next_dev = 0; + last_dev = gpu_max - 1; + } else { + next_dev = last_dev = dev_id; + } + + rte_rwlock_write_lock(&gpu_callback_lock); + do { + callbacks = &gpus[next_dev].callbacks; + RTE_TAILQ_FOREACH_SAFE(callback, callbacks, next, nextcb) { + if (callback->event != event || + callback->function != function || + (callback->user_data != user_data && + user_data != (void *)-1)) + continue; + TAILQ_REMOVE(callbacks, callback, next); + free(callback); + } + } while (++next_dev <= last_dev); + rte_rwlock_write_unlock(&gpu_callback_lock); + + return 0; +} + +static void +gpu_free_callbacks(struct rte_gpu *dev) +{ + struct rte_gpu_callback_list *callbacks; + struct rte_gpu_callback *callback, *nextcb; + + callbacks = &dev->callbacks; + rte_rwlock_write_lock(&gpu_callback_lock); + RTE_TAILQ_FOREACH_SAFE(callback, callbacks, next, nextcb) { + TAILQ_REMOVE(callbacks, callback, next); + free(callback); + } + rte_rwlock_write_unlock(&gpu_callback_lock); +} + +void +rte_gpu_notify(struct rte_gpu *dev, enum rte_gpu_event event) +{ + int16_t dev_id; + struct rte_gpu_callback *callback; + + dev_id = dev->info.dev_id; + rte_rwlock_read_lock(&gpu_callback_lock); + TAILQ_FOREACH(callback, &dev->callbacks, next) { + if (callback->event != event || callback->function == NULL) + continue; + callback->function(dev_id, event, callback->user_data); + } + rte_rwlock_read_unlock(&gpu_callback_lock); +} + int rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info) { diff --git a/lib/gpudev/gpudev_driver.h b/lib/gpudev/gpudev_driver.h index 9e096e3b64..2a7089aa52 100644 --- a/lib/gpudev/gpudev_driver.h +++ b/lib/gpudev/gpudev_driver.h @@ -12,6 +12,7 @@ #define RTE_GPUDEV_DRIVER_H #include +#include #include @@ -43,6 +44,8 @@ struct rte_gpu { struct rte_gpu_info info; /* Driver functions. */ struct rte_gpu_ops ops; + /* Event callback list. */ + TAILQ_HEAD(rte_gpu_callback_list, rte_gpu_callback) callbacks; /* Current state (used or not) in the running process. */ enum rte_gpu_state state; /* Updated by this library. */ /* Driver-specific private data for the running process. */ @@ -64,4 +67,8 @@ void rte_gpu_complete_new(struct rte_gpu *dev); __rte_internal int rte_gpu_release(struct rte_gpu *dev); +/* Call registered callbacks. No multi-process event. */ +__rte_internal +void rte_gpu_notify(struct rte_gpu *dev, enum rte_gpu_event); + #endif /* RTE_GPUDEV_DRIVER_H */ diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h index eb7cfa8c59..e1702fbfe4 100644 --- a/lib/gpudev/rte_gpudev.h +++ b/lib/gpudev/rte_gpudev.h @@ -31,6 +31,11 @@ extern "C" { /** Empty device ID. */ #define RTE_GPU_ID_NONE -1 +/** Catch-all device ID. */ +#define RTE_GPU_ID_ANY INT16_MIN + +/** Catch-all callback data. */ +#define RTE_GPU_CALLBACK_ANY_DATA ((void *)-1) /** Store device info. */ struct rte_gpu_info { @@ -46,6 +51,18 @@ struct rte_gpu_info { int16_t numa_node; }; +/** Flags passed in notification callback. */ +enum rte_gpu_event { + /** Device is just initialized. */ + RTE_GPU_EVENT_NEW, + /** Device is going to be released. */ + RTE_GPU_EVENT_DEL, +}; + +/** Prototype of event callback function. */ +typedef void (rte_gpu_callback_t)(int16_t dev_id, + enum rte_gpu_event event, void *user_data); + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. @@ -141,6 +158,59 @@ int16_t rte_gpu_find_next(int16_t dev_id); __rte_experimental int rte_gpu_close(int16_t dev_id); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Register a function as event callback. + * A function may be registered multiple times for different events. + * + * @param dev_id + * Device ID to get notified about. + * RTE_GPU_ID_ANY means all devices. + * @param event + * Device event to be registered for. + * @param function + * Callback function to be called on event. + * @param user_data + * Optional parameter passed in the callback. + * + * @return + * 0 on success, -rte_errno otherwise: + * - ENODEV if invalid dev_id + * - EINVAL if NULL function + * - ENOMEM if out of memory + */ +__rte_experimental +int rte_gpu_callback_register(int16_t dev_id, enum rte_gpu_event event, + rte_gpu_callback_t *function, void *user_data); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Unregister for an event. + * + * @param dev_id + * Device ID to be silenced. + * RTE_GPU_ID_ANY means all devices. + * @param event + * Registered event. + * @param function + * Registered function. + * @param user_data + * Optional parameter as registered. + * RTE_GPU_CALLBACK_ANY_DATA is a catch-all. + * + * @return + * 0 on success, -rte_errno otherwise: + * - ENODEV if invalid dev_id + * - EINVAL if NULL function + */ +__rte_experimental +int rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event, + rte_gpu_callback_t *function, void *user_data); + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. diff --git a/lib/gpudev/version.map b/lib/gpudev/version.map index 6ac6b327e2..b3b6b76c1c 100644 --- a/lib/gpudev/version.map +++ b/lib/gpudev/version.map @@ -2,6 +2,8 @@ EXPERIMENTAL { global: # added in 21.11 + rte_gpu_callback_register; + rte_gpu_callback_unregister; rte_gpu_close; rte_gpu_count_avail; rte_gpu_find_next; @@ -16,5 +18,6 @@ INTERNAL { rte_gpu_allocate; rte_gpu_complete_new; rte_gpu_get_by_name; + rte_gpu_notify; rte_gpu_release; }; -- 2.17.1