From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE313A0C4D; Mon, 8 Nov 2021 11:48:01 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D30141157; Mon, 8 Nov 2021 11:47:14 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08on2073.outbound.protection.outlook.com [40.107.101.73]) by mails.dpdk.org (Postfix) with ESMTP id 3F42941145 for ; Mon, 8 Nov 2021 11:47:11 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ExNYGSF1RAI3zzlfXA8bFdU9M7unX4s1p+TGwMEQDHJLg0mI0XrA+ZIdWdDGL3wsN5bZJWsOMwOP3Nfw2NdJZgWRyYRLnr21S4OnxIr+aGRjRdGQ79qOxO2HEt2K/vHgs+ECOwlQcZFNkzQA6EtuE6/lQ1GDwV5QoNpcb0oPSJoJveGFxllzs+31Z3aDTn+7vkxT2tqRxogwOQvEXtzzoYxzHRJqEXLaBBPfwuQZ8jAm/Q8fJxlpn6T1cIH7sjQkCeyE7glAJH90g2pjAMxDxQHtAYaQKSH7DfUTE3VDTq4jjuZLJWSioewna1r0yiFqrrwwvMMMoI8qTP/nbeIxlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qCVTJrRpKsMxoI4JKFRbIYu48qyr6fMfernRBPvHMIU=; b=Exp9LZkQisvHbxD9LtFbXyNQt1BDCeTfPlyqriP0gSkMjKBVVCQzNVxcia3qMQAEYy902B5HLqL+dr5iCJ0pwEGCxLguRfUIWw0WINof+R4LgtB8Y2YFYoeCiuM8WWuy1hLTYXCMz4xdyVprTrJ/gFmfMg5xkh2qitw5wb7wLgfa7bgjG6A2mpgK9qunDmhrR7sN5V9lKcZgHzsg/vd8f+M8YYKgSiV1zacNOJf9gY6ZwtvdzvXHg2Mo3+PGcm2mGvnmNikE2KNvJ/uh39tMiAePnUgmP7kkwHJJlVGrPU4QRMPskfGyeSdJkqvozspt0/zVy+Q5W8sMb5ZZ9e2cpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qCVTJrRpKsMxoI4JKFRbIYu48qyr6fMfernRBPvHMIU=; b=T7YYsdtuMh117700QOWrr8zniGf08oy6TTN0Z8ex93nwm5SluodH3TZbC0cDRJ0l4+Qj/H6WRci3faJIql4tPuxAXz1jXe+Su2zf7mMLxdDNoP6eKcvYau/iY31fH41CfHpufQqz/A2Ij2i22JJUfZn6+RPq7dUAjm/vk3/CAFYMSnwlQPTrq4YVkEncZX+ICFrGNftG99o8Efy0+UeHDAi9olmStPQUuPpNecrKqQKkUaxrIQI39W4u/OVINstzmypsoXmvjyVPTGKXSiZ8Nqs5ShMOUrPafQMW1PtfFgBcTl759V6dI2K5p+jFH3nB1BJQWXaJCzUzuXycJ8d1uA== Received: from MW2PR16CA0042.namprd16.prod.outlook.com (2603:10b6:907:1::19) by MN2PR12MB4301.namprd12.prod.outlook.com (2603:10b6:208:1d4::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Mon, 8 Nov 2021 10:47:09 +0000 Received: from CO1NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:907:1:cafe::eb) by MW2PR16CA0042.outlook.office365.com (2603:10b6:907:1::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Mon, 8 Nov 2021 10:47:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT064.mail.protection.outlook.com (10.13.175.77) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Mon, 8 Nov 2021 10:47:08 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 8 Nov 2021 10:47:01 +0000 From: To: CC: Elena Agostini Date: Mon, 8 Nov 2021 18:58:03 +0000 Message-ID: <20211108185805.3887-8-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211108185805.3887-1-eagostini@nvidia.com> References: <20210602203531.2288645-1-thomas@monjalon.net> <20211108185805.3887-1-eagostini@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0c5cf90a-0028-4ef8-20b6-08d9a2a51cef X-MS-TrafficTypeDiagnostic: MN2PR12MB4301: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:117; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YYTpV2O/rtsL4R9u8e0B57FzXMOqwGyGrfIe3K+B8OCJu4krxmRHS3cCbM3gWKhkqc3mgcyM4Yw8QPvUYbEIFnKFjI4i+Kjxk4QCsC3926cVFXSEUCk+TN+cvtR4xJgFGRTKg4A5+2ySsHlffx9ntIjjthg6xjk/spz0jdMIq9jX31WVNuuZmEUkhA023g2/8Fsjt7xngV/nSC2FNEstm7HmzczMY3s8MDHG5PsywCo9cGm+BqjwFMxreFvX0g8oJxpXxhY5LSxUnKRQXC76FJUdOuuMC8JuYz62DeayXI9M9mpLh/Oi4MYUoT3fgsdkGjiv0sEam+mqmR7Pwkb0Plp3A5Cvld4FLGNCXZSte+42QJSSizVPfOhC7iQqTDpFS5Miu+fYxAhRFUOHfl2D2YRopXxpwoxV471g3MMvcntosEQTceBrXkV/SZM7brUSHdskm7Ck+HOAuXc4KVYhtwjAzwC8UL+c0A1WYHqZQt4I2qkuBGNZBWliU7uxEM/WFcAUfHhk4s9cA9KUZFcjYkskIjULCgvIzHUsSXqMTR68H2jG93UvREqk5Deqo7yo++wOVC+5jVh5rltOurDJn3VI0NcifgTgSvue1gevJV9R2+HTr7QVsecWxZ7Sk/ij2gZugfAWLSUjXc9RYU360ouoLChww03FEeByUhHxGk8y208/GlF6rqRduBQafgkVEFI2PoqkX1meKNBSFrdBVg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(30864003)(6916009)(5660300002)(2876002)(1076003)(36756003)(36860700001)(107886003)(83380400001)(426003)(2906002)(2616005)(26005)(356005)(7636003)(55016002)(47076005)(6286002)(86362001)(336012)(508600001)(8936002)(4326008)(70586007)(186003)(316002)(8676002)(7696005)(16526019)(70206006)(82310400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2021 10:47:08.7353 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0c5cf90a-0028-4ef8-20b6-08d9a2a51cef X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4301 Subject: [dpdk-dev] [PATCH v5 7/9] gpudev: add communication flag X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Elena Agostini In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations. The purpose of this flag is to allow the CPU and the GPU to exchange ACKs. A possible use-case is described below. CPU: - Trigger some task on the GPU - Prepare some data - Signal to the GPU the data is ready updating the communication flag GPU: - Do some pre-processing - Wait for more data from the CPU polling on the communication flag - Consume the data prepared by the CPU Signed-off-by: Elena Agostini --- app/test-gpudev/main.c | 60 ++++++++++++++ doc/guides/prog_guide/gpudev.rst | 13 +++ doc/guides/rel_notes/release_21_11.rst | 1 + lib/gpudev/gpudev.c | 92 +++++++++++++++++++++ lib/gpudev/rte_gpudev.h | 108 +++++++++++++++++++++++++ lib/gpudev/version.map | 4 + 6 files changed, 278 insertions(+) diff --git a/app/test-gpudev/main.c b/app/test-gpudev/main.c index e3aca2225a..516a01b927 100644 --- a/app/test-gpudev/main.c +++ b/app/test-gpudev/main.c @@ -154,6 +154,61 @@ register_cpu_memory(uint16_t gpu_id) return 0; } +static int +create_update_comm_flag(uint16_t gpu_id) +{ + struct rte_gpu_comm_flag devflag; + int ret = 0; + uint32_t set_val; + uint32_t get_val; + + printf("\n=======> TEST: Communication flag\n"); + + ret = rte_gpu_comm_create_flag(gpu_id, &devflag, RTE_GPU_COMM_FLAG_CPU); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_create_flag returned error %d\n", ret); + return -1; + } + + set_val = 25; + ret = rte_gpu_comm_set_flag(&devflag, set_val); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_set_flag returned error %d\n", ret); + return -1; + } + + ret = rte_gpu_comm_get_flag_value(&devflag, &get_val); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_get_flag_value returned error %d\n", ret); + return -1; + } + + printf("Communication flag value at 0x%p was set to %d and current value is %d\n", devflag.ptr, set_val, get_val); + + set_val = 38; + ret = rte_gpu_comm_set_flag(&devflag, set_val); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_set_flag returned error %d\n", ret); + return -1; + } + + ret = rte_gpu_comm_get_flag_value(&devflag, &get_val); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_get_flag_value returned error %d\n", ret); + return -1; + } + + printf("Communication flag value at 0x%p was set to %d and current value is %d\n", devflag.ptr, set_val, get_val); + + ret = rte_gpu_comm_destroy_flag(&devflag); + if (ret < 0) { + fprintf(stderr, "rte_gpu_comm_destroy_flags returned error %d\n", ret); + return -1; + } + + return 0; +} + int main(int argc, char **argv) { @@ -204,6 +259,11 @@ main(int argc, char **argv) alloc_gpu_memory(gpu_id); register_cpu_memory(gpu_id); + /** + * Communication items test + */ + create_update_comm_flag(gpu_id); + /* clean up the EAL */ rte_eal_cleanup(); printf("Bye...\n"); diff --git a/doc/guides/prog_guide/gpudev.rst b/doc/guides/prog_guide/gpudev.rst index eb5f0af817..e0db627aed 100644 --- a/doc/guides/prog_guide/gpudev.rst +++ b/doc/guides/prog_guide/gpudev.rst @@ -32,6 +32,10 @@ This library provides a number of features: - Interoperability with device-specific library through generic handlers. - Allocate and free memory on the device. - Register CPU memory to make it visible from the device. +- Communication between the CPU and the device. + +The whole CPU - GPU communication is implemented +using CPU memory visible from the GPU. API Overview @@ -73,3 +77,12 @@ Some GPU drivers may need, under certain conditions, to enforce the coherency of external devices writes (e.g. NIC receiving packets) into the GPU memory. gpudev abstracts and exposes this capability. + +Communication Flag +~~~~~~~~~~~~~~~~~~ + +Considering an application with some GPU task +that's waiting to receive a signal from the CPU +to move forward with the execution. +The communication flag allocates a CPU memory GPU-visible ``uint32_t`` flag +that can be used by the CPU to communicate with a GPU task. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index a4d07bda9b..78b29d9a25 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -105,6 +105,7 @@ New Features * Device information * Memory management + * Communication flag * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.** diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c index 49526b335f..f887f3dd93 100644 --- a/lib/gpudev/gpudev.c +++ b/lib/gpudev/gpudev.c @@ -643,3 +643,95 @@ rte_gpu_mbw(int16_t dev_id) } return GPU_DRV_RET(dev->ops.mbw(dev)); } + +int +rte_gpu_comm_create_flag(uint16_t dev_id, struct rte_gpu_comm_flag *devflag, + enum rte_gpu_comm_flag_type mtype) +{ + size_t flag_size; + int ret; + + if (devflag == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + if (mtype != RTE_GPU_COMM_FLAG_CPU) { + rte_errno = EINVAL; + return -rte_errno; + } + + flag_size = sizeof(uint32_t); + + devflag->ptr = rte_zmalloc(NULL, flag_size, 0); + if (devflag->ptr == NULL) { + rte_errno = ENOMEM; + return -rte_errno; + } + + ret = rte_gpu_register(dev_id, flag_size, devflag->ptr); + if (ret < 0) { + rte_errno = ENOMEM; + return -rte_errno; + } + + devflag->mtype = mtype; + devflag->dev_id = dev_id; + + return 0; +} + +int +rte_gpu_comm_destroy_flag(struct rte_gpu_comm_flag *devflag) +{ + int ret; + + if (devflag == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + ret = rte_gpu_unregister(devflag->dev_id, devflag->ptr); + if (ret < 0) { + rte_errno = EINVAL; + return -1; + } + + rte_free(devflag->ptr); + + return 0; +} + +int +rte_gpu_comm_set_flag(struct rte_gpu_comm_flag *devflag, uint32_t val) +{ + if (devflag == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + if (devflag->mtype != RTE_GPU_COMM_FLAG_CPU) { + rte_errno = EINVAL; + return -rte_errno; + } + + RTE_GPU_VOLATILE(*devflag->ptr) = val; + + return 0; +} + +int +rte_gpu_comm_get_flag_value(struct rte_gpu_comm_flag *devflag, uint32_t *val) +{ + if (devflag == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + if (devflag->mtype != RTE_GPU_COMM_FLAG_CPU) { + rte_errno = EINVAL; + return -rte_errno; + } + + *val = RTE_GPU_VOLATILE(*devflag->ptr); + + return 0; +} diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h index 650ebfd700..1466ac164b 100644 --- a/lib/gpudev/rte_gpudev.h +++ b/lib/gpudev/rte_gpudev.h @@ -38,6 +38,9 @@ extern "C" { /** Catch-all callback data. */ #define RTE_GPU_CALLBACK_ANY_DATA ((void *)-1) +/** Access variable as volatile. */ +#define RTE_GPU_VOLATILE(x) (*(volatile typeof(x) *)&(x)) + /** Store device info. */ struct rte_gpu_info { /** Unique identifier name. */ @@ -68,6 +71,22 @@ enum rte_gpu_event { typedef void (rte_gpu_callback_t)(int16_t dev_id, enum rte_gpu_event event, void *user_data); +/** Memory where communication flag is allocated. */ +enum rte_gpu_comm_flag_type { + /** Allocate flag on CPU memory visible from device. */ + RTE_GPU_COMM_FLAG_CPU = 0, +}; + +/** Communication flag to coordinate CPU with the device. */ +struct rte_gpu_comm_flag { + /** Device that will use the device flag. */ + uint16_t dev_id; + /** Pointer to flag memory area. */ + uint32_t *ptr; + /** Type of memory used to allocate the flag. */ + enum rte_gpu_comm_flag_type mtype; +}; + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. @@ -405,6 +424,95 @@ int rte_gpu_unregister(int16_t dev_id, void *ptr); __rte_experimental int rte_gpu_mbw(int16_t dev_id); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Create a communication flag that can be shared + * between CPU threads and device workload to exchange some status info + * (e.g. work is done, processing can start, etc..). + * + * @param dev_id + * Reference device ID. + * @param devflag + * Pointer to the memory area of the devflag structure. + * @param mtype + * Type of memory to allocate the communication flag. + * + * @return + * 0 on success, -rte_errno otherwise: + * - ENODEV if invalid dev_id + * - EINVAL if invalid inputs + * - ENOTSUP if operation not supported by the driver + * - ENOMEM if out of space + * - EPERM if driver error + */ +__rte_experimental +int rte_gpu_comm_create_flag(uint16_t dev_id, + struct rte_gpu_comm_flag *devflag, + enum rte_gpu_comm_flag_type mtype); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Deallocate a communication flag. + * + * @param devflag + * Pointer to the memory area of the devflag structure. + * + * @return + * 0 on success, -rte_errno otherwise: + * - ENODEV if invalid dev_id + * - EINVAL if NULL devflag + * - ENOTSUP if operation not supported by the driver + * - EPERM if driver error + */ +__rte_experimental +int rte_gpu_comm_destroy_flag(struct rte_gpu_comm_flag *devflag); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Set the value of a communication flag as the input value. + * Flag memory area is treated as volatile. + * The flag must have been allocated with RTE_GPU_COMM_FLAG_CPU. + * + * @param devflag + * Pointer to the memory area of the devflag structure. + * @param val + * Value to set in the flag. + * + * @return + * 0 on success, -rte_errno otherwise: + * - EINVAL if invalid input params + */ +__rte_experimental +int rte_gpu_comm_set_flag(struct rte_gpu_comm_flag *devflag, + uint32_t val); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Get the value of the communication flag. + * Flag memory area is treated as volatile. + * The flag must have been allocated with RTE_GPU_COMM_FLAG_CPU. + * + * @param devflag + * Pointer to the memory area of the devflag structure. + * @param val + * Flag output value. + * + * @return + * 0 on success, -rte_errno otherwise: + * - EINVAL if invalid input params + */ +__rte_experimental +int rte_gpu_comm_get_flag_value(struct rte_gpu_comm_flag *devflag, + uint32_t *val); + #ifdef __cplusplus } #endif diff --git a/lib/gpudev/version.map b/lib/gpudev/version.map index d72d470d8e..2fc039373a 100644 --- a/lib/gpudev/version.map +++ b/lib/gpudev/version.map @@ -6,6 +6,10 @@ EXPERIMENTAL { rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close; + rte_gpu_comm_create_flag; + rte_gpu_comm_destroy_flag; + rte_gpu_comm_get_flag_value; + rte_gpu_comm_set_flag; rte_gpu_count_avail; rte_gpu_find_next; rte_gpu_free; -- 2.17.1