From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E063F45F89; Mon, 30 Dec 2024 16:33:13 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB12340272; Mon, 30 Dec 2024 16:33:11 +0100 (CET) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2080.outbound.protection.outlook.com [40.107.22.80]) by mails.dpdk.org (Postfix) with ESMTP id 7FCE240263 for ; Mon, 30 Dec 2024 16:33:10 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=SFZWYT986uBZoy27GtrTTRxqKs865/7AsjZf4DUy32AYVxDm0ca3ird51vIGFhoEdI9RbhVQymTsduz5P7/DljK0klefAqi34Mt/b9rvVf4Yd/9ANh0ccGrMDTiPL1y+PIIyiQxse/QkPexj2Tfi7I4qYwGZOh4SY116B0jg4/jnfi9PLVz730+Z09LSssn9lilcPAs3ATPj9xPTwoamfTxmm9b5/YycWY7Pa7AHOnPA7KB/YT+uIs5APYpmMYwGQeSTu+aWbX0VdGA0+xYUa5OEADh31KdxUWK9bs2QiBankP1Mc4IJxRu793pslpQL6XmL2JE2JEYoSZK4+6iO7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6KTPeaThmVows3XKtFmyyujPeaUn2bQxw1AhIpPxTbE=; b=DWGr05ZuqQWu3D36XmZwQc5LG7jj2hwBVtX5p87aZZBdx/Q2KWt197oHvUqOq2w4Ap0PhZ0lYBOJaX//Ea94rLw89erZnt+R41c7b4eaNwFlsvuuBHv1DDBkXe7akm5x50nKHlFPKBz3OCIRsfSClJvZhp+ero/w5UNtt/LC3vNXzKqtvttLxJyhdn8evDkwG+mIvPGI0w/jx9mSo9NVHftQAk2L2W9YJIncN3Nc2mS+aTOen27mJDJOPpqcUQljsXKG55qfJ1HIH8aO3gFHo4Ef82o0UpUnGRq5mTfmE08zhBSjZjS7BPtEs5rrokwzEPgaHvAIvHVg+OHdW7Q9cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6KTPeaThmVows3XKtFmyyujPeaUn2bQxw1AhIpPxTbE=; b=IMZ6HH8bFSKraZh1IuRnms6rp4dlRR82LFVFR71+lXtiD8pWs2v+ukei/zFVMk5mLoDJv+u0mA6Jh6nzO074tCIFojjCbxq2YUH1274xm6z5e5sv6esKwQgORdJRxKC6qbgPBAd746viJ0gQmJwtRYkKqWWC1+2LSMbPrE8wwKJzU9W38NF3U/khorcVWO506xlvaIb9UzJuZw/REUfYnthl3wALLQIu6THtK/O/Hq4mSIXOrKHq8I58PTojkcBApeCgtbscfDjH4XPc3TA0XGne7M4Va0P061vCrGc6l8XLZBfRvhoPX8bYaiiscD9alBled3Katjm86dsOnpiOJg== Received: from DU2PR04CA0344.eurprd04.prod.outlook.com (2603:10a6:10:2b4::17) by VI1PR07MB6317.eurprd07.prod.outlook.com (2603:10a6:800:13f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8293.19; Mon, 30 Dec 2024 15:33:03 +0000 Received: from DB5PEPF00014B8E.eurprd02.prod.outlook.com (2603:10a6:10:2b4:cafe::7f) by DU2PR04CA0344.outlook.office365.com (2603:10a6:10:2b4::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8293.18 via Frontend Transport; Mon, 30 Dec 2024 15:33:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB5PEPF00014B8E.mail.protection.outlook.com (10.167.8.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11 via Frontend Transport; Mon, 30 Dec 2024 15:33:03 +0000 Received: from seliius19819.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Mon, 30 Dec 2024 16:33:02 +0100 Received: from seliiuts03667.seli.gic.ericsson.se (seliiuts03667.seli.gic.ericsson.se [100.77.109.197]) by seliius19819.seli.gic.ericsson.se (Postfix) with ESMTP id 6065D4020605; Mon, 30 Dec 2024 16:33:01 +0100 (CET) Received: by seliiuts03667.seli.gic.ericsson.se (Postfix, from userid 125215) id 44BB560FE98B; Mon, 30 Dec 2024 16:33:01 +0100 (CET) From: Piotr Krzewinski To: CC: , Piotr Krzewinski Subject: [PATCH] service: add service maintenance callback Date: Mon, 30 Dec 2024 16:32:22 +0100 Message-ID: <20241230153222.484894-1-piotr.krzewinski@ericsson.com> X-Mailer: git-send-email 2.36.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB5PEPF00014B8E:EE_|VI1PR07MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: 018049f6-1f64-4215-75b9-08dd28e74003 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?8M0gknh7JxGY8D5vdrrWTHsWfuLZHXgqfYv06MDbOsnXjg/RQvpSJ3E3NZ5u?= =?us-ascii?Q?AKrNxft4KVku0AJOXZOEvIxM0PAu/HIZ18trru09QqDYfGqDQ5slf+nCkgHV?= =?us-ascii?Q?4QTvxSmclHM/7k6dHu5qdJ+5G9bnS4Mmlnar0gzYQkdrQVB55DMwqYDBE7Iu?= =?us-ascii?Q?gkasFN9nj07dnB4UQJfV1xx9GAYcSZ4Tq1r+V1TLX9EyuiCxpbbYR2yydC0J?= =?us-ascii?Q?M3TIVN65nDxV4W3MwoIB7dBII/ZeuebfS/GJ0WxqEop9yRgWm/KG+eEyHG3Q?= =?us-ascii?Q?KIhgECfH7UUOxjl206ePFNWjobe1UdpU9i6Pk9cNVPV5oxZKIoXL58kg1d00?= =?us-ascii?Q?wKDk1f8Ow69q9TX/bIdu9+1hEO62l8M+qOQODV0db3oMFDKwLLnXgYA27hYd?= =?us-ascii?Q?jyxaU2L07SSvZRz8S5jqcR+u812nK0D6Dlr4C0rdGtXyUoYRReLdUxQkRfBQ?= =?us-ascii?Q?Zh+M2cdh9dyDtgL0858BGX1hFzaJm+lu6VvTZM4RhkaNZNAhBU3blANmeuxN?= =?us-ascii?Q?61YRo4QPICCd7ENNtvS7juevJ+ZPVkP+2ilASND0c+okeelCsnMg6uiHD4lb?= =?us-ascii?Q?b/yw7yLv7XBm45s/n3gxmKJiRMWub144goCZCcWSgZRMOhf4wCKGZDMAfCtM?= =?us-ascii?Q?V97OsIGeCvd5L62rA/uNUqebTHdz/LwKvXZVeCDRIsWNNBPxS//ABbW3VNk8?= =?us-ascii?Q?0sUci+WrjHNVObzuppw7O6MHNgACEirsNhefGvspoIulDm0SqrFz3CerwCaa?= =?us-ascii?Q?6z5xY/UZckhNEbgZIqvrJKkSdl3uX4s6bjk7xl24Z/yBCiPDGosGrVHwvwEj?= =?us-ascii?Q?MalBPXPzIgiY8iGs3yWnhYRCphpTdSKs8Q7T8ccE8+TFyVtTTR4XCcpQC6l8?= =?us-ascii?Q?55bomwpZrZFK0GLI5CgGE5sC0LypfXEQcj97jPhJeMH1ta7ZxVztLKFdT8It?= =?us-ascii?Q?OrVSMq2T1y40sUw2sQiSXMfNYpRetTSv5HZjUMLQuOUr0mf0bQpAGFnzgR9e?= =?us-ascii?Q?Hqi4QnfSI1Y6SSkLqTF0XfyZRYTdLcWPXfJzjQSVhPXQNgUxHKz6wk2A/+5e?= =?us-ascii?Q?ipnB+Q0ni4HJKpShgrh9MWqiWDrT7HZvigzMBYwHcEGEugOnDcoMbINVKnVU?= =?us-ascii?Q?7duFUClywAtYvSaqRoKF+I+mvHeVRuzZcD7agRa/FYTNhNBfXZvqmdDlLYek?= =?us-ascii?Q?gnFvubwwMXQh4wzq3UVRhDHWaM3vAQm32oWUZfTmQuwrGSyIxL5kpx52BQYd?= =?us-ascii?Q?5p/L1cDO79rCBHdHCGpCRCu8NoQCaB2FosLM0hWqcTxFs0Xxl2k5JPPzfez1?= =?us-ascii?Q?VKdTyCf2WyknJNXzq4p0b09vxdVjxNnbEU6hF+KWgCcZA92T9DvyoTX+eDfq?= =?us-ascii?Q?+JescqQjdSvaWhRqWejJQxRwyFknbFD9AGVMxJWvy3icWYRfmdH2ZvCX5dgY?= =?us-ascii?Q?fIgFG8FCCpx/+RE6AViNV8sffxekYd0PMY4x8PvjlI+RhPKA5tAyN1Xa4qO6?= =?us-ascii?Q?yIfTgfGCtIbVVH8=3D?= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Dec 2024 15:33:03.2222 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 018049f6-1f64-4215-75b9-08dd28e74003 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB5PEPF00014B8E.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB6317 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add option to register a callback running on service lcores along regular services, which gets information about the service loop. It enables doing maintenance work or power saving during periods when all registered services are idling. As an example application that is doing dequeue from multiple event ports using single service lcore (e.g. using rte dispatcher library) may want to wait for new events inside the maintenance callback when there is no work available on ANY of the ports. This is not possible using non zero dequeue timeout without increasing latency of work that is scheduled to other event ports. Signed-off-by: Piotr Krzewinski --- app/test/test_service_cores.c | 54 +++++++++++++++++++++++++++++++++++ lib/eal/common/rte_service.c | 48 ++++++++++++++++++++++++++----- lib/eal/include/rte_service.h | 33 +++++++++++++++++++++ lib/eal/version.map | 4 +++ 4 files changed, 132 insertions(+), 7 deletions(-) diff --git a/app/test/test_service_cores.c b/app/test/test_service_cores.c index 46ed9a1868..f4abbd2dfe 100644 --- a/app/test/test_service_cores.c +++ b/app/test/test_service_cores.c @@ -1011,6 +1011,59 @@ service_may_be_active(void) return unregister_all(); } +static uint64_t busy_loops; +static uint64_t idle_loops; + +static void maint_callback(bool work_done) +{ + if (work_done) + busy_loops++; + else + idle_loops++; +} + +/* test registering and unregistering of service maint callback*/ +static int +service_maintenance(void) +{ + const uint32_t sid = 0; + busy_loops = 0; + idle_loops = 0; + /* expected failure cases */ + TEST_ASSERT_EQUAL(-EINVAL, rte_service_may_be_active(10000), + "Invalid service may be active check did not fail"); + + /* start the service */ + TEST_ASSERT_EQUAL(0, rte_service_runstate_set(sid, 1), + "Starting valid service failed"); + TEST_ASSERT_EQUAL(0, rte_service_lcore_add(slcore_id), + "Add service core failed when not in use before"); + TEST_ASSERT_EQUAL(0, rte_service_map_lcore_set(sid, slcore_id, 1), + "Enabling valid service on valid core failed"); + TEST_ASSERT_EQUAL(0, rte_service_maint_callback_register(maint_callback, slcore_id), + "Registering maintenance callback failed"); + TEST_ASSERT_EQUAL(0, rte_service_lcore_start(slcore_id), + "Service core start after add failed"); + + /* ensures core really is running the service function */ + TEST_ASSERT_EQUAL(1, service_lcore_running_check(), + "Service core expected to poll service but it didn't"); + + /* ensures service maintenance callback received information about work done */ + TEST_ASSERT(busy_loops > 0, "No busy loops detected"); + TEST_ASSERT(idle_loops > 0, "No idle loops detected"); + + /* stop the service, and wait for not-active with timeout */ + TEST_ASSERT_EQUAL(0, rte_service_runstate_set(sid, 0), + "Error: Service stop returned non-zero"); + TEST_ASSERT_EQUAL(0, service_ensure_stopped_with_timeout(sid), + "Error: Service not stopped after timeout period."); + TEST_ASSERT_EQUAL(0, rte_service_maint_callback_unregister(slcore_id), + "Unregistering service maintenance callback failed"); + + return unregister_all(); +} + /* check service may be active when service is running on a second lcore */ static int service_active_two_cores(void) @@ -1071,6 +1124,7 @@ static struct unit_test_suite service_tests = { TEST_CASE_ST(dummy_register, NULL, service_mt_unsafe_poll), TEST_CASE_ST(dummy_register, NULL, service_mt_safe_poll), TEST_CASE_ST(dummy_register, NULL, service_may_be_active), + TEST_CASE_ST(dummy_register, NULL, service_maintenance), TEST_CASE_ST(dummy_register, NULL, service_active_two_cores), TEST_CASES_END() /**< NULL terminate unit test array */ } diff --git a/lib/eal/common/rte_service.c b/lib/eal/common/rte_service.c index dad3150df9..850f9a2fb5 100644 --- a/lib/eal/common/rte_service.c +++ b/lib/eal/common/rte_service.c @@ -74,6 +74,7 @@ struct __rte_cache_aligned core_state { RTE_BITSET_DECLARE(service_active_on_lcore, RTE_SERVICE_NUM_MAX); RTE_ATOMIC(uint64_t) loops; RTE_ATOMIC(uint64_t) cycles; + rte_maint_func maint_callback; struct service_stats service_stats[RTE_SERVICE_NUM_MAX]; }; @@ -317,6 +318,30 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate) return 0; } +int32_t +rte_service_maint_callback_register(rte_maint_func callback, uint32_t lcore) +{ + struct core_state *cs = RTE_LCORE_VAR_LCORE(lcore, lcore_states); + if (lcore >= RTE_MAX_LCORE || !cs->is_service_core + || callback == NULL) + return -EINVAL; + + cs->maint_callback = callback; + return 0; +} + +int32_t +rte_service_maint_callback_unregister(uint32_t lcore) +{ + struct core_state *cs = RTE_LCORE_VAR_LCORE(lcore, lcore_states); + if (lcore >= RTE_MAX_LCORE || !cs->is_service_core + || !cs->maint_callback) + return -EINVAL; + + cs->maint_callback = NULL; + return 0; +} + int32_t rte_service_runstate_set(uint32_t id, uint32_t runstate) { @@ -378,16 +403,17 @@ service_counter_add(RTE_ATOMIC(uint64_t) *counter, uint64_t operand) rte_memory_order_relaxed); } -static inline void +static inline int32_t service_runner_do_callback(struct rte_service_spec_impl *s, struct core_state *cs, uint32_t service_idx) { rte_eal_trace_service_run_begin(service_idx, rte_lcore_id()); void *userdata = s->spec.callback_userdata; + int32_t rc; if (service_stats_enabled(s)) { uint64_t start = rte_rdtsc(); - int rc = s->spec.callback(userdata); + rc = s->spec.callback(userdata); struct service_stats *service_stats = &cs->service_stats[service_idx]; @@ -407,9 +433,10 @@ service_runner_do_callback(struct rte_service_spec_impl *s, service_counter_add(&service_stats->cycles, cycles); } } else { - s->spec.callback(userdata); + rc = s->spec.callback(userdata); } rte_eal_trace_service_run_end(service_idx, rte_lcore_id()); + return rc; } @@ -436,16 +463,17 @@ service_run(uint32_t i, struct core_state *cs, const uint64_t *mapped_services, rte_bitset_set(cs->service_active_on_lcore, i); + int32_t ret; if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) { if (!rte_spinlock_trylock(&s->execute_lock)) return -EBUSY; - service_runner_do_callback(s, cs, i); + ret = service_runner_do_callback(s, cs, i); rte_spinlock_unlock(&s->execute_lock); } else - service_runner_do_callback(s, cs, i); + ret = service_runner_do_callback(s, cs, i); - return 0; + return ret; } int32_t @@ -505,13 +533,19 @@ service_runner_func(void *arg) */ while (rte_atomic_load_explicit(&cs->runstate, rte_memory_order_acquire) == RUNSTATE_RUNNING) { + bool work_done = false; ssize_t id; RTE_BITSET_FOREACH_SET(id, cs->mapped_services, RTE_SERVICE_NUM_MAX) { /* return value ignored as no change to code flow */ - service_run(id, cs, cs->mapped_services, service_get(id), 1); + int32_t ret = service_run(id, cs, cs->mapped_services, service_get(id), 1); + if ((ret != -EAGAIN) && (ret != -EINVAL) && (ret != -ENOEXEC)) + work_done = true; } + if (cs->maint_callback != NULL) + cs->maint_callback(work_done); + rte_atomic_store_explicit(&cs->loops, cs->loops + 1, rte_memory_order_relaxed); } diff --git a/lib/eal/include/rte_service.h b/lib/eal/include/rte_service.h index 1236fe2dc6..b7eafd4443 100644 --- a/lib/eal/include/rte_service.h +++ b/lib/eal/include/rte_service.h @@ -336,6 +336,39 @@ int32_t rte_service_lcore_reset_all(void); */ int32_t rte_service_set_stats_enable(uint32_t id, int32_t enable); +/** + * Signature of callback function to run after all services are done + * + * If registered, service framework will execute the callback + * indicating if any of the services run on the core was performing work. + * This may be used e.g. for maintenance or power saving when no work done. + */ +typedef void (*rte_maint_func)(bool work_done); + +/** + * Register maintenance callback service. + * @b EXPERIMENTAL: this API may change without prior notice. + * + * @param callback Function callback to register + * @retval 0 Successfully registered the callback. + * -EINVAL Attempted to register an invalid callback or the + * lcore is not registered as service lcore + */ +__rte_experimental +int32_t rte_service_maint_callback_register(rte_maint_func callback, uint32_t lcore); + +/** + * Uregister maintenance callback service. + * @b EXPERIMENTAL: this API may change without prior notice. + * + * @param callback Function callback to register + * @retval 0 Successfully unregistered the callback. + * -EINVAL Attempted to unregister a callback from a core which + * is not a service lcore or for which no maint callback was registered + */ +__rte_experimental +int32_t rte_service_maint_callback_unregister(uint32_t lcore); + /** * Retrieve the list of currently enabled service cores. * diff --git a/lib/eal/version.map b/lib/eal/version.map index a20c713eb1..e5355e956e 100644 --- a/lib/eal/version.map +++ b/lib/eal/version.map @@ -398,6 +398,10 @@ EXPERIMENTAL { # added in 24.11 rte_bitset_to_str; rte_lcore_var_alloc; + + # added in 25.03 + rte_service_maint_callback_register; + rte_service_maint_callback_unregister; }; INTERNAL { -- 2.34.1