From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A5F1A034C; Fri, 8 Apr 2022 16:25:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 30EC94067E; Fri, 8 Apr 2022 16:25:48 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60076.outbound.protection.outlook.com [40.107.6.76]) by mails.dpdk.org (Postfix) with ESMTP id 013374003F for ; Fri, 8 Apr 2022 16:25:46 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dhU+nNbIYKRFOXIVtfFzv+1HEziz67oXpB/R4iFHMhAx7MopG84chvq3zKCLlmJpIBFmgBXWBB4DRlLndoXcaLKKOFMzkL8c7bWlZWPX/BPGxeYDlgFIXYA9qylF+A2ryiZSdfU3E/xR+kttVat06v0QXaENME8ze/fmSOd9d3x8wM7HTM6ghs/IiFrtaLX25jh4hOME1Bog7NgjPrp3Y3fvH4NmdXSPgy3z4wVLgGH82KTSt1bjx6OPjilJmbLKk8m7LS9M7tW5b+igVrDOKkvbbxWZFPnYsZaJpUv05k9jL0pik65SoN1IrvJYmqjwjpOpsYZx7sVWXcY6BwtO2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q6SlM2A5PeHuLks6QsdubO6RvRyR0w3EFzgL/MkI0Gg=; b=DeFxAVgcg/cm1cVyQ5eUU9HAXAHlqnXa+1CGAnD5XspS+bh7K5nz2v5lp81hCEfrVbE+4Sm/oHWGAxH7AgI8hFqd93qy48+CNLhZDx0KThZce1ob0R0xB3gm8fB6sy+9w+F5cU/qVqmG4yVo+EhgI5kULe6cUvRFDLWVrUauv4/jX+6ScK5FX3/JYUgBaaj/HY3fG907apJm8OngYWB/3uiQQyhz6UeNgqeHL7ZajDnevnguw284zbPzRdIRleQLXBUAMCBlv9+LaXuvHpTX/k7Rpq+tdmQehn/9ur+Ju+fi18kCkTNLQluXLY0CSpoCfUG3kEdznIEaLzd0fRSmoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=arm.com smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q6SlM2A5PeHuLks6QsdubO6RvRyR0w3EFzgL/MkI0Gg=; b=Jler6MZbtw6/DnofI904hECYHGxnnAJBFoVJlfR/Vo4oXUlfa6k0A9JBrmAvQR0p0GYwx7LKnENApQxEM5tzq2piwmv283jjFLe0J0V+YTczVznHx2g0T6WsSnvbChgSGIoY5EMyUcW0xgupQE8cU3owE40+EG636mqZ3DLpd/E= Received: from SV0P279CA0030.NORP279.PROD.OUTLOOK.COM (2603:10a6:f10:12::17) by AM0PR07MB4771.eurprd07.prod.outlook.com (2603:10a6:208:6c::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5164.8; Fri, 8 Apr 2022 14:25:45 +0000 Received: from HE1EUR02FT087.eop-EUR02.prod.protection.outlook.com (2603:10a6:f10:12:cafe::e9) by SV0P279CA0030.outlook.office365.com (2603:10a6:f10:12::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.22 via Frontend Transport; Fri, 8 Apr 2022 14:25:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; Received: from oa.msg.ericsson.com (192.176.1.74) by HE1EUR02FT087.mail.protection.outlook.com (10.152.11.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.5144.20 via Frontend Transport; Fri, 8 Apr 2022 14:25:45 +0000 Received: from ESESBMB501.ericsson.se (153.88.183.168) by ESESSMR501.ericsson.se (153.88.183.108) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.27; Fri, 8 Apr 2022 16:25:43 +0200 Received: from ESESBMB502.ericsson.se (153.88.183.169) by ESESBMB501.ericsson.se (153.88.183.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.27; Fri, 8 Apr 2022 16:25:43 +0200 Received: from seliiuapp00218.seli.gic.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.185) with Microsoft SMTP Server id 15.1.2308.27 via Frontend Transport; Fri, 8 Apr 2022 16:25:42 +0200 Received: from localhost.localdomain (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliiuapp00218.seli.gic.ericsson.se (Postfix) with ESMTP id 1F85B60195; Fri, 8 Apr 2022 16:25:43 +0200 (CEST) From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= To: CC: Thomas Monjalon , David Marchand , , , , , , , , =?UTF-8?q?Mattias=20R=C3=B6nnblom?= , Ola Liljedahl Subject: [PATCH v4] eal: add seqlock Date: Fri, 8 Apr 2022 16:24:42 +0200 Message-ID: <20220408142442.157192-1-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f30b7e45-c62f-4672-b00c-08da196bab47 X-MS-TrafficTypeDiagnostic: AM0PR07MB4771:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yvW0ohfZxjdxMFTYdnmPlcP+E4zSeAMjb91ZNor1lU8Bp6Rn1vZllfRaFWJhqiYdhlhRF1BmKSXg7+d16ipaxXJmt8WCy6BWt0FIOA7OeCvVupl3m2e73Pmy/bDbvR53R6fBaAH8cYMRaWWEnp4kEqueuKMikfn18La86GfauGl+S69065n0jnKYzhTFf4CHk+uFApYbbH7S3c/nwi9z6Yz8vt0ttv6TxTVvHIernX9I8KbbDm0urdPeoGpa+qcDS6W17A2XT2E/nG9r1FqR9xT08F7BJCRKT26QJM/A7NE6fINVnfYMRQcvT7hvQrc2V+XhqgjrhbfP0JZ1POEkfJgRn808PLvP+F7GPjdu9eRKxT1QXc9m97/6VhrA0vhv1KB1lsIaHZuPMb7I4KUt5ZxX2Y07SEFfGBI5dX07f/KlnLe0N8POqRnTyAjE6RTDtBmghR0eP2/Pu5ndk1NakwDXU5Fe8oXbQIsC+r1RpaeEfzHecYoWkeD3h8FztvjGRTxh3qPny/97zhfwgv0UCGj0BZVPHefmaCL7RjcPewCoKBpEF5eH5vFuGab3DevFbQBrvyABadziS94ZqoAQ68oHdex5+C/5Zv8hzVpasrrqaALWUXEUJZBPKd47AB5kH9S8AWznseVPKsgR4ghAak3giFyPpGM+1rg0TK7G93j0N2gOIruNIFOqjVIbMhd6CTANjYd95gPAiJnJDjo0l+BTR1WuJhsfeQCuFSCZqZbsNmMzzqB7iWOzVbhZVuJrjlL61aUAh5qtQU19IzwYKrVj5zF77xLvJ1l3Td2Keok= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(36860700001)(1076003)(66574015)(4326008)(82310400005)(7416002)(2616005)(26005)(6266002)(186003)(86362001)(36756003)(316002)(30864003)(34020700004)(40460700003)(8936002)(8676002)(336012)(47076005)(5660300002)(70206006)(7636003)(82960400001)(6916009)(54906003)(70586007)(83380400001)(356005)(2906002)(508600001)(966005); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2022 14:25:45.1829 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f30b7e45-c62f-4672-b00c-08da196bab47 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: HE1EUR02FT087.eop-EUR02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR07MB4771 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org A sequence lock (seqlock) is synchronization primitive which allows for data-race free, low-overhead, high-frequency reads, especially for data structures shared across many cores and which are updated relatively infrequently. A seqlock permits multiple parallel readers. The variant of seqlock implemented in this patch supports multiple writers as well. A spinlock is used for writer-writer serialization. To avoid resource reclamation and other issues, the data protected by a seqlock is best off being self-contained (i.e., no pointers [except to constant data]). One way to think about seqlocks is that they provide means to perform atomic operations on data objects larger what the native atomic machine instructions allow for. DPDK seqlocks are not preemption safe on the writer side. A thread preemption affects performance, not correctness. A seqlock contains a sequence number, which can be thought of as the generation of the data it protects. A reader will 1. Load the sequence number (sn). 2. Load, in arbitrary order, the seqlock-protected data. 3. Load the sn again. 4. Check if the first and second sn are equal, and even numbered. If they are not, discard the loaded data, and restart from 1. The first three steps need to be ordered using suitable memory fences. A writer will 1. Take the spinlock, to serialize writer access. 2. Load the sn. 3. Store the original sn + 1 as the new sn. 4. Perform load and stores to the seqlock-protected data. 5. Store the original sn + 2 as the new sn. 6. Release the spinlock. Proper memory fencing is required to make sure the first sn store, the data stores, and the second sn store appear to the reader in the mentioned order. The sn loads and stores must be atomic, but the data loads and stores need not be. The original seqlock design and implementation was done by Stephen Hemminger. This is an independent implementation, using C11 atomics. For more information on seqlocks, see https://en.wikipedia.org/wiki/Seqlock PATCH v4: * Reverted to Linux kernel style naming on the read side. * Bail out early from the retry function if an odd sequence number is encountered. * Added experimental warnings in the API documentation. * Static initializer now uses named field initialization. * Various tweaks to API documentation (including the example). PATCH v3: * Renamed both read and write-side critical section begin/end functions to better match rwlock naming, per Ola Liljedahl's suggestion. * Added 'extern "C"' guards for C++ compatibility. * Refer to the main lcore as the main lcore, and nothing else. PATCH v2: * Skip instead of fail unit test in case too few lcores are available. * Use main lcore for testing, reducing the minimum number of lcores required to run the unit tests to four. * Consistently refer to sn field as the "sequence number" in the documentation. * Fixed spelling mistakes in documentation. Updates since RFC: * Added API documentation. * Added link to Wikipedia article in the commit message. * Changed seqlock sequence number field from uint64_t (which was overkill) to uint32_t. The sn type needs to be sufficiently large to assure no reader will read a sn, access the data, and then read the same sn, but the sn has been incremented enough times to have wrapped during the read, and arrived back at the original sn. * Added RTE_SEQLOCK_INITIALIZER macro for static initialization. * Removed the rte_seqlock struct + separate rte_seqlock_t typedef with an anonymous struct typedef:ed to rte_seqlock_t. Acked-by: Morten Brørup Reviewed-by: Ola Liljedahl Signed-off-by: Mattias Rönnblom --- app/test/meson.build | 2 + app/test/test_seqlock.c | 202 +++++++++++++++++++++ lib/eal/common/meson.build | 1 + lib/eal/common/rte_seqlock.c | 12 ++ lib/eal/include/meson.build | 1 + lib/eal/include/rte_seqlock.h | 319 ++++++++++++++++++++++++++++++++++ lib/eal/version.map | 3 + 7 files changed, 540 insertions(+) create mode 100644 app/test/test_seqlock.c create mode 100644 lib/eal/common/rte_seqlock.c create mode 100644 lib/eal/include/rte_seqlock.h diff --git a/app/test/meson.build b/app/test/meson.build index 5fc1dd1b7b..5e418e8766 100644 --- a/app/test/meson.build +++ b/app/test/meson.build @@ -125,6 +125,7 @@ test_sources = files( 'test_rwlock.c', 'test_sched.c', 'test_security.c', + 'test_seqlock.c', 'test_service_cores.c', 'test_spinlock.c', 'test_stack.c', @@ -214,6 +215,7 @@ fast_tests = [ ['rwlock_rde_wro_autotest', true], ['sched_autotest', true], ['security_autotest', false], + ['seqlock_autotest', true], ['spinlock_autotest', true], ['stack_autotest', false], ['stack_lf_autotest', false], diff --git a/app/test/test_seqlock.c b/app/test/test_seqlock.c new file mode 100644 index 0000000000..3f1ce53678 --- /dev/null +++ b/app/test/test_seqlock.c @@ -0,0 +1,202 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Ericsson AB + */ + +#include + +#include +#include +#include + +#include + +#include "test.h" + +struct data { + rte_seqlock_t lock; + + uint64_t a; + uint64_t b __rte_cache_aligned; + uint64_t c __rte_cache_aligned; +} __rte_cache_aligned; + +struct reader { + struct data *data; + uint8_t stop; +}; + +#define WRITER_RUNTIME (2.0) /* s */ + +#define WRITER_MAX_DELAY (100) /* us */ + +#define INTERRUPTED_WRITER_FREQUENCY (1000) +#define WRITER_INTERRUPT_TIME (1) /* us */ + +static int +writer_run(void *arg) +{ + struct data *data = arg; + uint64_t deadline; + + deadline = rte_get_timer_cycles() + + WRITER_RUNTIME * rte_get_timer_hz(); + + while (rte_get_timer_cycles() < deadline) { + bool interrupted; + uint64_t new_value; + unsigned int delay; + + new_value = rte_rand(); + + interrupted = rte_rand_max(INTERRUPTED_WRITER_FREQUENCY) == 0; + + rte_seqlock_write_lock(&data->lock); + + data->c = new_value; + + /* These compiler barriers (both on the test reader + * and the test writer side) are here to ensure that + * loads/stores *usually* happen in test program order + * (always on a TSO machine). They are arrange in such + * a way that the writer stores in a different order + * than the reader loads, to emulate an arbitrary + * order. A real application using a seqlock does not + * require any compiler barriers. + */ + rte_compiler_barrier(); + data->b = new_value; + + if (interrupted) + rte_delay_us_block(WRITER_INTERRUPT_TIME); + + rte_compiler_barrier(); + data->a = new_value; + + rte_seqlock_write_unlock(&data->lock); + + delay = rte_rand_max(WRITER_MAX_DELAY); + + rte_delay_us_block(delay); + } + + return 0; +} + +#define INTERRUPTED_READER_FREQUENCY (1000) +#define READER_INTERRUPT_TIME (1000) /* us */ + +static int +reader_run(void *arg) +{ + struct reader *r = arg; + int rc = 0; + + while (__atomic_load_n(&r->stop, __ATOMIC_RELAXED) == 0 && rc == 0) { + struct data *data = r->data; + bool interrupted; + uint32_t sn; + uint64_t a; + uint64_t b; + uint64_t c; + + interrupted = rte_rand_max(INTERRUPTED_READER_FREQUENCY) == 0; + + do { + sn = rte_seqlock_read_begin(&data->lock); + + a = data->a; + /* See writer_run() for an explanation why + * these barriers are here. + */ + rte_compiler_barrier(); + + if (interrupted) + rte_delay_us_block(READER_INTERRUPT_TIME); + + c = data->c; + + rte_compiler_barrier(); + b = data->b; + + } while (rte_seqlock_read_retry(&data->lock, sn)); + + if (a != b || b != c) { + printf("Reader observed inconsistent data values " + "%" PRIu64 " %" PRIu64 " %" PRIu64 "\n", + a, b, c); + rc = -1; + } + } + + return rc; +} + +static void +reader_stop(struct reader *reader) +{ + __atomic_store_n(&reader->stop, 1, __ATOMIC_RELAXED); +} + +#define NUM_WRITERS (2) /* main lcore + one worker */ +#define MIN_NUM_READERS (2) +#define MAX_READERS (RTE_MAX_LCORE - NUM_WRITERS - 1) +#define MIN_LCORE_COUNT (NUM_WRITERS + MIN_NUM_READERS) + +/* Only a compile-time test */ +static rte_seqlock_t __rte_unused static_init_lock = RTE_SEQLOCK_INITIALIZER; + +static int +test_seqlock(void) +{ + struct reader readers[MAX_READERS]; + unsigned int num_readers; + unsigned int num_lcores; + unsigned int i; + unsigned int lcore_id; + unsigned int reader_lcore_ids[MAX_READERS]; + unsigned int worker_writer_lcore_id = 0; + int rc = 0; + + num_lcores = rte_lcore_count(); + + if (num_lcores < MIN_LCORE_COUNT) { + printf("Too few cores to run test. Skipping.\n"); + return 0; + } + + num_readers = num_lcores - NUM_WRITERS; + + struct data *data = rte_zmalloc(NULL, sizeof(struct data), 0); + + i = 0; + RTE_LCORE_FOREACH_WORKER(lcore_id) { + if (i == 0) { + rte_eal_remote_launch(writer_run, data, lcore_id); + worker_writer_lcore_id = lcore_id; + } else { + unsigned int reader_idx = i - 1; + struct reader *reader = &readers[reader_idx]; + + reader->data = data; + reader->stop = 0; + + rte_eal_remote_launch(reader_run, reader, lcore_id); + reader_lcore_ids[reader_idx] = lcore_id; + } + i++; + } + + if (writer_run(data) != 0 || + rte_eal_wait_lcore(worker_writer_lcore_id) != 0) + rc = -1; + + for (i = 0; i < num_readers; i++) { + reader_stop(&readers[i]); + if (rte_eal_wait_lcore(reader_lcore_ids[i]) != 0) + rc = -1; + } + + return rc; +} + +REGISTER_TEST_COMMAND(seqlock_autotest, test_seqlock); diff --git a/lib/eal/common/meson.build b/lib/eal/common/meson.build index 917758cc65..a41343bfed 100644 --- a/lib/eal/common/meson.build +++ b/lib/eal/common/meson.build @@ -35,6 +35,7 @@ sources += files( 'rte_malloc.c', 'rte_random.c', 'rte_reciprocal.c', + 'rte_seqlock.c', 'rte_service.c', 'rte_version.c', ) diff --git a/lib/eal/common/rte_seqlock.c b/lib/eal/common/rte_seqlock.c new file mode 100644 index 0000000000..d4fe648799 --- /dev/null +++ b/lib/eal/common/rte_seqlock.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Ericsson AB + */ + +#include + +void +rte_seqlock_init(rte_seqlock_t *seqlock) +{ + seqlock->sn = 0; + rte_spinlock_init(&seqlock->lock); +} diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build index 9700494816..48df5f1a21 100644 --- a/lib/eal/include/meson.build +++ b/lib/eal/include/meson.build @@ -36,6 +36,7 @@ headers += files( 'rte_per_lcore.h', 'rte_random.h', 'rte_reciprocal.h', + 'rte_seqlock.h', 'rte_service.h', 'rte_service_component.h', 'rte_string_fns.h', diff --git a/lib/eal/include/rte_seqlock.h b/lib/eal/include/rte_seqlock.h new file mode 100644 index 0000000000..961816aa10 --- /dev/null +++ b/lib/eal/include/rte_seqlock.h @@ -0,0 +1,319 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Ericsson AB + */ + +#ifndef _RTE_SEQLOCK_H_ +#define _RTE_SEQLOCK_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * @file + * RTE Seqlock + * + * A sequence lock (seqlock) is a synchronization primitive allowing + * multiple, parallel, readers to efficiently and safely (i.e., in a + * data-race free manner) access lock-protected data. The RTE seqlock + * permits multiple writers as well. A spinlock is used to + * writer-writer synchronization. + * + * A reader never blocks a writer. Very high frequency writes may + * prevent readers from making progress. + * + * A seqlock is not preemption-safe on the writer side. If a writer is + * preempted, it may block readers until the writer thread is allowed + * to continue. Heavy computations should be kept out of the + * writer-side critical section, to avoid delaying readers. + * + * Seqlocks are useful for data which are read by many cores, at a + * high frequency, and relatively infrequently written to. + * + * One way to think about seqlocks is that they provide means to + * perform atomic operations on objects larger than what the native + * machine instructions allow for. + * + * To avoid resource reclamation issues, the data protected by a + * seqlock should typically be kept self-contained (e.g., no pointers + * to mutable, dynamically allocated data). + * + * Example usage: + * @code{.c} + * #define MAX_Y_LEN (16) + * // Application-defined example data structure, protected by a seqlock. + * struct config { + * rte_seqlock_t lock; + * int param_x; + * char param_y[MAX_Y_LEN]; + * }; + * + * // Accessor function for reading config fields. + * void + * config_read(const struct config *config, int *param_x, char *param_y) + * { + * uint32_t sn; + * + * do { + * sn = rte_seqlock_read_begin(&config->lock); + * + * // Loads may be atomic or non-atomic, as in this example. + * *param_x = config->param_x; + * strcpy(param_y, config->param_y); + * // An alternative to an immediate retry is to abort and + * // try again at some later time, assuming progress is + * // possible without the data. + * } while (rte_seqlock_read_retry(&config->lock)); + * } + * + * // Accessor function for writing config fields. + * void + * config_update(struct config *config, int param_x, const char *param_y) + * { + * rte_seqlock_write_lock(&config->lock); + * // Stores may be atomic or non-atomic, as in this example. + * config->param_x = param_x; + * strcpy(config->param_y, param_y); + * rte_seqlock_write_unlock(&config->lock); + * } + * @endcode + * + * @see + * https://en.wikipedia.org/wiki/Seqlock. + */ + +#include +#include + +#include +#include +#include + +/** + * The RTE seqlock type. + */ +typedef struct { + uint32_t sn; /**< A sequence number for the protected data. */ + rte_spinlock_t lock; /**< Spinlock used to serialize writers. */ +} rte_seqlock_t; + +/** + * A static seqlock initializer. + */ +#define RTE_SEQLOCK_INITIALIZER \ + { \ + .sn = 0, \ + .lock = RTE_SPINLOCK_INITIALIZER \ + } + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Initialize the seqlock. + * + * This function initializes the seqlock, and leaves the writer-side + * spinlock unlocked. + * + * @param seqlock + * A pointer to the seqlock. + */ +__rte_experimental +void +rte_seqlock_init(rte_seqlock_t *seqlock); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Begin a read-side critical section. + * + * A call to this function marks the beginning of a read-side critical + * section, for @p seqlock. + * + * rte_seqlock_read_begin() returns a sequence number, which is later + * used in rte_seqlock_read_retry() to check if the protected data + * underwent any modifications during the read transaction. + * + * After (in program order) rte_seqlock_read_begin() has been called, + * the calling thread reads the protected data, for later use. The + * protected data read *must* be copied (either in pristine form, or + * in the form of some derivative), since the caller may only read the + * data from within the read-side critical section (i.e., after + * rte_seqlock_read_begin() and before rte_seqlock_read_retry()), + * but must not act upon the retrieved data while in the critical + * section, since it does not yet know if it is consistent. + * + * The protected data may be read using atomic and/or non-atomic + * operations. + * + * After (in program order) all required data loads have been + * performed, rte_seqlock_read_retry() should be called, marking + * the end of the read-side critical section. + * + * If rte_seqlock_read_retry() returns true, the just-read data is + * inconsistent and should be discarded. The caller has the option to + * either restart the whole procedure right away (i.e., calling + * rte_seqlock_read_begin() again), or do the same at some later time. + * + * If rte_seqlock_read_retry() returns false, the data was read + * atomically and the copied data is consistent. + * + * @param seqlock + * A pointer to the seqlock. + * @return + * The seqlock sequence number for this critical section, to + * later be passed to rte_seqlock_read_retry(). + * + * @see rte_seqlock_read_retry() + */ +__rte_experimental +static inline uint32_t +rte_seqlock_read_begin(const rte_seqlock_t *seqlock) +{ + /* __ATOMIC_ACQUIRE to prevent loads after (in program order) + * from happening before the sn load. Synchronizes-with the + * store release in rte_seqlock_write_unlock(). + */ + return __atomic_load_n(&seqlock->sn, __ATOMIC_ACQUIRE); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * End a read-side critical section. + * + * A call to this function marks the end of a read-side critical + * section, for @p seqlock. The application must supply the sequence + * number produced by the corresponding rte_seqlock_read_begin() call. + * + * After this function has been called, the caller should not access + * the protected data. + * + * In case rte_seqlock_read_retry() returns true, the just-read data + * was modified as it was being read and may be inconsistent, and thus + * should be discarded. + * + * In case this function returns false, the data is consistent and the + * set of atomic and non-atomic load operations performed between + * rte_seqlock_read_begin() and rte_seqlock_read_retry() were atomic, + * as a whole. + * + * @param seqlock + * A pointer to the seqlock. + * @param begin_sn + * The seqlock sequence number returned by rte_seqlock_read_begin(). + * @return + * true or false, if the just-read seqlock-protected data was + * inconsistent or consistent, respectively, at the time it was + * read. + * + * @see rte_seqlock_read_begin() + */ +__rte_experimental +static inline bool +rte_seqlock_read_retry(const rte_seqlock_t *seqlock, uint32_t begin_sn) +{ + uint32_t end_sn; + + /* An odd sequence number means the protected data was being + * modified already at the point of the rte_seqlock_read_begin() + * call. + */ + if (unlikely(begin_sn & 1)) + return true; + + /* make sure the data loads happens before the sn load */ + rte_atomic_thread_fence(__ATOMIC_ACQUIRE); + + end_sn = __atomic_load_n(&seqlock->sn, __ATOMIC_RELAXED); + + /* A writer pegged the sequence number during the read operation. */ + if (unlikely(begin_sn != end_sn)) + return true; + + return false; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Begin a write-side critical section. + * + * A call to this function acquires the write lock associated @p + * seqlock, and marks the beginning of a write-side critical section. + * + * After having called this function, the caller may go on to modify + * (both read and write) the protected data, in an atomic or + * non-atomic manner. + * + * After the necessary updates have been performed, the application + * calls rte_seqlock_write_unlock(). + * + * This function is not preemption-safe in the sense that preemption + * of the calling thread may block reader progress until the writer + * thread is rescheduled. + * + * Unlike rte_seqlock_read_begin(), each call made to + * rte_seqlock_write_lock() must be matched with an unlock call. + * + * @param seqlock + * A pointer to the seqlock. + * + * @see rte_seqlock_write_unlock() + */ +__rte_experimental +static inline void +rte_seqlock_write_lock(rte_seqlock_t *seqlock) +{ + uint32_t sn; + + /* to synchronize with other writers */ + rte_spinlock_lock(&seqlock->lock); + + sn = seqlock->sn + 1; + + __atomic_store_n(&seqlock->sn, sn, __ATOMIC_RELAXED); + + /* __ATOMIC_RELEASE to prevent stores after (in program order) + * from happening before the sn store. + */ + rte_atomic_thread_fence(__ATOMIC_RELEASE); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * End a write-side critical section. + * + * A call to this function marks the end of the write-side critical + * section, for @p seqlock. After this call has been made, the protected + * data may no longer be modified. + * + * @param seqlock + * A pointer to the seqlock. + * + * @see rte_seqlock_write_lock() + */ +__rte_experimental +static inline void +rte_seqlock_write_unlock(rte_seqlock_t *seqlock) +{ + uint32_t sn; + + sn = seqlock->sn + 1; + + /* synchronizes-with the load acquire in rte_seqlock_read_begin() */ + __atomic_store_n(&seqlock->sn, sn, __ATOMIC_RELEASE); + + rte_spinlock_unlock(&seqlock->lock); +} + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_SEQLOCK_H_ */ diff --git a/lib/eal/version.map b/lib/eal/version.map index b53eeb30d7..4a9d0ed899 100644 --- a/lib/eal/version.map +++ b/lib/eal/version.map @@ -420,6 +420,9 @@ EXPERIMENTAL { rte_intr_instance_free; rte_intr_type_get; rte_intr_type_set; + + # added in 22.07 + rte_seqlock_init; }; INTERNAL { -- 2.25.1