From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A7C545952; Tue, 10 Sep 2024 08:31:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 70B40427A7; Tue, 10 Sep 2024 08:30:26 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2045.outbound.protection.outlook.com [40.107.21.45]) by mails.dpdk.org (Postfix) with ESMTP id 692DE42759 for ; Tue, 10 Sep 2024 08:30:20 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wH8hxK8+tfu2A6enENFIA43BuTuzChK6Ga8s0LssQgpWCczCghm46nkQU8LMbVYuGHRbn8X20DCuHFTthQ7IYFOyk1SeBfe2vVxb6Nd+rnt+S/faJq/OplNcyP66lKa8OjPx4bNQvLFv6oCjjSNdGbuBYnjLZRj3yG6aidsSYCWR3Dgs1S8Xipekcwma0B6wm7O/HrMs5Us/iZxfdg27lSWSJEJk7jJW689Br8O9MVPIU6b3HuBHC1Edp7OCFkk/elvBZEVADCUJo3jSu8HCue3bypFg/GsDfvXgXa57yVncFb6Vr9mudd3fiVd+MCyTwmDAWgUx9Kk2eDTkBKwm4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f1GVgNVv8UHqCUFk9HxcWAxSXvibN/FETLdhUNajtSQ=; b=Xem3iEHKWaAvM0COjWHl125aiMzL1otwzof7vkGxzPPVutPXs141DBa0E40vJWSUC6N4BxA+u7qjq8+zDDY3FSqeuH4+A40aWOuu0pJo9H7ivx+Ds5yDFO9ld3cAJH3qdo0fwTHQYRjS+nJE8u8U1lWgNTaIcZg8A284sCOiMXMe42ntGj0a5JTRBBsWYdkjnEkR/CfAMYIbd72aMmKjS673RqByqOYi9nCF/wXQ/2cvhLczmDIq5zb6KfA7INJnMcZDcylt8goQ1JEiAdMYzuYraddDES3GBXodAAc2Nb0lCoBrQjY8XOZ4lOzSVqzsLtQ1rE2sYHYMkLLWqBEGIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f1GVgNVv8UHqCUFk9HxcWAxSXvibN/FETLdhUNajtSQ=; b=FOnUg1LY9DDIyit8gtl3FpEc+21CqiK3o8+IeO5Ytn0qU+vZsvCs+YNmJxb/UGW73HLse8eSHoMIlry1Ie+MKP7qyZVTay7r612o9DjEoblayH2Csqfszav7iuSvwh3P1I7wyDAZPEu1ndz996b9H3QGg6vKYbPtstQuC9Ai2j/R55aDB6p4gzCwl7nq2VuYrmB5qsv1EHaolHvXwyShfuUINGSVOBkpHLb9V1uUrINHjRj9+rjYjpsYPFyQv42+jT8OZB5rSSLxRIVkkfIh1coXnPGIT1XE/gxTAxaT0RmkU2zp0b5pbE2zn3zbvMWbz9CxkvfjADD6K86hQ1aWAw== Received: from AS4P191CA0012.EURP191.PROD.OUTLOOK.COM (2603:10a6:20b:5d5::11) by DUZPR07MB10037.eurprd07.prod.outlook.com (2603:10a6:10:4ae::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24; Tue, 10 Sep 2024 06:30:14 +0000 Received: from AM4PEPF00027A64.eurprd04.prod.outlook.com (2603:10a6:20b:5d5:cafe::45) by AS4P191CA0012.outlook.office365.com (2603:10a6:20b:5d5::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Tue, 10 Sep 2024 06:30:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AM4PEPF00027A64.mail.protection.outlook.com (10.167.16.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Tue, 10 Sep 2024 06:30:14 +0000 Received: from seliicinfr00050.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.64) with Microsoft SMTP Server id 15.2.1544.11; Tue, 10 Sep 2024 08:30:12 +0200 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00050.seli.gic.ericsson.se (Postfix) with ESMTP id C9C021C006B; Tue, 10 Sep 2024 08:30:12 +0200 (CEST) From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= To: CC: , Heng Wang , "Stephen Hemminger" , Tyler Retzlaff , =?UTF-8?q?Morten=20Br=C3=B8rup?= , Jack Bond-Preston , David Marchand , Chengwen Feng , =?UTF-8?q?Mattias=20R=C3=B6nnblom?= Subject: [PATCH v5 4/6] eal: add atomic bit operations Date: Tue, 10 Sep 2024 08:20:49 +0200 Message-ID: <20240910062051.699096-5-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240910062051.699096-1-mattias.ronnblom@ericsson.com> References: <20240909145743.697342-2-mattias.ronnblom@ericsson.com> <20240910062051.699096-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00027A64:EE_|DUZPR07MB10037:EE_ X-MS-Office365-Filtering-Correlation-Id: d5befaad-c4d4-4b6f-87cc-08dcd1620763 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?B?UWFjN2dKajY1U29yemg3SXovcXJmZmg1RXdWU0dZeUQxeCtTbkRmdENWeDBD?= =?utf-8?B?d1EwcXBYVW9DZmZNb3dCaTdMakFnUXQwbXRPenF6ZmxDNmdhSkE2ZHF3cTk4?= =?utf-8?B?WlA4d1YrdWhjZEV4a0c2ck5wTXpBVjNuMGQ5TXNYYkVpT2FLZkdFb1VvNzFs?= =?utf-8?B?eFpiVUFxWVZWYURBcGo0WWVzL2RwTmxMNVQyM3ZZWkw5VUw2dWVDQ1NHUFA3?= =?utf-8?B?UUxqcEhYZmJVUkRRSVVVUTBDdVhEMjJhTVdPQkFuRFJTbG9BVVlqcFRtSXZo?= =?utf-8?B?UHM4MEZVSHNJRFNteXJMeFVwMFN1NmZvRlhPOXA3KzY2RC9GWW9IK3ljZXBQ?= =?utf-8?B?a3ZDWlc4VzBhb3RleDhVOTZmUnZLTWNDbDdmN2tEN0xvUFUwQXBMa1VLNWoy?= =?utf-8?B?VUJWN0pEaUlOb0FLNXZJSGVLU2Vsd3dYRVdqWU9kS2p0Y3pTcUdDQjdZUlZn?= =?utf-8?B?Umt5SGNVbEhiTE41QWxTbHZsaTJHM1IrQ2o3Wmh4NUw0OGpxR1VNc3lnUnZI?= =?utf-8?B?bGJvdjh1WXg1QzBDWEZtWVNrUEp0OUlGcmJvckVLbDVrR09oeC9STG0vR01J?= =?utf-8?B?RHZkN28xWC9DSHVvQllTc05oSkQreStObkM3eTFiMGV1Q0Jqdm9HQ1owby85?= =?utf-8?B?U253VS9nNUpGWXlYRnFRdHZ4Q0E4T2pjT1p4R3Y1LzExUmNyYlptc2NRU3hS?= =?utf-8?B?RVRaQ1Jjd1krRHdkY0pKVFVjTjJmUEtQcjYvSWNna1Y5WTVpTWdYMS95T09s?= =?utf-8?B?SjlTUTFQVkVQeGJJbFdlU1NVNmFXQURvdnJiaDBlcHNJMDRQYUtyTlB6WER1?= =?utf-8?B?Tm1HcE8zUktUM2NNb0xlYkwydFFjZ3J5M1ZVYUVaMVNFQXViWHNMdkhScEVV?= =?utf-8?B?N1FuNW51ZmpPR2ZCcDRoQ3E3b1ppQ2JrU2NnMU01b1JJY1ZScWtTZU92dXpu?= =?utf-8?B?ZXY4YTVkRzFneHh2OTNHQStSd2xDTENtd2NVa0lkTkN2eXZFVWgxTVlva0Nv?= =?utf-8?B?MHBlSXFmWHl1d3p2SDFxdG5lZnpFejZKbU0wbytlYVRUaFFIVVJ6NWVrSlo4?= =?utf-8?B?bVZrZFFFYkR5SUFuemlGWDNTc29waU5oTnFIOGlYbEtMQ0RVcUNZOHdFcnBC?= =?utf-8?B?NG5abnVtcUdLOThpTldqZlArZ1hlUDRQZ3c0NkpDTG1HRFEzZzRyN1BCQ1RH?= =?utf-8?B?TU12SVFLai81Z3pXTWVJNVE0WGt4Q0pJN2MxODlwTVNuWkowYVhwY0kzN2M1?= =?utf-8?B?NmxOdzBkakZZSlhqZUU1ZktuRXpCZjMrdnA4Wm9jeXRCQ2V2WkdJU0RXY2lq?= =?utf-8?B?R0hkdjRZVGQ5T1oyUmxYWXRoQ0lKb1o4L2dmaEVlUnNIaEFwOGdBbmE0ZDJp?= =?utf-8?B?Z1dKZEdScTAvOGR2dDM2b2N3Wk9RMEJiS3VqeEQ3eFNicXdvRXAwbmtOSHZo?= =?utf-8?B?SkFCcElDcWp0K2hOUDEzbCsya3lVbjZ4SHhBMXQvTnpqQkhoTktScEJnK0to?= =?utf-8?B?SW1MOEQwTlJLV3ZaL1dNTGowbFJTdTlxeis3cTlNY1pnTHhSMHBCcWJ0V1ZC?= =?utf-8?B?TjFjTWpiek40TVRiMkRpRXpCM0hpU3RJWG1XazRnaCtJQ29pSUZVUGxLbmtl?= =?utf-8?B?b0tDSjBKTXlyNUZyMlRrZGFuK29LTzQwN3B2dCswcnEwVEF0WExSbWFTNFhO?= =?utf-8?B?VUU4WEJ1WmN6d3BUYzNFdzlSNFgveWJyZDE4STEvT24vQWZIZ2NIMWl0SG5q?= =?utf-8?B?ci8vUU1xZ0lOSENRSmVJZENLYkVMTWFEU1lWamRPa3ZZZ00wZmpTRTcvSU5Z?= =?utf-8?B?cFh2bFFhMGVYT2NmVGJNQjRyRXZ1NHBFT3RJNTh3V2tTV0hlWkN4SlAxRXUy?= =?utf-8?B?L3FBaEtIK3ZHK3RkRGJEZ2g5Qk9SN09ITlVNQnRMdjVKSTBITVB1VklBWjdz?= =?utf-8?Q?KVpvbZMT4FRPd6fRxNjD4KF6eXIG/0S/?= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Sep 2024 06:30:14.0262 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d5befaad-c4d4-4b6f-87cc-08dcd1620763 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00027A64.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DUZPR07MB10037 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add atomic bit test/set/clear/assign/flip and test-and-set/clear/assign/flip functions. All atomic bit functions allow (and indeed, require) the caller to specify a memory order. Signed-off-by: Mattias Rönnblom Acked-by: Morten Brørup Acked-by: Tyler Retzlaff Acked-by: Jack Bond-Preston -- PATCH v3: * Introduce __RTE_GEN_BIT_ATOMIC_*() 'qualifier' argument already in this patch (Jack Bond-Preston). * Refer to volatile bit op functions as variants instead of families (macro parameter naming). * Update release notes. PATCH: * Add missing macro #undef for C++ version of atomic bit flip. RFC v7: * Replace compare-exchange-based rte_bitset_atomic_test_and_*() and flip() with implementations that use the previous value as returned by the atomic fetch function. * Reword documentation to match the non-atomic macro variants. * Remove pointer to for memory model documentation, since there is no documentation for that API. RFC v6: * Have rte_bit_atomic_test() accept const-marked bitsets. RFC v4: * Add atomic bit flip. * Mark macro-generated private functions experimental. RFC v3: * Work around lack of C++ support for _Generic (Tyler Retzlaff). RFC v2: o Add rte_bit_atomic_test_and_assign() (for consistency). o Fix bugs in rte_bit_atomic_test_and_[set|clear](). o Use to support MSVC. --- doc/guides/rel_notes/release_24_11.rst | 17 + lib/eal/include/rte_bitops.h | 415 +++++++++++++++++++++++++ 2 files changed, 432 insertions(+) diff --git a/doc/guides/rel_notes/release_24_11.rst b/doc/guides/rel_notes/release_24_11.rst index 0ff70d9057..3111b1e4c0 100644 --- a/doc/guides/rel_notes/release_24_11.rst +++ b/doc/guides/rel_notes/release_24_11.rst @@ -56,6 +56,23 @@ New Features ======================================================= +* **Extended bit operations API.** + + The support for bit-level operations on single 32- and 64-bit words + in has been extended with two families of + semantically well-defined functions. + + rte_bit_[test|set|clear|assign|flip]() functions provide excellent + performance (by avoiding restricting the compiler and CPU), but give + no guarantees in regards to memory ordering or atomicity. + + rte_bit_atomic_*() provides atomic bit-level operations, including + the possibility to specifying memory ordering constraints. + + The new public API elements are polymorphic, using the _Generic- + based macros (for C) and function overloading (in C++ translation + units). + Removed Items ------------- diff --git a/lib/eal/include/rte_bitops.h b/lib/eal/include/rte_bitops.h index 6915b945ba..3ad6795fd1 100644 --- a/lib/eal/include/rte_bitops.h +++ b/lib/eal/include/rte_bitops.h @@ -21,6 +21,7 @@ #include #include +#include #ifdef __cplusplus extern "C" { @@ -226,6 +227,204 @@ extern "C" { uint32_t *: __rte_bit_flip32, \ uint64_t *: __rte_bit_flip64)(addr, nr) +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Test if a particular bit in a word is set with a particular memory + * order. + * + * Test a bit with the resulting memory load ordered as per the + * specified memory order. + * + * @param addr + * A pointer to the word to query. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit is set, and false otherwise. + */ +#define rte_bit_atomic_test(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test32, \ + const uint32_t *: __rte_bit_atomic_test32, \ + uint64_t *: __rte_bit_atomic_test64, \ + const uint64_t *: __rte_bit_atomic_test64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically set bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in + * the word pointed to by @c addr to '1', with the memory ordering as + * specified by @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_set32, \ + uint64_t *: __rte_bit_atomic_set64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically clear bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in + * the word pointed to by @c addr to '0', with the memory ordering as + * specified by @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_clear32, \ + uint64_t *: __rte_bit_atomic_clear64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically assign a value to bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in the + * word pointed to by @c addr to the value indicated by @c value, with + * the memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_assign32, \ + uint64_t *: __rte_bit_atomic_assign64)(addr, nr, value, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically flip bit in word. + * + * Generic selection macro to atomically negate the value of the bit + * specified by @c nr in the word pointed to by @c addr to the value + * indicated by @c value, with the memory ordering as specified with + * @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_flip(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_flip32, \ + uint64_t *: __rte_bit_atomic_flip64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and set a bit in word. + * + * Generic selection macro to atomically test and set bit specified by + * @c nr in the word pointed to by @c addr to '1', with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_set32, \ + uint64_t *: __rte_bit_atomic_test_and_set64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and clear a bit in word. + * + * Generic selection macro to atomically test and clear bit specified + * by @c nr in the word pointed to by @c addr to '0', with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_clear32, \ + uint64_t *: __rte_bit_atomic_test_and_clear64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and assign a bit in word. + * + * Generic selection macro to atomically test and assign bit specified + * by @c nr in the word pointed to by @c addr the value specified by + * @c value, with the memory ordering as specified with @c + * memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_assign32, \ + uint64_t *: __rte_bit_atomic_test_and_assign64)(addr, nr, \ + value, \ + memory_order) + #define __RTE_GEN_BIT_TEST(variant, qualifier, size) \ __rte_experimental \ static inline bool \ @@ -299,6 +498,146 @@ extern "C" { __RTE_GEN_BIT_OPS_SIZE(32) __RTE_GEN_BIT_OPS_SIZE(64) +#define __RTE_GEN_BIT_ATOMIC_TEST(variant, qualifier, size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_ ## variant ## test ## size(const qualifier uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + const qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (const qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + return rte_atomic_load_explicit(a_addr, memory_order) & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_SET(variant, qualifier, size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_ ## variant ## set ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_or_explicit(a_addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_CLEAR(variant, qualifier, size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_ ## variant ## clear ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_and_explicit(a_addr, ~mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_FLIP(variant, qualifier, size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_ ## variant ## flip ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_xor_explicit(a_addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_ASSIGN(variant, qualifier, size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_## variant ## assign ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, bool value, \ + int memory_order) \ + { \ + if (value) \ + __rte_bit_atomic_ ## variant ## set ## size(addr, nr, memory_order); \ + else \ + __rte_bit_atomic_ ## variant ## clear ## size(addr, nr, \ + memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_SET(variant, qualifier, size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_ ## variant ## test_and_set ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + uint ## size ## _t prev; \ + \ + prev = rte_atomic_fetch_or_explicit(a_addr, mask, \ + memory_order); \ + \ + return prev & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_CLEAR(variant, qualifier, size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_ ## variant ## test_and_clear ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + qualifier RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (qualifier RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + uint ## size ## _t prev; \ + \ + prev = rte_atomic_fetch_and_explicit(a_addr, ~mask, \ + memory_order); \ + \ + return prev & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(variant, qualifier, size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_ ## variant ## test_and_assign ## size(qualifier uint ## size ## _t *addr, \ + unsigned int nr, \ + bool value, \ + int memory_order) \ + { \ + if (value) \ + return __rte_bit_atomic_ ## variant ## test_and_set ## size(addr, nr, memory_order); \ + else \ + return __rte_bit_atomic_ ## variant ## test_and_clear ## size(addr, nr, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_OPS(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_TEST(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_SET(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_CLEAR(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_ASSIGN(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_SET(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_CLEAR(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(variant, qualifier, size) \ + __RTE_GEN_BIT_ATOMIC_FLIP(variant, qualifier, size) + +#define __RTE_GEN_BIT_ATOMIC_OPS_SIZE(size) \ + __RTE_GEN_BIT_ATOMIC_OPS(,, size) + +__RTE_GEN_BIT_ATOMIC_OPS_SIZE(32) +__RTE_GEN_BIT_ATOMIC_OPS_SIZE(64) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** @@ -994,6 +1333,15 @@ rte_log2_u64(uint64_t v) #undef rte_bit_assign #undef rte_bit_flip +#undef rte_bit_atomic_test +#undef rte_bit_atomic_set +#undef rte_bit_atomic_clear +#undef rte_bit_atomic_assign +#undef rte_bit_atomic_flip +#undef rte_bit_atomic_test_and_set +#undef rte_bit_atomic_test_and_clear +#undef rte_bit_atomic_test_and_assign + #define __RTE_BIT_OVERLOAD_SZ_2(fun, qualifier, size, arg1_type, arg1_name) \ static inline void \ rte_bit_ ## fun(qualifier uint ## size ## _t *addr, \ @@ -1037,12 +1385,79 @@ rte_log2_u64(uint64_t v) __RTE_BIT_OVERLOAD_SZ_3(fun, qualifier, 64, arg1_type, arg1_name, \ arg2_type, arg2_name) +#define __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name); \ + } + +#define __RTE_BIT_OVERLOAD_3R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) + +#define __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, size, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + static inline void \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4(fun, qualifier, arg1_type, arg1_name, arg2_type, \ + arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 32, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 64, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) + +#define __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) + __RTE_BIT_OVERLOAD_2R(test, const, bool, unsigned int, nr) __RTE_BIT_OVERLOAD_2(set,, unsigned int, nr) __RTE_BIT_OVERLOAD_2(clear,, unsigned int, nr) __RTE_BIT_OVERLOAD_3(assign,, unsigned int, nr, bool, value) __RTE_BIT_OVERLOAD_2(flip,, unsigned int, nr) +__RTE_BIT_OVERLOAD_3R(atomic_test, const, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_set,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_clear,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_4(atomic_assign,, unsigned int, nr, bool, value, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_flip,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_set,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_clear,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_4R(atomic_test_and_assign,, bool, unsigned int, nr, + bool, value, int, memory_order) + #endif #endif /* _RTE_BITOPS_H_ */ -- 2.34.1