From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 2202A43F04;
	Thu, 25 Apr 2024 11:10:22 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 443BE436DD;
	Thu, 25 Apr 2024 11:10:05 +0200 (CEST)
Received: from EUR03-DBA-obe.outbound.protection.outlook.com
 (mail-dbaeur03on2067.outbound.protection.outlook.com [40.107.104.67])
 by mails.dpdk.org (Postfix) with ESMTP id 0BAEE436C8
 for <dev@dpdk.org>; Thu, 25 Apr 2024 11:10:02 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=dwYCK1/aIjUjCBbv9sggvwNApATG6KAWBLGcUDlVqoN0dw0kcUBraRs2sgU+A/vC97FX5YmyiRP+fn7fvM83TAQRcWICJsg8Mx4d5qC0J7RpTwsyl6vuRk4CVDTsYN55vm+CJ7IuLF2cDz+wkujN7wlh0FuVq3/0UMPjrcKsyrDH+tYaI1e2hfh1S3W/xQIc+8wq844/KWkWZ/xa420brwJap2VemKKsnzLW9terlWJ7lX6B+oj+xmDdyZmcdvy1Ei3aDBjOXhUxytwzBjYSwmSUjMPOVrPgcR3nz1ZdJRzSFTkJtIyLxvYb9lvI1scdxa/KI0acFdO0OlQ6gw/TrQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=m127R3HU9pNG3bWTrW0IgoJ9+Gojo1qtktVZxd6EF04=;
 b=WqyyjqfglsJmRJPc3nQXD5vW9H6lpTQ+RlFiQtrCMpS3+ACKanXUH/HYG1hBjlR9VHK+QU+F/WmiUKG0liffLjQXOrdkuj2DLBarMN7mPHv6iTGsPR65qZubMnuTuM/qJ0drOlBOjiv1qWG2LalXuWcoqLsw+rhHsorQFm80wUTphqN7tDNVCkZWf00CF3Lpo2hJO7v4e9r7ykHw4zK+UmbcAZv6+3wTrwxdnp1596OBvkiMinMw5KkXsD9L1alMambUIg6/xtBgXRIkQbrjoM0snvZX3bAjSGLsh8rEns/pPFokgWMV/d5kiVuJ4LjxLL7gMrZaW0VbtH21baPjDA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com;
 dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; 
 dkim=none (message not signed); arc=none (0)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=m127R3HU9pNG3bWTrW0IgoJ9+Gojo1qtktVZxd6EF04=;
 b=EXcjhBGE39dmj6/PA/QAGws6O2jMxFVoUcR9xDqrPJ/Ez3PDZPRSlYBAF+zvWoqD4JTibpo+Ed3Stig0IwAVSed+1GfsEFjNCY32vITAxMr9UeofAu+ziv21mpjGtCONOYT9ZP3+pD+O67a9OWjkNjS9FjAAaeB+Tm13Pp7S7nHk8wHy705EyD2zY9kLO1IJ+jw9lI4Nh/psMa2p7X0nl+duEbeCM09Il5G5L1+BKi9Bfhv4w5mjj49X84YH5WT+mN+zFggMVM1nHlPaA3HvPEFvkz/1gyoEw1NhQm39Ru17bkAeXU/wLaKkq6aE8rEe6/gNP5G1CLilBSnKVCCkmA==
Received: from DUZP191CA0028.EURP191.PROD.OUTLOOK.COM (2603:10a6:10:4f8::24)
 by DBAPR07MB6807.eurprd07.prod.outlook.com (2603:10a6:10:19f::18) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Thu, 25 Apr
 2024 09:09:59 +0000
Received: from DU2PEPF0001E9C2.eurprd03.prod.outlook.com
 (2603:10a6:10:4f8:cafe::52) by DUZP191CA0028.outlook.office365.com
 (2603:10a6:10:4f8::24) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.25 via Frontend
 Transport; Thu, 25 Apr 2024 09:09:59 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74)
 smtp.mailfrom=ericsson.com; dkim=none (message not signed)
 header.d=none;dmarc=pass action=none header.from=ericsson.com;
Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates
 192.176.1.74 as permitted sender)
 receiver=protection.outlook.com; 
 client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C
Received: from oa.msg.ericsson.com (192.176.1.74) by
 DU2PEPF0001E9C2.mail.protection.outlook.com (10.167.8.71) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.7519.19 via Frontend Transport; Thu, 25 Apr 2024 09:09:59 +0000
Received: from seliicinfr00050.seli.gic.ericsson.se (153.88.142.248) by
 smtp-central.internal.ericsson.com (100.87.178.65) with Microsoft SMTP Server
 id 15.2.1258.12; Thu, 25 Apr 2024 11:09:59 +0200
Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100])
 by seliicinfr00050.seli.gic.ericsson.se (Postfix) with ESMTP id
 E238F1C006A; Thu, 25 Apr 2024 11:09:58 +0200 (CEST)
From: =?UTF-8?q?Mattias=20R=C3=B6nnblom?= <mattias.ronnblom@ericsson.com>
To: <dev@dpdk.org>
CC: <hofors@lysator.liu.se>, Heng Wang <heng.wang@ericsson.com>, "Stephen
 Hemminger" <stephen@networkplumber.org>, Tyler Retzlaff
 <roretzla@linux.microsoft.com>, =?UTF-8?q?Mattias=20R=C3=B6nnblom?=
 <mattias.ronnblom@ericsson.com>
Subject: [RFC v2 5/6] eal: add atomic bit operations
Date: Thu, 25 Apr 2024 10:58:52 +0200
Message-ID: <20240425085853.97888-6-mattias.ronnblom@ericsson.com>
X-Mailer: git-send-email 2.34.1
In-Reply-To: <20240425085853.97888-1-mattias.ronnblom@ericsson.com>
References: <20240302135328.531940-2-mattias.ronnblom@ericsson.com>
 <20240425085853.97888-1-mattias.ronnblom@ericsson.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DU2PEPF0001E9C2:EE_|DBAPR07MB6807:EE_
X-MS-Office365-Filtering-Correlation-Id: a656cc31-d8ad-4bb5-5e8f-08dc65077c01
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: =?utf-8?B?SSs4VHlIb2V0ZG5kUE1GOE5MNTM0c0dVazNZMWpkWEgyQzZSUGhnV29kMGM2?=
 =?utf-8?B?L0R2RERWMXY3YlNyR21aVGs1MWpONDVRMEVoMk1Yd0hYOGNFZ1FjNmJuYTJy?=
 =?utf-8?B?MFFTY215WGQvK2V2Mm05UVp2Smptck9pMjJibzF3YkJLTTFrOHlZYUVEbGZy?=
 =?utf-8?B?bTNCN0p2VktIZHlIaS9SQkRoSU1CeHhKNmhKYUV6T2s0Y1VFSGc5M0k1LzZ3?=
 =?utf-8?B?Vm00QXBwZWlHRkQvbnFXaDhxT1BLaVB2eDk2cjNkR3FxbTdxRWUzWUNpVlYv?=
 =?utf-8?B?U2N3ZmxpVm5FcC9mRC9Da0poWCtUYWFwUlE3YS9SRWc3RHBQUDJEdy9ldnMv?=
 =?utf-8?B?cTJFM0hJTkQ5MlUzbXlKaWZtQ2NSNlFmUlUza2FOQzdwUmNZcjgxSU9UR3VF?=
 =?utf-8?B?TG4vdVFnMURkNDh0blBsekJXWW5EQk1Ia0wxeWsvRlZvTmhMNFFCM3pEZCtq?=
 =?utf-8?B?bmRRLys4VVhObXBIQ1BEc1lLdmRhdWhGN2xzWTJ3d3AzWCtQYy8wNkFMZWY3?=
 =?utf-8?B?YkZHaGdOZzlUWUE1VWxISFE5RE80OUR6aGkvckVjcGZ0YWxJYzJVSHUrMkNE?=
 =?utf-8?B?bFVRUE1tZ0FKTU96ZjZURVE0OEJnTlRvNEFkeEpuK3hqZFFmMWlPdjFuMS9U?=
 =?utf-8?B?a3JXdXArYmdoMWZEcjIrUWNOWldaVGtKaEw3dGRXVDkzQ2NSSUVrUGRQMXor?=
 =?utf-8?B?bkpSaVlNNmlWZVE0eFRFb2hScGlPa0k1ZUV1SzRFbDN6b3d2bjJPSlUzR3pR?=
 =?utf-8?B?M1RLanRuaDBBRmlDWGp1UnBPV2tvQVJYMXVMWVZiZThaTTVRT3NPdEN1QWxh?=
 =?utf-8?B?bnp5c3FqUzFXZzFzNzZSeU9PUGNHL2NsaEt6TWJYRW03aGRtR0F4T1NrcFZZ?=
 =?utf-8?B?OHFRSEpveFJkYjVmbENJMzV4L0ZERHN5WU1tMG5mMS9Gc1dqdkhuVVVwYyt1?=
 =?utf-8?B?VUdNZHNWYU5Xc3BCRHoyRjlwYnRWbTkvbFR2VGhHZnFDdGxmTUsyNmxuSWhi?=
 =?utf-8?B?SEl2MTIycHBKd3k4NmxkbU5kWXUxLzdkQXlnUHBsQm5XQ3dSTDlRV0Z5Ymdr?=
 =?utf-8?B?cGw2UjBlZnI0VFJ3MnNOaUZPSmc1OFcxZWNEN1ZlSTdFbll2K3BLOHhKRkdl?=
 =?utf-8?B?MURSVXhFNzBSQXRRYzgvdzJKL2tyUHBDYW1nSkN3NFVPbHFFb2phVHNKWkhO?=
 =?utf-8?B?cWJ6Y2liKzdNOGEraGxhVitUWUg2dEpvQlJZVlF0UlB4R2syTFdBRHg4YS9W?=
 =?utf-8?B?VC83cWNvczhkSngyK2RyWlN5S21zOGZBQlVKVnljdVlTOUx1MFJ5NS9Eakox?=
 =?utf-8?B?WjVES0hlaGFFSkNZdDBzdTdsWkNUbGc5QjJuMk1CTXhEdHZXY2ZienBjMzJh?=
 =?utf-8?B?R1BaTXhzNXVTbUpQakFkUHZBRVZ0eWRhYlRadmJMaUsycWQwM2VkYjNsdDlN?=
 =?utf-8?B?bWhmTUJpYStTR3VYNUhIcUMzTkRVTEtUSkhlRjRvYU5rbHhJdGRMOHdzUGUz?=
 =?utf-8?B?c0NobGhsWGlBY015RENIVmE4cEpWcSs3NGQvTGZ1ZEh2VDl2NmpQRTJkaDFs?=
 =?utf-8?B?YTBHSkJSaG40M2JEbGFIWlBrVy9udmNRaXdnL01IQkU4cUJ2dFdzbm81T0Jw?=
 =?utf-8?B?TnV1UnkvNE9rcjFHOVFBREUvT0NoVkpOMmVEQWZ4Qm1UNCs5MC85Y2hqV3Fz?=
 =?utf-8?B?NVBGOVo3azg3MXorenNWaGsya1hLaEozTExuQ1gzR1dhdjJaM1pkbEFPcHpE?=
 =?utf-8?B?ek90cWdGaGV3Qkg2b28rN3NvYVBrS2JhZVlJY254K0h1Mi85eUFTU3pLc3g5?=
 =?utf-8?B?QktoRmh4NGRFTXZnbkVFUmE1d2lwT055cXlSTk9wbnNxc0FWTG9GY2MwQXlZ?=
 =?utf-8?Q?HRhJ7lU6B5Dw8?=
X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net;
 CAT:NONE; SFS:(13230031)(376005)(36860700004)(82310400014)(1800799015);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: ericsson.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2024 09:09:59.8471 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: a656cc31-d8ad-4bb5-5e8f-08dc65077c01
X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74];
 Helo=[oa.msg.ericsson.com]
X-MS-Exchange-CrossTenant-AuthSource: DU2PEPF0001E9C2.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR07MB6807
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

Add atomic bit test/set/clear/assign and test-and-set/clear functions.

All atomic bit functions allow (and indeed, require) the caller to
specify a memory order.

RFC v2:
 o Add rte_bit_atomic_test_and_assign() (for consistency).
 o Fix bugs in rte_bit_atomic_test_and_[set|clear]().
 o Use <rte_stdatomics.h> to support MSVC.

Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
---
 lib/eal/include/rte_bitops.h | 297 +++++++++++++++++++++++++++++++++++
 1 file changed, 297 insertions(+)

diff --git a/lib/eal/include/rte_bitops.h b/lib/eal/include/rte_bitops.h
index a2746e657f..8c38a1ac03 100644
--- a/lib/eal/include/rte_bitops.h
+++ b/lib/eal/include/rte_bitops.h
@@ -21,6 +21,7 @@
 
 #include <rte_compat.h>
 #include <rte_debug.h>
+#include <rte_stdatomic.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -342,6 +343,177 @@ extern "C" {
 		 uint32_t *: __rte_bit_once_assign32,			\
 		 uint64_t *: __rte_bit_once_assign64)(addr, nr, value)
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Test if a particular bit in a word is set with a particular memory
+ * order.
+ *
+ * Test a bit with the resulting memory load ordered as per the
+ * specified memory order.
+ *
+ * @param addr
+ *   A pointer to the word to query.
+ * @param nr
+ *   The index of the bit.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ * @return
+ *   Returns true if the bit is set, and false otherwise.
+ */
+#define rte_bit_atomic_test(addr, nr, memory_order)			\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_test32,			\
+		 uint64_t *: __rte_bit_atomic_test64)(addr, nr, memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically set bit in word.
+ *
+ * Atomically set bit specified by @c nr in the word pointed to by @c
+ * addr to '1', with the memory ordering as specified by @c
+ * memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ */
+#define rte_bit_atomic_set(addr, nr, memory_order)			\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_set32,			\
+		 uint64_t *: __rte_bit_atomic_set64)(addr, nr, memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically clear bit in word.
+ *
+ * Atomically set bit specified by @c nr in the word pointed to by @c
+ * addr to '0', with the memory ordering as specified by @c
+ * memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ */
+#define rte_bit_atomic_clear(addr, nr, memory_order)			\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_clear32,			\
+		 uint64_t *: __rte_bit_atomic_clear64)(addr, nr, memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically assign a value to bit in word.
+ *
+ * Atomically set bit specified by @c nr in the word pointed to by @c
+ * addr to the value indicated by @c value, with the memory ordering
+ * as specified with @c memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param value
+ *   The new value of the bit - true for '1', or false for '0'.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ */
+#define rte_bit_atomic_assign(addr, nr, value, memory_order)		\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_assign32,			\
+		 uint64_t *: __rte_bit_atomic_assign64)(addr, nr, value, \
+							memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically test and set a bit in word.
+ *
+ * Atomically test and set bit specified by @c nr in the word pointed
+ * to by @c addr to '1', with the memory ordering as specified with @c
+ * memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ * @return
+ *   Returns true if the bit was set, and false otherwise.
+ */
+#define rte_bit_atomic_test_and_set(addr, nr, memory_order)		\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_test_and_set32,		\
+		 uint64_t *: __rte_bit_atomic_test_and_set64)(addr, nr,	\
+							      memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically test and clear a bit in word.
+ *
+ * Atomically test and clear bit specified by @c nr in the word
+ * pointed to by @c addr to '0', with the memory ordering as specified
+ * with @c memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ * @return
+ *   Returns true if the bit was set, and false otherwise.
+ */
+#define rte_bit_atomic_test_and_clear(addr, nr, memory_order)		\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_test_and_clear32,		\
+		 uint64_t *: __rte_bit_atomic_test_and_clear64)(addr, nr, \
+								memory_order)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Atomically test and assign a bit in word.
+ *
+ * Atomically test and assign bit specified by @c nr in the word
+ * pointed to by @c addr the value specified by @c value, with the
+ * memory ordering as specified with @c memory_order.
+ *
+ * @param addr
+ *   A pointer to the word to modify.
+ * @param nr
+ *   The index of the bit.
+ * @param value
+ *   The new value of the bit - true for '1', or false for '0'.
+ * @param memory_order
+ *   The memory order to use. See <rte_stdatomics.h> for details.
+ * @return
+ *   Returns true if the bit was set, and false otherwise.
+ */
+#define rte_bit_atomic_test_and_assign(addr, nr, value, memory_order)	\
+	_Generic((addr),						\
+		 uint32_t *: __rte_bit_atomic_test_and_assign32,	\
+		 uint64_t *: __rte_bit_atomic_test_and_assign64)(addr, nr, \
+								 value, \
+								 memory_order)
+
 #define __RTE_GEN_BIT_TEST(name, size, qualifier)			\
 	static inline bool						\
 	name(const qualifier uint ## size ## _t *addr, unsigned int nr)	\
@@ -429,6 +601,131 @@ __rte_bit_once_assign64(volatile uint64_t *addr, unsigned int nr, bool value)
 		__rte_bit_once_clear64(addr, nr);
 }
 
+#define __RTE_GEN_BIT_ATOMIC_TEST(size)					\
+	static inline bool						\
+	__rte_bit_atomic_test ## size(const uint ## size ## _t *addr,	\
+				      unsigned int nr, int memory_order) \
+	{								\
+		RTE_ASSERT(nr < size);					\
+									\
+		const RTE_ATOMIC(uint ## size ## _t) *a_addr =		\
+			(const RTE_ATOMIC(uint ## size ## _t) *)addr;	\
+		uint ## size ## _t mask = (uint ## size ## _t)1 << nr;	\
+		return rte_atomic_load_explicit(a_addr, memory_order) & mask; \
+	}
+
+#define __RTE_GEN_BIT_ATOMIC_SET(size)					\
+	static inline void						\
+	__rte_bit_atomic_set ## size(uint ## size ## _t *addr,		\
+				     unsigned int nr, int memory_order)	\
+	{								\
+		RTE_ASSERT(nr < size);					\
+									\
+		RTE_ATOMIC(uint ## size ## _t) *a_addr =		\
+			(RTE_ATOMIC(uint ## size ## _t) *)addr;		\
+		uint ## size ## _t mask = (uint ## size ## _t)1 << nr;	\
+		rte_atomic_fetch_or_explicit(a_addr, mask, memory_order); \
+	}
+
+#define __RTE_GEN_BIT_ATOMIC_CLEAR(size)				\
+	static inline void						\
+	__rte_bit_atomic_clear ## size(uint ## size ## _t *addr,	\
+				       unsigned int nr, int memory_order) \
+	{								\
+		RTE_ASSERT(nr < size);					\
+									\
+		RTE_ATOMIC(uint ## size ## _t) *a_addr =		\
+			(RTE_ATOMIC(uint ## size ## _t) *)addr;		\
+		uint ## size ## _t mask = (uint ## size ## _t)1 << nr;	\
+		rte_atomic_fetch_and_explicit(a_addr, ~mask, memory_order); \
+	}
+
+#define __RTE_GEN_BIT_ATOMIC_ASSIGN(size)				\
+	static inline void						\
+	__rte_bit_atomic_assign ## size(uint ## size ## _t *addr,	\
+					unsigned int nr, bool value,	\
+					int memory_order)		\
+	{								\
+		if (value)						\
+			__rte_bit_atomic_set ## size(addr, nr, memory_order); \
+		else							\
+			__rte_bit_atomic_clear ## size(addr, nr,	\
+						       memory_order);	\
+	}
+
+#define __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size)			\
+	static inline bool						\
+	__rte_bit_atomic_test_and_assign ## size(uint ## size ## _t *addr, \
+						 unsigned int nr,	\
+						 bool value,		\
+						 int memory_order)	\
+	{								\
+		RTE_ASSERT(nr < size);					\
+									\
+		RTE_ATOMIC(uint ## size ## _t) *a_addr =		\
+			(RTE_ATOMIC(uint ## size ## _t) *)addr;		\
+		uint ## size ## _t before;				\
+		uint ## size ## _t target;				\
+									\
+		before = rte_atomic_load_explicit(a_addr,		\
+						  rte_memory_order_relaxed); \
+									\
+		do {							\
+			target = before;				\
+			__rte_bit_assign ## size(&target, nr, value);	\
+		} while (!rte_atomic_compare_exchange_weak_explicit(	\
+				a_addr, &before, target,		\
+				rte_memory_order_relaxed,		\
+				memory_order));				\
+		return __rte_bit_test ## size(&before, nr);		\
+	}
+
+#define __RTE_GEN_BIT_ATOMIC_OPS(size)			\
+	__RTE_GEN_BIT_ATOMIC_TEST(size)			\
+	__RTE_GEN_BIT_ATOMIC_SET(size)			\
+	__RTE_GEN_BIT_ATOMIC_CLEAR(size)		\
+	__RTE_GEN_BIT_ATOMIC_ASSIGN(size)		\
+	__RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size)
+
+__RTE_GEN_BIT_ATOMIC_OPS(32)
+__RTE_GEN_BIT_ATOMIC_OPS(64)
+
+__rte_experimental
+static inline bool
+__rte_bit_atomic_test_and_set32(uint32_t *addr, unsigned int nr,
+			      int memory_order)
+{
+	return __rte_bit_atomic_test_and_assign32(addr, nr, true,
+						  memory_order);
+}
+
+__rte_experimental
+static inline bool
+__rte_bit_atomic_test_and_clear32(uint32_t *addr, unsigned int nr,
+				int memory_order)
+{
+	return __rte_bit_atomic_test_and_assign32(addr, nr, false,
+						  memory_order);
+}
+
+__rte_experimental
+static inline bool
+__rte_bit_atomic_test_and_set64(uint64_t *addr, unsigned int nr,
+			      int memory_order)
+{
+	return __rte_bit_atomic_test_and_assign64(addr, nr, true,
+						  memory_order);
+}
+
+__rte_experimental
+static inline bool
+__rte_bit_atomic_test_and_clear64(uint64_t *addr, unsigned int nr,
+			      int memory_order)
+{
+	return __rte_bit_atomic_test_and_assign64(addr, nr, false,
+						  memory_order);
+}
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.34.1