From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A84CF470E5 for ; Thu, 25 Dec 2025 10:24:47 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6E54540658; Thu, 25 Dec 2025 10:24:39 +0100 (CET) Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazon11012038.outbound.protection.outlook.com [52.101.43.38]) by mails.dpdk.org (Postfix) with ESMTP id 2FF4A40658 for ; Thu, 25 Dec 2025 10:24:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iCl3TUTr3Cs68AUh+I+HYtvktoig4aNDT/6OQb050GGnfrdfZQcaYsXYs9raf0/Got0YFudKoCuN5WjVW7bUYSeTk/hLkg1d9Z12R8Um9jY6FWoe9RSn9ttGbGTG+tnDiOT/pqy7+LdqJUt6VBv09eqxhk+buQXgB7HZm+U2BZQWXjNezcKF9AS++eG/q51GzdKsyNrdG53Vorqi2ZW28Fk1+nOsSE17rC2o/rBn6HH2sVG/lzF4gEdovRDCZG18KKcONAnKgfCoHJ1xENmucdooG/TM2APSYpSW0+SN2zKjeXUSH/9HWfffBv6yPoKL/Cc/2ht8pAlcBdm4NKWnLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wCtW8+qN7vLTLkwkNF26xkWPHFNDcaZwEaD+JJacs48=; b=EX0ERfR/xLrO7/PeSI848b2fl5CkzXA9S4OoYzC4cxFxwX3SqGkgQ0ANDS4ptpA2n67vK3rr8ZaG3JtsmBzKoN8zvRFPhkqG3x329vsYN4tAondi/FaAZYXXkzZ3dAagIo8gszBOEj/3EaXzzdTNFca7LAitbluexlRBZGGaQavvwJAYGpZtwQw4ATwUzkp/0O62xSO2be04EZwQJAxk5txmVC+mWuu8mZvJL5fzZHoacz613LG6vVPbH+OoI4ZcD+y4k5IdalsKWTdZ2GT/PwgaIoMa62vRpnnDLL8tVpU3SLxCJv5k9B90Ay80vrXarBpkSOKnqpW9zwM35Urf+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=arm.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wCtW8+qN7vLTLkwkNF26xkWPHFNDcaZwEaD+JJacs48=; b=IYeMIkDhiGMy6Wpa6vd1cI8dB/sMGAaUGXqjJ7UgpS+FeEKiw2L2EhaOU3xOAOs07sh8GfYdCK4yH9igfqeeF6x1EkclfNhCOk5XVteM33DGMnjJxrt58/2bp86Eshsm/YuTljS7yN1YnU9gVJdbNuobA8ga34IsgSQXVn//GlmZsaA7iqP1SxIIOI+/5CnhR+cRPLnIMEzhvL0plRI87pSvMb45E2vpKCCkJCq1dvw2AXildIzHI+Swdr+GSoi/PMffOL1kDWLKBqIwi+5ZF621aYNLXcWSVMMv8jE1z7VsQSMbNfAoddZ+FzkMYvcIfSQCQL52leFk3OVpD+yEKg== Received: from SN6PR01CA0027.prod.exchangelabs.com (2603:10b6:805:b6::40) by LV8PR12MB9134.namprd12.prod.outlook.com (2603:10b6:408:180::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9456.11; Thu, 25 Dec 2025 09:24:31 +0000 Received: from SA2PEPF00003AEA.namprd02.prod.outlook.com (2603:10b6:805:b6:cafe::9d) by SN6PR01CA0027.outlook.office365.com (2603:10b6:805:b6::40) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9456.12 via Frontend Transport; Thu, 25 Dec 2025 09:24:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SA2PEPF00003AEA.mail.protection.outlook.com (10.167.248.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9456.9 via Frontend Transport; Thu, 25 Dec 2025 09:24:30 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 25 Dec 2025 01:24:22 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 25 Dec 2025 01:24:21 -0800 Received: from nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Thu, 25 Dec 2025 01:24:20 -0800 From: Shani Peretz To: Wathsala Vithanage CC: Ola Liljedahl , dpdk stable Subject: patch 'ring: establish a safe partial order in hts-ring' has been queued to stable release 23.11.6 Date: Thu, 25 Dec 2025 11:18:31 +0200 Message-ID: <20251225091938.345892-70-shperetz@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251225091938.345892-1-shperetz@nvidia.com> References: <20251221145746.763179-93-shperetz@nvidia.com> <20251225091938.345892-1-shperetz@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00003AEA:EE_|LV8PR12MB9134:EE_ X-MS-Office365-Filtering-Correlation-Id: 1a900a23-d08f-4580-16a2-08de439768c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|36860700013|30052699003|376014|82310400026|7053199007|13003099007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Iz/vPd8IkbUYKSOeTKkx8W+B5lZ/RdRtkP0StNtNVlo9o5lP9dqAXjKnKpj9?= =?us-ascii?Q?q5pQYCkRFPnl0+xOFJbwmaTZV5hUTl6OpNJitElZjLgeNUohZLG9i3ib6NeL?= =?us-ascii?Q?zzBdb1+L4idR6KX52i7HrH9Om/oMQU/Im5EFYcbXF+zVjQwKtA7qWPmA3mI0?= =?us-ascii?Q?NINB8TCxFtT70SMR7G4nEIxgWBK0H8AG24i7q9RgocsrQ1KlfIEaBZZWrykr?= =?us-ascii?Q?9dbHJARyT+NYF4ACTCCFHdv5fl8JQhNoR04XLUCumXmY3vc1dhzNdstu/A6e?= =?us-ascii?Q?htB8a1BQM3FA9XowMWk+PO+gT0c2IhDBUAN4mjq6TPsG0jCIQHRG1NaFgRP0?= =?us-ascii?Q?g6uuSVP+aGSXRiWxp7sS66s+MOMxaGvAqtgvYDhu0R5XCa7lPU6YoxC4f15U?= =?us-ascii?Q?O0uuPqYbtFAyS/ehHT+vx+mdnrJgnNDMTLIOhOUiu8vCdKV8J3Gsx+asP41P?= =?us-ascii?Q?zR4bydgReVjB1oTqVxlcbn8XtIf9GxmUZQyofEnrLwnRu0EjJnmxCU033676?= =?us-ascii?Q?qxWLHr9QLoht8aIB6AGYqDkbUgvEj6lhd+ujeIIpI5cgN4gCBULMX4UtbnBp?= =?us-ascii?Q?+TbRF/HihJd94JPcuhYF8UGnpg4UIgPalJDcKwlEY0QQBQTQCQa7PjfJCiyh?= =?us-ascii?Q?rhg+vO0yD1WOdkMXBevTLPQfv17IpwNYYUincBpj3E3i8NnDjQpn1GUJCbLW?= =?us-ascii?Q?fRM7sHVsu+A3xts7L2djzn1p0gq8XvHPAEc3tqFFlweha0FisV+w+4DQxr0t?= =?us-ascii?Q?R4EdrtMwo/KXtVtDPrWbgWcKMDkL5BLwxddvq40AipE4K4mF7f4RoVNmgUPL?= =?us-ascii?Q?F3E/Q3TUq2AN9arjnYl9758/p32tv6gEw70w6qBwXbcw+YO+GjupSnbKKFly?= =?us-ascii?Q?udkxnOT0706wlScFPDJebxaeCYnbsHK2JY8yJPy9g3ysoz7Wso/QVKCBzmYt?= =?us-ascii?Q?BMwyytINoYmSKtRCDTOZJYS+snyocoEeHGmWP1scvV835hSx4EqcGIinZIJb?= =?us-ascii?Q?chSfxAovSvoNErxosaNXwgYMJ+YL5hA0rOMguU+7/knXrYB7AeSVJRpFEDxC?= =?us-ascii?Q?9AgRZxZEsHG2swzj3LDiJxCduaSxXjCVQs2Xa3+5qjvyJH9KefIDmjlrlqYY?= =?us-ascii?Q?2sC4Y2Kwmggk04DftEDOHStaMjpU8rceH23VIRZ6zw9ar7VIsxMNFYOhMYn8?= =?us-ascii?Q?QcgVwrJTZZ3eSUtpOq+troB5Gp/lA8ZeFkv5wYKuwRvij4XvabY0vZ0eNpg/?= =?us-ascii?Q?GoPf9kCGY396JGosZF/ou8TF3ClJCj2JgQsN0uzk2HcNRTDXpkCgQDYWFB6F?= =?us-ascii?Q?008kG65xd0jRnZV5PiBnpjWkASVPC9ZVAeklPXgyclw+zjg8yg5W1yXEBbvK?= =?us-ascii?Q?1xchunQ5/XjjP2iVgfkubvbHOg4Q4dYg/3zLxb/OvJmbEhai8dvwuIGpb5CH?= =?us-ascii?Q?b+OezTxJyCUUrKm3HQk3XXSbyREhRlNZbm4NLO1os79AiFIsKEchEA89mnV9?= =?us-ascii?Q?L17vuu2tCRlRrwUQpYg1v3pv6qqMa01QDhYolJI1JsZ9OP1lhvkUcj2wlArx?= =?us-ascii?Q?/HLECOUBiS3/5sFKCLc=3D?= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230040)(1800799024)(36860700013)(30052699003)(376014)(82310400026)(7053199007)(13003099007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Dec 2025 09:24:30.8739 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1a900a23-d08f-4580-16a2-08de439768c7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00003AEA.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9134 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.6 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 12/30/25. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/shanipr/dpdk-stable This queued commit can be viewed at: https://github.com/shanipr/dpdk-stable/commit/60f8d1394655899888f2a17c934ab04d9c62e934 Thanks. Shani --- >From 60f8d1394655899888f2a17c934ab04d9c62e934 Mon Sep 17 00:00:00 2001 From: Wathsala Vithanage Date: Tue, 2 Dec 2025 20:39:27 +0000 Subject: [PATCH] ring: establish a safe partial order in hts-ring [ upstream commit 66d5f962780694f6aebf000907fc3ce7a72584f9 ] Enforce a safe partial order by making the CAS and the preceding head load use release and acquire semantics. This creates a pairwise happens-before relationship between threads of the same role. Combine the two load-acquire operations of ht.raw, which were previously split across the two paths of a conditional branch, into __rte_ring_hts_head_wait. This simplifies the branching logic and makes the synchronization behavior easier to understand. Add comments to explain synchronizes with edges in detail. Signed-off-by: Wathsala Vithanage Signed-off-by: Ola Liljedahl --- lib/ring/rte_ring_hts_elem_pvt.h | 96 +++++++++++++++++++++++--------- 1 file changed, 71 insertions(+), 25 deletions(-) diff --git a/lib/ring/rte_ring_hts_elem_pvt.h b/lib/ring/rte_ring_hts_elem_pvt.h index 91f5eeccb9..3c27197b6b 100644 --- a/lib/ring/rte_ring_hts_elem_pvt.h +++ b/lib/ring/rte_ring_hts_elem_pvt.h @@ -32,22 +32,40 @@ __rte_ring_hts_update_tail(struct rte_ring_hts_headtail *ht, uint32_t old_tail, RTE_SET_USED(enqueue); tail = old_tail + num; + + /* + * R0: Release the tail update. Establishes a synchronization edge with + * the load-acquire at A1/A3. This release ensures that all updates to + * *ht and the ring array made by this thread become visible to the + * opposing thread once the tail value written here is observed. + */ rte_atomic_store_explicit(&ht->ht.pos.tail, tail, rte_memory_order_release); } /** - * @internal waits till tail will become equal to head. - * Means no writer/reader is active for that ring. - * Suppose to work as serialization point. + * @internal + * Waits until the tail becomes equal to the head. + * This indicates that another thread has finished its transaction, and there + * is a chance that we could be the next writer or reader in line. + * + * Returns ht.raw at this point. The value may be imprecise, since another + * thread might change the state before we observe ht.raw, but that does not + * matter. The function __rte_ring_hts_move_head() can detect and recall this + * function when it reaches the linearization point (CAS). */ -static __rte_always_inline void +static __rte_always_inline union __rte_ring_hts_pos __rte_ring_hts_head_wait(const struct rte_ring_hts_headtail *ht, - union __rte_ring_hts_pos *p) + int memorder) { - while (p->pos.head != p->pos.tail) { + union __rte_ring_hts_pos p; + p.raw = rte_atomic_load_explicit(&ht->ht.raw, memorder); + + while (p.pos.head != p.pos.tail) { rte_pause(); - p->raw = rte_atomic_load_explicit(&ht->ht.raw, rte_memory_order_acquire); + p.raw = rte_atomic_load_explicit(&ht->ht.raw, memorder); } + + return p; } /** @@ -58,13 +76,11 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, enum rte_ring_queue_behavior behavior, uint32_t *old_head, uint32_t *free_entries) { - uint32_t n; + uint32_t n, cons_tail; union __rte_ring_hts_pos np, op; const uint32_t capacity = r->capacity; - op.raw = rte_atomic_load_explicit(&r->hts_prod.ht.raw, rte_memory_order_acquire); - do { /* Reset n to the initial burst count */ n = num; @@ -74,7 +90,20 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, * make sure that we read prod head/tail *before* * reading cons tail. */ - __rte_ring_hts_head_wait(&r->hts_prod, &op); + /* + * A0: Synchronizes with the CAS at R1. + * Establishes a happens-before relationship with a thread of the same + * type that released the ht.raw, ensuring this thread observes all of + * its memory effects needed to maintain a safe partial order. + */ + op = __rte_ring_hts_head_wait(&r->hts_prod, rte_memory_order_acquire); + + /* + * A1: Establish a synchronizes-with edge using a store-release at R0. + * This ensures that all memory effects from the preceding opposing + * thread are observed. + */ + cons_tail = rte_atomic_load_explicit(&r->cons.tail, rte_memory_order_acquire); /* * The subtraction is done between two unsigned 32bits value @@ -82,7 +111,7 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, * *old_head > cons_tail). So 'free_entries' is always between 0 * and capacity (which is < size). */ - *free_entries = capacity + r->cons.tail - op.pos.head; + *free_entries = capacity + cons_tail - op.pos.head; /* check that we have enough room in ring */ if (unlikely(n > *free_entries)) @@ -96,13 +125,16 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, np.pos.head = op.pos.head + n; /* - * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: - * - OOO reads of cons tail value - * - OOO copy of elems from the ring + * R1: Establishes a synchronizes-with edge with the load-acquire + * of ht.raw at A0. This makes sure that the store-release to the + * tail by this thread, if it was of the opposite type, becomes + * visible to another thread of the current type. That thread will + * then observe the updates in the same order, keeping a safe + * partial order. */ } while (rte_atomic_compare_exchange_strong_explicit(&r->hts_prod.ht.raw, (uint64_t *)(uintptr_t)&op.raw, np.raw, - rte_memory_order_acquire, rte_memory_order_acquire) == 0); + rte_memory_order_release, rte_memory_order_relaxed) == 0); *old_head = op.pos.head; return n; @@ -116,11 +148,9 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, enum rte_ring_queue_behavior behavior, uint32_t *old_head, uint32_t *entries) { - uint32_t n; + uint32_t n, prod_tail; union __rte_ring_hts_pos np, op; - op.raw = rte_atomic_load_explicit(&r->hts_cons.ht.raw, rte_memory_order_acquire); - /* move cons.head atomically */ do { /* Restore n as it may change every loop */ @@ -131,14 +161,27 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, * make sure that we read cons head/tail *before* * reading prod tail. */ - __rte_ring_hts_head_wait(&r->hts_cons, &op); + /* + * A2: Synchronizes with the CAS at R2. + * Establishes a happens-before relationship with a thread of the same + * type that released the ht.raw, ensuring this thread observes all of + * its memory effects needed to maintain a safe partial order. + */ + op = __rte_ring_hts_head_wait(&r->hts_cons, rte_memory_order_acquire); + + /* + * A3: Establish a synchronizes-with edge using a store-release at R0. + * This ensures that all memory effects from the preceding opposing + * thread are observed. + */ + prod_tail = rte_atomic_load_explicit(&r->prod.tail, rte_memory_order_acquire); /* The subtraction is done between two unsigned 32bits value * (the result is always modulo 32 bits even if we have * cons_head > prod_tail). So 'entries' is always between 0 * and size(ring)-1. */ - *entries = r->prod.tail - op.pos.head; + *entries = prod_tail - op.pos.head; /* Set the actual entries for dequeue */ if (n > *entries) @@ -151,13 +194,16 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, np.pos.head = op.pos.head + n; /* - * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: - * - OOO reads of prod tail value - * - OOO copy of elems from the ring + * R2: Establishes a synchronizes-with edge with the load-acquire + * of ht.raw at A2. This makes sure that the store-release to the + * tail by this thread, if it was of the opposite type, becomes + * visible to another thread of the current type. That thread will + * then observe the updates in the same order, keeping a safe + * partial order. */ } while (rte_atomic_compare_exchange_strong_explicit(&r->hts_cons.ht.raw, (uint64_t *)(uintptr_t)&op.raw, np.raw, - rte_memory_order_acquire, rte_memory_order_acquire) == 0); + rte_memory_order_release, rte_memory_order_relaxed) == 0); *old_head = op.pos.head; return n; -- 2.43.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2025-12-25 11:16:39.845569579 +0200 +++ 0070-ring-establish-a-safe-partial-order-in-hts-ring.patch 2025-12-25 11:16:36.073824000 +0200 @@ -1 +1 @@ -From 66d5f962780694f6aebf000907fc3ce7a72584f9 Mon Sep 17 00:00:00 2001 +From 60f8d1394655899888f2a17c934ab04d9c62e934 Mon Sep 17 00:00:00 2001 @@ -3,2 +3,4 @@ -Date: Tue, 11 Nov 2025 18:37:18 +0000 -Subject: [PATCH] ring: establish safe partial order in HTS mode +Date: Tue, 2 Dec 2025 20:39:27 +0000 +Subject: [PATCH] ring: establish a safe partial order in hts-ring + +[ upstream commit 66d5f962780694f6aebf000907fc3ce7a72584f9 ] @@ -17,3 +18,0 @@ -Fixes: 1cc363b8ce06e ("ring: introduce HTS ring mode") -Cc: stable@dpdk.org - @@ -23,2 +22,2 @@ - lib/ring/rte_ring_hts_elem_pvt.h | 66 ++++++++++++++++++++++++-------- - 1 file changed, 49 insertions(+), 17 deletions(-) + lib/ring/rte_ring_hts_elem_pvt.h | 96 +++++++++++++++++++++++--------- + 1 file changed, 71 insertions(+), 25 deletions(-) @@ -27 +26 @@ -index e2b82dd1e6..a01089d15d 100644 +index 91f5eeccb9..3c27197b6b 100644 @@ -37,3 +36,3 @@ -+ * the load-acquire at A1. This release ensures that all updates to *ht -+ * and the ring array made by this thread become visible to the opposing -+ * thread once the tail value written here is observed. ++ * the load-acquire at A1/A3. This release ensures that all updates to ++ * *ht and the ring array made by this thread become visible to the ++ * opposing thread once the tail value written here is observed. @@ -62 +61 @@ -+ rte_memory_order memorder) ++ int memorder) @@ -78 +77 @@ -@@ -80,11 +98,9 @@ __rte_ring_hts_move_head(struct rte_ring_hts_headtail *d, +@@ -58,13 +76,11 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, @@ -80 +79 @@ - uint32_t *entries) + uint32_t *free_entries) @@ -83 +82 @@ -+ uint32_t n, stail; ++ uint32_t n, cons_tail; @@ -86 +85,3 @@ -- op.raw = rte_atomic_load_explicit(&d->ht.raw, rte_memory_order_acquire); + const uint32_t capacity = r->capacity; + +- op.raw = rte_atomic_load_explicit(&r->hts_prod.ht.raw, rte_memory_order_acquire); @@ -91 +92 @@ -@@ -94,7 +110,20 @@ __rte_ring_hts_move_head(struct rte_ring_hts_headtail *d, +@@ -74,7 +90,20 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, @@ -95 +96 @@ -- __rte_ring_hts_head_wait(d, &op); +- __rte_ring_hts_head_wait(&r->hts_prod, &op); @@ -102 +103 @@ -+ op = __rte_ring_hts_head_wait(d, rte_memory_order_acquire); ++ op = __rte_ring_hts_head_wait(&r->hts_prod, rte_memory_order_acquire); @@ -109 +110 @@ -+ stail = rte_atomic_load_explicit(&s->tail, rte_memory_order_acquire); ++ cons_tail = rte_atomic_load_explicit(&r->cons.tail, rte_memory_order_acquire); @@ -113,2 +114,2 @@ -@@ -102,7 +131,7 @@ __rte_ring_hts_move_head(struct rte_ring_hts_headtail *d, - * *old_head > cons_tail). So 'entries' is always between 0 +@@ -82,7 +111,7 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, + * *old_head > cons_tail). So 'free_entries' is always between 0 @@ -117,2 +118,2 @@ -- *entries = capacity + s->tail - op.pos.head; -+ *entries = capacity + stail - op.pos.head; +- *free_entries = capacity + r->cons.tail - op.pos.head; ++ *free_entries = capacity + cons_tail - op.pos.head; @@ -121,2 +122,2 @@ - if (unlikely(n > *entries)) -@@ -116,14 +145,17 @@ __rte_ring_hts_move_head(struct rte_ring_hts_headtail *d, + if (unlikely(n > *free_entries)) +@@ -96,13 +125,16 @@ __rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num, @@ -136 +137,65 @@ - } while (rte_atomic_compare_exchange_strong_explicit(&d->ht.raw, + } while (rte_atomic_compare_exchange_strong_explicit(&r->hts_prod.ht.raw, + (uint64_t *)(uintptr_t)&op.raw, np.raw, +- rte_memory_order_acquire, rte_memory_order_acquire) == 0); ++ rte_memory_order_release, rte_memory_order_relaxed) == 0); + + *old_head = op.pos.head; + return n; +@@ -116,11 +148,9 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, + enum rte_ring_queue_behavior behavior, uint32_t *old_head, + uint32_t *entries) + { +- uint32_t n; ++ uint32_t n, prod_tail; + union __rte_ring_hts_pos np, op; + +- op.raw = rte_atomic_load_explicit(&r->hts_cons.ht.raw, rte_memory_order_acquire); +- + /* move cons.head atomically */ + do { + /* Restore n as it may change every loop */ +@@ -131,14 +161,27 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, + * make sure that we read cons head/tail *before* + * reading prod tail. + */ +- __rte_ring_hts_head_wait(&r->hts_cons, &op); ++ /* ++ * A2: Synchronizes with the CAS at R2. ++ * Establishes a happens-before relationship with a thread of the same ++ * type that released the ht.raw, ensuring this thread observes all of ++ * its memory effects needed to maintain a safe partial order. ++ */ ++ op = __rte_ring_hts_head_wait(&r->hts_cons, rte_memory_order_acquire); ++ ++ /* ++ * A3: Establish a synchronizes-with edge using a store-release at R0. ++ * This ensures that all memory effects from the preceding opposing ++ * thread are observed. ++ */ ++ prod_tail = rte_atomic_load_explicit(&r->prod.tail, rte_memory_order_acquire); + + /* The subtraction is done between two unsigned 32bits value + * (the result is always modulo 32 bits even if we have + * cons_head > prod_tail). So 'entries' is always between 0 + * and size(ring)-1. + */ +- *entries = r->prod.tail - op.pos.head; ++ *entries = prod_tail - op.pos.head; + + /* Set the actual entries for dequeue */ + if (n > *entries) +@@ -151,13 +194,16 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num, + np.pos.head = op.pos.head + n; + + /* +- * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: +- * - OOO reads of prod tail value +- * - OOO copy of elems from the ring ++ * R2: Establishes a synchronizes-with edge with the load-acquire ++ * of ht.raw at A2. This makes sure that the store-release to the ++ * tail by this thread, if it was of the opposite type, becomes ++ * visible to another thread of the current type. That thread will ++ * then observe the updates in the same order, keeping a safe ++ * partial order. + */ + } while (rte_atomic_compare_exchange_strong_explicit(&r->hts_cons.ht.raw, @@ -138,4 +203,2 @@ -- rte_memory_order_acquire, -- rte_memory_order_acquire) == 0); -+ rte_memory_order_release, -+ rte_memory_order_relaxed) == 0); +- rte_memory_order_acquire, rte_memory_order_acquire) == 0); ++ rte_memory_order_release, rte_memory_order_relaxed) == 0);