From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EE9F0470E5 for ; Thu, 25 Dec 2025 10:24:52 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C62B740649; Thu, 25 Dec 2025 10:24:52 +0100 (CET) Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012040.outbound.protection.outlook.com [40.93.195.40]) by mails.dpdk.org (Postfix) with ESMTP id 95DB840649 for ; Thu, 25 Dec 2025 10:24:51 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tnSXerQbcpgLglswoGaM0NoX4MZHv3eO/tgke9Dyo+Z7r1bzrtqgYligIMCMc+TmmxbzjewIn+K2IPSTsjOty/kop1QhEJDWy/yEDdnyaZl7q/D6uPffnASDkoPTFORG9znZXG9EHu3HaQhRceQn959he8+wuWg98jtgZPuvTs13gFG7Po9IHRVgv2jtQ6TTYikR/GmXknlVQjpBd0L5hewwh48Q1nqQrGhTt+bIbV7v8Ko3Ix0Rg5iY4wEyCBLcTrYBhNV5+alg0Hij1RG0WxLVXGt1B4/W/bGi0sn5na9kApqvhROtaF89u4QGCqRrrgBgSzzUcXVV3wQPO9WqFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=krhN4cnT0ByzlIVUB8oqodBvCk5H/WOiJpeihHnLjhs=; b=ZgMG1GWmYoXlFlwR11JFiVbnprma6Vu2OHk9njpXR4ECBxnsR3U1IUk+1X9t8K/EXS7WwSwAI2JRE9SPc129Nh950Purj4zdmRojm6T2DLgzcynQfeswoW1aD1KV96A0MMzym+2EKFD9L1KxbKgORLApqhTMSknVQIcKLqMsjJjIQ8NvFydLOIgfnFLPQmHa98aa9PfLQVVQ0P3ecCCba84p+vWaWiFamn41h1Uy2yrgee+8lqZJizl+hNVDxrbkkUHpa0hrft3Ar6NPe/ftdYSRcVsl45CM/E8toidKdXwBn4h2OA/KXlTsD2jhQfwEa0lV/UcdXKH/2dlQIqJDKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=arm.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=krhN4cnT0ByzlIVUB8oqodBvCk5H/WOiJpeihHnLjhs=; b=Q5xfV7dT5PKwJ/+uG6M/rObSLD9fe5urGN2jmw3Ev552GqVEgxZtecZiFwQI0B8vJDdcnLTCW0aRBxGp3qvwhukTd8M1QCWxvwvP/7+jtI1tl+g2Tk1pUK/9poi/x6m4gTBp4nSncTi+Wyt2kf0vSE6xZuxvWq/2Uu90OpP9woRY6VlTH41zGearIqk48zhvfVcOi1xFcgYpnjstsUBethh5xCKSFkk6rZqSzgrgNfZvAciXrTj2qhDv0IcHewZAPrDHWPk6fQsQ4Piowt4JCGoYSqndXBAAsd82AOTE5qzpKSHOHPBfTu6mNWlqH0g9CGFSXvXBNKCdHAwZpQrLXw== Received: from CH0P221CA0005.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:11c::13) by SJ2PR12MB8135.namprd12.prod.outlook.com (2603:10b6:a03:4f3::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9456.11; Thu, 25 Dec 2025 09:24:45 +0000 Received: from CH2PEPF00000141.namprd02.prod.outlook.com (2603:10b6:610:11c:cafe::c0) by CH0P221CA0005.outlook.office365.com (2603:10b6:610:11c::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9456.11 via Frontend Transport; Thu, 25 Dec 2025 09:24:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF00000141.mail.protection.outlook.com (10.167.244.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9456.9 via Frontend Transport; Thu, 25 Dec 2025 09:24:44 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 25 Dec 2025 01:24:36 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 25 Dec 2025 01:24:35 -0800 Received: from nvidia.com (10.127.8.12) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20 via Frontend Transport; Thu, 25 Dec 2025 01:24:34 -0800 From: Shani Peretz To: Wathsala Vithanage CC: Ola Liljedahl , dpdk stable Subject: patch 'ring: establish safe partial order in RTS mode' has been queued to stable release 23.11.6 Date: Thu, 25 Dec 2025 11:18:32 +0200 Message-ID: <20251225091938.345892-71-shperetz@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251225091938.345892-1-shperetz@nvidia.com> References: <20251221145746.763179-93-shperetz@nvidia.com> <20251225091938.345892-1-shperetz@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000141:EE_|SJ2PR12MB8135:EE_ X-MS-Office365-Filtering-Correlation-Id: 1dc38933-7aae-4639-f2f3-08de439770f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|82310400026|36860700013|7053199007|13003099007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?d0aXnsDwOpUylj/2YKxiypUiGK+5XvFkTZ6WCS7EOVQocVQ3Yhcg6XLtsDD/?= =?us-ascii?Q?zrXzh8elMkgeeUg05D4xIQFtKSWCUFIx77rLHwH32lI/WKhJrkYh2HY76174?= =?us-ascii?Q?hrTagjbw+oyQyFD6B6E6VEzRd8naIfG2/UU2VazVag/BxcsqWM1Ga3+WIwaG?= =?us-ascii?Q?SHZLYGCy3fr+QR1QUix1VCbyaIT/HxPN9NPscfgKdwSQFxRozHg3gxvRIcF5?= =?us-ascii?Q?O8x54tLdgNEdV76zm+6I2/STHKzwxvAOHDVO8fM9Emxb2b67LtCyhtMbsXg7?= =?us-ascii?Q?iEgouNDcawUVVdHibSr4vHmPJrmo797CBhMLoyL/vqIGc7QyKSUA5szNRu14?= =?us-ascii?Q?H28Jfstolyv6eaB8aISQuHnbJMWKU+a54NKCitvWP5Nfr80FIMcO36Ts3+Ju?= =?us-ascii?Q?E2DG6hfnJeeVwxdqUee48sHCX0fZSO5StxDjNUBgB4EXKC1oRx545vDmGW+f?= =?us-ascii?Q?AFppNhX24oSttpIzpn4ClEEx+b8N9IxtzesL0cqoIq1gP8porbxMMfWigNNU?= =?us-ascii?Q?QBbboptQf1VFgQuq9sydsX7c6Ppt7qK74e0aVHrA91JYbxUt6ECJtMDCim/z?= =?us-ascii?Q?rHhPX5ydwmC6jGrTyOU2VLxckLHRDZG4IfIhU9GJh5f0BQ9lc7QLK03zgVg+?= =?us-ascii?Q?2P+c2mZfFkY1J/2PrBBYjpTTkT5QFselGgJRQsW3YH33bbF+5wUiw149vqfy?= =?us-ascii?Q?wXcKQMIzc9oBZDXknyNbFtstlDGAnSu6iiWsV0JvzHnQdLm2TCv80moFXyIl?= =?us-ascii?Q?7WfPc3WqUMoJIzDSnraUxVDfjahHza1zNKDMv7ONTPQlCaalm33QbOGou5qi?= =?us-ascii?Q?lyYCA1/HnzDuBJ0tbiwTDgBzMIWokJJc7BfQ90mV4/FVueFhmqv4CHhmAtoP?= =?us-ascii?Q?0fVu1GURT2/DWxgwBNb4GtImpz7KmvNk6uhFaOdpN6EfQIRqkXBkynkgkyCV?= =?us-ascii?Q?WgQKfljfQNDyWGU+yFcKlNrJnbpKNaNVBCxyE0JziBzYpfjGDJ7Xob0a2qKP?= =?us-ascii?Q?E7bpqD6R4GUPn1iix3nqkG/6Zej/nhF0iUa0eqmA/+uQmuqyVjABCiZ5JhDZ?= =?us-ascii?Q?+vSUoQOrO8mXCgboxHPEyWfeq91VGEl/0NFrXd+sTm/Qy6alXsfsfC1L7Qk3?= =?us-ascii?Q?dKgIMFhDIArlA04a9wzb+AX3NNYFzx0n8E0q02bFxq/yInV1LSkn6wzLYPkC?= =?us-ascii?Q?xKuF7d4MRHE0skZnQ/n+C8YOE7zQQvTchdrNd0I/XskmKM+ia2l34WLamRP3?= =?us-ascii?Q?Xm/+1CJpSI+dkR4f/H+t0lkqUix1gGQWg2bJhwA0XgKopYeWTaP/QcFScreW?= =?us-ascii?Q?8pR2qnLsqSE1ZuIcHyVJYj011SoytKyOGhtdKLSNS4HRLpNDKEkpSprMUY+d?= =?us-ascii?Q?oub4c+hcSd4mrElXRWKdoFB2eCCxiTvq+tjCe3V2NNOTi6HgEeZlLivTuDSE?= =?us-ascii?Q?6a+2FDy4BzCwhCdZyEVnc71DnEDe96hTzf8sxgF9iNbyyDOOReJApRVybVlp?= =?us-ascii?Q?+2vCptOkp+2j2Ubs05O8rxB/Xd3ZDSBRlZqcSnOsH+w37VcZJa/kfzL552j3?= =?us-ascii?Q?gXMjyIv5FAVUlNBRaJ0=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(82310400026)(36860700013)(7053199007)(13003099007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Dec 2025 09:24:44.5688 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1dc38933-7aae-4639-f2f3-08de439770f3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000141.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8135 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.6 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 12/30/25. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/shanipr/dpdk-stable This queued commit can be viewed at: https://github.com/shanipr/dpdk-stable/commit/0916a3572d6e6f5ce60a16390889aabbc48003d0 Thanks. Shani --- >From 0916a3572d6e6f5ce60a16390889aabbc48003d0 Mon Sep 17 00:00:00 2001 From: Wathsala Vithanage Date: Tue, 2 Dec 2025 20:39:28 +0000 Subject: [PATCH] ring: establish safe partial order in RTS mode [ upstream commit 36b69b5f958e10eb5beb4292ade57199a722a045 ] Enforce a safe partial order by making the CAS and the preceding head load use release and acquire semantics. This creates a pairwise happens-before relationship between threads of the same kind (i.e. between consumers or between producers). Combine the two load-acquire operations of ht->head.raw, which were previously split across the two paths of a conditional branch, into __rte_ring_rts_head_wait. This simplifies the branching logic and makes the synchronization behavior easier to understand. Add comments to explain synchronizes with edges in detail. Signed-off-by: Wathsala Vithanage Signed-off-by: Ola Liljedahl --- lib/ring/rte_ring_rts_elem_pvt.h | 96 ++++++++++++++++++++++++-------- 1 file changed, 72 insertions(+), 24 deletions(-) diff --git a/lib/ring/rte_ring_rts_elem_pvt.h b/lib/ring/rte_ring_rts_elem_pvt.h index 122650346b..867175e15f 100644 --- a/lib/ring/rte_ring_rts_elem_pvt.h +++ b/lib/ring/rte_ring_rts_elem_pvt.h @@ -31,6 +31,17 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) * might preceded us, then don't update tail with new value. */ + /* + * A0 = {A0.a, A0.b}: Synchronizes with the CAS at R0. + * The CAS at R0 in same typed thread establishes a happens-before + * relationship with this load acquire. Ensures that this thread + * observes the same or later values for h.raw/h.val.cnt + * observed by the other thread when it updated ht->tail.raw. + * If not, ht->tail.raw may get updated out of sync (e.g. getting + * updated to the same value twice). A0.a makes sure this condition + * holds when CAS succeeds and A0.b when it fails. + */ + /* A0.a */ ot.raw = rte_atomic_load_explicit(&ht->tail.raw, rte_memory_order_acquire); do { @@ -41,6 +52,11 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) if (++nt.val.cnt == h.val.cnt) nt.val.pos = h.val.pos; + /* + * R0: Synchronizes with A2 of a different thread of the opposite type and A0.b + * of a different thread of the same type. + */ + /* A0.b */ } while (rte_atomic_compare_exchange_strong_explicit(&ht->tail.raw, (uint64_t *)(uintptr_t)&ot.raw, nt.raw, rte_memory_order_release, rte_memory_order_acquire) == 0); @@ -50,18 +66,22 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) * @internal This function waits till head/tail distance wouldn't * exceed pre-defined max value. */ -static __rte_always_inline void +static __rte_always_inline union __rte_ring_rts_poscnt __rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht, - union __rte_ring_rts_poscnt *h) + int memorder) { - uint32_t max; + union __rte_ring_rts_poscnt h; + uint32_t max = ht->htd_max; - max = ht->htd_max; - while (h->val.pos - ht->tail.val.pos > max) { + h.raw = rte_atomic_load_explicit(&ht->head.raw, memorder); + + while (h.val.pos - ht->tail.val.pos > max) { rte_pause(); - h->raw = rte_atomic_load_explicit(&ht->head.raw, rte_memory_order_acquire); + h.raw = rte_atomic_load_explicit(&ht->head.raw, memorder); } + + return h; } /** @@ -72,13 +92,11 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, enum rte_ring_queue_behavior behavior, uint32_t *old_head, uint32_t *free_entries) { - uint32_t n; + uint32_t n, cons_tail; union __rte_ring_rts_poscnt nh, oh; const uint32_t capacity = r->capacity; - oh.raw = rte_atomic_load_explicit(&r->rts_prod.head.raw, rte_memory_order_acquire); - do { /* Reset n to the initial burst count */ n = num; @@ -88,7 +106,20 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, * make sure that we read prod head *before* * reading cons tail. */ - __rte_ring_rts_head_wait(&r->rts_prod, &oh); + /* + * A1 Synchronizes with the CAS at R1. + * Establishes a happens-before relationship with a thread of the same + * type that released the ht.raw, ensuring this thread observes all of + * its memory effects needed to maintain a safe partial order. + */ + oh = __rte_ring_rts_head_wait(&r->rts_prod, rte_memory_order_acquire); + + /* + * A2: Establish a synchronizes-with edge using a store-release at R0. + * This ensures that all memory effects from the preceding opposing + * thread are observed. + */ + cons_tail = rte_atomic_load_explicit(&r->cons.tail, rte_memory_order_acquire); /* * The subtraction is done between two unsigned 32bits value @@ -96,7 +127,7 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, * *old_head > cons_tail). So 'free_entries' is always between 0 * and capacity (which is < size). */ - *free_entries = capacity + r->cons.tail - oh.val.pos; + *free_entries = capacity + cons_tail - oh.val.pos; /* check that we have enough room in ring */ if (unlikely(n > *free_entries)) @@ -110,13 +141,16 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, nh.val.cnt = oh.val.cnt + 1; /* - * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: - * - OOO reads of cons tail value - * - OOO copy of elems to the ring + * R1: Establishes a synchronizes-with edge with the load-acquire + * of ht.raw at A1. Ensures that the store-release to the tail by + * this thread, if it was of the opposite type, becomes + * visible to another thread of the current type. That thread will + * then observe the updates in the same order, keeping a safe + * partial order. */ } while (rte_atomic_compare_exchange_strong_explicit(&r->rts_prod.head.raw, (uint64_t *)(uintptr_t)&oh.raw, nh.raw, - rte_memory_order_acquire, rte_memory_order_acquire) == 0); + rte_memory_order_release, rte_memory_order_relaxed) == 0); *old_head = oh.val.pos; return n; @@ -130,11 +164,9 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, enum rte_ring_queue_behavior behavior, uint32_t *old_head, uint32_t *entries) { - uint32_t n; + uint32_t n, prod_tail; union __rte_ring_rts_poscnt nh, oh; - oh.raw = rte_atomic_load_explicit(&r->rts_cons.head.raw, rte_memory_order_acquire); - /* move cons.head atomically */ do { /* Restore n as it may change every loop */ @@ -145,14 +177,27 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, * make sure that we read cons head *before* * reading prod tail. */ - __rte_ring_rts_head_wait(&r->rts_cons, &oh); + /* + * A3: Synchronizes with the CAS at R2. + * Establishes a happens-before relationship with a thread of the same + * type that released the ht.raw, ensuring this thread observes all of + * its memory effects needed to maintain a safe partial order. + */ + oh = __rte_ring_rts_head_wait(&r->rts_cons, rte_memory_order_acquire); + + /* + * A4: Establish a synchronizes-with edge using a store-release at R0. + * This ensures that all memory effects from the preceding opposing + * thread are observed. + */ + prod_tail = rte_atomic_load_explicit(&r->prod.tail, rte_memory_order_acquire); /* The subtraction is done between two unsigned 32bits value * (the result is always modulo 32 bits even if we have * cons_head > prod_tail). So 'entries' is always between 0 * and size(ring)-1. */ - *entries = r->prod.tail - oh.val.pos; + *entries = prod_tail - oh.val.pos; /* Set the actual entries for dequeue */ if (n > *entries) @@ -165,13 +210,16 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, nh.val.cnt = oh.val.cnt + 1; /* - * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: - * - OOO reads of prod tail value - * - OOO copy of elems from the ring + * R2: Establishes a synchronizes-with edge with the load-acquire + * of ht.raw at A3. Ensures that the store-release to the tail by + * this thread, if it was of the opposite type, becomes + * visible to another thread of the current type. That thread will + * then observe the updates in the same order, keeping a safe + * partial order. */ } while (rte_atomic_compare_exchange_strong_explicit(&r->rts_cons.head.raw, (uint64_t *)(uintptr_t)&oh.raw, nh.raw, - rte_memory_order_acquire, rte_memory_order_acquire) == 0); + rte_memory_order_release, rte_memory_order_relaxed) == 0); *old_head = oh.val.pos; return n; -- 2.43.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2025-12-25 11:16:39.892313987 +0200 +++ 0071-ring-establish-safe-partial-order-in-RTS-mode.patch 2025-12-25 11:16:36.077854000 +0200 @@ -1 +1 @@ -From 36b69b5f958e10eb5beb4292ade57199a722a045 Mon Sep 17 00:00:00 2001 +From 0916a3572d6e6f5ce60a16390889aabbc48003d0 Mon Sep 17 00:00:00 2001 @@ -3 +3 @@ -Date: Tue, 11 Nov 2025 18:37:19 +0000 +Date: Tue, 2 Dec 2025 20:39:28 +0000 @@ -5,0 +6,2 @@ +[ upstream commit 36b69b5f958e10eb5beb4292ade57199a722a045 ] + @@ -18,3 +19,0 @@ -Fixes: e6ba4731c0f3a ("ring: introduce RTS ring mode") -Cc: stable@dpdk.org - @@ -24,2 +23,2 @@ - lib/ring/rte_ring_rts_elem_pvt.h | 67 +++++++++++++++++++++++--------- - 1 file changed, 49 insertions(+), 18 deletions(-) + lib/ring/rte_ring_rts_elem_pvt.h | 96 ++++++++++++++++++++++++-------- + 1 file changed, 72 insertions(+), 24 deletions(-) @@ -28 +27 @@ -index 96825931f8..0280369b21 100644 +index 122650346b..867175e15f 100644 @@ -49,2 +48 @@ -@@ -40,7 +51,11 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) - nt.raw = ot.raw; +@@ -41,6 +52,11 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) @@ -53 +51 @@ -- + @@ -62 +60 @@ -@@ -50,18 +65,21 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) +@@ -50,18 +66,22 @@ __rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht) @@ -70 +68 @@ -+ rte_memory_order memorder) ++ int memorder) @@ -77 +74,0 @@ -+ h.raw = rte_atomic_load_explicit(&ht->head.raw, memorder); @@ -79,0 +77,2 @@ ++ h.raw = rte_atomic_load_explicit(&ht->head.raw, memorder); ++ @@ -90 +89 @@ -@@ -94,12 +112,9 @@ __rte_ring_rts_move_head(struct rte_ring_rts_headtail *d, +@@ -72,13 +92,11 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, @@ -92 +91 @@ - uint32_t *entries) + uint32_t *free_entries) @@ -95 +94 @@ -+ uint32_t n, stail; ++ uint32_t n, cons_tail; @@ -98,2 +97,3 @@ -- oh.raw = rte_atomic_load_explicit(&d->head.raw, -- rte_memory_order_acquire); + const uint32_t capacity = r->capacity; + +- oh.raw = rte_atomic_load_explicit(&r->rts_prod.head.raw, rte_memory_order_acquire); @@ -104 +104 @@ -@@ -109,7 +124,20 @@ __rte_ring_rts_move_head(struct rte_ring_rts_headtail *d, +@@ -88,7 +106,20 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, @@ -108 +108 @@ -- __rte_ring_rts_head_wait(d, &oh); +- __rte_ring_rts_head_wait(&r->rts_prod, &oh); @@ -115 +115 @@ -+ oh = __rte_ring_rts_head_wait(d, rte_memory_order_acquire); ++ oh = __rte_ring_rts_head_wait(&r->rts_prod, rte_memory_order_acquire); @@ -122 +122 @@ -+ stail = rte_atomic_load_explicit(&s->tail, rte_memory_order_acquire); ++ cons_tail = rte_atomic_load_explicit(&r->cons.tail, rte_memory_order_acquire); @@ -126,2 +126,2 @@ -@@ -117,7 +145,7 @@ __rte_ring_rts_move_head(struct rte_ring_rts_headtail *d, - * *old_head > cons_tail). So 'entries' is always between 0 +@@ -96,7 +127,7 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, + * *old_head > cons_tail). So 'free_entries' is always between 0 @@ -130,2 +130,2 @@ -- *entries = capacity + s->tail - oh.val.pos; -+ *entries = capacity + stail - oh.val.pos; +- *free_entries = capacity + r->cons.tail - oh.val.pos; ++ *free_entries = capacity + cons_tail - oh.val.pos; @@ -134,2 +134,2 @@ - if (unlikely(n > *entries)) -@@ -131,14 +159,17 @@ __rte_ring_rts_move_head(struct rte_ring_rts_headtail *d, + if (unlikely(n > *free_entries)) +@@ -110,13 +141,16 @@ __rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num, @@ -149 +149,65 @@ - } while (rte_atomic_compare_exchange_strong_explicit(&d->head.raw, + } while (rte_atomic_compare_exchange_strong_explicit(&r->rts_prod.head.raw, + (uint64_t *)(uintptr_t)&oh.raw, nh.raw, +- rte_memory_order_acquire, rte_memory_order_acquire) == 0); ++ rte_memory_order_release, rte_memory_order_relaxed) == 0); + + *old_head = oh.val.pos; + return n; +@@ -130,11 +164,9 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, + enum rte_ring_queue_behavior behavior, uint32_t *old_head, + uint32_t *entries) + { +- uint32_t n; ++ uint32_t n, prod_tail; + union __rte_ring_rts_poscnt nh, oh; + +- oh.raw = rte_atomic_load_explicit(&r->rts_cons.head.raw, rte_memory_order_acquire); +- + /* move cons.head atomically */ + do { + /* Restore n as it may change every loop */ +@@ -145,14 +177,27 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, + * make sure that we read cons head *before* + * reading prod tail. + */ +- __rte_ring_rts_head_wait(&r->rts_cons, &oh); ++ /* ++ * A3: Synchronizes with the CAS at R2. ++ * Establishes a happens-before relationship with a thread of the same ++ * type that released the ht.raw, ensuring this thread observes all of ++ * its memory effects needed to maintain a safe partial order. ++ */ ++ oh = __rte_ring_rts_head_wait(&r->rts_cons, rte_memory_order_acquire); ++ ++ /* ++ * A4: Establish a synchronizes-with edge using a store-release at R0. ++ * This ensures that all memory effects from the preceding opposing ++ * thread are observed. ++ */ ++ prod_tail = rte_atomic_load_explicit(&r->prod.tail, rte_memory_order_acquire); + + /* The subtraction is done between two unsigned 32bits value + * (the result is always modulo 32 bits even if we have + * cons_head > prod_tail). So 'entries' is always between 0 + * and size(ring)-1. + */ +- *entries = r->prod.tail - oh.val.pos; ++ *entries = prod_tail - oh.val.pos; + + /* Set the actual entries for dequeue */ + if (n > *entries) +@@ -165,13 +210,16 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num, + nh.val.cnt = oh.val.cnt + 1; + + /* +- * this CAS(ACQUIRE, ACQUIRE) serves as a hoist barrier to prevent: +- * - OOO reads of prod tail value +- * - OOO copy of elems from the ring ++ * R2: Establishes a synchronizes-with edge with the load-acquire ++ * of ht.raw at A3. Ensures that the store-release to the tail by ++ * this thread, if it was of the opposite type, becomes ++ * visible to another thread of the current type. That thread will ++ * then observe the updates in the same order, keeping a safe ++ * partial order. + */ + } while (rte_atomic_compare_exchange_strong_explicit(&r->rts_cons.head.raw, @@ -151,4 +215,2 @@ -- rte_memory_order_acquire, -- rte_memory_order_acquire) == 0); -+ rte_memory_order_release, -+ rte_memory_order_relaxed) == 0); +- rte_memory_order_acquire, rte_memory_order_acquire) == 0); ++ rte_memory_order_release, rte_memory_order_relaxed) == 0);