From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27E3F427F0; Wed, 22 Mar 2023 17:40:41 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 064A340E09; Wed, 22 Mar 2023 17:40:41 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2072.outbound.protection.outlook.com [40.107.21.72]) by mails.dpdk.org (Postfix) with ESMTP id 9387040A84 for ; Wed, 22 Mar 2023 17:40:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHiad5AqdSrP1Z1M1VTQtfVV5IfzScx2XR47NR9kAoo=; b=aASBAb+XKL2S/cQljluRUU3Qyhtk/h+qyjb2S/pK0olw9ZNg6etLojR7bxULrkoScNmcSE6CKtzr2qjoSsouzWPRvl9OS8QBVCRjDjsqRBnljpCL1oYmrsMWIqthouUczMsI5J0dZOU47QYpUTVWORIvBxy9oETQjU2DAT0shM0= Received: from DB6PR0801CA0057.eurprd08.prod.outlook.com (2603:10a6:4:2b::25) by PAVPR08MB9332.eurprd08.prod.outlook.com (2603:10a6:102:302::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37; Wed, 22 Mar 2023 16:40:37 +0000 Received: from DBAEUR03FT006.eop-EUR03.prod.protection.outlook.com (2603:10a6:4:2b:cafe::77) by DB6PR0801CA0057.outlook.office365.com (2603:10a6:4:2b::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37 via Frontend Transport; Wed, 22 Mar 2023 16:40:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT006.mail.protection.outlook.com (100.127.142.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6222.17 via Frontend Transport; Wed, 22 Mar 2023 16:40:37 +0000 Received: ("Tessian outbound cfb430c87a1e:v135"); Wed, 22 Mar 2023 16:40:37 +0000 X-CR-MTA-TID: 64aa7808 Received: from a3ebda249ac5.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 356D7FCF-6068-47C5-A1B4-79CB69328624.1; Wed, 22 Mar 2023 16:40:26 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id a3ebda249ac5.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 22 Mar 2023 16:40:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cxkf1N6+mapAZ2NeTt8SymNvkWeTWR4mjmgRPMPKsHy046ZiYeWC4n8v5sdmOOw9/UdS7DDXDdvoJ1HH4aUxDGy12VqckF5cYqqUXUOrF6Xn39TBaxPl1RikhL9NbdoTCLXeexeJRMfrufX9Yslq8B2eARrzL9plDJbLyA0ehEVwRCefYM8xZojXS1RfggbzIXNxX5U2T3eyCXtrnEo2GDA4/kQxH4lM7NhN6NyixwcsrKvBuKhx2CpU7d1J4KuC5iLc7uT8bhYewMqXIn+Bwf9m4c+IHRmdRJ8I1HD0n633dGoNp76kcu4tO4uRpGSSg6l0jhS1cHvv/iGPPUzuBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pHiad5AqdSrP1Z1M1VTQtfVV5IfzScx2XR47NR9kAoo=; b=Q2IWcTYFKw2dwQ9GYcuOqkcGEFQoC5g7Xja7/Gxzvb3svfWqxOHXxv1+YRQFJomVwGYEfp8Cg9AaSkhhV9MS52JL1Iy13s/Jx4LPCFfadmsb/jXwGWcYQlXOMkWjAlmdILpf3runZ7xGIS6V0ug/uUVF7wRCmCd0thP8UZ+RoJ8nQc8FdC1Gz8v/lJeOuSFso0RNP94vYP8EQ9pG6auacRUeUjKGdyw7tGvEoSkm9x0IDVVVRcChe5yPgdztGItSzNrBx7ENnTn4kwfdgDwVsUO43A3wnUi8iw9H3YsQ0sxJu5T17AoPGp9FFsZUiOXlxgqIpVyy3Lmq8uXi6myn0w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHiad5AqdSrP1Z1M1VTQtfVV5IfzScx2XR47NR9kAoo=; b=aASBAb+XKL2S/cQljluRUU3Qyhtk/h+qyjb2S/pK0olw9ZNg6etLojR7bxULrkoScNmcSE6CKtzr2qjoSsouzWPRvl9OS8QBVCRjDjsqRBnljpCL1oYmrsMWIqthouUczMsI5J0dZOU47QYpUTVWORIvBxy9oETQjU2DAT0shM0= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB9PR08MB8436.eurprd08.prod.outlook.com (2603:10a6:10:3d4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37; Wed, 22 Mar 2023 16:40:15 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9%5]) with mapi id 15.20.6178.037; Wed, 22 Mar 2023 16:40:15 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , Tyler Retzlaff CC: Stephen Hemminger , "dev@dpdk.org" , Ruifeng Wang , "thomas@monjalon.net" , nd , nd Subject: RE: [PATCH 0/7] replace rte atomics with GCC builtin atomics Thread-Topic: [PATCH 0/7] replace rte atomics with GCC builtin atomics Thread-Index: Adlc0x2jp4hZ/JHqTt+4XE4Z0dKiCQAAqo6gAAGcVLA= Date: Wed, 22 Mar 2023 16:40:14 +0000 Message-ID: References: <1679084388-19267-1-git-send-email-roretzla@linux.microsoft.com> <20230317144226.2f26bad1@hermes.local> <20230317214910.GA31884@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> <98CBD80474FA8B44BF855DF32C47DC35D877E1@smartserver.smartshare.dk> <20230322142134.GA29057@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> <98CBD80474FA8B44BF855DF32C47DC35D877E6@smartserver.smartshare.dk> <20230322152932.GB29057@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> <98CBD80474FA8B44BF855DF32C47DC35D877E8@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D877E8@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 67839EB3BF68B342929AADF97A7EF398.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DBAPR08MB5814:EE_|DB9PR08MB8436:EE_|DBAEUR03FT006:EE_|PAVPR08MB9332:EE_ X-MS-Office365-Filtering-Correlation-Id: 8976f050-ae08-43e7-0416-08db2af42a7d x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: VltSn27lUnyodh9apcTuZyMVAVFUDosMebSAR+0fQwnQEnEu3aOAWboqTVRm4DuTn3tQPEbm/P5V3PN3csEkwXp5jp4j1b2DqLyGc4903MuK1Tzkpu/XAInAlzbgmCHp7Tc/qqDflYaVhiJmUZzi/EC4+efwiSPPQDo8+VSwZRty/grg1G/mGZzFQ1KCZ+cUSsUYS5TMlLPztNUJbObZPrJhrhzgTJKoHIi/Ts2LC8oQN5NIaC2Sd8JZXu9/nwCN9Kzj6tMyJZ7EDXq/bT4Z0JvnQ1Gf4TcGq1+pp6nfpm5Zhz8SFeXiTYRoC23RW4uV6aylZdkwADbsglQae9yZ2llVa0TVFvl5ycStQoXMLHAYa4Gf0UfPIF4NinuQ9j96dN/CsvMyJSsMat56W7T5oZT8z1Z73XaVKPJUlqlQq8Qg504DJ6mrybLcJ65f2cjM18kNkNGL2xK2cWBIuN4G+SqYJvfwBnDAuzWBbHFfldloe3K7VZ6oiohDx7FtKWrMSnX2+BncaBkTdacIxdEagU6bMshBCJqiyKK3EVtldJiJ4LOi0rJxiMLdC/K4VqNvDeZpcsxSSuMZjVoC9R2bi91eQKXJIxUCZayrNzmrcR/br8Bn2ZRyuEu+v2pO6XFNjApBxyX8DFkyb/0Zkd76l4OHLeVh/aZjh2cryOn5hxzOuikmL/G2meNnoXPz3yHdxbIiI/+OQDZGYIu/2duqLQ== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(396003)(346002)(376002)(136003)(39860400002)(366004)(451199018)(8676002)(52536014)(7696005)(66556008)(4326008)(66476007)(66446008)(2906002)(66946007)(5660300002)(8936002)(64756008)(76116006)(38070700005)(86362001)(38100700002)(33656002)(122000001)(6506007)(316002)(478600001)(71200400001)(110136005)(53546011)(54906003)(55016003)(9686003)(83380400001)(41300700001)(66574015)(186003); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB8436 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT006.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 63f6c2d4-0d25-4bc7-0729-08db2af41cd0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hNirfaknF7WxyLS1wD44pZ3EWVlI0dMkjlby8++lhlEHnWfCOz6RCmRfeDeIV54nfkmI+ASVfzk+WLYsMlbRcxUza0vYryA44aVJoUfaP4tkcpJ70ym+PeqUdzqcAO2O42EY4YopZ2SLznoYmdL8dy/W+wf9K+7j9FeHqqUq/0LLVm90AjN16n46yw3tPEw5apY40FMzxhOjy+idjWO5d0mDCioq0hMWVqM8pVziy+Qdez+SPN3gzTDwYxiY6QJbaUDisLKOCP5QVea+4grgRRuyMcbYKs5SEavv0urOoa8HedYV/lZACpvtf6NJK8SuknUuBFoAR9jyVHYbtRA/oJBgBPdZYDR6YdoVu4jagxydm1G42aIsVUzKaF4LbHrPNVugjLXcvrqsiB13PQO5xPpqGKoyhDI46TTyJ1yuQ4/IVh8oZQla4vv0lwFQiSTtYNHebg9hdal3aLOfIl4GoIQ86ZUVlPcS5cdaSoVCI8zjXMFRf4vgB78sfW8e37xLnSRg4W70VZN9CH4fcMIf6LXwWf4mZzvltOBcnL+WKDbR9mhYD/tZ3RaJTeuz9gsdu4yD8aNELnTGh91IOFUPszkZ4DSwqSChgOP8r6GD44FCk6IBjhCbUcsZyzDHVrT6yWs2bmvj1r9GF4GecQPq0Yg16Two/KfpAjIsJA2YImHuSUlSr0PI1uS/goKZfZ4mztXqdsd8euyyDwLpkM9OWA== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(136003)(346002)(39860400002)(396003)(451199018)(36840700001)(46966006)(40470700004)(6506007)(9686003)(186003)(53546011)(26005)(7696005)(47076005)(66574015)(54906003)(316002)(478600001)(83380400001)(110136005)(336012)(82740400003)(4326008)(8936002)(70206006)(41300700001)(52536014)(36860700001)(70586007)(8676002)(40460700003)(5660300002)(2906002)(81166007)(86362001)(356005)(82310400005)(55016003)(40480700001)(33656002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2023 16:40:37.6123 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8976f050-ae08-43e7-0416-08db2af42a7d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT006.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR08MB9332 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Morten Br=F8rup > Sent: Wednesday, March 22, 2023 11:14 AM > To: Tyler Retzlaff > Cc: Stephen Hemminger ; dev@dpdk.org; > Honnappa Nagarahalli ; Ruifeng Wang > ; thomas@monjalon.net > Subject: RE: [PATCH 0/7] replace rte atomics with GCC builtin atomics >=20 > > From: Tyler Retzlaff [mailto:roretzla@linux.microsoft.com] > > Sent: Wednesday, 22 March 2023 16.30 > > > > On Wed, Mar 22, 2023 at 03:58:07PM +0100, Morten Br=F8rup wrote: > > > > From: Tyler Retzlaff [mailto:roretzla@linux.microsoft.com] > > > > Sent: Wednesday, 22 March 2023 15.22 > > > > > > > > On Wed, Mar 22, 2023 at 12:28:44PM +0100, Morten Br=F8rup wrote: > > > > > > From: Tyler Retzlaff [mailto:roretzla@linux.microsoft.com] > > > > > > Sent: Friday, 17 March 2023 22.49 > > > > > > > > > > > > On Fri, Mar 17, 2023 at 02:42:26PM -0700, Stephen Hemminger > wrote: > > > > > > > On Fri, 17 Mar 2023 13:19:41 -0700 Tyler Retzlaff > > > > > > > wrote: > > > > > > > > > > > > > > > Replace the use of rte_atomic.h types and functions, > > > > > > > > instead use > > GCC > > > > > > > > supplied C++11 memory model builtins. > > > > > > > > > > > > > > > > This series covers the libraries and drivers that are > > > > > > > > built on > > > > Windows. > > > > > > > > > > > > > > > > The code has be converted to use the __atomic builtins but > > > > > > > > there > > are > > > > > > > > additional during conversion i notice that there may be > > > > > > > > some > > issues > > > > > > > > that need to be addressed. > > > > > > > > > > > > > > I don't think all these cmpset need to use SEQ_CST. > > > > > > > Especially for the places where it is used a loop, might be > > > > > > > more efficient with some of the other memory models. > > > > > > > > > > > > i agree. > > > > > > > > > > > > however, i'm not trying to improve the code with this change, > > > > > > just decouple it from rte_atomics.h so trying my best to avoid > > > > > > any unnecessary semantic change. > > > > > > > > > > > > certainly if the maintainers of this code wish to weaken the > > > > > > ordering where appropriate after the change is merged they can > > > > > > do so and > > handily > > > > > > this change has enabled them to do so easily allowing them to > > > > > > test > > just > > > > > > their change in isolation. > > > > > > > > > > I agree with the two-step approach, where this first step is a > > > > > simple > > > > search-and-replacement; but I insist that you add a FIXME or > > > > similar note where you have blindly used SEQ_CST, indicating that > > > > the memory order > > needs to > > > > be reviewed and potentially corrected. > > > > > > > > i think the maintainers need to take some responsibility, if they > > > > see optimizations they missed when previously writing the code > > > > they need to follow up with a patch themselves. i can't do > > > > everything for them and marking things i'm not sure about will > > > > only lead to me having to churn patch series to remove the unwanted > comments later. > > > > > > The previous atomic functions didn't have the "memory order" > > > parameter, so > > the maintainers didn't have to think about it - and thus they didn't > > miss any optimizations when accepting the code. > > > > > > I also agree 100 % that it is not your responsibility to consider or > > determine which memory order is appropriate! > > > > > > But I think you should mark the locations where you are changing > > > from the > > old rte_atomic functions (where no memory order optimization was > > available) to the new functions - to highlight where the option of > > memory ordering has been introduced and knowingly ignored (by you). > > > > > > > first, i have to apologize i confused myself about which of the many > > patch series i have up right now that you were commenting on. >=20 > No worries... you are rushing through quite an effort for this, so a litt= le > confusion is perfectly understandable. Especially when I'm replying to an= ageing > email. :-) >=20 > > > > let me ask for clarification in relation to this series. > > > > isn't that every single usage of the rte_atomic APIs? >=20 > Probably, yes. >=20 > > i mean are you > > literally asking for the entire patch series to look like the > > following patch snippet with the expectation that maintainers will > > come along and clean up/review after this series is merged? > > > > -rte_atomic_add32(&o, v); > > +//FIXME: opportunity for relaxing ordering constraint, please review > > +__atomic_fetch_add(&o, v, order); >=20 > Exactly. And something similar for the rte_atomicXX_t variables changed t= o > intXX_t, such as the packet counters. >=20 > Realistically, I don't expect the maintainers to clean them up anytime so= on. The > purpose is to make the FIXMEs stick until someone eventually cleans them = up, so > they are not forgotten as time passes. Cleaning up the rte_atomic APIs is a different effort. There is already lot= of effort that has gone into this and there is more effort happening (rte_= ring being a painful one) Instead of having FIXME, why not just send a separate patch with SEQ_CST (s= till a search and replace)? We can leave the tougher ones like rte_ring as = they are being worked on. >=20 > > > > this would just be a mechanical addition to this series so i can > > certainly accomodate that, i thought something more complicated was > > being asked for. if this is all, then sure no problem. >=20 > Great. >=20 > > > > > > keep in mind i have to touch each of these again when converting > > > > to standard so that's a better time to review ~everything in more > > > > detail because when converting to standard that's when suddenly > > > > you get a bunch of code generation that is "fallback" to seq_cst th= at isn't > happening now. > > > > > > I think you should to do it when replacing the rte_atomic functions > > > with the > > __atomic functions. It will make it easier to see where the memory > > order was knowingly ignored, and should be reviewed for optimization. > > > > > > > > > > > the series that converts to standard needs to be up for review as > > > > soon as possible to maximize available time for feedback before > > > > 23.11 so it would be better to get the simpler cut & paste > > > > normalizing the code out of the way to unblock that series submissi= on. > > > > > > > > > > > > > > Also, in a couple of the drivers, you are using int64_t for > > > > > packet > > counters. > > > > These cannot be negative and should be uint64_t. And AFAIK, such > > > > counters > > can > > > > use RELAXED memory order. > > > > > > > > i know you don't mean to say i selected the types and rather that > > > > the types that were selected are not quite correct for their usage. > > > > > > Yes; the previous types were also signed, and you didn't change that. > > > > > > > again > > > > on the review that actually adopts std atomics is a better place > > > > to make any potential type changes since we are "breaking" the API > > > > for 23.11 anyway. further, the std atomics series technically > > > > changes all the types so it's probably better to make one type > > > > change then rather than one now and one later. > > > > > > > > i think it would be best to get these validated and merged asap so > > > > we can get to the std atomics review. when that series is up let's > > > > discuss further how i can mark areas of concern, with that series > > > > i expect there will have to be some changes in order to avoid minor > regressions. > > > > > > > > thanks! > > > > > > I thought it would be better to catch these details (i.e. memory > > > ordering > > and signedness) early on, but I now understand that you planned to do > > it in a later step. So I'll let you proceed as you have planned. > > > > > > Thanks for all your work on this, Tyler. It is much appreciated! > > > > again, sorry for the confusion the sooner i can get some of these > > merged the easier it will be for me to manage the final series. i hope > > david/thomas can merge the simple normalization patches as soon as > > 23.03 cycle is complete. >=20 > Yes. An early merge would also provide more time for reviewing and optimi= zing > the memory order of the most important atomic operations.