From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C83C6A04B1; Mon, 23 Nov 2020 16:06:18 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AA08F37AF; Mon, 23 Nov 2020 16:06:17 +0100 (CET) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50069.outbound.protection.outlook.com [40.107.5.69]) by dpdk.org (Postfix) with ESMTP id DE6E4160 for ; Mon, 23 Nov 2020 16:06:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T0qXGp/JU+yf9E4x09r68G3k+CDeDBbpUqeL25VUnGU=; b=EDneGrSyvsEQ767a4gRsUg0hLveBnucWczPPNHupOKjBY3sqPLhJhuHboftRJJzTwI1IR9BOAstlx1YniqZGWMG4NwjnEfrDkPTdbSZw5DHc1py74mpkRFvggXPsvgHjzeD5AXihe+G79jR/RmQdwcWoZqr9n2lg+PFLItqndek= Received: from AM5PR0701CA0002.eurprd07.prod.outlook.com (2603:10a6:203:51::12) by AM7PR08MB5463.eurprd08.prod.outlook.com (2603:10a6:20b:106::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.20; Mon, 23 Nov 2020 15:06:13 +0000 Received: from AM5EUR03FT049.eop-EUR03.prod.protection.outlook.com (2603:10a6:203:51:cafe::fa) by AM5PR0701CA0002.outlook.office365.com (2603:10a6:203:51::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.9 via Frontend Transport; Mon, 23 Nov 2020 15:06:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT049.mail.protection.outlook.com (10.152.17.130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.20 via Frontend Transport; Mon, 23 Nov 2020 15:06:13 +0000 Received: ("Tessian outbound e0cdfd2b0406:v71"); Mon, 23 Nov 2020 15:06:13 +0000 X-CR-MTA-TID: 64aa7808 Received: from 13f03815a2b3.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 88856005-EFA3-442D-88BB-B909FD68F090.1; Mon, 23 Nov 2020 15:06:08 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 13f03815a2b3.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 23 Nov 2020 15:06:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c2IMx71/GXeM4d9xCI8jMPI/7WheKIJhTe1vbZa1Ob7diuIpgMQXDkur2tsqKrg8OhvvYiYjaAPDdbUtFB1/OEc6bwn7gViMCcfXpoBN5ubDwGn17GpuDDPhuvRomvWsYdcaKr/DZ9BU+n8IuE345vBZK3QNiv4e7v+PcfZ31x3O4ScIn7BiwPYYkgSM/dcne78VFkmVN3aqyLFTHrZK9E92ueuf/VSbjpenlYTzUYiQquL4iNT/6M6Q2ieu3gm4TqdgAOKXv3PRkWqvYNcG8TQzOVMvCohpLEKoe2mfhgY/7QMnXgjFCvgTZbWuCVIPqiQgd6T+HGm+CPDe2/jwuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T0qXGp/JU+yf9E4x09r68G3k+CDeDBbpUqeL25VUnGU=; b=N4CSb/bQiLMDpCsqmCXGquu7Kqgefx1Om74TlrVqr2ZzcYKBRltI0xpXd5H3qnN4Bg0f0AezMKSD4yj1gVrsBmrydA0RjgVRV3k01ev+W4Q65mGb9NL5FMKcqxLPmyUkuCpPCUelN4KG0odX1kX8vKdAdizt4Xr4GAhd4RKs8NN8OvjAbOMmMMPAYp4OD4PhTEyjSw9IwbwhPLa/0Q38CP4eb5mvApu3zE9a2a+LmcH8gyuVbCNAxtSFd9jI2nXOpFraPW/tMCOJJxOWwKLOXdUg9J7Q+RMdzul3fgVny0QTvJAA9GHZrgHSwXGjsvAFlKnEO4ZuQNHj03xcfk0+/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T0qXGp/JU+yf9E4x09r68G3k+CDeDBbpUqeL25VUnGU=; b=EDneGrSyvsEQ767a4gRsUg0hLveBnucWczPPNHupOKjBY3sqPLhJhuHboftRJJzTwI1IR9BOAstlx1YniqZGWMG4NwjnEfrDkPTdbSZw5DHc1py74mpkRFvggXPsvgHjzeD5AXihe+G79jR/RmQdwcWoZqr9n2lg+PFLItqndek= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB7PR08MB2954.eurprd08.prod.outlook.com (2603:10a6:5:1c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Mon, 23 Nov 2020 15:06:06 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7814:9c1:781f:475d]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7814:9c1:781f:475d%4]) with mapi id 15.20.3589.030; Mon, 23 Nov 2020 15:06:06 +0000 From: Honnappa Nagarahalli To: "thomas@monjalon.net" CC: "dev@dpdk.org" , nd , Diogo Behrens , "david.marchand@redhat.com" , Honnappa Nagarahalli , nd Thread-Topic: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory Thread-Index: AQHWpthJjZGD7vozU0KcDD8t04pzMKmgvuDggDPoFICAAVzDIA== Date: Mon, 23 Nov 2020 15:06:06 +0000 Message-ID: References: <20200826092002.19395-1-diogo.behrens@huawei.com> <7423385.l6g0CaCsxP@thomas> <3356496.NAzAc6GACp@thomas> In-Reply-To: <3356496.NAzAc6GACp@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: F46845D3D2E2504F985191CA0E824DCC.0 x-checkrecipientchecked: true Authentication-Results-Original: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; x-originating-ip: [217.140.110.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 2a6aae34-f47f-46a4-4a17-08d88fc151ab x-ms-traffictypediagnostic: DB7PR08MB2954:|AM7PR08MB5463: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: nE1KDvBKgegpN51EOt/ZF69rFyTOXGh1uXvo4TvvP5Rk1fRxxvAuvZd/B+w79ITxrKB7Km9YvR68qvP0DnIhS6sHu1DbjIfM4hdLW7IetKfs8cibUJp9hQJyv/qUF1DJcZW/4CRNkMzFieu4MdYSE3xGMLpKiwljvuYDIJ02J/nMP2V5PWNIjo7AL1bjyczjWrcaMWXJnNkKXzSKX0iQjFtBLD2xsSHAF2GcLSZHunAQLEEqR7EXjyGA9vDtokIRRjqu4zeuCot5N5VETUkFb9yiYRJPztZtOXg5G1Kq/6om5qLxt7j1I6qgahfSCsVphZ8QInyMVY6xiKYqcF57yXmq3XOVVifjx1r5+pyDfPY2wPO8tCzj5EVopk8UjPqjhThRWytZmkLMl2Vr3HcjRQ== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(396003)(366004)(136003)(376002)(346002)(66446008)(64756008)(66946007)(66476007)(5660300002)(66556008)(55016002)(9686003)(2906002)(478600001)(86362001)(966005)(54906003)(76116006)(26005)(186003)(6916009)(52536014)(6506007)(71200400001)(316002)(83380400001)(53546011)(8676002)(33656002)(4326008)(8936002)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: aM7kbQ1SjMR3TFFDSdr5UJformOILizolPvTnBMxurqJLp0ENo9rC33V4+Q0GFSOdst9XlTGL/Qz24sQX5i6a7UwMMiWYgrAUO/crOeNIte2yP3tcFSNjP0HmcRSWlaseYTNL4lR34O5LnAyAq5rF0VUybNzbfKISlO8W4hwpEejEVJlmlCjKhxS0WpFQ+Uz8vlOoQJDGcXB3kdtck7deH95uEIJCfm1lKKWwlYgYKRC571PNkvUHwg2pWvefsXQxe10azfTfsyZPrjaYCDlMhn2/9tREyVmKhIwTG/R8O4mrmouA8T4lxmkoD/n2XMOZDo61fzDvJz4AOAnWV5b0CXvskkcxApV2iH4XkAKv55zfa4bY7m/ocC0XwgIzbxXtBDM8c7HlnkPaLGL0hLsiBSE3x4rlx1RsDKIZnmzezgTqwt7b+v3mTqR7HQnKA/+1EkFP8h+KtJB1hV/WCV1wGAI0lU4TjBaYB592MeYHKoIrx1XMgfTou+N7glhI1mai2Hgb4AUWjkyNvyCo6m0D3v/j5juJomVFbDvhDd4LlfDo/3ddkAEI36MqzBQ2RghY+YMyI71lTB7YKJNJhlYICXeiv5v+AFelLRIRXEckJaBIzxHd6RoK26mEptlBTNJn2ljdTTp1YaaWtmKCoCuADPbmN32WIjMQq0Gjb5RikLJeo5j33M4LvF3ZsUXDGNY3L5FzSkepJXwLoAr3MwGFv7y4cW98haPDJCkEHoOj1+8g1EiyrbvUkjbuYnfEK1wd51TvH9O6AH0rN/sMceJcJU+WXIeBvZO6ZvG8W7fpemzhdGxFskuVtsjRZLwDkPBNxsF7dDxz0jRIGeoQq9XRBAYfZGSwnl94zAm9o0RJ6uC+9WC/E1g+qJqxipRFS7Sn59s7XmQJm3xbSBNQhQYqw== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR08MB2954 Original-Authentication-Results: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT049.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 7784c587-144d-4fcd-0238-08d88fc14d57 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TyKqXELj0QznL6TpE19fCdAB2fIzWT9A7HHtuCdnilr93rdvC4Kbh5AEsO4XEyrAoX7NQBnH4dVRsHLJ07pk1RNy+SHFQ9KFA5hWxLtIpIijO4sG0jP3L5XNovz79THVHvSKaUUNXC/j0O5BPan+4hNTVHypcswGP4GaIKjqT3J7/BKnL1xK6v4SdW/Y2uzMMKJ42Mg5Jb5T0LiKop9R+kEs3YzyD5B7B2ftPFnDhiyMkpDLRkflZdwRZ2l86ktaoCJvAyUhlEWKp/le3vRquGfVJ6pu7rWDxfk2GFsUjat05KP7AElHYo2qk0s/DRtI7ogy0ZwpfF2O3kHDWG3aANBNAz1f2+MkqtCOHXyJDP0CqmfkfhhfYEkWdLVrKEwwjN7vVCfA94ug0mdZynQp99cgYXDryxOINi3iIATUstnZfsjYm7YKlQxW8v6p/bzzCyakdrV2CZ3Y1Y+ayJCmtGTfD0m2jHaxhp5Y+JeQvro= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(346002)(136003)(46966005)(47076004)(82740400003)(6506007)(7696005)(70586007)(53546011)(86362001)(52536014)(36906005)(478600001)(81166007)(5660300002)(356005)(336012)(316002)(54906003)(2906002)(6862004)(33656002)(186003)(4326008)(26005)(70206006)(966005)(83380400001)(55016002)(9686003)(8936002)(82310400003)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2020 15:06:13.3905 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2a6aae34-f47f-46a4-4a17-08d88fc151ab X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT049.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR08MB5463 Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > 07/10/2020 11:55, Diogo Behrens: > > > > Hi Thomas, > > > > > > > > we are still waiting for the comments from Honnappa. In our > > > > understanding, the missing barrier is a bug according to the > > > > model. We reproduced the scenario in herd7, which represents the > > > > authoritative memory model: > > > > https://developer.arm.com/architectures/cpu-architecture/a-profile > > > > /mem > > > > ory-model-tool > > > > > > > > Here is a litmus code that shows that the XCHG (when compiled to > > > > LDAXR > > > and STLR) is not atomic wrt memory updates to other locations: > > > > ----- > > > > AArch64 XCHG-nonatomic > > > > { > > > > 0:X1=3Dlocked; 0:X3=3Dnext; > > > > 1:X1=3Dlocked; 1:X3=3Dnext; 1:X5=3Dtail; } > > > > P0 | P1; > > > > LDR W0, [X3] | MOV W0, #1; > > > > CBZ W0, end | STR W0, [X1]; (* init locked *) > > > > MOV W2, #2 | MOV W2, #0; > > > > STR W2, [X1] | xchg:; > > > > end: | LDAXR W6, [X5]; > > > > NOP | STLXR W4, W0, [X5]; > > > > NOP | CBNZ W4, xchg; > > > > NOP | STR W0, [X3]; (* set next *) > > > > exists > > > > (0:X2=3D2 /\ locked=3D1) > > > > ----- > > > > (web version of herd7: http://diy.inria.fr/www/?record=3Daarch64) > > > > > > > > P1 is trying to acquire the lock: > > > > - initializes locked > > > > - does the xchg on the tail of the mcslock > > > > - sets the next > > > > > > > > P0 is releasing the lock: > > > > - if next is not set, just terminates > > > > - if next is set, stores 2 in locked > > > > > > > > The initialization of locked should never overwrite the store 2 to > > > > locked, but > > > it does. > > > > To avoid that reordering to happen, one should make the last store > > > > of P1 to > > > have a "release" barrier, ie, STLR. > > > > > > > > This is equivalent to the reordering occurring in the mcslock of li= brte_eal. > > > > > > > > Best regards, > > > > -Diogo > > > > > > > > -----Original Message----- > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > Sent: Tuesday, October 6, 2020 11:50 PM > > > > To: Phil Yang ; Diogo Behrens > > > > ; Honnappa Nagarahalli > > > > > > > > Cc: dev@dpdk.org; nd > > > > Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on > > > > weak memory > > > > > > > > 31/08/2020 20:45, Honnappa Nagarahalli: > > > > > > > > > > Hi Diogo, > > > > > > > > > > Thanks for your explanation. > > > > > > > > > > As documented in > > > https://developer.arm.com/documentation/ddi0487/fc B2.9.5 Load- > > > Exclusive and Store-Exclusive instruction usage restrictions: > > > > > " Between the Load-Exclusive and the Store-Exclusive, there are > > > > > no explicit memory accesses, preloads, direct or indirect System > > > > > register writes, address translation instructions, cache or TLB > > > maintenance instructions, exception generating instructions, > > > exception returns, or indirect branches." > > > > > [Honnappa] This is a requirement on the software, not on the > > > > > micro- > > > architecture. > > > > > We are having few discussions internally, will get back soon. > > > > > > > > > > So it is not allowed to insert (1) & (4) between (2, 3). The > > > > > cmpxchg > > > operation is atomic. > > > > > > > > > > > > Please what is the conclusion? > > Apologies for not updating on this sooner. > > > > Unfortunately, memory ordering questions are hard topics. I have been > discussing this internally with few experts and it is still ongoing, hope= to > conclude soon. > > > > My focus has been to replace __atomic_exchange_n(msl, me, > __ATOMIC_ACQ_REL) with __atomic_exchange_n(msl, me, > __ATOMIC_SEQ_CST). However, the generated code is the same in the second > case as well (for load-store exclusives), which I am not sure if it is co= rrect. > > > > I think we have 2 choices here: > > 1) Accept the patch - when my internal discussion concludes, I can make= the > change and backport according to the conclusion. > > 2) Wait till the discussion is over - it might take another couple of > > weeks >=20 > One month passed since this last update. > We are keeping this issue in DPDK 20.11.0 I guess. >=20 I can accept this patch and move forward for 20.11. It is a stronger barrie= r and I do not see any issues from the code perspective. I will run tests o= n few platforms and provide my ACK. It is work in progress with few changes for me to make sure we have an opti= mal solution for all platforms. Those changes can go into 21.02.