From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 58BAAA04DD; Tue, 20 Oct 2020 23:50:09 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6CB76AC84; Tue, 20 Oct 2020 23:50:06 +0200 (CEST) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80089.outbound.protection.outlook.com [40.107.8.89]) by dpdk.org (Postfix) with ESMTP id 0469EAC6C for ; Tue, 20 Oct 2020 23:50:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sefOfOdRElf0Ol47nULaqtwuAQHIsHfGQmbnmNj3D+k=; b=H4vFhJlX+mg0l7EvBLIFQNXwgzVn8WnEptuk9dOFaGi+3XImk9YllHeHMdQFZ95eJBs4B4/uUdTNBL4VHXQZNDDu5BeFxnAno1sIDJ/tohd3ZHSzBUZ+ZZUXge5y1JgmlY52a5Z+k6eFqKreSwYItXzoGXykDBzhNtp8elNWirU= Received: from AM5PR0701CA0024.eurprd07.prod.outlook.com (2603:10a6:203:51::34) by AM4PR08MB2643.eurprd08.prod.outlook.com (2603:10a6:205:5::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3477.20; Tue, 20 Oct 2020 21:49:54 +0000 Received: from AM5EUR03FT022.eop-EUR03.prod.protection.outlook.com (2603:10a6:203:51:cafe::7) by AM5PR0701CA0024.outlook.office365.com (2603:10a6:203:51::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.9 via Frontend Transport; Tue, 20 Oct 2020 21:49:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT022.mail.protection.outlook.com (10.152.16.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.18 via Frontend Transport; Tue, 20 Oct 2020 21:49:54 +0000 Received: ("Tessian outbound d5e343850048:v64"); Tue, 20 Oct 2020 21:49:54 +0000 X-CR-MTA-TID: 64aa7808 Received: from 65835c57dbe2.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 052056D8-4EBF-4EE5-9B7D-5CB9DD7D9398.1; Tue, 20 Oct 2020 21:49:49 +0000 Received: from EUR03-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 65835c57dbe2.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 20 Oct 2020 21:49:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ltH7Qe+IIz0ygJew5yYwfDOVXISP72dC5pdWIy9TGcLVm2Xuo5rSElevU4QOUJDR79mXS8QbC5keOvEPLXKQNmcYyi78Y5OXnmcbHk7P8VWsy9hf7bT5m37bEXu8GC+SaJPmmt0kY3liZFWhCwVpp3XUHML+jXLTRd/yDO807EuySdZyRllBYQBIVIa+Zaq/KbjymH9vS/VfVGd1fIDjPlc+1QZ9/NCtaH6jUFj9xezK/bABJuTxgdiZJnmFsciC8qPp5AuTUfuGb2M8aYsF7dOpohKf1qE6qIaB5GAExZWgqrB2xg6bh/86D78VQ0PnTlApBz51uSSkIJvFcdq8LQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sefOfOdRElf0Ol47nULaqtwuAQHIsHfGQmbnmNj3D+k=; b=OIH1LUDAotLZk5SKPFtXa7GsjPEluidy7AgCBUFv6iojzydlsOPdVWNHOPFR3Fwkr/vTxOgFL5PNn3zYTgOXPoA2N8uxHv/6nHNkWGsiVcmg7MzZ+sjC+Bx5zetX5H/7KfpzzmgFGZlgJ4qdfNfV5UmgF80XdI5dRDWtKAmeP0/i0b9ArOEqgld15MHe9ypfEtqZ21zCt6qFgMlQb9RWaWKyBv3ShhDtj+3jj74qiAGEUKx4a3Q1j1NHU/bWK5T4KJWiSZKi85ZEGZapTIYXv/G4HTBNC5D8WC3c8k+uLXVAc/m6nwY2iU4d6nCRqi1zlWqIXO72S0lSqG8dnAZPxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sefOfOdRElf0Ol47nULaqtwuAQHIsHfGQmbnmNj3D+k=; b=H4vFhJlX+mg0l7EvBLIFQNXwgzVn8WnEptuk9dOFaGi+3XImk9YllHeHMdQFZ95eJBs4B4/uUdTNBL4VHXQZNDDu5BeFxnAno1sIDJ/tohd3ZHSzBUZ+ZZUXge5y1JgmlY52a5Z+k6eFqKreSwYItXzoGXykDBzhNtp8elNWirU= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB6PR0801MB1990.eurprd08.prod.outlook.com (2603:10a6:4:6c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3477.27; Tue, 20 Oct 2020 21:49:46 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7814:9c1:781f:475d]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7814:9c1:781f:475d%4]) with mapi id 15.20.3499.018; Tue, 20 Oct 2020 21:49:46 +0000 From: Honnappa Nagarahalli To: "thomas@monjalon.net" CC: Phil Yang , "dev@dpdk.org" , nd , Diogo Behrens , "david.marchand@redhat.com" , Honnappa Nagarahalli , nd Thread-Topic: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory Thread-Index: AQHWpthJjZGD7vozU0KcDD8t04pzMKmgvuDg Date: Tue, 20 Oct 2020 21:49:46 +0000 Message-ID: References: <20200826092002.19395-1-diogo.behrens@huawei.com> <1947647.zX4bR4m4Ni@thomas> <05e6a3569608493abbcc4dba618c5c2c@huawei.com> <7423385.l6g0CaCsxP@thomas> In-Reply-To: <7423385.l6g0CaCsxP@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 5FC501231FF22D43B43DB10BF47C1783.0 x-checkrecipientchecked: true Authentication-Results-Original: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.113.13.105] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: c612a993-4e6c-49f4-7679-08d8754214ab x-ms-traffictypediagnostic: DB6PR0801MB1990:|AM4PR08MB2643: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: I+wjTC5/hOlaf7n20oXzWysTh2fEzFFdBGyeUBdNUBqrgt73LQlK4FAPAQjy0yZWTzxV1uHBc2JYkmxLeQtpDQV1bWjDqvEVCIm/2GZOO6zODzRsvSNXuprxc94h7fMmMq4bX2eBNJzmUfzkM85WgroD/InJ45RkcQwBrbD/UeObaKML0OcVo6O460ueYWA50qnYUY7Jjlbkd5HHMUbVHlDVOOvy1EEigogI7hnPfPTbvmJn7YNgDjnby/2qiDQ2c6JNizHEX2BEmZR+y91C99KIoPC0P3J/YlTN7e5NMQjLw+EuRxBg8SlFwyNxwk9D2UBHBeWs8pLVhs9eWXem76CQ2ysdHV7dP06xwqn/Grn+b9lxlYNdon9WNJeFQ1tztLDNiNwggIMBEvDI9XcpbA== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(376002)(346002)(39850400004)(55016002)(966005)(478600001)(52536014)(33656002)(66446008)(2906002)(66556008)(66476007)(64756008)(66946007)(8936002)(5660300002)(4326008)(83380400001)(8676002)(76116006)(6916009)(86362001)(9686003)(53546011)(6506007)(71200400001)(7696005)(54906003)(186003)(316002)(26005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: nty8qDyMhMHigY0Rnd/6HFYL8uBhA3WiLoKhvsPbU+P0cVWs1PXIMYL79p37CHPYVAN8AZB28nBK57t7UzM0AJrgLV56JHmzVAULK3AeVl33EyegYdQP2RJwYNF5/0LxN7rjkBH2my5QZzLq3YSmX4VOLFjjyylI5m20Z+sqU2BXUoSp8J3YXCFV7Ck5jYyEu7KAdQypeaTV7pnaFMkUY5QM3P3TC03Kw8wvs+YMH8c1/fp+zEIU/IgjG8CnrjJzeQMceaeoAccMiwuPvIoX7XXXN59sF/5HuFlJaKzMeB7EkvPpnFpX36kqKqgR1nNTeFd0gzCbZuEx3F3xvRixmoV0meKscGPiX8+nlGa68CQ3jwBMym0CP6XqlKvIAZkwVid+jBaPicgKXNuBCC3TPgtXg1iFHj2NjRZKiyNDMb/0mloATT50CYDxl+SIVFPIU65RBOdlCG00kLCsVjoV2oHSg5tyD0QKYU1WJEBImO5m7alCjQLkFmDq9xJLzJKI13gpkU6Ly7Q9UHFqV48Aq33531fplwcgOTy2bXLzfDdoPDLOrsP4hKnPdXGJEP9iqsFv3z7J+5bI30RZ1TYnWD0uLv6d8S0NsDA4nBAQkAgYb9XBRRhpGczgHy9HLDS4c5cjOs3uB4tjY+4kjUtfVQ== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0801MB1990 Original-Authentication-Results: monjalon.net; dkim=none (message not signed) header.d=none; monjalon.net; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT022.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 42fa56c2-520e-469d-2152-08d875420f90 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EjAki5jFmwqkT2CPoTsc462qP7qXG82b7bJeTJEaY60gMoC19vfJUT60DCme0zkQuXv2uLH0IbuHmUr2Kq0Qm1bL9IPAMc3oWAJpooa0P3x058HINesvYjdllnZPZ2um+eq3NKSF3f3QO/vngQC20zyR9aQXBqXM2t5GA+pY8IAX0hjKsV4GO1ww7TBwMlRD9Rc/UROAU6/w4G4Bhi6K0G2Za51Lb/z2wXmRwH4iEc/rGevvI/4l8eVc01JYQ9Ud0vVDr8l8DTahS//JlxfyHYn+2KjeR/h35Hzpj9kjk3y96owEBPDOp2rMCxVZJM7KZANX9Y4HtplppVA+51kenA36dI5PXr7amZywMe1/O6/2RWkKGkHm6uNuEO9ZC9V3UgSb8dmaE83zYQ95Cw2msqgY2O+SLwc7LdKWFIfUYBJCVKXqcaOB1xQu0V5FuYVs5tPeLjcaMprVNS9TQdgYvszVlu6kqij6p175zoroqTc= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(39850400004)(376002)(396003)(136003)(346002)(46966005)(186003)(81166007)(82310400003)(83380400001)(8936002)(47076004)(356005)(82740400003)(6862004)(4326008)(5660300002)(316002)(36906005)(9686003)(336012)(54906003)(33656002)(86362001)(55016002)(8676002)(26005)(70586007)(2906002)(7696005)(70206006)(53546011)(478600001)(6506007)(52536014)(966005); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2020 21:49:54.7428 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c612a993-4e6c-49f4-7679-08d8754214ab X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT022.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR08MB2643 Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > Honnappa? >=20 > 07/10/2020 11:55, Diogo Behrens: > > Hi Thomas, > > > > we are still waiting for the comments from Honnappa. In our > > understanding, the missing barrier is a bug according to the model. We > > reproduced the scenario in herd7, which represents the authoritative > > memory model: > > https://developer.arm.com/architectures/cpu-architecture/a-profile/mem > > ory-model-tool > > > > Here is a litmus code that shows that the XCHG (when compiled to LDAXR > and STLR) is not atomic wrt memory updates to other locations: > > ----- > > AArch64 XCHG-nonatomic > > { > > 0:X1=3Dlocked; 0:X3=3Dnext; > > 1:X1=3Dlocked; 1:X3=3Dnext; 1:X5=3Dtail; > > } > > P0 | P1; > > LDR W0, [X3] | MOV W0, #1; > > CBZ W0, end | STR W0, [X1]; (* init locked *) > > MOV W2, #2 | MOV W2, #0; > > STR W2, [X1] | xchg:; > > end: | LDAXR W6, [X5]; > > NOP | STLXR W4, W0, [X5]; > > NOP | CBNZ W4, xchg; > > NOP | STR W0, [X3]; (* set next *) > > exists > > (0:X2=3D2 /\ locked=3D1) > > ----- > > (web version of herd7: http://diy.inria.fr/www/?record=3Daarch64) > > > > P1 is trying to acquire the lock: > > - initializes locked > > - does the xchg on the tail of the mcslock > > - sets the next > > > > P0 is releasing the lock: > > - if next is not set, just terminates > > - if next is set, stores 2 in locked > > > > The initialization of locked should never overwrite the store 2 to lock= ed, but > it does. > > To avoid that reordering to happen, one should make the last store of P= 1 to > have a "release" barrier, ie, STLR. > > > > This is equivalent to the reordering occurring in the mcslock of librte= _eal. > > > > Best regards, > > -Diogo > > > > -----Original Message----- > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > Sent: Tuesday, October 6, 2020 11:50 PM > > To: Phil Yang ; Diogo Behrens > > ; Honnappa Nagarahalli > > > > Cc: dev@dpdk.org; nd > > Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak > > memory > > > > 31/08/2020 20:45, Honnappa Nagarahalli: > > > > > > Hi Diogo, > > > > > > Thanks for your explanation. > > > > > > As documented in > https://developer.arm.com/documentation/ddi0487/fc B2.9.5 Load- > Exclusive and Store-Exclusive instruction usage restrictions: > > > " Between the Load-Exclusive and the Store-Exclusive, there are no > > > explicit memory accesses, preloads, direct or indirect System > > > register writes, address translation instructions, cache or TLB > maintenance instructions, exception generating instructions, exception > returns, or indirect branches." > > > [Honnappa] This is a requirement on the software, not on the micro- > architecture. > > > We are having few discussions internally, will get back soon. > > > > > > So it is not allowed to insert (1) & (4) between (2, 3). The cmpxchg > operation is atomic. > > > > > > Please what is the conclusion? Apologies for not updating on this sooner. Unfortunately, memory ordering questions are hard topics. I have been discu= ssing this internally with few experts and it is still ongoing, hope to con= clude soon. My focus has been to replace __atomic_exchange_n(msl, me, __ATOMIC_ACQ_REL)= with __atomic_exchange_n(msl, me, __ATOMIC_SEQ_CST). However, the generate= d code is the same in the second case as well (for load-store exclusives), = which I am not sure if it is correct. I think we have 2 choices here: 1) Accept the patch - when my internal discussion concludes, I can make the= change and backport according to the conclusion. 2) Wait till the discussion is over - it might take another couple of weeks >=20 >=20