From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BB93AA0526; Wed, 25 Nov 2020 05:51:15 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 68FB9C940; Wed, 25 Nov 2020 05:51:13 +0100 (CET) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80054.outbound.protection.outlook.com [40.107.8.54]) by dpdk.org (Postfix) with ESMTP id 20ABCC93C for ; Wed, 25 Nov 2020 05:51:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=; b=0UfNEG9JSb3VjjfP25XXIZRHeP0p1DBfb5YZPCerFctTXqo2n837rS63K+NhyN/2Ijub9mk8zEIy3/1S7wS7RWLKmfU/QMkjkx4uywhVVDbIDJXUWkLzXw/0Mj6fZqbmhmTVJQ4pNuCnApzSVoZCNf5nd9iq669/osUbqiYEZUQ= Received: from AM5P194CA0019.EURP194.PROD.OUTLOOK.COM (2603:10a6:203:8f::29) by AM5PR0801MB1811.eurprd08.prod.outlook.com (2603:10a6:203:39::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20; Wed, 25 Nov 2020 04:51:05 +0000 Received: from VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com (2603:10a6:203:8f:cafe::b9) by AM5P194CA0019.outlook.office365.com (2603:10a6:203:8f::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20 via Frontend Transport; Wed, 25 Nov 2020 04:51:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT004.mail.protection.outlook.com (10.152.18.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.20 via Frontend Transport; Wed, 25 Nov 2020 04:51:04 +0000 Received: ("Tessian outbound e0cdfd2b0406:v71"); Wed, 25 Nov 2020 04:51:04 +0000 X-CR-MTA-TID: 64aa7808 Received: from d43d171384c6.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 5A22A322-8235-482D-9F3D-4A85A84130CC.1; Wed, 25 Nov 2020 04:50:59 +0000 Received: from EUR01-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d43d171384c6.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 25 Nov 2020 04:50:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ITof73Pn/mZzBNnOzzhHQh/EFY1VBYAFEWfFS8DQgLKHe5wVThPg0objlxwNsynA7s7GmflpdjR0h5QtqMQOn5Tz3xF2a4KA4NxVqe1vBkcvx3Dyz8sk/F36m+xz0fIMJa4NPZ+83r4r/JMWmZKS4+MAdifb3bJMWd/00qm+x7Ms6mvnd0Mk3XHOx9n4Nv5tOM823L3DLLARg8BQQXU+qMdb1Vhw5P2QIQQtMRuBFMc5QM+gXzD3VoCrcDza8tBfvkUrsflnTn6ccNYU3J74xujpmIVMkk22mWLNUVwk+xqScsRLcDzZrpuOHp8NjdWaxv3XA6HGr7XOy7nD9H9IxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=; b=bNuNwr0L+AuQ/ed9Qp7EG2iwWgNZgeCUdHgyhDLivpsRBUOWHC8t10HL4V4WQulAi75PLLe/JfiULuXXZYscw2fxhB/rMb4t3iMCZ7lycOs4pWdvcjXX99YkXm9RswiAHWLTi3dLfEYcyjNpnKCysI2cp8OlLF38zJiZXfgN4/vNhMJONFpDsN0qS+i+rI0UlrVW7JgvYOstpLddbXV+gYy1kXapc7/31nHjitdTN1PzmCZV94DYBIOZ8hv63HUZvDHncwXrhbw/tUmZe0/CdkVg5efJAXJVxo4S1TDLr+w8hUm5/rMl/Oq8kGvo0ogtQH7nkmCLQtAiRf/berfX9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=; b=0UfNEG9JSb3VjjfP25XXIZRHeP0p1DBfb5YZPCerFctTXqo2n837rS63K+NhyN/2Ijub9mk8zEIy3/1S7wS7RWLKmfU/QMkjkx4uywhVVDbIDJXUWkLzXw/0Mj6fZqbmhmTVJQ4pNuCnApzSVoZCNf5nd9iq669/osUbqiYEZUQ= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB8PR08MB5484.eurprd08.prod.outlook.com (2603:10a6:10:111::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Wed, 25 Nov 2020 04:50:54 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::9b8:e230:9bbf:c61f]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::9b8:e230:9bbf:c61f%7]) with mapi id 15.20.3611.021; Wed, 25 Nov 2020 04:50:54 +0000 From: Honnappa Nagarahalli To: Stephen Hemminger CC: Diogo Behrens , "thomas@monjalon.net" , "david.marchand@redhat.com" , "dev@dpdk.org" , nd , Honnappa Nagarahalli , nd Thread-Topic: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory Thread-Index: AQHWwcaSjZGD7vozU0KcDD8t04pzMKnWHG2AgAIq3yA= Date: Wed, 25 Nov 2020 04:50:54 +0000 Message-ID: References: <20200826092002.19395-1-diogo.behrens@huawei.com> <20201123113651.46323c54@hermes.local> In-Reply-To: <20201123113651.46323c54@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 2B78BF7590720D4DB17A735C43A96C28.0 x-checkrecipientchecked: true Authentication-Results-Original: networkplumber.org; dkim=none (message not signed) header.d=none;networkplumber.org; dmarc=none action=none header.from=arm.com; x-originating-ip: [217.140.110.7] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 356ac048-1ca5-4e15-d41a-08d890fdb72c x-ms-traffictypediagnostic: DB8PR08MB5484:|AM5PR0801MB1811: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 2mOF+wn2Z9EOqzhTAsWc5KdP/HADPteWH/ueMXKUvOy47D+J2+H1d+0xsJuKN07N1yIjc3cSKKPdoh0mMdMl7apjQrIQyGVFCswFOm+dId/L57YakOJEvyIvndfVg7grzffhSDYKLlNfpf73SNDdti5XiSaRB8bd4R89r+jfMy//Vc36ea77mFVHtHIbJ/C9MB9e6shnLdjvVvkY9sYPfzMogRlAYXreBtJHl/AVc0kwhCYK8rL5eVj278HqRtSF0Qhh3rW1/lHed6R+IlAF3EGqMAOzIlCCN1PkdTJ+YYmT+ovA4sZntub6+CQBjKme X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(376002)(136003)(39860400002)(346002)(66446008)(66476007)(7696005)(5660300002)(83380400001)(64756008)(54906003)(76116006)(66946007)(316002)(86362001)(4326008)(33656002)(52536014)(55016002)(186003)(26005)(6916009)(6506007)(71200400001)(66556008)(478600001)(9686003)(2906002)(8676002)(8936002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?WgQ13fqzUa770Ek8K5ZQUwZQbkcQ6+AWoAtImjHkRd4NZ8pqXCA4f6EhiL03?= =?us-ascii?Q?yUIdbDGuYwiKWOV43zS1NNA5vmD+OkwweX0sQDxIZqvxlRWJmojnRXMHtWkg?= =?us-ascii?Q?jblFSjDt+1AT1ZCK6+Tab1kJCo23AWg1OxvHFkfiKN+LOs8R9ghLXSoWKYTw?= =?us-ascii?Q?rZRNedxY2Grx9EbPfR86OXZm/Pae3qRWQPVhB7JbtTjJ6i2Wa8DXhElzpLOS?= =?us-ascii?Q?PgkxfE8mU2LDnHjy4H5WymuWGM8iF0oVKStguYBSYcUPpAzIceod5pAb6x6P?= =?us-ascii?Q?+VaxgCFKHen1c37hQ2xzVjMigYTEbVJhje1kT2qYXXkGCeVnWNZxvErAa3we?= =?us-ascii?Q?+IIIz/DqdmzCA5hXnUi2v6fJHT4+0x7n/t1ASL7sdAAbnHt5hj5BOYMvIrMk?= =?us-ascii?Q?hrGMLsqCTVVPn4p9FTa/0rCriqYf5EnqRttlTjkBTwBq/0YFDHqvfQsQ7E8+?= =?us-ascii?Q?5kCF98DSqGux3kQWKOTbesdM36IGqObu6V10XpD0Lem/OBNZhY0hUlR1eL/z?= =?us-ascii?Q?A1/l3EQCi1od3HroHVPn/fE7FsEHmAvwAsN3gePsKzyXcuWydTGM7F8TlB1w?= =?us-ascii?Q?l+Ks+KZY/jzl2zoSbecKRFE3Eq/3K3bdErnWFW8F+wNfThOOjoDgVIfX8Hw1?= =?us-ascii?Q?37KbTuFyrj8acemLz1THl8kikuJa69Ees+SuXxmxF9/cc/RQ5fpv9icPnXvO?= =?us-ascii?Q?rIVJRCkysAWgE5qriZpJUE04IA/DJTWqCysOXiofDjI1BkVw/2NpOpQhscDN?= =?us-ascii?Q?sb9++qtnw5Q/DhPL40I/rxcywXEUnxKMpFMiIqEXG8o3dLTU6pgmL4SoSPyV?= =?us-ascii?Q?SVqU6/lYWptwwhhPNytsAo150Qs8tAbU3RGqJVe+VSDZQkK4jmmK4G+7tQ0Q?= =?us-ascii?Q?tiCVuFNukTOYdEArUIFgasjZ9azugI9k5LVtv/IiHvtgQWMgedRYkqpDrUcw?= =?us-ascii?Q?v/OFMX7sDd64IYglofqNtaQq/C8pPZKRzAAgFOjBldg=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR08MB5484 Original-Authentication-Results: networkplumber.org; dkim=none (message not signed) header.d=none; networkplumber.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: a05c2f1d-9dcd-4e0a-0787-08d890fdb125 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vnVaSsUGg4y+5q5FAhMtkdYr9LGSnIZHw6rnc89WZLLpyAPGIH0bqmTVcWu44pk30XuW3UU+uTJ9hTCq1VK5s25sq4BQJz3Q6IYSVV1AnSFApuhykm7PLZWMXu/5l/DZviwJhnEyXY05rtjByR+UPcmmsdLpE4rsZzVJRiSEAn+Mvcz8nzXskxBcsIyaDqBecDkc7ZF1aIh8ZzczfRiezsYxF5ex7PnqShN3bb2sWkV7d2cwEtyjDfk2Zon0D/g7ey4wUhBd+uwRbwDfzTjg0vJvyAKiCUg//lw+75EW78zImuUGO08a3+1Yi6MfPYxTJoLXqQ29Yju/gqh1Fqx5Ug1bOFDhxRFFb0LOpNipdrQuAYxRKc8/FvnTxcp/dVsNgU+2XIqdyX2NzjTpVCOf8g== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(396003)(346002)(136003)(376002)(39860400002)(46966005)(54906003)(316002)(8676002)(356005)(82740400003)(5660300002)(55016002)(33656002)(52536014)(9686003)(70206006)(81166007)(47076004)(83380400001)(8936002)(6862004)(2906002)(4326008)(70586007)(336012)(186003)(26005)(7696005)(6506007)(86362001)(478600001)(82310400003); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2020 04:51:04.5929 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 356ac048-1ca5-4e15-d41a-08d890fdb72c X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0801MB1811 Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > > > The initialization me->locked=3D1 in lock() must happen before > > > next->locked=3D0 in unlock(), otherwise a thread may hang forever= , > > > waiting me->locked become 0. On weak memory systems (such as > ARMv8), > > > the current implementation allows me->locked=3D1 to be reordered = with > > > announcing the node (pred->next=3Dme) and, consequently, to be > > > reordered with next->locked=3D0 in unlock(). > > > > > > This fix adds a release barrier to pred->next=3Dme, forcing > > > me->locked=3D1 to happen before this operation. > > > > > > Signed-off-by: Diogo Behrens > > The change looks fine to me. I have tested this on few x86 and Arm mac= hines. > > Acked-by: Honnappa Nagarahalli >=20 > Maybe a simpler alternative would be as fast and safer. Why is this safer? > By using compare_exchange you can get same effect in one operation. > Like the following UNTESTED. >=20 > diff --git a/lib/librte_eal/include/generic/rte_mcslock.h > b/lib/librte_eal/include/generic/rte_mcslock.h > index 78b0df295e2d..9c537ce577e6 100644 > --- a/lib/librte_eal/include/generic/rte_mcslock.h > +++ b/lib/librte_eal/include/generic/rte_mcslock.h > @@ -48,23 +48,23 @@ rte_mcslock_lock(rte_mcslock_t **msl, rte_mcslock_t > *me) > rte_mcslock_t *prev; >=20 > /* Init me node */ > - __atomic_store_n(&me->locked, 1, __ATOMIC_RELAXED); > - __atomic_store_n(&me->next, NULL, __ATOMIC_RELAXED); > + me->locked =3D 1; >=20 > - /* If the queue is empty, the exchange operation is enough to acquire > - * the lock. Hence, the exchange operation requires acquire semantics. > - * The store to me->next above should complete before the node is > - * visible to other CPUs/threads. Hence, the exchange operation > requires > - * release semantics as well. > + /* > + * Atomic insert into single linked list > */ > - prev =3D __atomic_exchange_n(msl, me, __ATOMIC_ACQ_REL); > + do { > + prev =3D __atomic_load_n(msl, __ATOMIC_RELAXED); > + me->next =3D prev; This needs to be __atomic_store_n(__ATOMIC_RELEASE) as it can sink below th= e following line. > + } while (!__atomic_compare_exchange_n(&msl, me, prev, > + __ATOMIC_ACQUIRE, > __ATOMIC_RELAXED)); > + > if (likely(prev =3D=3D NULL)) { > /* Queue was empty, no further action required, > * proceed with lock taken. > */ > return; > } > - __atomic_store_n(&prev->next, me, __ATOMIC_RELAXED); >=20 > /* The while-load of me->locked should not move above the previous > * store to prev->next. Otherwise it will cause a deadlock. Need a