From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id BB93AA0526;
	Wed, 25 Nov 2020 05:51:15 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 68FB9C940;
	Wed, 25 Nov 2020 05:51:13 +0100 (CET)
Received: from EUR04-VI1-obe.outbound.protection.outlook.com
 (mail-eopbgr80054.outbound.protection.outlook.com [40.107.8.54])
 by dpdk.org (Postfix) with ESMTP id 20ABCC93C
 for <dev@dpdk.org>; Wed, 25 Nov 2020 05:51:11 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; 
 s=selector2-armh-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=;
 b=0UfNEG9JSb3VjjfP25XXIZRHeP0p1DBfb5YZPCerFctTXqo2n837rS63K+NhyN/2Ijub9mk8zEIy3/1S7wS7RWLKmfU/QMkjkx4uywhVVDbIDJXUWkLzXw/0Mj6fZqbmhmTVJQ4pNuCnApzSVoZCNf5nd9iq669/osUbqiYEZUQ=
Received: from AM5P194CA0019.EURP194.PROD.OUTLOOK.COM (2603:10a6:203:8f::29)
 by AM5PR0801MB1811.eurprd08.prod.outlook.com (2603:10a6:203:39::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20; Wed, 25 Nov
 2020 04:51:05 +0000
Received: from VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com
 (2603:10a6:203:8f:cafe::b9) by AM5P194CA0019.outlook.office365.com
 (2603:10a6:203:8f::29) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20 via Frontend
 Transport; Wed, 25 Nov 2020 04:51:04 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123)
 smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified)
 header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none
 header.from=arm.com;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by
 VE1EUR03FT004.mail.protection.outlook.com (10.152.18.106) with
 Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3589.20 via Frontend Transport; Wed, 25 Nov 2020 04:51:04 +0000
Received: ("Tessian outbound e0cdfd2b0406:v71");
 Wed, 25 Nov 2020 04:51:04 +0000
X-CR-MTA-TID: 64aa7808
Received: from d43d171384c6.2
 by 64aa7808-outbound-1.mta.getcheckrecipient.com id
 5A22A322-8235-482D-9F3D-4A85A84130CC.1; 
 Wed, 25 Nov 2020 04:50:59 +0000
Received: from EUR01-HE1-obe.outbound.protection.outlook.com
 by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d43d171384c6.2
 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384);
 Wed, 25 Nov 2020 04:50:59 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=ITof73Pn/mZzBNnOzzhHQh/EFY1VBYAFEWfFS8DQgLKHe5wVThPg0objlxwNsynA7s7GmflpdjR0h5QtqMQOn5Tz3xF2a4KA4NxVqe1vBkcvx3Dyz8sk/F36m+xz0fIMJa4NPZ+83r4r/JMWmZKS4+MAdifb3bJMWd/00qm+x7Ms6mvnd0Mk3XHOx9n4Nv5tOM823L3DLLARg8BQQXU+qMdb1Vhw5P2QIQQtMRuBFMc5QM+gXzD3VoCrcDza8tBfvkUrsflnTn6ccNYU3J74xujpmIVMkk22mWLNUVwk+xqScsRLcDzZrpuOHp8NjdWaxv3XA6HGr7XOy7nD9H9IxQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=;
 b=bNuNwr0L+AuQ/ed9Qp7EG2iwWgNZgeCUdHgyhDLivpsRBUOWHC8t10HL4V4WQulAi75PLLe/JfiULuXXZYscw2fxhB/rMb4t3iMCZ7lycOs4pWdvcjXX99YkXm9RswiAHWLTi3dLfEYcyjNpnKCysI2cp8OlLF38zJiZXfgN4/vNhMJONFpDsN0qS+i+rI0UlrVW7JgvYOstpLddbXV+gYy1kXapc7/31nHjitdTN1PzmCZV94DYBIOZ8hv63HUZvDHncwXrhbw/tUmZe0/CdkVg5efJAXJVxo4S1TDLr+w8hUm5/rMl/Oq8kGvo0ogtQH7nkmCLQtAiRf/berfX9g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass
 header.d=arm.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; 
 s=selector2-armh-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=J4Gc4s1ytjolMFasY0McPXBGfhY5Py3Jt8XK2N8Uvak=;
 b=0UfNEG9JSb3VjjfP25XXIZRHeP0p1DBfb5YZPCerFctTXqo2n837rS63K+NhyN/2Ijub9mk8zEIy3/1S7wS7RWLKmfU/QMkjkx4uywhVVDbIDJXUWkLzXw/0Mj6fZqbmhmTVJQ4pNuCnApzSVoZCNf5nd9iq669/osUbqiYEZUQ=
Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6)
 by DB8PR08MB5484.eurprd08.prod.outlook.com (2603:10a6:10:111::15) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Wed, 25 Nov
 2020 04:50:54 +0000
Received: from DBAPR08MB5814.eurprd08.prod.outlook.com
 ([fe80::9b8:e230:9bbf:c61f]) by DBAPR08MB5814.eurprd08.prod.outlook.com
 ([fe80::9b8:e230:9bbf:c61f%7]) with mapi id 15.20.3611.021; Wed, 25 Nov 2020
 04:50:54 +0000
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: Stephen Hemminger <stephen@networkplumber.org>
CC: Diogo Behrens <diogo.behrens@huawei.com>, "thomas@monjalon.net"
 <thomas@monjalon.net>, "david.marchand@redhat.com"
 <david.marchand@redhat.com>, "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>,
 Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, nd <nd@arm.com>
Thread-Topic: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory
Thread-Index: AQHWwcaSjZGD7vozU0KcDD8t04pzMKnWHG2AgAIq3yA=
Date: Wed, 25 Nov 2020 04:50:54 +0000
Message-ID: <DBAPR08MB581461EC03BD251C6E403EB898FA0@DBAPR08MB5814.eurprd08.prod.outlook.com>
References: <20200826092002.19395-1-diogo.behrens@huawei.com>
 <DBAPR08MB581416E1D0835A1BBCAB042998FC0@DBAPR08MB5814.eurprd08.prod.outlook.com>
 <20201123113651.46323c54@hermes.local>
In-Reply-To: <20201123113651.46323c54@hermes.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-ts-tracking-id: 2B78BF7590720D4DB17A735C43A96C28.0
x-checkrecipientchecked: true
Authentication-Results-Original: networkplumber.org; dkim=none (message not
 signed) header.d=none;networkplumber.org; dmarc=none action=none
 header.from=arm.com;
x-originating-ip: [217.140.110.7]
x-ms-publictraffictype: Email
X-MS-Office365-Filtering-HT: Tenant
X-MS-Office365-Filtering-Correlation-Id: 356ac048-1ca5-4e15-d41a-08d890fdb72c
x-ms-traffictypediagnostic: DB8PR08MB5484:|AM5PR0801MB1811:
x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr
x-ms-exchange-transport-forked: True
X-Microsoft-Antispam-PRVS: <AM5PR0801MB1811DA07A5D912BC5138167298FA0@AM5PR0801MB1811.eurprd08.prod.outlook.com>
x-checkrecipientrouted: true
nodisclaimer: true
x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam-Untrusted: BCL:0;
X-Microsoft-Antispam-Message-Info-Original: 2mOF+wn2Z9EOqzhTAsWc5KdP/HADPteWH/ueMXKUvOy47D+J2+H1d+0xsJuKN07N1yIjc3cSKKPdoh0mMdMl7apjQrIQyGVFCswFOm+dId/L57YakOJEvyIvndfVg7grzffhSDYKLlNfpf73SNDdti5XiSaRB8bd4R89r+jfMy//Vc36ea77mFVHtHIbJ/C9MB9e6shnLdjvVvkY9sYPfzMogRlAYXreBtJHl/AVc0kwhCYK8rL5eVj278HqRtSF0Qhh3rW1/lHed6R+IlAF3EGqMAOzIlCCN1PkdTJ+YYmT+ovA4sZntub6+CQBjKme
X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en;
 SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com;
 PTR:; CAT:NONE;
 SFS:(4636009)(366004)(396003)(376002)(136003)(39860400002)(346002)(66446008)(66476007)(7696005)(5660300002)(83380400001)(64756008)(54906003)(76116006)(66946007)(316002)(86362001)(4326008)(33656002)(52536014)(55016002)(186003)(26005)(6916009)(6506007)(71200400001)(66556008)(478600001)(9686003)(2906002)(8676002)(8936002);
 DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata: =?us-ascii?Q?WgQ13fqzUa770Ek8K5ZQUwZQbkcQ6+AWoAtImjHkRd4NZ8pqXCA4f6EhiL03?=
 =?us-ascii?Q?yUIdbDGuYwiKWOV43zS1NNA5vmD+OkwweX0sQDxIZqvxlRWJmojnRXMHtWkg?=
 =?us-ascii?Q?jblFSjDt+1AT1ZCK6+Tab1kJCo23AWg1OxvHFkfiKN+LOs8R9ghLXSoWKYTw?=
 =?us-ascii?Q?rZRNedxY2Grx9EbPfR86OXZm/Pae3qRWQPVhB7JbtTjJ6i2Wa8DXhElzpLOS?=
 =?us-ascii?Q?PgkxfE8mU2LDnHjy4H5WymuWGM8iF0oVKStguYBSYcUPpAzIceod5pAb6x6P?=
 =?us-ascii?Q?+VaxgCFKHen1c37hQ2xzVjMigYTEbVJhje1kT2qYXXkGCeVnWNZxvErAa3we?=
 =?us-ascii?Q?+IIIz/DqdmzCA5hXnUi2v6fJHT4+0x7n/t1ASL7sdAAbnHt5hj5BOYMvIrMk?=
 =?us-ascii?Q?hrGMLsqCTVVPn4p9FTa/0rCriqYf5EnqRttlTjkBTwBq/0YFDHqvfQsQ7E8+?=
 =?us-ascii?Q?5kCF98DSqGux3kQWKOTbesdM36IGqObu6V10XpD0Lem/OBNZhY0hUlR1eL/z?=
 =?us-ascii?Q?A1/l3EQCi1od3HroHVPn/fE7FsEHmAvwAsN3gePsKzyXcuWydTGM7F8TlB1w?=
 =?us-ascii?Q?l+Ks+KZY/jzl2zoSbecKRFE3Eq/3K3bdErnWFW8F+wNfThOOjoDgVIfX8Hw1?=
 =?us-ascii?Q?37KbTuFyrj8acemLz1THl8kikuJa69Ees+SuXxmxF9/cc/RQ5fpv9icPnXvO?=
 =?us-ascii?Q?rIVJRCkysAWgE5qriZpJUE04IA/DJTWqCysOXiofDjI1BkVw/2NpOpQhscDN?=
 =?us-ascii?Q?sb9++qtnw5Q/DhPL40I/rxcywXEUnxKMpFMiIqEXG8o3dLTU6pgmL4SoSPyV?=
 =?us-ascii?Q?SVqU6/lYWptwwhhPNytsAo150Qs8tAbU3RGqJVe+VSDZQkK4jmmK4G+7tQ0Q?=
 =?us-ascii?Q?tiCVuFNukTOYdEArUIFgasjZ9azugI9k5LVtv/IiHvtgQWMgedRYkqpDrUcw?=
 =?us-ascii?Q?v/OFMX7sDd64IYglofqNtaQq/C8pPZKRzAAgFOjBldg=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR08MB5484
Original-Authentication-Results: networkplumber.org;
 dkim=none (message not signed)
 header.d=none; networkplumber.org; dmarc=none action=none header.from=arm.com; 
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com
X-MS-Office365-Filtering-Correlation-Id-Prvs: a05c2f1d-9dcd-4e0a-0787-08d890fdb125
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: vnVaSsUGg4y+5q5FAhMtkdYr9LGSnIZHw6rnc89WZLLpyAPGIH0bqmTVcWu44pk30XuW3UU+uTJ9hTCq1VK5s25sq4BQJz3Q6IYSVV1AnSFApuhykm7PLZWMXu/5l/DZviwJhnEyXY05rtjByR+UPcmmsdLpE4rsZzVJRiSEAn+Mvcz8nzXskxBcsIyaDqBecDkc7ZF1aIh8ZzczfRiezsYxF5ex7PnqShN3bb2sWkV7d2cwEtyjDfk2Zon0D/g7ey4wUhBd+uwRbwDfzTjg0vJvyAKiCUg//lw+75EW78zImuUGO08a3+1Yi6MfPYxTJoLXqQ29Yju/gqh1Fqx5Ug1bOFDhxRFFb0LOpNipdrQuAYxRKc8/FvnTxcp/dVsNgU+2XIqdyX2NzjTpVCOf8g==
X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:;
 IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com;
 PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE;
 SFS:(4636009)(396003)(346002)(136003)(376002)(39860400002)(46966005)(54906003)(316002)(8676002)(356005)(82740400003)(5660300002)(55016002)(33656002)(52536014)(9686003)(70206006)(81166007)(47076004)(83380400001)(8936002)(6862004)(2906002)(4326008)(70586007)(336012)(186003)(26005)(7696005)(6506007)(86362001)(478600001)(82310400003);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2020 04:51:04.5929 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 356ac048-1ca5-4e15-d41a-08d890fdb72c
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123];
 Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT004.eop-EUR03.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0801MB1811
Subject: Re: [dpdk-dev] [PATCH] librte_eal: fix mcslock hang on weak memory
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

<snip>

> >
> > >
> > >     The initialization me->locked=3D1 in lock() must happen before
> > >     next->locked=3D0 in unlock(), otherwise a thread may hang forever=
,
> > >     waiting me->locked become 0. On weak memory systems (such as
> ARMv8),
> > >     the current implementation allows me->locked=3D1 to be reordered =
with
> > >     announcing the node (pred->next=3Dme) and, consequently, to be
> > >     reordered with next->locked=3D0 in unlock().
> > >
> > >     This fix adds a release barrier to pred->next=3Dme, forcing
> > >     me->locked=3D1 to happen before this operation.
> > >
> > > Signed-off-by: Diogo Behrens <diogo.behrens@huawei.com>
> > The change looks fine to me.  I have tested this on few x86 and Arm mac=
hines.
> > Acked-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>=20
> Maybe a simpler alternative would be as fast and safer.
Why is this safer?

> By using compare_exchange you can get same effect in one operation.
> Like the following UNTESTED.
>=20
> diff --git a/lib/librte_eal/include/generic/rte_mcslock.h
> b/lib/librte_eal/include/generic/rte_mcslock.h
> index 78b0df295e2d..9c537ce577e6 100644
> --- a/lib/librte_eal/include/generic/rte_mcslock.h
> +++ b/lib/librte_eal/include/generic/rte_mcslock.h
> @@ -48,23 +48,23 @@ rte_mcslock_lock(rte_mcslock_t **msl, rte_mcslock_t
> *me)
>  	rte_mcslock_t *prev;
>=20
>  	/* Init me node */
> -	__atomic_store_n(&me->locked, 1, __ATOMIC_RELAXED);
> -	__atomic_store_n(&me->next, NULL, __ATOMIC_RELAXED);
> +	me->locked =3D 1;
>=20
> -	/* If the queue is empty, the exchange operation is enough to acquire
> -	 * the lock. Hence, the exchange operation requires acquire semantics.
> -	 * The store to me->next above should complete before the node is
> -	 * visible to other CPUs/threads. Hence, the exchange operation
> requires
> -	 * release semantics as well.
> +	/*
> +	 * Atomic insert into single linked list
>  	 */
> -	prev =3D __atomic_exchange_n(msl, me, __ATOMIC_ACQ_REL);
> +	do {
> +		prev =3D __atomic_load_n(msl, __ATOMIC_RELAXED);
> +		me->next =3D prev;
This needs to be __atomic_store_n(__ATOMIC_RELEASE) as it can sink below th=
e following line.

> +	} while (!__atomic_compare_exchange_n(&msl, me, prev,
> +					    __ATOMIC_ACQUIRE,
> __ATOMIC_RELAXED));
> +
>  	if (likely(prev =3D=3D NULL)) {
>  		/* Queue was empty, no further action required,
>  		 * proceed with lock taken.
>  		 */
>  		return;
>  	}
> -	__atomic_store_n(&prev->next, me, __ATOMIC_RELAXED);
>=20
>  	/* The while-load of me->locked should not move above the previous
>  	 * store to prev->next. Otherwise it will cause a deadlock. Need a