From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1472CA0577; Mon, 6 Apr 2020 06:22:27 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 143182B96; Mon, 6 Apr 2020 06:22:26 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-eopbgr130081.outbound.protection.outlook.com [40.107.13.81]) by dpdk.org (Postfix) with ESMTP id 220A8F12 for ; Mon, 6 Apr 2020 06:22:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JBLwUSJlQPbrtJv/RC6nhbkfYkAVtrDqOlHBkEo/EpM=; b=umbYfLzj9H9EQEhuksUeTh3sNGI6MsoiroiY+zSDI77oB5+x9drSsFJRNoGx/tsE7D1oq+7khbEzbAKuxbMX0C6u41y6J6LPrdx5LOeJwjyMSVKk3mn7cH/2gbtzYa1Kj2b2BJUghIzVV0r3ArqHeP4EMKPdL0riZll+muUivxM= Received: from VI1PR08CA0242.eurprd08.prod.outlook.com (2603:10a6:803:dc::15) by VI1PR08MB2688.eurprd08.prod.outlook.com (2603:10a6:802:19::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.21; Mon, 6 Apr 2020 04:22:18 +0000 Received: from VE1EUR03FT045.eop-EUR03.prod.protection.outlook.com (2603:10a6:803:dc:cafe::57) by VI1PR08CA0242.outlook.office365.com (2603:10a6:803:dc::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.15 via Frontend Transport; Mon, 6 Apr 2020 04:22:18 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT045.mail.protection.outlook.com (10.152.19.51) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.17 via Frontend Transport; Mon, 6 Apr 2020 04:22:18 +0000 Received: ("Tessian outbound 5345ff401cf8:v50"); Mon, 06 Apr 2020 04:22:18 +0000 X-CR-MTA-TID: 64aa7808 Received: from 7f0356aab0cf.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 13AE0DFA-8F34-4CBB-A307-577916E12AC6.1; Mon, 06 Apr 2020 04:22:13 +0000 Received: from EUR02-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 7f0356aab0cf.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Apr 2020 04:22:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MvTxqTnIPHL7DCyX7I8otkqlSXrqk9ywOqpcrBFzTAlV44WF2AUFPejIjfKgllfND01gdkzUqICQqug2GRKtZd0qs1Emf2IueoI3G44RpQYD9yQdTBHJoOQ++JBHfuHE9bEZTFxbORSgIWgciaR0RecACoMJiujhZa79oSFsGwz5zQ9cV7W7hr8P0jLJ/02tUxbgZtPGnxDq3gHp27XCeijuROyizqnFAA0qqQrvIQknKRE4hJwCtvFoPzfh02JS4ldFIX7632quZ04f7feXVw00DVZ5kl/kyVsRdLdCX6hCFc2rfB5cd0O3W6fUHISMHet47y7v6eHxJNiQZidDXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JBLwUSJlQPbrtJv/RC6nhbkfYkAVtrDqOlHBkEo/EpM=; b=ZZz1vIeV1Zv3f5xWgu5rgGe0XeSKzx47PaboySuCUfNGJZT4MYXQUChRa77HMDIJJz0y+CDBd++PvrujtUNjVpVgj7s5kaFzDwlXnB96PJBQGF9simf0KOeSU+BEl9Fb+MkHRxaAdaclq9+CSy8e1EmCj57neZCxMd2T8JoBnO0HuHUtxKFUH0FRmWqocZgsmwteSO9bXURg40Y5klSfSB1Xn/Ac4L0uPRZNyPoo/2nOoBAdL3HhdPihmX5A81fe/cutSnqLQTnbmatMDLrS+Nv7FFjvkyAVqI68dtZGCjIXBSbr6d+K3ZQqF6nIHQ2pwa252Fa1EW4kg1Dkt3hd5g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JBLwUSJlQPbrtJv/RC6nhbkfYkAVtrDqOlHBkEo/EpM=; b=umbYfLzj9H9EQEhuksUeTh3sNGI6MsoiroiY+zSDI77oB5+x9drSsFJRNoGx/tsE7D1oq+7khbEzbAKuxbMX0C6u41y6J6LPrdx5LOeJwjyMSVKk3mn7cH/2gbtzYa1Kj2b2BJUghIzVV0r3ArqHeP4EMKPdL0riZll+muUivxM= Received: from DBBPR08MB4646.eurprd08.prod.outlook.com (10.255.79.144) by DBBPR08MB4364.eurprd08.prod.outlook.com (20.179.42.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.16; Mon, 6 Apr 2020 04:22:09 +0000 Received: from DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d]) by DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d%5]) with mapi id 15.20.2878.021; Mon, 6 Apr 2020 04:22:09 +0000 From: Honnappa Nagarahalli To: "Van Haaren, Harry" , Phil Yang , "thomas@monjalon.net" , "Ananyev, Konstantin" , "stephen@networkplumber.org" , "maxime.coquelin@redhat.com" , "dev@dpdk.org" CC: "david.marchand@redhat.com" , "jerinj@marvell.com" , "hemant.agrawal@nxp.com" , Gavin Hu , Ruifeng Wang , Joyce Kong , nd , Honnappa Nagarahalli , nd Thread-Topic: [PATCH v3 11/12] service: optimize with c11 one-way barrier Thread-Index: AQHWCa8yVV8Gb9nxO06Cv/93v3MHoahqyOgA Date: Mon, 6 Apr 2020 04:22:09 +0000 Message-ID: References: <1583999071-22872-1-git-send-email-phil.yang@arm.com> <1584407863-774-1-git-send-email-phil.yang@arm.com> <1584407863-774-12-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 00041410-202d-4436-8585-5580b6bbf82e.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [70.113.25.165] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: a2cd54bc-00bb-475c-97c6-08d7d9e2181e x-ms-traffictypediagnostic: DBBPR08MB4364:|DBBPR08MB4364:|VI1PR08MB2688: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:5797;OLM:5797; x-forefront-prvs: 0365C0E14B X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBBPR08MB4646.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(396003)(346002)(39850400004)(366004)(136003)(376002)(26005)(8676002)(478600001)(9686003)(110136005)(55016002)(81156014)(81166006)(8936002)(52536014)(54906003)(186003)(5660300002)(33656002)(71200400001)(966005)(7696005)(64756008)(66556008)(66476007)(66946007)(76116006)(66446008)(4326008)(316002)(2906002)(86362001)(6506007)(87944003); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: Jx9cpKwQQKR1xSTsXI4FMYaala04Iv+Jj4+AWM9E7IUgxa7w1orlwNt2IM4+h7kGoKkFn0COkEGcGm9n/TVCjcCHOInohclEGVpHffUo/PGh/2XxqfSKdliTD85LzTfCRWcA+qlHklEDcN3BBMqDl2lbFB/eG/YiKl88V+gR5pM45zBgVaC2GNJhSBtWjhZ+IvwyMu1QujMnQI3ICJ0rOiaQtxZvtaOEhQrBMkP5v4s0SAn0U1QXc9y2QP9OZdeRFE7ocRuHsgDL31/ZGoQTvp2LYq9KfBowm0dy/8lcrQDswGiBALyGFmkWzKeN9qzuRdtopMR+vgn/cyDPooEceNqZYdXjpBuMuvMs806vAQcucwZQOrHeDTkyB5IvWHCqBEa8PyKeC6eDCTus3fkFklf3k3JRfcruOMRE/Vt/nH6qvGk+BRQYukb5pHMiAtBbdORqa351jiMYJXiKmbA/MXy6k/75LL6TOpVr7Bxk46VujpBTo3fw1jngB8VyVdd1+4Ybq4WYAFRTCQ0PDBkKxGPZ0Eiv8ZANIpRYF2SH7HPhN+ZoZzbUzsJGUo16tLrG x-ms-exchange-antispam-messagedata: A3mw+5TI13n1WPh8gwizfMB+JfgRxeLPnrrTBU/VQcwjQQf4c0VzPia+QCAYnkGHEEMFQ7HQuTXejMsltRk1yU9ntj6Ix31aLecO1GBLEWCly3WWcm+3YG8YoZQam/A3hMeJUYpUSIXIUNqME+wH1A== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4364 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT045.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(346002)(376002)(136003)(39850400004)(396003)(46966005)(86362001)(4326008)(2906002)(5660300002)(52536014)(8676002)(70206006)(6506007)(70586007)(8936002)(81156014)(81166006)(26005)(54906003)(47076004)(33656002)(186003)(36906005)(9686003)(316002)(7696005)(110136005)(356004)(55016002)(966005)(82740400003)(478600001)(336012)(26826003)(87944003); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: c4bc3866-7e46-429b-ee0f-08d7d9e212e8 X-Forefront-PRVS: 0365C0E14B X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D9uwy8Gqw8wD5rktZhuOGFscw2ipM3L2jv7U4nt52KHsvmbFMGPOfX8sL6WPcdFpHSJwgcZfq7snsw88ye1PsQKebFhQSEnSOoKu/ejmeTYBuTpsTyeJHLpiwql0o/ZjAZIuMCCTXSWPTZVYa05GRp6C6UXAFinOyL/Z+CmQ/P4IGudi/UJHIrgYCWWPtS8vIDvZv1ZEABGZd5GLZaqA62MzAre0l1SeiE1he/eUDGsA7M3ry+ivDfcqLh0E3pbyWoX3it2/sB6PEsicqiSboTLM304Pd/ISKAj1ORpHwerh+OaaZFpWW80sE+MOtGXrWGqvspR/8NnJKuxkN+vJRVhkBtyIy+1EScNLPKmqpDnRe23/S+pS0XJgkM519X5ohE4hfUDk41tZy3+CggPDRziIC/s0QnS4kthb22EVbryJGNPhJo/54vBpv9/ay0cUzlKhoBv7H7g0/LzRvs3ylrLUhYccFKJ63Zh5LJ/gjhepSP+kSfvWUWkcijF9NtUyNgeZQZYhF/lrW1oUdMU+cYqpbK8xFZdUY61TRWECTiYzFnlvtktZL5lNqCtW5RhPbbP+LcgXtXIF+CD43qosDZq1t3IAMS4fOf+5410QaD2/qgsVGDBn/2vlv241F4Xv X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Apr 2020 04:22:18.5259 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a2cd54bc-00bb-475c-97c6-08d7d9e2181e X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB2688 Subject: Re: [dpdk-dev] [PATCH v3 11/12] service: optimize with c11 one-way barrier X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > Subject: [PATCH v3 11/12] service: optimize with c11 one-way barrier > > > > The num_mapped_cores and execute_lock are synchronized with > > rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This > > patch optimized it with > > c11 atomic one-way barrier. > > > > Signed-off-by: Phil Yang > > Reviewed-by: Ruifeng Wang > > Reviewed-by: Gavin Hu > > Reviewed-by: Honnappa Nagarahalli >=20 > Based on discussion on-list, it seems the consensus is to not use GCC bui= ltins, > but instead use C11 APIs "proper"? If my conclusion is correct, the v+1 o= f this > patchset would require updates to that style API. >=20 > Inline comments for context below, -Harry >=20 >=20 > > --- > > lib/librte_eal/common/rte_service.c | 50 > > ++++++++++++++++++++++++++---------- > > - > > 1 file changed, 35 insertions(+), 15 deletions(-) > > > > diff --git a/lib/librte_eal/common/rte_service.c > > b/lib/librte_eal/common/rte_service.c > > index 0843c3c..c033224 100644 > > --- a/lib/librte_eal/common/rte_service.c > > +++ b/lib/librte_eal/common/rte_service.c > > @@ -42,7 +42,7 @@ struct rte_service_spec_impl { > > * running this service callback. When not set, a core may take the > > * lock and then run the service callback. > > */ > > - rte_atomic32_t execute_lock; > > + uint32_t execute_lock; > > > > /* API set/get-able variables */ > > int8_t app_runstate; > > @@ -54,7 +54,7 @@ struct rte_service_spec_impl { > > * It does not indicate the number of cores the service is running > > * on currently. > > */ > > - rte_atomic32_t num_mapped_cores; > > + int32_t num_mapped_cores; >=20 > Any reason why "int32_t" or "uint32_t" is used over another? > execute_lock is a uint32_t above, num_mapped_cores is an int32_t? >=20 >=20 > > uint64_t calls; > > uint64_t cycles_spent; > > } __rte_cache_aligned; > > @@ -332,7 +332,8 @@ rte_service_runstate_get(uint32_t id) > > rte_smp_rmb(); > > > > int check_disabled =3D !(s->internal_flags & SERVICE_F_START_CHECK); > > - int lcore_mapped =3D (rte_atomic32_read(&s->num_mapped_cores) > > 0); > > + int lcore_mapped =3D (__atomic_load_n(&s->num_mapped_cores, > > + __ATOMIC_RELAXED) > 0); > > > > return (s->app_runstate =3D=3D RUNSTATE_RUNNING) && > > (s->comp_runstate =3D=3D RUNSTATE_RUNNING) && @@ -375,11 > +376,20 @@ > > service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, > > cs->service_active_on_lcore[i] =3D 1; > > > > if ((service_mt_safe(s) =3D=3D 0) && (serialize_mt_unsafe =3D=3D 1)) = { > > - if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1)) > > + uint32_t expected =3D 0; > > + /* ACQUIRE ordering here is to prevent the callback > > + * function from hoisting up before the execute_lock > > + * setting. > > + */ > > + if (!__atomic_compare_exchange_n(&s->execute_lock, > &expected, 1, > > + 0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED)) > > return -EBUSY; >=20 > Let's try improve the magic "1" and "0" constants, I believe the "1" here= is the > desired "new value on success", and the 0 is "bool weak", where our 0/fal= se > constant implies a strongly ordered compare exchange? >=20 > "Weak is true for weak compare_exchange, which may fail spuriously, and > false for the strong variation, which never fails spuriously.", from > https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html >=20 > const uint32_t on_success_value =3D 1; > const bool weak =3D 0; > __atomic_compare_exchange_n(&s->execute_lock, &expected, > on_success_value, weak, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED); >=20 >=20 > Although a bit more verbose, I feel this documents usage a lot better, > particularly for those who aren't as familiar with the C11 function argum= ents > order. >=20 > Admittedly with the API change to not use __builtins, perhaps this commen= t is > moot. Suggest changing the execute_lock to rte_spinlock_t and use rte_spinlock_tr= ylock API. >=20 >=20 > > > > service_runner_do_callback(s, cs, i); > > - rte_atomic32_clear(&s->execute_lock); > > + /* RELEASE ordering here is used to pair with ACQUIRE > > + * above to achieve lock semantic. > > + */ > > + __atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE); > > } else > > service_runner_do_callback(s, cs, i); > > > > @@ -415,11 +425,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, > > uint32_t > > serialize_mt_unsafe) > > /* Increment num_mapped_cores to indicate that the service > > * is running on a core. > > */ > > - rte_atomic32_inc(&s->num_mapped_cores); > > + __atomic_add_fetch(&s->num_mapped_cores, 1, > __ATOMIC_ACQUIRE); > > > > int ret =3D service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe); > > > > - rte_atomic32_dec(&s->num_mapped_cores); > > + __atomic_sub_fetch(&s->num_mapped_cores, 1, > __ATOMIC_RELEASE); > > > > return ret; > > } > > @@ -552,24 +562,32 @@ service_update(uint32_t sid, uint32_t lcore, > > > > uint64_t sid_mask =3D UINT64_C(1) << sid; > > if (set) { > > - uint64_t lcore_mapped =3D lcore_states[lcore].service_mask & > > - sid_mask; > > + /* When multiple threads try to update the same lcore > > + * service concurrently, e.g. set lcore map followed > > + * by clear lcore map, the unsynchronized service_mask > > + * values have issues on the num_mapped_cores value > > + * consistency. So we use ACQUIRE ordering to pair with > > + * the RELEASE ordering to synchronize the service_mask. > > + */ > > + uint64_t lcore_mapped =3D __atomic_load_n( > > + &lcore_states[lcore].service_mask, > > + __ATOMIC_ACQUIRE) & sid_mask; >=20 > Thanks for the comment - it helps me understand things a bit better. > Some questions/theories to validate; > 1) The service_mask ACQUIRE avoids other loads being hoisted above it, > correct? >=20 > 2) There are non-atomic stores to service_mask. Is it correct that the st= ores > themselves aren't the issue, but relative visibility of service_mask stor= es vs > num_mapped_cores? (Detail in (3) below) >=20 >=20 > > if (*set && !lcore_mapped) { > > lcore_states[lcore].service_mask |=3D sid_mask; > > - > rte_atomic32_inc(&rte_services[sid].num_mapped_cores); > > + > __atomic_add_fetch(&rte_services[sid].num_mapped_cores, > > + 1, __ATOMIC_RELEASE); > > } > > if (!*set && lcore_mapped) { > > lcore_states[lcore].service_mask &=3D ~(sid_mask); > > - > rte_atomic32_dec(&rte_services[sid].num_mapped_cores); > > + > __atomic_sub_fetch(&rte_services[sid].num_mapped_cores, > > + 1, __ATOMIC_RELEASE); > > } >=20 > 3) Here we update the core-local service_mask, and then update the > num_mapped_cores with an ATOMIC_RELEASE. The RELEASE here ensures > that the previous store to service_mask is guaranteed to be visible on al= l cores > if this store is visible. Why do we care about this property? > The service_mask is core local anway. We are working on concurrency between the reader and writer. The service_ma= sk is local to the core, but it is accessed by a reader and writer. I think we should wait to conclude on the meaning of 'num_mapped_cores', th= at will dictate what the order should be. For ex: if it is just for statist= ics purpose, then we could use just RELAXED memory order and then the order= for service_mask will also change. >=20 > 4) Even with the load ACQ service_mask, and REL num_mapped_cores store, > is there not still a race-condition possible where 2 lcores simultaneousl= y load- > ACQ the service_mask, and then both do atomic add/sub_fetch with REL? >=20 > 5) Assuming 4 above race is true, it raises the real question - the servi= ce-cores > control APIs are not designed to be multi-thread-safe. Orchestration of > service/lcore mappings is not meant to be done by multiple threads at the > same time. Documenting this loudly may help, I'm happy to send a patch to= do > so if we're agreed on the above? I completely agree here. writer-writer concurrency is another topic and we = should (for now at least) say that the control plane APIs are not thread sa= fe. >=20 >=20 >=20 >=20 > > } > > > > if (enabled) > > *enabled =3D !!(lcore_states[lcore].service_mask & (sid_mask)); > > > > - rte_smp_wmb(); > > - > > return 0; > > } > > > > @@ -625,7 +643,8 @@ rte_service_lcore_reset_all(void) > > } > > } > > for (i =3D 0; i < RTE_SERVICE_NUM_MAX; i++) > > - rte_atomic32_set(&rte_services[i].num_mapped_cores, 0); > > + __atomic_store_n(&rte_services[i].num_mapped_cores, 0, > > + __ATOMIC_RELAXED); > > > > rte_smp_wmb(); > > > > @@ -708,7 +727,8 @@ rte_service_lcore_stop(uint32_t lcore) > > int32_t enabled =3D service_mask & (UINT64_C(1) << i); > > int32_t service_running =3D rte_service_runstate_get(i); > > int32_t only_core =3D (1 =3D=3D > > - > rte_atomic32_read(&rte_services[i].num_mapped_cores)); > > + > __atomic_load_n(&rte_services[i].num_mapped_cores, > > + __ATOMIC_RELAXED)); > > > > /* if the core is mapped, and the service is running, and this > > * is the only core that is mapped, the service would cease to > > -- > > 2.7.4