From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 56D4FA0577; Mon, 6 Apr 2020 19:06:22 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6FF211BED8; Mon, 6 Apr 2020 19:06:21 +0200 (CEST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2073.outbound.protection.outlook.com [40.107.22.73]) by dpdk.org (Postfix) with ESMTP id ADDDF2BE9 for ; Mon, 6 Apr 2020 19:06:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3wcUWLphiJKPd3NMCfxXo9Ci0/uauuduGkOPW8icE8Y=; b=3BWe3BjVGX0Jmc90iMbGm50GFAGbJ31Wy4LJPkFb29W+2z2XAhDuJOKXqHUxqFvKFY0tWbO/9OcY41y/P0trQ7aGKdbodZpS8fcm6ShuZe/IPvOYV9sU1t2aMshAKTn0/HvfjP0yfu3dIEfS2RHt4kpehtw06c7pN33AyMJ7D5w= Received: from DB7PR05CA0055.eurprd05.prod.outlook.com (2603:10a6:10:2e::32) by DB6PR08MB2646.eurprd08.prod.outlook.com (2603:10a6:6:20::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2835.24; Mon, 6 Apr 2020 17:06:16 +0000 Received: from DB5EUR03FT006.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:2e:cafe::c3) by DB7PR05CA0055.outlook.office365.com (2603:10a6:10:2e::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.21 via Frontend Transport; Mon, 6 Apr 2020 17:06:16 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT006.mail.protection.outlook.com (10.152.20.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.17 via Frontend Transport; Mon, 6 Apr 2020 17:06:16 +0000 Received: ("Tessian outbound e2c88df8bbbe:v50"); Mon, 06 Apr 2020 17:06:16 +0000 X-CR-MTA-TID: 64aa7808 Received: from b90e93a26cf5.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id D9032180-0995-4592-9EC1-F2325951156E.1; Mon, 06 Apr 2020 17:06:11 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id b90e93a26cf5.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Apr 2020 17:06:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BhjgkK+ggONPTotl+Kf1RrI2U92NWxiFSek6BumyrsfPbF4gNy6vJbu82Q1vCozZYyX3n8AWTFgGx3DiHrdTQWn1wqXyZfAmCi767H1SMx+E6VlgNRSp0LR3e3cYH4SbZEy0s2RjswoKKzOSHTWqiUvZVPKKYm4I2Sl5X+aP+FpsHdcWREwy6Zu2tZCqd0sVw4YwYfWff1LOaavEv8vc16g7ZCCugag78ThjAyvduJdSC21VBpEVcM/nluonV0b7HQLabMPntBsNhG34F32DO0QSz3fm9egx5Gg91lHF/r6Q0cfTKqU+Qswj1xPdNgMZkPlovt5oSOrTWauYU2B93Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3wcUWLphiJKPd3NMCfxXo9Ci0/uauuduGkOPW8icE8Y=; b=bOHB2ov0bC/z4JtFKVOzCM/8ctE7UvdhGhjYjerBiNHkoiHMaXa/alh53hnRz26lXM28s1QXMi5mKra1bERxs5FpMAK1uGZqmtCmLh8Ig+xRJWc3L7wH2GbHMkAIoGHZlVAsBbF6QPmlAClIrU1V7GS85WONRo9dKE4pNXgpQcyD6kxWI8pilPzOT4TNC+D8ULl/oJc+GjYYJcEVMtmMzBiwfk43JsZuRsoOyNyVDtrtPq916Z5X52ftIsHKPJE8LPYFWJu/7AK8ltXiZkgmU46NAI1mUkI572jecM01HdAintG+9ywWgF40rkyYuD+WRk8Blakcnga/mTt2mPJPLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3wcUWLphiJKPd3NMCfxXo9Ci0/uauuduGkOPW8icE8Y=; b=3BWe3BjVGX0Jmc90iMbGm50GFAGbJ31Wy4LJPkFb29W+2z2XAhDuJOKXqHUxqFvKFY0tWbO/9OcY41y/P0trQ7aGKdbodZpS8fcm6ShuZe/IPvOYV9sU1t2aMshAKTn0/HvfjP0yfu3dIEfS2RHt4kpehtw06c7pN33AyMJ7D5w= Received: from DBBPR08MB4646.eurprd08.prod.outlook.com (10.255.79.144) by DBBPR08MB4505.eurprd08.prod.outlook.com (20.179.43.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.16; Mon, 6 Apr 2020 17:06:10 +0000 Received: from DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d]) by DBBPR08MB4646.eurprd08.prod.outlook.com ([fe80::1870:afc4:b90f:609d%5]) with mapi id 15.20.2878.021; Mon, 6 Apr 2020 17:06:10 +0000 From: Honnappa Nagarahalli To: "Van Haaren, Harry" , Phil Yang , "thomas@monjalon.net" , "Ananyev, Konstantin" , "stephen@networkplumber.org" , "maxime.coquelin@redhat.com" , "dev@dpdk.org" CC: "david.marchand@redhat.com" , "jerinj@marvell.com" , "hemant.agrawal@nxp.com" , Gavin Hu , Ruifeng Wang , Joyce Kong , nd , Honnappa Nagarahalli , nd Thread-Topic: [PATCH v3 12/12] service: relax barriers with C11 atomic operations Thread-Index: AQHWCa84WS0a/sReJUuOBLNWx5/BzKhrgrBA Date: Mon, 6 Apr 2020 17:06:09 +0000 Message-ID: References: <1583999071-22872-1-git-send-email-phil.yang@arm.com> <1584407863-774-1-git-send-email-phil.yang@arm.com> <1584407863-774-13-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 2d195ff0-845f-4fd5-995a-bb977e796e83.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [70.113.25.165] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: f94a5e57-651b-469b-a060-08d7da4cd1b2 x-ms-traffictypediagnostic: DBBPR08MB4505:|DBBPR08MB4505:|DB6PR08MB2646: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:813;OLM:813; x-forefront-prvs: 0365C0E14B X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBBPR08MB4646.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(346002)(376002)(39860400002)(396003)(366004)(136003)(55016002)(54906003)(478600001)(86362001)(316002)(110136005)(71200400001)(4326008)(33656002)(186003)(66446008)(64756008)(7696005)(5660300002)(9686003)(66556008)(26005)(8936002)(66476007)(76116006)(53546011)(6506007)(66946007)(2906002)(52536014)(8676002)(81156014)(81166006); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: JYtgOF4sDbB2tXtpg1kmiREhpCww8v/H8q9QYxM4711W+cmrauZvETOOrBojIYqZmiz7NzAo9+vvrsM3BnIKvCgJgv9D/Zqq+Ew+ZwgMc91+UrDqMH3yl6H/flN8AiBc0Qkn32JCILSvCaeaUs7QbtWoeC2LBMPsjrSthijz0Oclx6xbgG7eN79VUTxfWP4hdNTKzM1QwETjkUfsOPHW49YhKs1KD+bJd+xMpnv7lr0gNm/6VmHfnnm/P3TfiN5nKPIvUgFexBCqKsNQQFUU57p8OEh3rM2kCfJBd3bcYPJTe3/4FGunH/0QJCpYbNhtJIfj/40hX606lT7gm3ygGfVsGzAS8CWgF2Q98qFF1UATEJPnvkGk0tXaCr6CNNVj9ogi1/X+lt/WcnYbOjSCuU8DVFyH0b8s+osXsr1c3E9G92+fFVJA+yfw/whw6Wvh x-ms-exchange-antispam-messagedata: mx2omyWQC7K/tR6KSaomfsFeQJ0TMi7CVB08WEN4igXKuQTn4qyKOwM6gY9qebm1WU5KA0I1sKtyepLXxGFmse487XFuVvj1gD/aUoOfDb+snYSyv1OontVpO/xLVg0BUZMsXGz84aSDcbLgOMEucg== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4505 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT006.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(136003)(396003)(376002)(39860400002)(346002)(46966005)(4326008)(356004)(55016002)(9686003)(47076004)(26005)(186003)(478600001)(70206006)(52536014)(336012)(30864003)(26826003)(70586007)(86362001)(6506007)(82740400003)(53546011)(5660300002)(8676002)(33656002)(54906003)(81156014)(8936002)(316002)(2906002)(81166006)(7696005)(110136005); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 43c115af-8d86-4034-56df-08d7da4ccdc7 X-Forefront-PRVS: 0365C0E14B X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E0ueIS33hS/YMAt9bKrtB3NfGt1rQmXlMDg7iIngTwWMxbTw9ITF5ldOLTl+id4TuLwvOwENypz42tuEBBq0+RiaCb+Ct63DITsurgyGkPaoNedXcwjw2jssr7J8yR6r8aDm8UR/AUr3CTRAEyqB4v8DA7ahcmzW0D8NUv7op7Y6BIPoyiyyR7QXlkQ082N3pQ0Q8GyjpE6Zl72WQNGe0tLA7+Ho9TcRkbspgiiUH9koIJYMEQgef4DUh6UkKDIAQTs9fLWmYiuc4DymIM+1zhobFI7sbAhDlBBLA2oOnNlv3nD00RE4rjg+3w3I6mXMND26fQy5nnbE7UbllLrFH3FKNG0zvtBhzkuPUxTzkI52G39VOTAtn9RQQ0pe2GoXP6hdeDdxwi+yhQGRbUrQbYKFus77NpwlPcyGsa7w5Fmow+BSibBB2CaOg9npPlQtfvleiSSqAhVm01QztzIz/NYZdUbPUjp/HyBeswj7jDARZT/iflTjQm9qU+Dn6R7Tx5q96fgQ4zkFY1rtbPK8eg== X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Apr 2020 17:06:16.6658 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f94a5e57-651b-469b-a060-08d7da4cd1b2 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR08MB2646 Subject: Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Just to get us on the same page on 'techniques to communicate data from wri= ter to reader' (apologies if it is too trivial) Let us say that the writer has 512B (key point is - cannot be written atomi= cally) that needs to be communicated to the reader. Since the data cannot be written atomically, we need a guard variable (whic= h can be written atomically, can be a flag or pointer to data). So, the wri= ter will store 512B in non-atomic way and write to guard variable with rele= ase memory order. So, if the guard variable is valid (set in the case of fl= ag or not null in the case of pointer), it guarantees that 512B is written. The reader will read the guard variable with acquire memory order and read = the 512B data only if the guard variable is valid. So, the acquire memory o= rder on the guard variable guarantees that the load of 512B does not happen= before the guard variable is read. The validity check on the guard variabl= e guarantees that 512B was written before it was read. The store(guard_variable, RELEASE) on the writer and the load(guard_variabl= e, ACQUIRE) can be said as synchronizing with each other. (the guard variable technique applies even if we are not using C11 atomics) Let us say that the writer has 4B (key point is - can be written atomically= ) that needs to be communicated to the reader. The writer is free to write = this atomically with no constraints on memory ordering as long as this data= is not acting as a guard variable for any other data. In my understanding, the sequence of APIs to call to start a service (write= r) are as follows: 1) rte_service_init 2) rte_service_component_register 3) 4) rte_service_component_runstate_set (the reader is allowed at this point = to read the information about the service - written by rte_service_componen= t_register API. This API should not be called before rte_service_component_= register) 5) 6) rte_service_runstate_set (the reader is allowed at this point to read th= e information about the service - written by rte_service_component_register= API and run the service. This API can be called anytime. But, the reader s= hould not attempt to run the service before this API is called) 7) rte_lcore_service_add (multiple of these probably, can be called before = this, can't be called later) 8) rte_service_map_lcore_set (this can be called anytime. Can be called eve= n if the service is not registered) 9) rte_service_lcore_start (again, this can be called anytime, even before = the service is registered) So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Only = these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in rea= der. We can write test cases with different orders of these API calls to prove t= hat the memory orders we use are sufficient. Few comments are inline based on this assessment. > Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic > operations >=20 > > From: Phil Yang > > Sent: Tuesday, March 17, 2020 1:18 AM > > To: thomas@monjalon.net; Van Haaren, Harry > > ; Ananyev, Konstantin > > ; stephen@networkplumber.org; > > maxime.coquelin@redhat.com; dev@dpdk.org > > Cc: david.marchand@redhat.com; jerinj@marvell.com; > > hemant.agrawal@nxp.com; Honnappa.Nagarahalli@arm.com; > > gavin.hu@arm.com; ruifeng.wang@arm.com; joyce.kong@arm.com; > nd@arm.com > > Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic > > operations > > > > To guarantee the inter-threads visibility of the shareable domain, it > > uses a lot of rte_smp_r/wmb in the service library. This patch relaxed > > these barriers for service by using c11 atomic one-way barrier operatio= ns. > > > > Signed-off-by: Phil Yang > > Reviewed-by: Ruifeng Wang > > Reviewed-by: Gavin Hu > > --- > > lib/librte_eal/common/rte_service.c | 45 > > ++++++++++++++++++++---------------- > > - > > 1 file changed, 25 insertions(+), 20 deletions(-) > > > > diff --git a/lib/librte_eal/common/rte_service.c > > b/lib/librte_eal/common/rte_service.c > > index c033224..d31663e 100644 > > --- a/lib/librte_eal/common/rte_service.c > > +++ b/lib/librte_eal/common/rte_service.c > > @@ -179,9 +179,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t > > enabled) > > SERVICE_VALID_GET_OR_ERR_RET(id, s, 0); > > > > if (enabled) > > - s->internal_flags |=3D SERVICE_F_STATS_ENABLED; > > + __atomic_or_fetch(&s->internal_flags, > SERVICE_F_STATS_ENABLED, > > + __ATOMIC_RELEASE); > > else > > - s->internal_flags &=3D ~(SERVICE_F_STATS_ENABLED); > > + __atomic_and_fetch(&s->internal_flags, > > + ~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE); >=20 > Not sure why these have to become stores with RELEASE memory ordering? > (More occurances of same Q below, just answer here?) Agree, 'internal_flags' is not acting as a guard variable, this should be R= ELAXED (similarly for the others below). Though I suggest keeping it atomic= . >=20 > > return 0; > > } > > @@ -193,9 +195,11 @@ rte_service_set_runstate_mapped_check(uint32_t > > id, int32_t enabled) > > SERVICE_VALID_GET_OR_ERR_RET(id, s, 0); > > > > if (enabled) > > - s->internal_flags |=3D SERVICE_F_START_CHECK; > > + __atomic_or_fetch(&s->internal_flags, > SERVICE_F_START_CHECK, > > + __ATOMIC_RELEASE); > > else > > - s->internal_flags &=3D ~(SERVICE_F_START_CHECK); > > + __atomic_and_fetch(&s->internal_flags, > ~(SERVICE_F_START_CHECK), > > + __ATOMIC_RELEASE); >=20 > Same as above, why do these require RELEASE? Agree >=20 >=20 > Remainder of patch below seems to make sense - there's a wmb() involved > hence RELEASE m/o. >=20 > > return 0; > > } > > @@ -264,8 +268,8 @@ rte_service_component_register(const struct > > rte_service_spec *spec, > > s->spec =3D *spec; > > s->internal_flags |=3D SERVICE_F_REGISTERED | > SERVICE_F_START_CHECK; > > > > - rte_smp_wmb(); > > - rte_service_count++; > > + /* make sure the counter update after the state change. */ > > + __atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE); >=20 > This makes sense to me - the RELEASE ensures that previous stores to the > s->internal_flags are visible to other cores before rte_service_count > increments atomically. rte_service_count is not a guard variable, does not need RELEASE order. It = is also not used by the reader. It looks like it is just a statistic being = maintained. >=20 >=20 > > if (id_ptr) > > *id_ptr =3D free_slot; > > @@ -281,9 +285,10 @@ rte_service_component_unregister(uint32_t id) > > SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); > > > > rte_service_count--; > > - rte_smp_wmb(); > > > > - s->internal_flags &=3D ~(SERVICE_F_REGISTERED); > > + /* make sure the counter update before the state change. */ > > + __atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED), > > + __ATOMIC_RELEASE); RELAXED is enough. > > > > /* clear the run-bit in all cores */ > > for (i =3D 0; i < RTE_MAX_LCORE; i++) > > @@ -301,11 +306,12 @@ rte_service_component_runstate_set(uint32_t id, > > uint32_t > > runstate) > > SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); > > > > if (runstate) > > - s->comp_runstate =3D RUNSTATE_RUNNING; > > + __atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING, > > + __ATOMIC_RELEASE); > > else > > - s->comp_runstate =3D RUNSTATE_STOPPED; > > + __atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED, > > + __ATOMIC_RELEASE); Here we need a thread_fence to prevent the memory operations from a subsequ= ent call to 'rte_service_component_unregister' from getting hoisted above t= his. The user should be forced to call rte_service_component_unregister bef= ore calling rte_service_component_runstate_set. I suggest adding a check in= rte_service_component_unregister to ensure that the state is set to RUNSTA= TE_STOPPED. In fact, the user needs to make sure that the service is stoppe= d for sure before calling rte_service_component_unregister. > > > > - rte_smp_wmb(); > > return 0; > > } > > > > > > @@ -316,11 +322,12 @@ rte_service_runstate_set(uint32_t id, uint32_t > runstate) > > SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); > > > > if (runstate) > > - s->app_runstate =3D RUNSTATE_RUNNING; > > + __atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING, > > + __ATOMIC_RELEASE); > > else > > - s->app_runstate =3D RUNSTATE_STOPPED; > > + __atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED, > > + __ATOMIC_RELEASE); > > > > - rte_smp_wmb(); > > return 0; > > } > > > > @@ -442,7 +449,8 @@ service_runner_func(void *arg) > > const int lcore =3D rte_lcore_id(); > > struct core_state *cs =3D &lcore_states[lcore]; > > > > - while (lcore_states[lcore].runstate =3D=3D RUNSTATE_RUNNING) { > > + while (__atomic_load_n(&cs->runstate, > > + __ATOMIC_ACQUIRE) =3D=3D RUNSTATE_RUNNING) { This can be RELAXED, lcore's runstate is not acting as a guard variable. However, note that the writer thread wants to communicate the 'runstate' (4= B) to the reader thread. This ordering needs to be handled in 'rte_eal_remo= te_launch' and 'eal_thread_loop' functions. We have to note that in some ot= her use case, the writer wants to communicate more than 4B to reader. Curre= ntly, the 'write' and 'read' system calls may have enough barriers to make = things work fine. But, I suggest using the ' lcore_config[lcore].f' as the = guard variable to make it explicit and not depend on 'write' and 'read'. We= can take up the EAL things in a later patch as it does not cause any issue= s right now. > > const uint64_t service_mask =3D cs->service_mask; > > > > for (i =3D 0; i < RTE_SERVICE_NUM_MAX; i++) { @@ -453,8 > +461,6 @@ > > service_runner_func(void *arg) > > } > > > > cs->loops++; > > - > > - rte_smp_rmb(); > > } > > > > lcore_config[lcore].state =3D WAIT; > > @@ -663,9 +669,8 @@ rte_service_lcore_add(uint32_t lcore) > > > > /* ensure that after adding a core the mask and state are defaults */ > > lcore_states[lcore].service_mask =3D 0; > > - lcore_states[lcore].runstate =3D RUNSTATE_STOPPED; > > - > > - rte_smp_wmb(); Agree. > > + __atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED, > > + __ATOMIC_RELEASE); This can be relaxed. > > > > return rte_eal_wait_lcore(lcore); > > } > > -- > > 2.7.4