From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 40495A0597;
	Wed,  8 Apr 2020 21:42:47 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 6C0341C202;
	Wed,  8 Apr 2020 21:42:46 +0200 (CEST)
Received: from mga09.intel.com (mga09.intel.com [134.134.136.24])
 by dpdk.org (Postfix) with ESMTP id E900D1C1F5
 for <dev@dpdk.org>; Wed,  8 Apr 2020 21:42:44 +0200 (CEST)
IronPort-SDR: X7jqKanpJdHUD+KVBCujZLCsg88iOMMp2gsHk+CSkfAANQLZGR+1ix74SQ5nzx0nRYrzdnpx2H
 aTYVuG3Nxy2A==
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from fmsmga003.fm.intel.com ([10.253.24.29])
 by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
 08 Apr 2020 12:42:43 -0700
IronPort-SDR: R1zhfKgl48PPinB7UcMJCI+2Sb6NIKXM7UpeaPCQg8VpjioMnXnSQe3Jp6n64UyUnhCWZb0evv
 TbPqCr08CnWQ==
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.72,359,1580803200"; d="scan'208";a="297319891"
Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201])
 by FMSMGA003.fm.intel.com with ESMTP; 08 Apr 2020 12:42:43 -0700
Received: from fmsmsx115.amr.corp.intel.com (10.18.116.19) by
 FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS)
 id 14.3.439.0; Wed, 8 Apr 2020 12:42:43 -0700
Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by
 fmsmsx115.amr.corp.intel.com (10.18.116.19) with Microsoft SMTP Server (TLS)
 id 14.3.439.0; Wed, 8 Apr 2020 12:42:42 -0700
Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.176)
 by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id
 14.3.439.0; Wed, 8 Apr 2020 12:42:42 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=FI6biZt55hu2cYe5gy463QtEp0rcG3b+2e44TTDZ/CsaEECAW6dm5NRNnjE5aTVFErt2eaZ38qyc2elpd3MVArmd6MTgTx65qNWNGd329v3z8GxF2C5IIFRvcUSy7ja6iImcU0MZtEXUsfxZMXPNifkhwVVUg6BYMf3UodVkNfAcvv45ysq5MAL+lLNZJi2tJf2DScjxcOsPiKRrlFPEJtlUwTyzJipBmUdEiNXa1XOph1c53PuTSzvk8m1LxIKo7JXSttOIDAfxsgWoTZjZHBns6P7gpmlM+Az4lW7r3x2RX4mVWJ5uihGSbQLYDYkdOVOyiI4Z/khAR4ouOK4a+w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=xzffp8MQoxfeMzYHlWFKOr6f39U3kINTkztFKfBMnC8=;
 b=kAKAvUr0R1Ov/Y7RMd6CTxYlfKa7y8w3uZWTzrZstst9bnUWe0GwSB7/hZQeaLpf7hslh0GcuXr0daQSo46YmehS0+QC9SIBMuVEkmMUs4QeJvOfjd792+YkT41CBY/Us/PU1xAFUoHp0rQ7c2VmY8N8xVn4ziHygdYbJcxXXcsp6vjBwHSbWDqgzhsBaSL1rs3Q6fzOIf+M6GYWz7N6oeIPApZ7eyly0bOkvb088tJW6XkdWc0/ykceicAA+PX8i0rHx0+1mAnID2FxK5AP7b9zbzqUpzKC7EuMSIjOmTm9z6qnCHLVbCGcH2YHYV4cKEm6ABflAp8OH+IewklNZA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com;
 dkim=pass header.d=intel.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; 
 s=selector2-intel-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=xzffp8MQoxfeMzYHlWFKOr6f39U3kINTkztFKfBMnC8=;
 b=qoGaFhNfZywkAtA2Tbv0N8l/L1nqp14ODg0YH8I2xxRg8gqBz9ZHj1lmEIsmPGPdTaQUEiMVc5IRO0X11cLQAK03YoXTFb6E0SlJudGDWvkSeNDhldm96uFZctzfiB5vh9K3a+E9KkZIm1fsJFxjp9cQ0qcDRe/2hVRqbi95UVI=
Received: from BYAPR11MB3143.namprd11.prod.outlook.com (2603:10b6:a03:92::32)
 by BYAPR11MB2919.namprd11.prod.outlook.com (2603:10b6:a03:8d::21)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.21; Wed, 8 Apr
 2020 19:42:41 +0000
Received: from BYAPR11MB3143.namprd11.prod.outlook.com
 ([fe80::5917:9757:49b0:3c49]) by BYAPR11MB3143.namprd11.prod.outlook.com
 ([fe80::5917:9757:49b0:3c49%5]) with mapi id 15.20.2878.021; Wed, 8 Apr 2020
 19:42:41 +0000
From: "Van Haaren, Harry" <harry.van.haaren@intel.com>
To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Phil Yang
 <Phil.Yang@arm.com>, "thomas@monjalon.net" <thomas@monjalon.net>, "Ananyev,
 Konstantin" <konstantin.ananyev@intel.com>, "stephen@networkplumber.org"
 <stephen@networkplumber.org>, "maxime.coquelin@redhat.com"
 <maxime.coquelin@redhat.com>, "dev@dpdk.org" <dev@dpdk.org>
CC: "david.marchand@redhat.com" <david.marchand@redhat.com>,
 "jerinj@marvell.com" <jerinj@marvell.com>, "hemant.agrawal@nxp.com"
 <hemant.agrawal@nxp.com>, Gavin Hu <Gavin.Hu@arm.com>, Ruifeng Wang
 <Ruifeng.Wang@arm.com>, Joyce Kong <Joyce.Kong@arm.com>, nd <nd@arm.com>, nd
 <nd@arm.com>
Thread-Topic: [PATCH v3 12/12] service: relax barriers with C11 atomic
 operations
Thread-Index: AQHV+/omNUSP5Ua9ZE6XZhRVe3IM2qhnXyCwgAUUQYCAA03xQA==
Date: Wed, 8 Apr 2020 19:42:41 +0000
Message-ID: <BYAPR11MB31430FE1E9BBC2638521D5BFD7C00@BYAPR11MB3143.namprd11.prod.outlook.com>
References: <1583999071-22872-1-git-send-email-phil.yang@arm.com>
 <1584407863-774-1-git-send-email-phil.yang@arm.com>
 <1584407863-774-13-git-send-email-phil.yang@arm.com>
 <BYAPR11MB314398FD7D65F3DEBC0D8147D7C70@BYAPR11MB3143.namprd11.prod.outlook.com>
 <DBBPR08MB464685DEDE3E5A564EAEDD0498C20@DBBPR08MB4646.eurprd08.prod.outlook.com>
In-Reply-To: <DBBPR08MB464685DEDE3E5A564EAEDD0498C20@DBBPR08MB4646.eurprd08.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
dlp-product: dlpe-windows
dlp-reaction: no-action
dlp-version: 11.2.0.6
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=harry.van.haaren@intel.com; 
x-originating-ip: [192.198.151.165]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 37a86157-990f-4579-9ada-08d7dbf50042
x-ms-traffictypediagnostic: BYAPR11MB2919:
x-ms-exchange-transport-forked: True
x-microsoft-antispam-prvs: <BYAPR11MB2919E3DA0AF1DC99E26828EBD7C00@BYAPR11MB2919.namprd11.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:8882;
x-forefront-prvs: 0367A50BB1
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:BYAPR11MB3143.namprd11.prod.outlook.com; PTR:; CAT:NONE;
 SFTY:;
 SFS:(10019020)(346002)(39850400004)(376002)(136003)(366004)(396003)(4326008)(8936002)(66476007)(2906002)(186003)(316002)(110136005)(54906003)(64756008)(6506007)(66946007)(53546011)(76116006)(86362001)(81166007)(71200400001)(26005)(66556008)(66446008)(55016002)(9686003)(33656002)(81156014)(7696005)(8676002)(7416002)(66574012)(5660300002)(52536014)(478600001);
 DIR:OUT; SFP:1102; 
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: ZanCUOWWL5iHMGYnGZdmpjIs+bgeOM2qAdGLG5eyZiSUH13xx5I18PjyKxze6K3TcwMwtpFVt9G59aQ47ixnSP6lzIX8K59a0KPGG3vUonUULJR3gSaFCIII4B8IRkAVvVP/M6+TAm4TjynW7mvF5qbfOdUxRVCuj2mgliEfDs2clUiNScOSCVUbXGF44hbBkE3Qf67LZI/rs9tFTLzRb1Xj739OJny2aVFyl/MbA3M1FXxsWxLMD3Iyxw6x5IZgZTUuEnIcgtPhp7B/URK2IRHa7gYWvdYUbG6uVFzb+K4l5AOc00KAzygq22bnJJLdwXBkVnlZW9KABlr14O+1KvTZf1wLqAdk69k7Vz05viGmDYkUHFi4/0MwAHjVgmJ810dpiA2+jGTB8KFxTf94SCD5K1WrT1pBULYiJZxfLz5xlbmejskbiZfa5+C7xezj
x-ms-exchange-antispam-messagedata: AEymCdRtXl6EYoOSn2/i5NIBbV/hHjOIKOzbov9qnhA4hbMgz6ic79Bsoaw227l99VlaK/OvHs5WmQFb4SESruDmQHUIeFD6ZQBPjVGb7l64dFaIHeqLO9g+XGpN+SY8+LQhzw6wtd19eKA36riewg==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 37a86157-990f-4579-9ada-08d7dbf50042
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2020 19:42:41.2850 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: NTSNpLUnxkgZFt6++5Me1kP6MMCRXE3RebJc4Py9tBbTgdjOQ1vI++1ihBqdehLKA5dlHIA/+OmPuFGCCXNTGAVgZdGC7EiXPzW7R/Ff68w=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2919
X-OriginatorOrg: intel.com
Subject: Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11
 atomic operations
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Monday, April 6, 2020 6:06 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>; Phil Yang
> <Phil.Yang@arm.com>; thomas@monjalon.net; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; stephen@networkplumber.org;
> maxime.coquelin@redhat.com; dev@dpdk.org
> Cc: david.marchand@redhat.com; jerinj@marvell.com; hemant.agrawal@nxp.com=
;
> Gavin Hu <Gavin.Hu@arm.com>; Ruifeng Wang <Ruifeng.Wang@arm.com>; Joyce K=
ong
> <Joyce.Kong@arm.com>; nd <nd@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic
> operations
>=20
> <snip>
> Just to get us on the same page on 'techniques to communicate data from w=
riter
> to reader' (apologies if it is too trivial)
>=20
> Let us say that the writer has 512B (key point is - cannot be written
> atomically) that needs to be communicated to the reader.
>=20
> Since the data cannot be written atomically, we need a guard variable (wh=
ich
> can be written atomically, can be a flag or pointer to data). So, the wri=
ter
> will store 512B in non-atomic way and write to guard variable with releas=
e
> memory order. So, if the guard variable is valid (set in the case of flag=
 or
> not null in the case of pointer), it guarantees that 512B is written.
>=20
> The reader will read the guard variable with acquire memory order and rea=
d the
> 512B data only if the guard variable is valid. So, the acquire memory ord=
er on
> the guard variable guarantees that the load of 512B does not happen befor=
e the
> guard variable is read. The validity check on the guard variable guarante=
es
> that 512B was written before it was read.
>=20
> The store(guard_variable, RELEASE) on the writer and the load(guard_varia=
ble,
> ACQUIRE) can be said as synchronizing with each other.
>=20
> (the guard variable technique applies even if we are not using C11 atomic=
s)

Yep agreed on the above.


> Let us say that the writer has 4B (key point is - can be written atomical=
ly)
> that needs to be communicated to the reader. The writer is free to write =
this
> atomically with no constraints on memory ordering as long as this data is=
 not
> acting as a guard variable for any other data.
>=20
> In my understanding, the sequence of APIs to call to start a service (wri=
ter)
> are as follows:
> 1) rte_service_init
> 2) rte_service_component_register
> 3) <possible configuration of the service>
> 4) rte_service_component_runstate_set (the reader is allowed at this poin=
t to
> read the information about the service - written by
> rte_service_component_register API. This API should not be called before
> rte_service_component_register)
> 5) <possible configuration of the service>
> 6) rte_service_runstate_set (the reader is allowed at this point to read =
the
> information about the service - written by rte_service_component_register=
 API
> and run the service. This API can be called anytime. But, the reader shou=
ld
> not attempt to run the service before this API is called)
> 7) rte_lcore_service_add (multiple of these probably, can be called befor=
e
> this, can't be called later)
> 8) rte_service_map_lcore_set (this can be called anytime. Can be called e=
ven
> if the service is not registered)
> 9) rte_service_lcore_start (again, this can be called anytime, even befor=
e the
> service is registered)

I think this can be simplified, if we look at calling threads:
 - one thread is the writer/config thread, and is allowed to call anything
 --- Any updates/changes must be atomically correct in how the other thread=
s can read state.

 - service lcores, which fundamentally spin in run(), and call services map=
ped to it
 --- here we need to ensure any service mapped to it is atomic, and the ser=
vice is valid to run.

 - other application threads using "run on app lcore" function
 --- similar to service lcore, check for service in valid state, and allow =
to run.


Services are not allowed to be unregistered e.g. while running.
I'd like to avoid the "explosion of combinations" by enforcing
simple limitations (and documenting them more clearly if/where required).


> So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Onl=
y
> these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in
> reader.
>=20
> We can write test cases with different orders of these API calls to prove=
 that
> the memory orders we use are sufficient.
>=20
> Few comments are inline based on this assessment.

Sure. As per other email thread, splitting changes into bugfix/cleanup/C11
would likely help to try keep track of changes etc required per patch, its
getting hard to follow the various topics being discussed in parallel.


> > Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic
> > operations
> >
> > > From: Phil Yang <phil.yang@arm.com>
> > > Sent: Tuesday, March 17, 2020 1:18 AM
> > > To: thomas@monjalon.net; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>; Ananyev, Konstantin
> > > <konstantin.ananyev@intel.com>; stephen@networkplumber.org;
> > > maxime.coquelin@redhat.com; dev@dpdk.org
> > > Cc: david.marchand@redhat.com; jerinj@marvell.com;
> > > hemant.agrawal@nxp.com; Honnappa.Nagarahalli@arm.com;
> > > gavin.hu@arm.com; ruifeng.wang@arm.com; joyce.kong@arm.com;
> > nd@arm.com
> > > Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic
> > > operations
> > >
> > > To guarantee the inter-threads visibility of the shareable domain, it
> > > uses a lot of rte_smp_r/wmb in the service library. This patch relaxe=
d
> > > these barriers for service by using c11 atomic one-way barrier operat=
ions.
> > >
> > > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > > Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > > Reviewed-by: Gavin Hu <gavin.hu@arm.com>
> > > ---
> > >  lib/librte_eal/common/rte_service.c | 45

<snip patch review comments>