From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 40495A0597; Wed, 8 Apr 2020 21:42:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6C0341C202; Wed, 8 Apr 2020 21:42:46 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id E900D1C1F5 for ; Wed, 8 Apr 2020 21:42:44 +0200 (CEST) IronPort-SDR: X7jqKanpJdHUD+KVBCujZLCsg88iOMMp2gsHk+CSkfAANQLZGR+1ix74SQ5nzx0nRYrzdnpx2H aTYVuG3Nxy2A== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2020 12:42:43 -0700 IronPort-SDR: R1zhfKgl48PPinB7UcMJCI+2Sb6NIKXM7UpeaPCQg8VpjioMnXnSQe3Jp6n64UyUnhCWZb0evv TbPqCr08CnWQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,359,1580803200"; d="scan'208";a="297319891" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by FMSMGA003.fm.intel.com with ESMTP; 08 Apr 2020 12:42:43 -0700 Received: from fmsmsx115.amr.corp.intel.com (10.18.116.19) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 8 Apr 2020 12:42:43 -0700 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx115.amr.corp.intel.com (10.18.116.19) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 8 Apr 2020 12:42:42 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.176) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 8 Apr 2020 12:42:42 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FI6biZt55hu2cYe5gy463QtEp0rcG3b+2e44TTDZ/CsaEECAW6dm5NRNnjE5aTVFErt2eaZ38qyc2elpd3MVArmd6MTgTx65qNWNGd329v3z8GxF2C5IIFRvcUSy7ja6iImcU0MZtEXUsfxZMXPNifkhwVVUg6BYMf3UodVkNfAcvv45ysq5MAL+lLNZJi2tJf2DScjxcOsPiKRrlFPEJtlUwTyzJipBmUdEiNXa1XOph1c53PuTSzvk8m1LxIKo7JXSttOIDAfxsgWoTZjZHBns6P7gpmlM+Az4lW7r3x2RX4mVWJ5uihGSbQLYDYkdOVOyiI4Z/khAR4ouOK4a+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xzffp8MQoxfeMzYHlWFKOr6f39U3kINTkztFKfBMnC8=; b=kAKAvUr0R1Ov/Y7RMd6CTxYlfKa7y8w3uZWTzrZstst9bnUWe0GwSB7/hZQeaLpf7hslh0GcuXr0daQSo46YmehS0+QC9SIBMuVEkmMUs4QeJvOfjd792+YkT41CBY/Us/PU1xAFUoHp0rQ7c2VmY8N8xVn4ziHygdYbJcxXXcsp6vjBwHSbWDqgzhsBaSL1rs3Q6fzOIf+M6GYWz7N6oeIPApZ7eyly0bOkvb088tJW6XkdWc0/ykceicAA+PX8i0rHx0+1mAnID2FxK5AP7b9zbzqUpzKC7EuMSIjOmTm9z6qnCHLVbCGcH2YHYV4cKEm6ABflAp8OH+IewklNZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xzffp8MQoxfeMzYHlWFKOr6f39U3kINTkztFKfBMnC8=; b=qoGaFhNfZywkAtA2Tbv0N8l/L1nqp14ODg0YH8I2xxRg8gqBz9ZHj1lmEIsmPGPdTaQUEiMVc5IRO0X11cLQAK03YoXTFb6E0SlJudGDWvkSeNDhldm96uFZctzfiB5vh9K3a+E9KkZIm1fsJFxjp9cQ0qcDRe/2hVRqbi95UVI= Received: from BYAPR11MB3143.namprd11.prod.outlook.com (2603:10b6:a03:92::32) by BYAPR11MB2919.namprd11.prod.outlook.com (2603:10b6:a03:8d::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.21; Wed, 8 Apr 2020 19:42:41 +0000 Received: from BYAPR11MB3143.namprd11.prod.outlook.com ([fe80::5917:9757:49b0:3c49]) by BYAPR11MB3143.namprd11.prod.outlook.com ([fe80::5917:9757:49b0:3c49%5]) with mapi id 15.20.2878.021; Wed, 8 Apr 2020 19:42:41 +0000 From: "Van Haaren, Harry" To: Honnappa Nagarahalli , Phil Yang , "thomas@monjalon.net" , "Ananyev, Konstantin" , "stephen@networkplumber.org" , "maxime.coquelin@redhat.com" , "dev@dpdk.org" CC: "david.marchand@redhat.com" , "jerinj@marvell.com" , "hemant.agrawal@nxp.com" , Gavin Hu , Ruifeng Wang , Joyce Kong , nd , nd Thread-Topic: [PATCH v3 12/12] service: relax barriers with C11 atomic operations Thread-Index: AQHV+/omNUSP5Ua9ZE6XZhRVe3IM2qhnXyCwgAUUQYCAA03xQA== Date: Wed, 8 Apr 2020 19:42:41 +0000 Message-ID: References: <1583999071-22872-1-git-send-email-phil.yang@arm.com> <1584407863-774-1-git-send-email-phil.yang@arm.com> <1584407863-774-13-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 authentication-results: spf=none (sender IP is ) smtp.mailfrom=harry.van.haaren@intel.com; x-originating-ip: [192.198.151.165] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 37a86157-990f-4579-9ada-08d7dbf50042 x-ms-traffictypediagnostic: BYAPR11MB2919: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 0367A50BB1 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3143.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10019020)(346002)(39850400004)(376002)(136003)(366004)(396003)(4326008)(8936002)(66476007)(2906002)(186003)(316002)(110136005)(54906003)(64756008)(6506007)(66946007)(53546011)(76116006)(86362001)(81166007)(71200400001)(26005)(66556008)(66446008)(55016002)(9686003)(33656002)(81156014)(7696005)(8676002)(7416002)(66574012)(5660300002)(52536014)(478600001); DIR:OUT; SFP:1102; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ZanCUOWWL5iHMGYnGZdmpjIs+bgeOM2qAdGLG5eyZiSUH13xx5I18PjyKxze6K3TcwMwtpFVt9G59aQ47ixnSP6lzIX8K59a0KPGG3vUonUULJR3gSaFCIII4B8IRkAVvVP/M6+TAm4TjynW7mvF5qbfOdUxRVCuj2mgliEfDs2clUiNScOSCVUbXGF44hbBkE3Qf67LZI/rs9tFTLzRb1Xj739OJny2aVFyl/MbA3M1FXxsWxLMD3Iyxw6x5IZgZTUuEnIcgtPhp7B/URK2IRHa7gYWvdYUbG6uVFzb+K4l5AOc00KAzygq22bnJJLdwXBkVnlZW9KABlr14O+1KvTZf1wLqAdk69k7Vz05viGmDYkUHFi4/0MwAHjVgmJ810dpiA2+jGTB8KFxTf94SCD5K1WrT1pBULYiJZxfLz5xlbmejskbiZfa5+C7xezj x-ms-exchange-antispam-messagedata: AEymCdRtXl6EYoOSn2/i5NIBbV/hHjOIKOzbov9qnhA4hbMgz6ic79Bsoaw227l99VlaK/OvHs5WmQFb4SESruDmQHUIeFD6ZQBPjVGb7l64dFaIHeqLO9g+XGpN+SY8+LQhzw6wtd19eKA36riewg== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 37a86157-990f-4579-9ada-08d7dbf50042 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2020 19:42:41.2850 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: NTSNpLUnxkgZFt6++5Me1kP6MMCRXE3RebJc4Py9tBbTgdjOQ1vI++1ihBqdehLKA5dlHIA/+OmPuFGCCXNTGAVgZdGC7EiXPzW7R/Ff68w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB2919 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v3 12/12] service: relax barriers with C11 atomic operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Honnappa Nagarahalli > Sent: Monday, April 6, 2020 6:06 PM > To: Van Haaren, Harry ; Phil Yang > ; thomas@monjalon.net; Ananyev, Konstantin > ; stephen@networkplumber.org; > maxime.coquelin@redhat.com; dev@dpdk.org > Cc: david.marchand@redhat.com; jerinj@marvell.com; hemant.agrawal@nxp.com= ; > Gavin Hu ; Ruifeng Wang ; Joyce K= ong > ; nd ; Honnappa Nagarahalli > ; nd > Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic > operations >=20 > > Just to get us on the same page on 'techniques to communicate data from w= riter > to reader' (apologies if it is too trivial) >=20 > Let us say that the writer has 512B (key point is - cannot be written > atomically) that needs to be communicated to the reader. >=20 > Since the data cannot be written atomically, we need a guard variable (wh= ich > can be written atomically, can be a flag or pointer to data). So, the wri= ter > will store 512B in non-atomic way and write to guard variable with releas= e > memory order. So, if the guard variable is valid (set in the case of flag= or > not null in the case of pointer), it guarantees that 512B is written. >=20 > The reader will read the guard variable with acquire memory order and rea= d the > 512B data only if the guard variable is valid. So, the acquire memory ord= er on > the guard variable guarantees that the load of 512B does not happen befor= e the > guard variable is read. The validity check on the guard variable guarante= es > that 512B was written before it was read. >=20 > The store(guard_variable, RELEASE) on the writer and the load(guard_varia= ble, > ACQUIRE) can be said as synchronizing with each other. >=20 > (the guard variable technique applies even if we are not using C11 atomic= s) Yep agreed on the above. > Let us say that the writer has 4B (key point is - can be written atomical= ly) > that needs to be communicated to the reader. The writer is free to write = this > atomically with no constraints on memory ordering as long as this data is= not > acting as a guard variable for any other data. >=20 > In my understanding, the sequence of APIs to call to start a service (wri= ter) > are as follows: > 1) rte_service_init > 2) rte_service_component_register > 3) > 4) rte_service_component_runstate_set (the reader is allowed at this poin= t to > read the information about the service - written by > rte_service_component_register API. This API should not be called before > rte_service_component_register) > 5) > 6) rte_service_runstate_set (the reader is allowed at this point to read = the > information about the service - written by rte_service_component_register= API > and run the service. This API can be called anytime. But, the reader shou= ld > not attempt to run the service before this API is called) > 7) rte_lcore_service_add (multiple of these probably, can be called befor= e > this, can't be called later) > 8) rte_service_map_lcore_set (this can be called anytime. Can be called e= ven > if the service is not registered) > 9) rte_service_lcore_start (again, this can be called anytime, even befor= e the > service is registered) I think this can be simplified, if we look at calling threads: - one thread is the writer/config thread, and is allowed to call anything --- Any updates/changes must be atomically correct in how the other thread= s can read state. - service lcores, which fundamentally spin in run(), and call services map= ped to it --- here we need to ensure any service mapped to it is atomic, and the ser= vice is valid to run. - other application threads using "run on app lcore" function --- similar to service lcore, check for service in valid state, and allow = to run. Services are not allowed to be unregistered e.g. while running. I'd like to avoid the "explosion of combinations" by enforcing simple limitations (and documenting them more clearly if/where required). > So, there are 2 guard variables - 'comp_runstate' and 'app_runstate'. Onl= y > these 2 need to have RELEASE ordering in writer and ACQUIRE ordering in > reader. >=20 > We can write test cases with different orders of these API calls to prove= that > the memory orders we use are sufficient. >=20 > Few comments are inline based on this assessment. Sure. As per other email thread, splitting changes into bugfix/cleanup/C11 would likely help to try keep track of changes etc required per patch, its getting hard to follow the various topics being discussed in parallel. > > Subject: RE: [PATCH v3 12/12] service: relax barriers with C11 atomic > > operations > > > > > From: Phil Yang > > > Sent: Tuesday, March 17, 2020 1:18 AM > > > To: thomas@monjalon.net; Van Haaren, Harry > > > ; Ananyev, Konstantin > > > ; stephen@networkplumber.org; > > > maxime.coquelin@redhat.com; dev@dpdk.org > > > Cc: david.marchand@redhat.com; jerinj@marvell.com; > > > hemant.agrawal@nxp.com; Honnappa.Nagarahalli@arm.com; > > > gavin.hu@arm.com; ruifeng.wang@arm.com; joyce.kong@arm.com; > > nd@arm.com > > > Subject: [PATCH v3 12/12] service: relax barriers with C11 atomic > > > operations > > > > > > To guarantee the inter-threads visibility of the shareable domain, it > > > uses a lot of rte_smp_r/wmb in the service library. This patch relaxe= d > > > these barriers for service by using c11 atomic one-way barrier operat= ions. > > > > > > Signed-off-by: Phil Yang > > > Reviewed-by: Ruifeng Wang > > > Reviewed-by: Gavin Hu > > > --- > > > lib/librte_eal/common/rte_service.c | 45