From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <Jerin.JacobKollanukkaran@cavium.com> Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0081.outbound.protection.outlook.com [104.47.34.81]) by dpdk.org (Postfix) with ESMTP id 5532D1B31D for <dev@dpdk.org>; Wed, 1 Nov 2017 20:04:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=1qEbPJ0lwhBt7ZqYfeC+vrQr0fPpIkWIo18kZDBRWBw=; b=VdA65NmehCQgurZrkzQBcuM0OfiH9UB+xIVw9zwyQmzfpdk6MgDcCZwJaZkG/uWNYdVfwECY2hdkRN8jDqpcJ5BogS6tln7ZFgEZw+6edPkYxXRsFJNft/kiw4Dw+/ovPIvtAL6LlFhR1qjjFRZw3TrFf6GupfILPACex57Qh08= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.JacobKollanukkaran@cavium.com; Received: from jerin (106.201.40.78) by BN3PR07MB2513.namprd07.prod.outlook.com (10.167.4.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.197.13; Wed, 1 Nov 2017 19:04:44 +0000 Date: Thu, 2 Nov 2017 00:34:21 +0530 From: Jerin Jacob <jerin.jacob@caviumnetworks.com> To: Jia He <hejianet@gmail.com> Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Zhao, Bing" <ilovethull@163.com>, Olivier MATZ <olivier.matz@6wind.com>, "dev@dpdk.org" <dev@dpdk.org>, "jia.he@hxt-semitech.com" <jia.he@hxt-semitech.com>, "jie2.liu@hxt-semitech.com" <jie2.liu@hxt-semitech.com>, "bing.zhao@hxt-semitech.com" <bing.zhao@hxt-semitech.com>, "Richardson, Bruce" <bruce.richardson@intel.com>, jianbo.liu@arm.com, hemant.agrawal@nxp.com Message-ID: <20171101190420.GA21407@jerin> References: <2601191342CEEE43887BDE71AB9772585FAAB570@IRSMSX103.ger.corp.intel.com> <3e580cd7-2854-d855-be9c-7c4ce06e3ed5@gmail.com> <20171020054319.GA4249@jerin> <ab7154a2-a9f8-f12e-b6a0-2805c2065e2e@gmail.com> <20171023100617.GA17957@jerin> <b0e6eae2-e61b-1946-ac44-d781225778e5@gmail.com> <20171025132642.GA13977@jerin> <b54ce545-b75d-9813-209f-4125bd76c7db@gmail.com> <20171031111433.GA21742@jerin> <69adfb00-4582-b362-0540-d1d9d6bcf6aa@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <69adfb00-4582-b362-0540-d1d9d6bcf6aa@gmail.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-Originating-IP: [106.201.40.78] X-ClientProxiedBy: BM1PR01CA0114.INDPRD01.PROD.OUTLOOK.COM (10.174.208.30) To BN3PR07MB2513.namprd07.prod.outlook.com (10.167.4.138) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c5498bd5-68fd-4f92-3f95-08d5215b6ca5 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(2017052603199); SRVR:BN3PR07MB2513; X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 3:wmQJRf+rW4TWUmS9H7+LpewU+h5RdIcB61ToLUJpDM8JlWFhtb46HPttIhvtqVFV86HcfmDE3UGeVk9FpmGC3PcV+0Z6oLfZo2Ow9u6xlrBif4I/NZsTOStEzcPt4pZunWv4Cot0DjaY3ruf/E5fVjXnrX9SGuxSJ3D3APyYWC7g/WAvFIqcKOBgMGPut5AYC1LRpIHn/Aay7wrg9ahUc5xNeYywCuCOi3fItFFxV9FTyKvJBPuAbyyC0KOgRYhP; 25:U9BpaYcNaNSd10O1NVgSuLnln6oPc8fUKtdoz+F0aQElVjK6O+4dVJ2SDoBpqzaxhEZGWJ+lvPOIYvrPd6JZl4GCvGODvADMTdYp9sEV+bqn82INOmM3gL+LHNL3Pc8pbqZASrLjaVnmD6t+/rQC7C5yBJaDhvImy2ABQMxZy1VZaRzH/wgR6GJZfIzr00ShLrDDlRJpqSckesYkj+Y117UidaPb8zoaOSO9bNTQQnLzCjHVWlwt0dkw3eDiqMUtULzgFMwJkHIs4pX71Van1K9QowrNZQYeSvQZHi5fWhba/zZv6jZ/y6OqORjU/SukHIFJSW+228YegP6G9BDKow==; 31:hZ5YqbJKCDh/ImhvlADA1WzTSRnNt4lgoEaNXm9hGBen+rVYjQUT8D1H78qFoSy2PyGWQlfE24Xml52enxjgFMAjmEwaFY/wGhpImYKCai2G1Co5rWEznCbrdOuvR8W/GET8sgQ7vrLRqM9X2r6Ztl7CxZWf2LU8gv2g7glgkNq6vz+HeHR5VnC7QK9cNpI6+a6riGXhjH6JL41+lWTDESgo56Mw2XJYJfoL9sdvaGI= X-MS-TrafficTypeDiagnostic: BN3PR07MB2513: X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 20:8iTyToYRJrkdbcK4JqyAcSR3O5HG5EC0rR8uX1xiVvwiIUkTbKyrQMfQNlX7kGCur34sk4GkKnx0Q5lcE+QerEPRoDnqGVEOKj0xiU1lKg2PduuFDcF41JmSPWRUKE9T0U4QzQql9VhtH9rxU8P7kfqv5NfPhJFOD18gBqnu1wJ1g6/SJ6gS+LronXwyyYv0y+eU7zKT328sEg/ZjWyRd/FH5iKgf9qJYyV+OId3TU+0epjZCJyV/pb/WmDWL2r96xIbIT6kTKaUIswCbTfwXZgPaKK718Rko7y3UXK6YWfTDQl0qVZDG+HZdXGtGcPrRLOKKP0NiLD+gznegOdnetIX9ZuPnJJAUJtFxV9aEr2USz0nrlnN2qf9AHYnV+sJHy2u9/KpeU8ou5CqNnW4uwI9nfyuYQsk4wmujmVsaaX6P+K+rEonHrhVnptfa/qrvwAjGs3fzWkKcyNfitxZu7bCN2npJBzj6eDXFvWjxworiT4MwKouPPVbu+3ObJnEF6U32XBPFeP7LgQUKWxTvBOA1ty11vWeMVVwL3fgp/VG6J3MqKW14DkeVuj1pCDjlsuckPp75Iml819UOSzstSDyS1Z/FUROLhXBfyZmkVU= X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(180628864354917)(166708455590820)(185117386973197)(130843839470238)(228905959029699); X-Microsoft-Antispam-PRVS: <BN3PR07MB2513A686AD68E25099655C5FE35F0@BN3PR07MB2513.namprd07.prod.outlook.com> X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(93006095)(100000703101)(100105400095)(3231020)(3002001)(6041248)(20161123562025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123560025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN3PR07MB2513; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN3PR07MB2513; X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 4:JzAX2E+WBM8HHnLlJHu+KcMtwqDL9XbfSGyE/WoLJRN9oBLwewnBcMrdLr1SWNspyaw8JJrEPumwFBET798CWu1gl50mQYi4oEYkqtdG68sEAflnaUUHScAdvZQVOpcU0PKm7WxhW4bDFpNNrieJUH4nlT8D2hpHuS1WkgrRhCxx0qOKR1aDRVpHcvJcFiLCr4Zk44T0TD375npOqrgDIj+Px+IuL88AEn5OgZm+n6zebzyx96bbS25kxg3f8e2OI7CIuZFnCGgDpIp5LTF0MXm+VegsvcJgec9xir0nGaYjx2w3W/uavPLxcNKBXHaEU4ekpZOyFL5UoeTWibv/JyDuBhx6wEaFiTM5kezQGyCaNceeJjsoOFkpeDcu3rUXXdTc7Xs6SYC8Y5hAsF9sTVfgYwhwvi4lBLCWtBk8XYgSDnomKUpzu0bNPN1KfhGavjqttV3AVM+Rqianyz1bUduyX95g5JkLtTNljA0J82g= X-Forefront-PRVS: 0478C23FE0 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(24454002)(13464003)(377424004)(199003)(51234002)(189002)(966005)(106356001)(50466002)(81156014)(3846002)(189998001)(72206003)(81166006)(6116002)(6496005)(8936002)(105586002)(42882006)(16526018)(97736004)(8656006)(6916009)(575784001)(2950100002)(5660300001)(478600001)(6666003)(2906002)(7416002)(23756003)(83506002)(93886005)(316002)(53936002)(33716001)(33656002)(68736007)(9686003)(55016002)(6306002)(54906003)(6246003)(4326008)(1411001)(1076002)(305945005)(8676002)(39060400002)(7736002)(2870700001)(53376002)(66066001)(50986999)(76176999)(54356999)(229853002)(25786009)(58126008)(5009440100003)(101416001)(47776003)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR07MB2513; H:jerin; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1; BN3PR07MB2513; 23:tV+KGifBfvuymFd8glzbrJ3xFI/G6qA8Xd4+QdZ?= =?iso-8859-1?Q?/zUMei0xYK1+Qy0eeiNWxD/sDPjRoBP4ZLkn/BkUnB1Jktd1JveMVFGIio?= =?iso-8859-1?Q?9l/kJZTF5Rfxa33EEBh5G3issj+Kq36LEHzqA9/aJZlPb3f8STg1S+bPs8?= =?iso-8859-1?Q?FKSE5I9QgwyRFEyByAOxoqG2Oy6HhsyK+pwWek1PJq56UiR93w0vOJXiqq?= =?iso-8859-1?Q?c9j+263MLiCTCdBXbXpH2CMvDJC4h2oVUlIp2Yn4Z5RdjjRv0WGslrNh9t?= =?iso-8859-1?Q?vLSRHyWZ98RZEvDguiLrkV7HwgDOCW2Vrv7RuSobXFXT0v8H33qZMJSKma?= =?iso-8859-1?Q?aGSwSNnSTyMJNq3Njs7K9Zecby81FSh7EsmLrPNmHJP6bHuVIhx6GijjAM?= =?iso-8859-1?Q?bLeNVqMxUZSHOyHrfVDtHORkGn1Y3OzHzg3/yGRCKBhAudHOM7hSWP6/8G?= =?iso-8859-1?Q?TlYmNpqfKkiSRTjklERNSOP8dnvsPtuC/vj87GLIAeIZQzy+SpTSpLtTia?= =?iso-8859-1?Q?RNC/+ZaOgObz/OodZHDq8MHElMJONuZ6x6L4Ld5UKz+d9Qz8lX6HI1EMOs?= =?iso-8859-1?Q?WvUn2NVZIXxf46VloN4HOe0mXJ9apNQrMGo0cEC7vRW+XAWgq+Soa5Yt7a?= =?iso-8859-1?Q?WlNo3Q+lIo6Jre5vSF4PcxMAPWXfV0CvdsC2i3OOoYRVH/N0rrY9l/te1x?= =?iso-8859-1?Q?sH+RS1QIx7jbM3Wh7ZnvCr+VGCV3t5W/DHjFODjgsWj1QFbnVOgxMUm3Os?= =?iso-8859-1?Q?F5DTvgmdIm2YtY8y0Bi167bMesUBlejOsvGryn0gln1aXXfn1/WqphuQ5d?= =?iso-8859-1?Q?FlcRbQtmZDt8Qkc0WLuQB6zslN6KE6/8zTqgr/bSmE0jKvLajwmu/TMpXf?= =?iso-8859-1?Q?zi/NtRou6FwUaOl1Oycv4rlzPA2WgIieP7s1bl9c9VIp3TfRff50cDkCyk?= =?iso-8859-1?Q?CiQPnOBDDOuSauAUCmfqT3T3CbfduVq7kg8Lu1Isbij9Z5UiJLMDZnv9Q+?= =?iso-8859-1?Q?P6q+J6cFjp1vKK9q63w1/JEsEW+43h0vBsvDbso45TkM18xYzsf5r+NiJL?= =?iso-8859-1?Q?p0k+GonYLL8U+9fS1ty6msl+M/2bc55Njbl6feN5m/bdgB30eArBj02U9h?= =?iso-8859-1?Q?buQpmZsyRf2mPf//nFqDzx/BSXri3JFP5G0YikahyVeObMnGvIhZwpBjb4?= =?iso-8859-1?Q?rYT+zahhtImS+iAoY0m2A78nZ5E0twRLZ+srSjdrDszY2EkImVqsIusA5Q?= =?iso-8859-1?Q?Rlp2utWNHPzi4oGwj/52fQGXiELoDVdK7taQOR83pG2g7cPS7eYBP0BOJ2?= =?iso-8859-1?Q?pVzK8Qg3Ww5YQRIKas9/JtHELLrr0vfvj0ip+SrMO8LM6okRhSVVehfAVE?= =?iso-8859-1?Q?kyQYNuUPAkLXjvVjXWzr5EmsBjwkiZ0w34JJe/KhIcaNADx50W0fVkygJu?= =?iso-8859-1?Q?Ja0Nl+d3Yvod8QcveEng3vtZfcaLpMsKAsxgYe1t0EuXQqU3WTkDOPX8/3?= =?iso-8859-1?Q?qUll62xedcVXRrHBHI4JCMtremyFUH3wo/5Qvz3dp980MFaMLLlVriRmVJ?= =?iso-8859-1?Q?4Zk5pPqIPnhwI1oeUbMcqJmj6Y=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2513; 6:AtNov4lecCfgDRmLN76gwy4pHd9IRK5hvUjuEDleP6gC9QT/ftu8J/w3Jgu+8uRzHg/UNFC16X/sWsMV7u3jTsHH2taQDQzF1wPHL1hU5RmO4NnUoe47rqyjpmnXXyQB4V9iOZky6E3q8kJ4ibHL9bNi2OpPRbeAf6v6zeTHP9X5SlX96tWopmuWle4oNY9ILH8htREWVQaq7kbFEL8FnVNonAerJKNCLHyls7LAKMYClF1hvlx54h4B9CC8b/BKet5vz2y4WWqMrl9kdrWZj4ucT1ZKGKGfUpTtuZa71xLAvyYlQG8X48VrMIG1o441LkZ5OPegTdhQllLnOr84jikeHQu8R/P5z/Lk/u8Tl6k=; 5:r/Z7QCeAEyENpeLnglJe52ZcceOIJ31kNLdDIZLbOpK7x2y+GwcmSjNFUFsh8Q7P4xZEXE7WNre61gXKe+DOZ2hyAs8ZRSVK2mNaWg9DiAp2vs7184BMC229mp+6UHBTvzMeAvMc/zd5sGlx4bixk7klfys6olpxmUbyDNVTcK8=; 24:2+HaryeX7jdnoIjhYoneNDQq3PGJQvxJ/asKhUNsCmqrsWxrrkjXd6nks21Y87XnYGrYQ090baGzMVG/409idiEhYIf+10e3XNyuaDdxNVs=; 7:Egmxx2Jv9uwgsDQ7AgKcuhWW4X4Zg5NRaUAqpT5aruuQQvXgbwjNcMNgh5ggG1itFmIX5eKOl+LeVXr7NmQnuFz+H9+m6Mr+Jj6aPamAHF+9SVqlxn3uNQcgAsCiX/EuQYrqk5LG/aHU10rxgXmx7B4AV2rvMPZ+WxTDpPtYOyd/RRRVwxrQdMxq5y3b4APDVCBnAxUof1e6/fbBFFmyt+o1XWmQ+XlmscpUg1lkUFAyW6HYECp/v9Y1u/ur9jlE SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2017 19:04:44.4426 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c5498bd5-68fd-4f92-3f95-08d5215b6ca5 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR07MB2513 Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Wed, 01 Nov 2017 19:04:53 -0000 Date: Thu, 2 Nov 2017 00:27:46 +0530 From: Jerin Jacob <jerin.jacob@caviumnetworks.com> To: Jia He <hejianet@gmail.com> Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Zhao, Bing" <ilovethull@163.com>, Olivier MATZ <olivier.matz@6wind.com>, "dev@dpdk.org" <dev@dpdk.org>, "jia.he@hxt-semitech.com" <jia.he@hxt-semitech.com>, "jie2.liu@hxt-semitech.com" <jie2.liu@hxt-semitech.com>, "bing.zhao@hxt-semitech.com" <bing.zhao@hxt-semitech.com>, "Richardson, Bruce" <bruce.richardson@intel.com>, jianbo.liu@arm.com, hemant.agrawal@nxp.com Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod loading when doing enqueue/dequeue Message-ID: <20171101185723.GA18759@jerin> References: <2601191342CEEE43887BDE71AB9772585FAAB570@IRSMSX103.ger.corp.intel.com> <3e580cd7-2854-d855-be9c-7c4ce06e3ed5@gmail.com> <20171020054319.GA4249@jerin> <ab7154a2-a9f8-f12e-b6a0-2805c2065e2e@gmail.com> <20171023100617.GA17957@jerin> <b0e6eae2-e61b-1946-ac44-d781225778e5@gmail.com> <20171025132642.GA13977@jerin> <b54ce545-b75d-9813-209f-4125bd76c7db@gmail.com> <20171031111433.GA21742@jerin> <69adfb00-4582-b362-0540-d1d9d6bcf6aa@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <69adfb00-4582-b362-0540-d1d9d6bcf6aa@gmail.com> User-Agent: Mutt/1.9.1 (2017-09-22) -----Original Message----- > Date: Wed, 1 Nov 2017 10:53:12 +0800 > From: Jia He <hejianet@gmail.com> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com> > Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Zhao, Bing" > <ilovethull@163.com>, Olivier MATZ <olivier.matz@6wind.com>, > "dev@dpdk.org" <dev@dpdk.org>, "jia.he@hxt-semitech.com" > <jia.he@hxt-semitech.com>, "jie2.liu@hxt-semitech.com" > <jie2.liu@hxt-semitech.com>, "bing.zhao@hxt-semitech.com" > <bing.zhao@hxt-semitech.com>, "Richardson, Bruce" > <bruce.richardson@intel.com>, jianbo.liu@arm.com, hemant.agrawal@nxp.com > Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod > loading when doing enqueue/dequeue > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 > Thunderbird/52.4.0 > > Hi Jerin > > Thanks for your suggestions. I will try to use config macro to let it be > chosen by user. It is better, if we avoid #ifdef's in the common code. I think, you can do the scheme like examples/l3fwd/l3fwd_em_hlm_neon.h. Where, the common code will have all generic routine, difference will be moved to a separate files to reduce #ifdef clutter. > > I need to point out one possible issue in your load_acq/store_rel patch > > at https://github.com/jerinjacobk/mytests/blob/master/ring/0001-ring-using-c11-memory-model.patch > > @@ -516,8 +541,13 @@ __rte_ring_move_cons_head(struct rte_ring *r, int > is_sc, > /* Restore n as it may change every loop */ > n = max; > > +#if 0 > *old_head = r->cons.head; > const uint32_t prod_tail = r->prod.tail; > +#else > + *old_head = __atomic_load_n(&r->cons.head, __ATOMIC_RELAXED); > --[1] > + const uint32_t prod_tail = __atomic_load_n(&r->prod.tail, > __ATOMIC_ACQUIRE); --[2] > +#endif > > line [1] __ATOMIC_RELAXED is not enough for this case(tested in our ARM64 > server). > > line [2] __ATOMIC_ACQUIRE guarantee the 2nd load will not be reorded before > the 1st load, but will not > > guarantee the 1st load will not be reordered after the 2nd load. Please also For me it looks same. [1] can not cross [2] is same as [2] cannot cross [1], if [1] are [2] at back to back. No ? > refer to your mentioned freebsd implementation. They use __ATOMIC_ACQUIRE at > line [1]. If I understand it correctly. in freebsd, they are planning to revisit the usage of first __ATOMIC_ACQUIRE in Freebsd-12 https://github.com/freebsd/freebsd/blob/master/sys/sys/buf_ring.h#L170 > > Should it be like instead? > > +#else > + *old_head = __atomic_load_n(&r->cons.head, __ATOMIC_ACQUIRE); > + const uint32_t prod_tail = __atomic_load_n(&r->prod.tail, > __ATOMIC_ACQUIRE); It would be nice to see how much overhead it gives.ie back to back __ATOMIC_ACQUIRE. > > > Cheers, > > Jia > > > > On 10/31/2017 7:14 PM, Jerin Jacob Wrote: > > -----Original Message----- > > > Date: Tue, 31 Oct 2017 10:55:15 +0800 > > > From: Jia He <hejianet@gmail.com> > > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com> > > > Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Zhao, Bing" > > > <ilovethull@163.com>, Olivier MATZ <olivier.matz@6wind.com>, > > > "dev@dpdk.org" <dev@dpdk.org>, "jia.he@hxt-semitech.com" > > > <jia.he@hxt-semitech.com>, "jie2.liu@hxt-semitech.com" > > > <jie2.liu@hxt-semitech.com>, "bing.zhao@hxt-semitech.com" > > > <bing.zhao@hxt-semitech.com>, "Richardson, Bruce" > > > <bruce.richardson@intel.com> > > > Subject: Re: [dpdk-dev] [PATCH] ring: guarantee ordering of cons/prod > > > loading when doing enqueue/dequeue > > > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 > > > Thunderbird/52.4.0 > > > > > > Hi Jerin > > Hi Jia, > > > > > Do you think next step whether I need to implement the load_acquire half > > > barrier as per freebsd > > I did a quick prototype using C11 memory model(ACQUIRE/RELEASE) schematics > > and tested on two arm64 platform in Cavium(Platform A: Non arm64 OOO machine) > > and Platform B: arm64 OOO machine) > > > > smp_rmb() performs better in Platform A: > > acquire/release semantics perform better in platform B: > > > > Here is the patch: > > https://github.com/jerinjacobk/mytests/blob/master/ring/0001-ring-using-c11-memory-model.patch > > > > In terms of next step: > > - I am not sure the cost associated with acquire/release semantics on x86 or ppc. > > IMO, We need to have both options under conditional compilation > > flags and let the target platform choose the best one. > > > > Thoughts? > > > > Here is the performance numbers: > > - Both platforms are running at different frequency, So absolute numbers does not > > matter, Just check the relative numbers. > > > > Platform A: Performance numbers: > > ================================ > > no patch(Non arm64 OOO machine) > > ------------------------------- > > > > SP/SC single enq/dequeue: 40 > > MP/MC single enq/dequeue: 282 > > SP/SC burst enq/dequeue (size: 8): 11 > > MP/MC burst enq/dequeue (size: 8): 42 > > SP/SC burst enq/dequeue (size: 32): 8 > > MP/MC burst enq/dequeue (size: 32): 16 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 8.01 > > MC empty dequeue: 11.01 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 11.30 > > MP/MC bulk enq/dequeue (size: 8): 42.85 > > SP/SC bulk enq/dequeue (size: 32): 8.25 > > MP/MC bulk enq/dequeue (size: 32): 16.46 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 20.62 > > MP/MC bulk enq/dequeue (size: 8): 56.30 > > SP/SC bulk enq/dequeue (size: 32): 10.94 > > MP/MC bulk enq/dequeue (size: 32): 18.66 > > Test OK > > > > # smp_rmb() patch((Non OOO arm64 machine) > > http://dpdk.org/dev/patchwork/patch/30029/ > > ----------------------------------------- > > > > SP/SC single enq/dequeue: 42 > > MP/MC single enq/dequeue: 291 > > SP/SC burst enq/dequeue (size: 8): 12 > > MP/MC burst enq/dequeue (size: 8): 44 > > SP/SC burst enq/dequeue (size: 32): 8 > > MP/MC burst enq/dequeue (size: 32): 16 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 13.01 > > MC empty dequeue: 15.01 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 11.60 > > MP/MC bulk enq/dequeue (size: 8): 44.32 > > SP/SC bulk enq/dequeue (size: 32): 8.60 > > MP/MC bulk enq/dequeue (size: 32): 16.50 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 20.95 > > MP/MC bulk enq/dequeue (size: 8): 56.90 > > SP/SC bulk enq/dequeue (size: 32): 10.90 > > MP/MC bulk enq/dequeue (size: 32): 18.78 > > Test OK > > RTE>> > > > > # c11 memory model patch((Non OOO arm64 machine) > > https://github.com/jerinjacobk/mytests/blob/master/ring/0001-ring-using-c11-memory-model.patch > > --------------------------------------------------------------------------------------------- > > ### Testing single element and burst enq/deq ### > > SP/SC single enq/dequeue: 197 > > MP/MC single enq/dequeue: 328 > > SP/SC burst enq/dequeue (size: 8): 31 > > MP/MC burst enq/dequeue (size: 8): 50 > > SP/SC burst enq/dequeue (size: 32): 13 > > MP/MC burst enq/dequeue (size: 32): 18 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 13.01 > > MC empty dequeue: 18.02 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 30.95 > > MP/MC bulk enq/dequeue (size: 8): 50.30 > > SP/SC bulk enq/dequeue (size: 32): 13.27 > > MP/MC bulk enq/dequeue (size: 32): 18.11 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 43.38 > > MP/MC bulk enq/dequeue (size: 8): 64.42 > > SP/SC bulk enq/dequeue (size: 32): 16.71 > > MP/MC bulk enq/dequeue (size: 32): 22.21 > > > > > > Platform B: Performance numbers: > > ============================== > > #no patch(OOO arm64 machine) > > ---------------------------- > > > > ### Testing single element and burst enq/deq ### > > SP/SC single enq/dequeue: 81 > > MP/MC single enq/dequeue: 207 > > SP/SC burst enq/dequeue (size: 8): 15 > > MP/MC burst enq/dequeue (size: 8): 31 > > SP/SC burst enq/dequeue (size: 32): 7 > > MP/MC burst enq/dequeue (size: 32): 11 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 3.00 > > MC empty dequeue: 5.00 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 15.38 > > MP/MC bulk enq/dequeue (size: 8): 30.64 > > SP/SC bulk enq/dequeue (size: 32): 7.25 > > MP/MC bulk enq/dequeue (size: 32): 11.06 > > > > ### Testing using two hyperthreads ### > > SP/SC bulk enq/dequeue (size: 8): 31.51 > > MP/MC bulk enq/dequeue (size: 8): 49.38 > > SP/SC bulk enq/dequeue (size: 32): 14.32 > > MP/MC bulk enq/dequeue (size: 32): 15.89 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 72.66 > > MP/MC bulk enq/dequeue (size: 8): 121.89 > > SP/SC bulk enq/dequeue (size: 32): 16.88 > > MP/MC bulk enq/dequeue (size: 32): 24.23 > > Test OK > > RTE>> > > > > > > # smp_rmb() patch((OOO arm64 machine) > > http://dpdk.org/dev/patchwork/patch/30029/ > > ------------------------------------------- > > > > ### Testing single element and burst enq/deq ### > > SP/SC single enq/dequeue: 152 > > MP/MC single enq/dequeue: 265 > > SP/SC burst enq/dequeue (size: 8): 24 > > MP/MC burst enq/dequeue (size: 8): 39 > > SP/SC burst enq/dequeue (size: 32): 9 > > MP/MC burst enq/dequeue (size: 32): 13 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 31.01 > > MC empty dequeue: 32.01 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 24.26 > > MP/MC bulk enq/dequeue (size: 8): 39.52 > > SP/SC bulk enq/dequeue (size: 32): 9.47 > > MP/MC bulk enq/dequeue (size: 32): 13.31 > > > > ### Testing using two hyperthreads ### > > SP/SC bulk enq/dequeue (size: 8): 40.29 > > MP/MC bulk enq/dequeue (size: 8): 59.57 > > SP/SC bulk enq/dequeue (size: 32): 17.34 > > MP/MC bulk enq/dequeue (size: 32): 21.58 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 79.05 > > MP/MC bulk enq/dequeue (size: 8): 153.46 > > SP/SC bulk enq/dequeue (size: 32): 26.41 > > MP/MC bulk enq/dequeue (size: 32): 38.37 > > Test OK > > RTE>> > > > > > > # c11 memory model patch((OOO arm64 machine) > > https://github.com/jerinjacobk/mytests/blob/master/ring/0001-ring-using-c11-memory-model.patch > > ---------------------------------------------------------------------------------------------- > > ### Testing single element and burst enq/deq ### > > SP/SC single enq/dequeue: 98 > > MP/MC single enq/dequeue: 130 > > SP/SC burst enq/dequeue (size: 8): 18 > > MP/MC burst enq/dequeue (size: 8): 22 > > SP/SC burst enq/dequeue (size: 32): 7 > > MP/MC burst enq/dequeue (size: 32): 9 > > > > ### Testing empty dequeue ### > > SC empty dequeue: 4.00 > > MC empty dequeue: 5.00 > > > > ### Testing using a single lcore ### > > SP/SC bulk enq/dequeue (size: 8): 17.40 > > MP/MC bulk enq/dequeue (size: 8): 22.88 > > SP/SC bulk enq/dequeue (size: 32): 7.62 > > MP/MC bulk enq/dequeue (size: 32): 8.96 > > > > ### Testing using two hyperthreads ### > > SP/SC bulk enq/dequeue (size: 8): 20.24 > > MP/MC bulk enq/dequeue (size: 8): 25.83 > > SP/SC bulk enq/dequeue (size: 32): 12.21 > > MP/MC bulk enq/dequeue (size: 32): 13.20 > > > > ### Testing using two physical cores ### > > SP/SC bulk enq/dequeue (size: 8): 67.54 > > MP/MC bulk enq/dequeue (size: 8): 124.63 > > SP/SC bulk enq/dequeue (size: 32): 21.13 > > MP/MC bulk enq/dequeue (size: 32): 28.44 > > Test OK > > RTE>>quit > > > > > > > or find any other performance test case to compare the performance impact? > > As far as I know, ring_perf_autotest is the better performance test. > > If you have trouble in using "High-resolution cycle counter" in your platform then also > > you can use ring_perf_auto test to compare the performance(as relative > > number matters) > > > > Jerin > > > > > Thanks for any suggestions. > > > > > > Cheers, > > > Jia >