From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0075.outbound.protection.outlook.com [104.47.36.75]) by dpdk.org (Postfix) with ESMTP id DCD8F7EBF; Thu, 19 Apr 2018 17:18:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=v7lqXq8EB17AtXF+SyO6GhnzXFcaDBE+L+AGrpk33c0=; b=HfFlI90+i+cmvx1714WRR2k37DrjvpOeF/rDqd27mqRUIV/cnuUa60OcmwEvzH20/bjB+mpNos0X6V6gkj7fy6taHyaaQ5ZrB5+YFdfJml6nObVlMkNMeG26HeUWhu7nRu0bw7pufwjc2ybTOis0o3X5sTFsL6aQWGg18BJ7g+o= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from ltp-pvn (111.93.218.67) by DM5PR07MB3468.namprd07.prod.outlook.com (2603:10b6:4:67::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.675.14; Thu, 19 Apr 2018 15:18:44 +0000 Date: Thu, 19 Apr 2018 20:48:33 +0530 From: Pavan Nikhilesh To: Bruce Richardson , Ferruh Yigit , thomas@monjalon.net, jerin.jacob@caviumnetworks.com, techboard@dpdk.org, dev@dpdk.org Cc: dev@dpdk.org Message-ID: <20180419151832.GA7962@ltp-pvn> References: <20180418153035.5972-1-pbhagavatula@caviumnetworks.com> <20180418175505.GA17954@ltp-pvn> <291a43da-6c2d-f65b-374d-206a0f674db6@intel.com> <20180419092051.GA8072@ltp-pvn> <20180419120958.GC11352@bricha3-MOBL.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180419120958.GC11352@bricha3-MOBL.ger.corp.intel.com> User-Agent: Mutt/1.9.5 (2018-04-13) X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BN6PR11CA0003.namprd11.prod.outlook.com (2603:10b6:405:2::13) To DM5PR07MB3468.namprd07.prod.outlook.com (2603:10b6:4:67::23) X-MS-PublicTrafficType: Email X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DM5PR07MB3468; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3468; 3:rxmUGop7D64nHI2a3flwvJ607ll9b6Hl4+KxRJ58+YX/0m7eAIP3r2YXud7OakXYDrLDYTtg3ljXNMuQVUNSw+aFBrk8d0cZr3b5N7tpLSNXX1Sn/nzJUIfMX9b2TO4yGZXNpJUlubhVeJU0F+L0Cyq0yMelHdRht6AxywqKJh0v8IjsYiwp+kYQ9FDKAezRLhZkXoZIzDfxyQHwScxj61TZgH9alol6FzgOCU4jc3KZOkp3IwSOgsXCCqdtnA6V; 25:QpoPapadNPg3FtbQKyQcU1bKCOLpfLf0xLCRbJwk9nKsNHnhV6H25H71nP/7bjAo87yCaBm/6hRoCD6mVwUjtoXh3fm9NumeLjurag/X1oX/l4WNLZjpkOkmVF589jYRcKNCMo7je2Y5RXvCBC8sgdgR3r6naAIYFN8xxoAIa2arzWTyWzT8HaYooVDNbSa6lX1NZTs8JH/HJ7WyP7eNwEZ4J/1cXBmY36pyqkk8Yg2A2iMw2Y7ykpvaZFv6SqDM5a1IM+K/IQi9+v3WOI0HFyKEyevzzDX+QEE2hVafWTd5idKuQAIxCWSnGHX8uIXYT1igcgYlUVHyAuD5Mtox1A==; 31:w9k74JWz0OcKKi8mkCqrSQgN2lcAfpxOcRadCOjKeDlTHWiHXOh6kLqtUkcE7rHZORTtAKC3dGUKCPGbrsrfs1/S/Gn1rCX0XlvbgIync728SDVQp5iuJktWuoxIyuZ3VT933EDP8y3PnOl0CN62nk3jbEMPTaafmeEw6YWhYMBDadIRGJBAv21cDVFai8zMIoCZq9ee/OHrdls3VatY/T4PWP9yl0DUa27Xe8KKyps= X-MS-TrafficTypeDiagnostic: DM5PR07MB3468: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3468; 20:rZ2wWPCtuC5GuN2Lh3Zaq2LcJeyzXLM23nIwhBKYXX1IIFgYWXDHTo3XDMgsrh6gi6o6tlkRFvRtxJA9mpL+O6yyUD3du96tYf/egaT2Zes0IxdY/w3R04H7VcHSAAZh+rU3fMGHtdIK4qpOeV0n9DjtvFaKRzQQ9I4Ll6bkrrgs9m3SyfcF+fzlpum3Og21R8D3RzB8MRr/5KhLIXuHI5jc3v50hGHg9LwvZUcPvUEeBAXExHPm8PWLD5B0xvYS6SqHT94FbBry0j4tcbajVFdBcygxhe+kZuPWTWYzuL6XVUN487Xnm+HRQyteByixXOnlaOv34/TRcKciSKoUuovPe3J6GTNPavxxNhfWpBURVlw4uvYjJUy+WDQvgeJZu1COQ415FsHvKkv0n7jA1VxpNxk70z5HEkh619zFOgVXZDb2GKyx9a90Wr7OQG0WQRyNmQNCR8dUkMlutK413c9N4z9Z1qTeGK3FltHvPqvfQ11np2fWrbuVmokEyiNMlVQEH3vowIqpmZax9fmBabXgwlNCRD3YZCBmcGbudiFpgqzCus0rytDSSzKGQfZWEy9W670YVH5eZ5pRMZJixt98vYbS/LO3mnEXq9U/dww=; 4:ErsJ6qHziitShjToVerwGtW20lZlQzPQ7XaXBmC9hpDKIuPbS0pKxCHZYWka5ZiMDsECUE6Mj0lIMuUtzgWoXD/QN2c84Ov85UXk/FkAmiY72ZUPNuE+G9OpwG7JxM5UQLRdrv/yTQvYEbMMr6NzlzqHcKjFjObygf2tnfS0WNCSQ8AydLQ0t9KHsV9NvKmZPiom2VxBsdsp/92BKso0wlZCWW29dpNRlkmf2d4DqlqI4smIQKCe6/84sTZUE8Qr6y6+P+r0SYkgTGF2dhWAI8fOiOai3e4Aa1xAcwQVBij0h783XMEK5Q0LLDhpEVZh X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(131327999870524); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231232)(944501383)(52105095)(93006095)(6041310)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(20161123564045)(6072148)(201708071742011); SRVR:DM5PR07MB3468; BCL:0; PCL:0; RULEID:; SRVR:DM5PR07MB3468; X-Forefront-PRVS: 0647963F84 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(366004)(39380400002)(376002)(346002)(39860400002)(396003)(5009440100003)(6306002)(7736002)(4326008)(16526019)(42882007)(186003)(55016002)(11346002)(8936002)(25786009)(9686003)(305945005)(966005)(446003)(26005)(33656002)(8676002)(6246003)(229853002)(81166006)(53936002)(33896004)(6496006)(72206003)(316002)(23726003)(59450400001)(50466002)(76176011)(53546011)(93886005)(3846002)(2906002)(58126008)(33716001)(1076002)(110136005)(478600001)(16586007)(5660300001)(6116002)(386003)(476003)(52116002)(956004)(47776003)(66066001)(6666003)(18370500001)(107986001)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3468; H:ltp-pvn; FPR:; SPF:None; LANG:en; MLV:nov; PTR:InfoNoRecords; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3468; 23:+aSRqYU/J7zm2ooGcHJMXQxTKtcOZXm3/+aI22+VX?= =?us-ascii?Q?6//l90Ky8VFAqi2tjA5sjpE4IbQKT6Dm1MUiZfaCt5EyjCA3Z2IQp08dJUrv?= =?us-ascii?Q?Ew+FHagOvlRdwfwAxnHyZKnDwpclbTB+DkzyvzKRAGUWKkQaaw77sriTv1yz?= =?us-ascii?Q?6/mgK+VpMReTvGAxm9XYUwHuFAbts/mRj6brThWn5cKVCBHxfCK8QfZK8NGM?= =?us-ascii?Q?5swb1LHVhPKcAK1ewiBoDQ79qOfvjhoP5GvzmnlBGTPYm0FX5Vko1CZHGQm/?= =?us-ascii?Q?m/WWINYzSAzAUNOCvjMy3Lt5XkH7dCMkrLQsUSxMfjVKO262nks5F7qVv1QG?= =?us-ascii?Q?u6nVCmH8nVb9eY7rP3ACUE28JV1hdVCuZdlsCpHU+YzYpt/vpJg8KRH37AZx?= =?us-ascii?Q?FBFQFWzLd+d+QMsrMoSI6bbwdYjaZvxoo+7EVIL1TZQbF0+hg7xsPxGK8stk?= =?us-ascii?Q?IupZBy0b6pddMkfN05QADaagPN0VQKNNpBUZGiyM+nFIiDQQDKs9lCdczXVE?= =?us-ascii?Q?pT3NrMYSayUc2lwg6riw2IEx6wZsjqWrriTE1zJzzJmA2ThPoSatxwT3Za8K?= =?us-ascii?Q?htzwcmjlH3hR8dsdsOFvfiMGez1L9VKeh4uokokW8/Eu5v47Hb2L6MRNT+0x?= =?us-ascii?Q?SKUCtLvj7nm0t6MUDC8U+CJru/BqQA/5cnmgx4jlV8UER+usZf4RUkCPaq5K?= =?us-ascii?Q?v9TmqanmFbFvuppBrz2keQIo8HIGFzDrS1KpPiMdA4KY47KwKPheTETBZPis?= =?us-ascii?Q?2jbiHLsh1J34VMGiCcfVsjrTuUD9qtrHMH0ymjY4BHPiKyVo1FijYEtQwybZ?= =?us-ascii?Q?WRIDQ2tyPM765PgHslNLW68Fz1j+PWyDmcNDiikfYW+AkugNYPN6FK92mcR9?= =?us-ascii?Q?IgGBwtTWzWwK68X4x9nR2qBzpWiV5c509EhlX8Rj6kVy16gErL2OOQE3rzwJ?= =?us-ascii?Q?esP2ZCIPWK171cGa6gzLxf0HTw1+7Iam06reJdWU+HcSdYR2/YklXxDErnTD?= =?us-ascii?Q?WQliBakktK1oBNt3YrzwHFMUf8UP8Ved5p1kDWqHm6uWB8yKdqKwDEgH277m?= =?us-ascii?Q?UpYuH3aak6Jk7NIH1XriwJou9JSQEb3oMcxSoUWVoxQR7TBfjoeCYoR3wPi5?= =?us-ascii?Q?NDJ3OYh2FOEB5uaXXn2h3/REwxer9BePsWySdNGJ+sE2rEAS0vPxV+0BBOrf?= =?us-ascii?Q?phv3+Woi4rO6dOXJLDgEmLeIXIDQTHa5m07rSwDYhGyBODN4cvqtbu59vs+h?= =?us-ascii?Q?tdgsPiOFA9CWwyxMElAEpk/wrPSV7t2hYly56Z1IdNnIFhL+V/Far9Whtnqf?= =?us-ascii?Q?q2vsVAzDyAuuLGpR71miII=3D?= X-Microsoft-Antispam-Message-Info: TDX9EZ69hADcKRGZRbL3XBcsQnfbKmlKWfXSEtuSAeTKtAMBorUknEp/a1nATa2LAu/k+ey4W4cFiral6srB6ym8JC39CtIwIdWjINXz1AzwlSWZDAR2aaSu/A2gL4llp54R6QFQyR77nomjw00bVlnAyAw0fwVaiDMARDUUzRyqQL3xSs01kJACfMREtB/a X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3468; 6:s3sGrZdtYLAJGxfIAaUGePFstItJLrwmZT+EXzlqGfurbYSemNOpWmtcdDBD13Xh2G45+MU8ZSveqg0YiWUjHHPybtT64GDW7udVfbqo2LN+4m76tWjOZzDMqdNMJRQj18MepmMfLmbwI4rVvA2yT7yU1MgOUEeNPW2GRzv4g1RwF4Y7zYgaRSlWh0K2OVwPiRG318hGyDLZKpKYgH8T5RpFzoNZUWMYS6d1UPQBjuG8STgbFHJjab61Hyg4j0A/G7hgJpkiXE36wjd0rV4hv8y0HIQufe5yaZZ0Mip6yGVGgAuk7XieJuIK79JvUBFQQDrVJoGftYAzzAGUWAGh0X92vGayqPqC/eEjxtfEYpfgHWHGYySJmQnk9Y7VT6oKhw+/PGKo7SZqtkvx0kCputbv7zzkuHd5D0uuh2BEn4ubQoo+/27UYPsVfGhSBfS+laGFzwAFUyM8QXHFKv89QA==; 5:J9ZN4clsoWa08GjkJMpxZ33tYL9OYm1FdxZCuue4EkUxxOXKTMj5LgycZjxZwc92DriYA8H0eTrOHDiy2TwEnpPL/IAHMw/ysCN7lShEx38cOyBFwk2QgWH90wZAi0Pa4FsmrCWD7iAlCSlPBM82h8We/jKH/Z0Q2olb+UvO1Sg=; 24:9Y8YNDKTFheiWDX/b7wTTSuzdpRul0rsQHbvOWX6NBs5d9Qc/jVL50C9kOyT0mDjeAsO5Rcr6touTrB1ivUrAPNQz8Tbs2crnz1sBDUoNQw= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3468; 7:D3iNpmu+DmrxrNV5GfStdCGcK9zBUpmK9EQJZWoCBXI57Nn7wmXe1oClJli/DrLVgDFJqabbpzChm8gNlFFsKM+3gWSbYjZ0AmW43PGsvn62nnfJA/mrRUN6qwyHcRebodtHRADElL4XcK6txRkuyN2m1ZRFnYAOKslYOkbKBzJNACfKd+wZyiUDxxRAQf8UyYcQcqbfbxf52Rzqns8P415vOubASCzl5ttXbBLYuG2ScorhedtyC56bz9jidsEH X-MS-Office365-Filtering-Correlation-Id: c4f34e07-d549-4f9b-4b44-08d5a608dbdd X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2018 15:18:44.3996 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c4f34e07-d549-4f9b-4b44-08d5a608dbdd X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3468 Subject: Re: [dpdk-dev] [PATCH 1/2] eal: add macro to mark variable mostly read only X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2018 15:18:54 -0000 On Thu, Apr 19, 2018 at 01:09:58PM +0100, Bruce Richardson wrote: > On Thu, Apr 19, 2018 at 02:50:52PM +0530, Pavan Nikhilesh wrote: > > On Wed, Apr 18, 2018 at 07:03:06PM +0100, Ferruh Yigit wrote: > > > On 4/18/2018 6:55 PM, Pavan Nikhilesh wrote: > > > > On Wed, Apr 18, 2018 at 06:43:11PM +0100, Ferruh Yigit wrote: > > > >> On 4/18/2018 4:30 PM, Pavan Nikhilesh wrote: > > > >>> Add macro to mark a variable to be mostly read only and place it in a > > > >>> separate section. > > > >>> > > > >>> Signed-off-by: Pavan Nikhilesh > > > >>> --- > > > >>> > > > >>> Group together mostly read only data to avoid cacheline bouncing, also > > > >>> useful for auditing purposes. > > > >>> > > > >>> lib/librte_eal/common/include/rte_common.h | 5 +++++ > > > >>> 1 file changed, 5 insertions(+) > > > >>> > > > >>> diff --git a/lib/librte_eal/common/include/rte_common.h b/lib/librte_eal/common/include/rte_common.h > > > >>> index 6c5bc5a76..f2ff2e9e6 100644 > > > >>> --- a/lib/librte_eal/common/include/rte_common.h > > > >>> +++ b/lib/librte_eal/common/include/rte_common.h > > > >>> @@ -114,6 +114,11 @@ static void __attribute__((constructor(prio), used)) func(void) > > > >>> */ > > > >>> #define __rte_noinline __attribute__((noinline)) > > > >>> > > > >>> +/** > > > >>> + * Mark a variable to be mostly read only and place it in a separate section. > > > >>> + */ > > > >>> +#define __rte_read_mostly __attribute__((__section__(".read_mostly"))) > > > >> > > > > > > > > Hi Ferruh, > > > > > > > >> Hi Pavan, > > > >> > > > >> Is the section ".read_mostly" treated specially [1] or is this just for grouping > > > >> symbols together (to reduce cacheline bouncing)? > > > > > > > > The section .read_mostly is not treated specially it's just for grouping > > > > symbols. > > > > > > I have encounter with a blog post claiming this is not working: > > > > > > " > > > The problem with the above approach is that once all the __read_mostly variables > > > are grouped into one section, the remaining "non-read-mostly" variables end-up > > > together too. This increases the chances that two frequently used elements (in > > > the "non-read-mostly" region) will end-up competing for the same position (or > > > cache-line, the basic fixed-sized block for memory<-->cache transfers) in the > > > cache. Thus frequent accesses will cause excessive cache thrashing on that > > > particular cache-line thereby degrading the overall system performance. > > > " > > > > > > https://thecodeartist.blogspot.com/2011/12/why-readmostly-does-not-work-as-it.html > > > > > > > The author is concerned about processors with less cache set-associativity, > > almost all modern processors have >= 16 way set associativity. And the above > > issue can happen even now when two frequently written global variables are > > placed next to each other. > > > > Currently, we don't have much control over how the global variables are > > arranged and a single addition/deletion to the global variables causes change > > in alignment and in some cases minor performance regression. > > Tagging them as __read_mostly we can easily identify the alignment changes > > across builds by comparing map files global variable section. > > > > I have verified the patch-set on arm64 (16-way set-associative) and didn't > > notice any performance regression. > > Did you have a chance to verify if there is any performance regression? > > > Is there a performance improvement? It's seems a relatively strange change > to me, so I'd like to know that it really improves performance in test > cases. We had a performance regression of ~200k between 17.11 and 18.02 due enabling dpaa/dpaa2 in default config this was due to new global variables being added and changing the alignment. Moving read mostly global variables (logtypes/device arrays) to a separate section helps when tracking performance regression between builds. > > /Bruce