From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0066.outbound.protection.outlook.com [104.47.40.66]) by dpdk.org (Postfix) with ESMTP id 0EC1C7260; Thu, 19 Apr 2018 11:21:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=1jKZWA6Hp+oqHjIbC8UBzNilIbB8wGWd/ALkCEjP6OA=; b=PRw0BJO/43I4uAi0dK75wMgGSqG+EyAJHfPG+7CfXA01THWn8A8fWPXbh9UtbqaLsql7QAEzoQ2mNuXNo8ctt9x590zx5fxoVO2ORtfD24vKLrLELa9Z+hQ2pgv+ndt8VgyJN2H3kkj9IjnFZRurmSqKm5qVyyu+oJdkxInCXnM= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from ltp-pvn (111.93.218.67) by DM5PR07MB3466.namprd07.prod.outlook.com (2603:10b6:4:67::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.675.11; Thu, 19 Apr 2018 09:21:03 +0000 Date: Thu, 19 Apr 2018 14:50:52 +0530 From: Pavan Nikhilesh To: Ferruh Yigit , thomas@monjalon.net, jerin.jacob@caviumnetworks.com, techboard@dpdk.org Cc: dev@dpdk.org Message-ID: <20180419092051.GA8072@ltp-pvn> References: <20180418153035.5972-1-pbhagavatula@caviumnetworks.com> <20180418175505.GA17954@ltp-pvn> <291a43da-6c2d-f65b-374d-206a0f674db6@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291a43da-6c2d-f65b-374d-206a0f674db6@intel.com> User-Agent: Mutt/1.9.5 (2018-04-13) X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BL0PR02CA0008.namprd02.prod.outlook.com (2603:10b6:207:3c::21) To DM5PR07MB3466.namprd07.prod.outlook.com (2603:10b6:4:67::21) X-MS-PublicTrafficType: Email X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DM5PR07MB3466; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 3:Qd1rsGo8dXAHhw+dRlASXLuwN2bxJBNAL4+Ujh6aYH5lGnhc/2GDyx2GyIohIAY5XIQhTNtEL5bDLsoi1Ix5NasOFwqB7M2uA2sRFIkz3P1XIM6XiZMs0akf0ba8gV5YoW0KWOzKxBnn2IBxPTYDgXVffV/8acNeh/OYAagSZUPsZem6j4doLfPMhFBYMZbAmyeY/p0K3e/KN5GW+6b99GpDC8mPk0Ey4caCN9F71A13EWGW+NLvAOTpsNc0ZMOu; 25:QhVe5Os+vDa1cAUt47ZXw8DVXKwQ/iTxX6ToGAAo63dn9JscnCSYhPViKFkhFqJw1N+BvdYa8LGW+Rhl3vmkDH1h01OjyBZsxVpC2ipvNzMqEGiR/QgYG1g91mhD5mUbY1BV2WGvOlWZQKybuf+yKmzUbU4WG4Ihr4rSdDTWuQnT53QUINVASjloYJJ2RlEhsGp9A0garSHFblT5H6HU8DSIA7spwRAxAU45pcnBs/VagiQ+xLUefkUMCgGf51JdOS4CbReYt8G4aLdLJBJ8lUZPUKIu5zNcAKYOrASlWlC+aC4Vkf8zGmXKdbU3hmdXad+WnYAxMR6jYyseDc9d8Q==; 31:gYMqcnTrRPCehrLaSwaaqDpSMeRXSH500H3z8RkaoKlftRprJ5KxOeZLDo7To2XDBQbFigaCMuzhr67vmAPqDJ0RZ6/LAKy3EFhurqR+DW+mhqwo0kgNa0HtHyF+O2/+J5JEiKgXUmiI7oDDfbN6/OitZKqQ6lbfmZta2Faj8hNBqdFiuzr6ZtgqRytErhACB62F+RWAh7kNbC5RAV35cAmj9HF9n+NCs+M9zmvXcMM= X-MS-TrafficTypeDiagnostic: DM5PR07MB3466: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 20:UcQ820O9qWLcMzjmVSNoy7gY/mlOKu5zq35jqR2i9Q7BG5id5Za2so04593tRII5+xDebxxUgan5LxbiorAHE0CscrK92k3DovNYhHRQ1jpN181FKx2rDLEcNKccXoD8c22Pk8JFPls6COMTSqSheQQWHqUpsjWhb15++SGKAXsaTYJh83yb3svspjRGVEaTVTWL6aWQHUYQEJhzKLfoN6Spr6xn9nW3XYDlj45MbwqbGi0pZY90uciaJMwWoAmnwyoPBZwBEje3zOdOVzXhurrTaf0KTnTnMJics9/9DXAi+IC0+4mQHQe3lDXh8GpkDjixPFiwJVgvSnK0qtOka5obVxN7LFQXjapRcvSyQZlRjvv1xcra4nzHRmrNAoYDy9G7ukFjjim+/TNT29mkAzR/uiV/MW2aZZkcOolcoxsyzUBrKkjStI4ISi1+eTz49+PhSX0UDMCkJ7bPG6AUmU8EbOYFCiM6rkNLaFem4XUcYDkps9/5lBS35IeLCmi2SQsLhmL1n9m0HyLYC8Yooc11AwnxSbwju4dOx0L13Srz/q429/TGCJVEvu+HyECtYeU2VlXAh8lbNNIrjUnCJjPWXiLOQLuwOVsEf7w4pdk= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820)(131327999870524); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(10201501046)(3231232)(944501327)(52105095)(3002001)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(20161123562045)(20161123560045)(6072148)(201708071742011); SRVR:DM5PR07MB3466; BCL:0; PCL:0; RULEID:; SRVR:DM5PR07MB3466; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 4:NShlwxRw3dbl/NWNlwbrJv4o8o+sb2Yd3+ZMm8ValcjgRkru3B4u5nkWGmuNAGmQp4QtouD1wCT/XGofnBjOE74X100JBsicHAP5j6svFb3oKaB4nHdNx3eL1yHYZoa+qXQwVLs2j1HsSczdIUxQU0sHDi+rhlKwjQlyDw1sS01li/AnCtrKADCJfyD8M3rUsfENLTKJRZLfVuTBdXgDLVoKlsbo1HNOOdjeAJSHAvC6lvIfAhjexut+hIlif8czIDMqcvXf48+/E84kpY7tRrxa8oKxxmUpojQ+kGSbsCAukK5ImA2k+TyF7pLQY81RhKmvMJbW0Wc+O/EmDdUl9n+x9+DyTbElnjehwPhyVLY= X-Forefront-PRVS: 0647963F84 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(376002)(39850400004)(396003)(366004)(346002)(39380400002)(58126008)(386003)(33896004)(11346002)(93886005)(478600001)(50466002)(16586007)(42882007)(53546011)(229853002)(476003)(59450400001)(8936002)(316002)(6666003)(47776003)(5660300001)(52116002)(6496006)(8676002)(7736002)(81166006)(76176011)(305945005)(53936002)(6116002)(1076002)(23726003)(3846002)(33656002)(25786009)(66066001)(33716001)(6246003)(2906002)(186003)(966005)(26005)(6306002)(446003)(4326008)(5009440100003)(72206003)(9686003)(16526019)(55016002)(956004)(18370500001)(107986001)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3466; H:ltp-pvn; FPR:; SPF:None; LANG:en; MLV:nov; PTR:InfoNoRecords; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3466; 23:fugJpvaQmQvwEmDy/zUlJLi7i5kHX8ADPt6/BwVjN?= =?us-ascii?Q?wcAvxXu+WWj8ldxnWb+fp3CENFFdXTTIqL0s+4e27C6p5JohWPAFfOmtsSOg?= =?us-ascii?Q?UE6S1v07j/ZWSEhJln86Jboq0GRx5lI9ynaBoUUM9HrMqNP/zJOOG6wZR7kA?= =?us-ascii?Q?8kV/vOWO+6vJp839JEDNACPM5GaDjBbfaUnhFA385dPJFdC+UcBHzr4teWz0?= =?us-ascii?Q?q2k1ryFkD97M4CEP4rzV6NZmvOUA9KaXzNC4j1zT9iUSJXMO2oBbAjCa9QSj?= =?us-ascii?Q?g5qh/gmYnwDgSf2tyNwKJw4Bxi2v3Wp/i9ZrnEZHnSw7e+govi6hX7uIfFGE?= =?us-ascii?Q?JQJjELA34VOCQubJKs8EokKPwpfr5HCtUtSS8Iy+OLYUUF3ezkPc9FE7YBHy?= =?us-ascii?Q?VU9njNTb+BoBTDa5R6tQiRb1f2oNzqiHV5OkXq66xTyg/5iOtv4OivVXudIP?= =?us-ascii?Q?ipkEm+GB+PDtp8aJro0VN+RWLK8cpT+awAumPgOzJ93GU1hXQoue9UDeSx6M?= =?us-ascii?Q?SiV/JpjiZC7NSLOyvvD3sJtesGas1hlbySMjdRGHkDX6kHLJ5445UQN5ZVgQ?= =?us-ascii?Q?cl35lwsleMybJErx0bg++82aWf5M/7unzdIA+QIZHB3kBJwC67zzqzvAghsM?= =?us-ascii?Q?Ue5gHj9p+UjnDYHOZhpmHtXRC5m43YSZUUtANnKQxEb3Nu6WN93n9UZTA4Ny?= =?us-ascii?Q?NAN3tdgljWIiO48cBmmaIO4SydtkruHbLcQ9dcxPiXf7bM4MjGi4S6eaYQ2/?= =?us-ascii?Q?jucNVIVkc9LSGayGM33ysSQU1UWwzOi6dEt2Ba8M0lXacPZ4rS8gu3fyRbHb?= =?us-ascii?Q?mRtKwVUW/l/KQuEA4nYqIZ+Codut2KPUpC6C5cvRSBYSxN4TsP3EaDMUBIVj?= =?us-ascii?Q?GMu5TgtGMUkP0l0BaVb+RogynyO9qu+hOd9oYg9Sva1UcDjKMAY4n0PqrKi5?= =?us-ascii?Q?eRctbHKiOOI5B217WrOSSKDWdF64IwrcIciRQeKpyifVqxu2OsYJWKkXaOKn?= =?us-ascii?Q?SGjfGgVbgIMrLxVzJNDiaU7YK3xLsEYOi/EqrYcWIq8MtdL92FfYKrMniN/I?= =?us-ascii?Q?uisbu3ohdyyZLJB33N5XcHIwzFkO/87OHoYyNcZMlZ7sIir65AlD21tIC+OZ?= =?us-ascii?Q?l0g5WnhFobE71KOvZsfXqp0z98p4QANM3vBunBbG6p/KIH42VrQ7zdEdO7lH?= =?us-ascii?Q?7uyuj7N8iuDu9nqR//F1YY/w2uibtDaR9PYtsL+YNGn/4j0qgyAEwJFYSP6W?= =?us-ascii?Q?VEAwJ+uwkxanO//bqbE5FI+rReyySRJNAsWmLUZtBhU6wLmXkDl7tJY39fyX?= =?us-ascii?B?QT09?= X-Microsoft-Antispam-Message-Info: ff19Z7uZNxMNnTX2u0MBvHlfY0mnW6I6Ys3+t1iHubKjMIWd8YZe1TVRy/ZYBoiKejciazOm8ezCPNGBH2cXksmLJw+Ba7UUES15qd4RXBSqVQHZXMl+BKjAEWPz+PXIKtqp8S0qS+g9oP1XIUdqPbATSyFQf2MDZKd5gqhn3kh7lVluZX4+1rECd4UDtxhb X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 6:/GXaOKlqDP5LWFMyD7Rxb/wwn7PI3hotWOdwGgcXn/Vo7uSlBo5cp39bV4v5oEy75mDnsoD6PAKXFw0vRx/uN9ckmAP2f7bQKF54sixdgVyEnAKZyMM/OeQJZB9dq4NB5xH+mjQKNIWeDwGfb8eZNaAijP5xMxFAh4dbWucEBSIbr1wT5zHvzpvUAFchd/GaPbUP2YySKvsdbbQQFClZLpubIcj/uGWX1EPBKMBpEV1Qj/jhsWzSEXzvvF7wIKxMXfmyepR+cX4hWq498ClYHC9PJJsbhVDjA+M/c+M0nzjBHY7NT+FdyZj1F7rgdSpfo4s3NanpEtoJsQn7bHAJZHQBNizze7QVSMzIjBOt/PiZV7JbomztUFfwiWWkZhoFq96VI7FIJPp27RmWrJ98kOapid/ziilHqcqBfdi35Z0D4efeuOnB9Rf15Hl9k1Y/1FIXZMijYSt4mFR6CXCPGA==; 5:Q4cvHq9m6Vm8Zpf8CP0YFkvgp8mYW0dHqXMwXgQcmh1m5AkF+TvPK+gcrLQJVI+NBoRNyK4stjU/lSQM9bdFcpexIHG+H2vP9PFywNQ5gZdOedEWlkVlvnTqxRGCNIhZGBZD6RsdICVk6XxF4ZPG+88Osyd+9TwFrAX1UT5HnEo=; 24:NQplSCG9Uf5KJhZQpCK++sqQ+i8B6mX+0Rq3VJceUxoKwwy/w5sWvp6IkAEUAU8xkOOexok5uRcs4W4BpBwJMZf4uAL06lDMSfUCJPQYemo= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 7:jhfx28VBtBSXxpGrCyyLw4eT/pfr2FkFQX1h5gyDKwBN9i0q7aG79TuhmPHXxXR4VllnUo5zako4JENUZS295P3GHVsdqa1+y1ZGrv7dZpO38biCXxy65awrgkAvKKUkhLFE23Lwa1ppyM9j17S4/Er4CTZN9vgE4MbK5AmgvBW8FO6FfryN9kEle876QJ7Y4617MAbbCECiKy0roJtqo4uo67g0VXnJlCeaJ/JmhO4oaWU7UjcV5WHUPzc6QaR+ X-MS-Office365-Filtering-Correlation-Id: bbee9ebe-f46e-4e37-d551-08d5a5d6e0ec X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2018 09:21:03.5963 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bbee9ebe-f46e-4e37-d551-08d5a5d6e0ec X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3466 Subject: Re: [dpdk-dev] [PATCH 1/2] eal: add macro to mark variable mostly read only X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2018 09:21:08 -0000 On Wed, Apr 18, 2018 at 07:03:06PM +0100, Ferruh Yigit wrote: > On 4/18/2018 6:55 PM, Pavan Nikhilesh wrote: > > On Wed, Apr 18, 2018 at 06:43:11PM +0100, Ferruh Yigit wrote: > >> On 4/18/2018 4:30 PM, Pavan Nikhilesh wrote: > >>> Add macro to mark a variable to be mostly read only and place it in a > >>> separate section. > >>> > >>> Signed-off-by: Pavan Nikhilesh > >>> --- > >>> > >>> Group together mostly read only data to avoid cacheline bouncing, also > >>> useful for auditing purposes. > >>> > >>> lib/librte_eal/common/include/rte_common.h | 5 +++++ > >>> 1 file changed, 5 insertions(+) > >>> > >>> diff --git a/lib/librte_eal/common/include/rte_common.h b/lib/librte_eal/common/include/rte_common.h > >>> index 6c5bc5a76..f2ff2e9e6 100644 > >>> --- a/lib/librte_eal/common/include/rte_common.h > >>> +++ b/lib/librte_eal/common/include/rte_common.h > >>> @@ -114,6 +114,11 @@ static void __attribute__((constructor(prio), used)) func(void) > >>> */ > >>> #define __rte_noinline __attribute__((noinline)) > >>> > >>> +/** > >>> + * Mark a variable to be mostly read only and place it in a separate section. > >>> + */ > >>> +#define __rte_read_mostly __attribute__((__section__(".read_mostly"))) > >> > > > > Hi Ferruh, > > > >> Hi Pavan, > >> > >> Is the section ".read_mostly" treated specially [1] or is this just for grouping > >> symbols together (to reduce cacheline bouncing)? > > > > The section .read_mostly is not treated specially it's just for grouping > > symbols. > > I have encounter with a blog post claiming this is not working: > > " > The problem with the above approach is that once all the __read_mostly variables > are grouped into one section, the remaining "non-read-mostly" variables end-up > together too. This increases the chances that two frequently used elements (in > the "non-read-mostly" region) will end-up competing for the same position (or > cache-line, the basic fixed-sized block for memory<-->cache transfers) in the > cache. Thus frequent accesses will cause excessive cache thrashing on that > particular cache-line thereby degrading the overall system performance. > " > > https://thecodeartist.blogspot.com/2011/12/why-readmostly-does-not-work-as-it.html > The author is concerned about processors with less cache set-associativity, almost all modern processors have >= 16 way set associativity. And the above issue can happen even now when two frequently written global variables are placed next to each other. Currently, we don't have much control over how the global variables are arranged and a single addition/deletion to the global variables causes change in alignment and in some cases minor performance regression. Tagging them as __read_mostly we can easily identify the alignment changes across builds by comparing map files global variable section. I have verified the patch-set on arm64 (16-way set-associative) and didn't notice any performance regression. Did you have a chance to verify if there is any performance regression? > > > >> > >> [1] > >> If this is special section, can you please point counter part in the kernel? > > > > The kernel has something similar[1] but they have a custom linker script to > > arrange symbols. > > > > [1] https://github.com/torvalds/linux/blob/a27fc14219f2e3c4a46ba9177b04d9b52c875532/arch/x86/include/asm/cache.h#L11 > > kernel commit id 54cb27a71f51d304342c79e62fd7667f2171062b > > > >> > >> > >>> + > >>> /*********** Macros for pointer arithmetic ********/ > >>> > >>> /** > >>> -- > >>> 2.17.0 > >>> > >> >