From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 506BB430FE; Fri, 25 Aug 2023 11:24:07 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 415CB41148; Fri, 25 Aug 2023 11:24:07 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id CFA1942D12 for ; Fri, 25 Aug 2023 11:24:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692955445; x=1724491445; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=+9/2px9Hap5z6tbSulxyITUghkc3Nikq9LkK3aSOWP4=; b=Ac7M61R2w4gAgCqCGzBAtNBrj8+EmJ6XzK9Q32xfoMNUyAeTyWW1C4Iq 9Sdx6AjMU3sQsla5gjxf/9vy4mVuZJ/5IVuRkOPZ3YPHNtLSDTXl8x1lH Yg1UIqgRGTsU6Jv4r/2jxV6PLUj2/zChkAH1ciPvONtzaq9IlFOAYOK1v DLtODG7AVt4Mx65/p9YLuPlrOT6j3Dpa6QTU1UHiL9Srp15WIlPaRWH16 HTUZ4m8RAKEzTiyE6Rph49mgO3A+ISfuciJMFkecRHB0ItoqBqs8lhwdf jQrhVmVB8dJ1/IFoq4cmXHwPOUS9b3eHiLnQJ7kC2VTh4OfXBb7tNKolf A==; X-IronPort-AV: E=McAfee;i="6600,9927,10812"; a="377413295" X-IronPort-AV: E=Sophos;i="6.02,195,1688454000"; d="scan'208";a="377413295" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Aug 2023 02:24:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10812"; a="911229159" X-IronPort-AV: E=Sophos;i="6.02,195,1688454000"; d="scan'208";a="911229159" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga005.jf.intel.com with ESMTP; 25 Aug 2023 02:24:03 -0700 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 25 Aug 2023 02:24:03 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 25 Aug 2023 02:24:03 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Fri, 25 Aug 2023 02:24:03 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.174) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Fri, 25 Aug 2023 02:24:03 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XZc6LYmLN+YxuJOweUBB6J3ZbxBL0k3VIgoOoc/hevibZSDPtEP3jXieYI/YCb/svcw0xdru7ZbOkEcclEzS8iccq5s+ps4GQH1mbPh+WCeft+E6KxkKtdqr+dcbTZUxgZy3LuvbfceC4cBAadCqm6eGlgloFb8Ua+u8F3zcOYegV+7Vcm7PfCZl2xYJI3/NJzt8YNE4Jymw6GkPbj7wdj6zinRPTXKIOVzBfVc1+58BiFkkOkJXAsPJgSKulJmdeN8b++1JgdQl8291ZC84eDsG5yYhDAQAJKgtGofK0U6u3eHZGjENH2U/GXFxZ1+9J7qjw3LFtfzXsqPjXD0iqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RO4DeNrE5AwNY59dv6jSQjONU/ZPqBIPkP3a4efkT30=; b=TxpaHd44C6W3jrtuMTG2Qsxf9D5L/JoZhP9xtBrrU7fFQcoUSHamxFaVa//Gzggg5jJk/vUZIIPQA+WPzk6UxE25iD0IWO4r3X+yn+UCqAZfg5qZLvRiD6QxQHDQfPoKys99Oww1hELjyVGZZPAiv0PGdvip4avKxQ7t0RWS8Bri2DUtANNbd+y9UQhF83JdIGmkL+xV5QbzvnidYDjpzL27wV4kAvQ7Ezy0cNiZb67fRNKuIfOht7EhdF8z6cGmuX1uMqvfevaWQhanZi3HLiaKhco9kdlZ13RMAckKQXVD1Mo/jxP3F/+5rX6EiyznZFW3/wQ+UutekVc4kJrI7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by LV3PR11MB8484.namprd11.prod.outlook.com (2603:10b6:408:1b4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.30; Fri, 25 Aug 2023 09:24:00 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347%5]) with mapi id 15.20.6699.027; Fri, 25 Aug 2023 09:24:00 +0000 Date: Fri, 25 Aug 2023 10:23:52 +0100 From: Bruce Richardson To: Morten =?iso-8859-1?Q?Br=F8rup?= CC: , , Subject: Re: cache thrashing question Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D87B39@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D87B3A@smartserver.smartshare.dk> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D87B3A@smartserver.smartshare.dk> X-ClientProxiedBy: DU7PR01CA0005.eurprd01.prod.exchangelabs.com (2603:10a6:10:50f::28) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|LV3PR11MB8484:EE_ X-MS-Office365-Filtering-Correlation-Id: d2213a6d-543a-4163-380d-08dba54d0407 X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3GL7cduje7EbzMnR63UwWZ722IJJJQA0UVjo4sfShoHoQTCa/rF6pv7zftUpvWKtsxG+G6+Xi/XSGiMK2L8tbft4yf6eIvSgIfJ3fYaGsMEL1/++RNz4wuxsjquB47BlfP5pN35ZaO1GPyhnKOkmiHFjLCfhO0sh2izH6+GnJhP2BBTV1KQLZlSpPQseyuj4sFMEJla8psKgn8uO7I/TUVyJldncMFvzMEJ2OWhkDrd01T55QXyx+8yiNuW7gtCcXUwogGPloFkUVQrTasraGo+bp6iMQDfXZb2xY2azb2DOFG3zfcPhMcV69NpliSI7BiwHBwxk2V/uXEiakdpgw38KPtBipa1OYCGap8fZiXUiIzsn76SQ7ipvtWaaZDzYpAN3ux2Gjw0JlkA7hXK5JAPgFgVyamq+J9/ARI25jsz62Xjh6nhjrle6UzxSjddlO1O+CbOGA2nKzhCucIQYLiRxZaJTI9OF2fSx36dieWO4sePfQf1NBjWBvWMhOt5zLFeRfLw0Sc4G81plYDva8Atfs02CVU19mVLX09lCB24= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(366004)(39860400002)(136003)(346002)(396003)(451199024)(1800799009)(186009)(82960400001)(38100700002)(8676002)(4326008)(8936002)(41300700001)(6486002)(6506007)(316002)(6666004)(6916009)(66476007)(66556008)(66946007)(86362001)(6512007)(26005)(66574015)(478600001)(44832011)(83380400001)(966005)(2906002)(7116003)(3480700007)(5660300002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?7XzB3/HIqsnS742wE6EkbjvQcOlYPMR+Br3IuXMZdYADYHstNKETk9njwQ?= =?iso-8859-1?Q?2MaxxrW3BRdAKmWeuaFQZpvv8TT595mfyOJcH+GpRFsKx2ISSkqvjniNyz?= =?iso-8859-1?Q?7e/uKK3pO5ZKlKHuGb3VO26Fs0PtUd5EDjWyIThc8HvoMx3w1mnn4KSykM?= =?iso-8859-1?Q?0nbT+tgXBvAP16OhVQBkWSLdwLZn7J9ZYu2TwZg9JtQGxtZNbuNqW0xLJB?= =?iso-8859-1?Q?MXM85gAVexOiLXWsX5zoMfrBjsrXYIMEEbw7Ht/FmQmPKpk1Bh840eA5HO?= =?iso-8859-1?Q?3MVLVHurmyDrgy+oCEAOE6zsD7Ez8Qi7B/LwYAgpOqYGdJfMpy6xY96Qhm?= =?iso-8859-1?Q?tHXvws8A3YKAEReUARsuSnA17V2GOvBPBepbXfBGhnWG1ksTtGJC0dxelj?= =?iso-8859-1?Q?KOe1/t3xr/5EgwxVddBYBPBlvflM92XK62Tg0MTjqaJootV57WfWlC+9l+?= =?iso-8859-1?Q?xmXWsmPGMWH3gZdvJIZ0IvGQJWXf4c9WMSC9JXgJgHu2a8zPYG4f9LLr5b?= =?iso-8859-1?Q?riz/wihznfHDGkuIckR3A0dLaqsXbkj9j9jW9+6mRKDj1oEGFi7jLsu1Ij?= =?iso-8859-1?Q?GCax65KZm8rncACbw/isYUVGgI4jccFa+sJSe6Y3ShPjJ3GHAwFZY/bhLt?= =?iso-8859-1?Q?sQJZdbDD0FKLk7Tae0+Vp5BPeCcF+xmyXQ/DZHPD1Yw9iAku6t5h4OJmL2?= =?iso-8859-1?Q?ObO6BdoubED60hzURcJpq3dlmgIx2fMtsyRmm3NCNlRxO1OeoBYBdKH/Oy?= =?iso-8859-1?Q?InV5NsjRoeCaaby3jRPMBqIkwkc2edq0iAGvHBFr/FpskZJGUw+q+yuAYV?= =?iso-8859-1?Q?ADeUpsS2aMzD5X0i1WKbTpl7MWppS5JlMOBVd0OEqnNPe/e9HqNknMcoma?= =?iso-8859-1?Q?okZPhsRtRFIGoFrmI243HcK0CxJN3kLOQoqImjv1fw4qCICJ5ohEmnSC/+?= =?iso-8859-1?Q?nVWQcQm+8HHIIQyFGFIAGfKcCZ6ESqkbxOPwHX1cWGsNEWEDohbT2tTTch?= =?iso-8859-1?Q?9l2em+ojth6dGXtR9fUmw1nWcaFO49UgrpAqaomrdVaTXpLQGX2h9A7vk2?= =?iso-8859-1?Q?JKNqLB2gBNb3TCwpwHhCJpX1tbp2tg/zl0q/oX7XUN6Uwfo1SaFpmw9Vrg?= =?iso-8859-1?Q?cGB5ImyY5tOQkea2lvyqEblqHDanhYbheFswT10iQt9OF706zKPom7EDGZ?= =?iso-8859-1?Q?4ox2rxtXDyBpX/c5iVxVZTebBS9my/ie06SkM0pBZWWaJ2/3cv/vrHYK8y?= =?iso-8859-1?Q?60uiVM6JWF7TItFNgG9WP5sYwgdiYsfNgULuedJvSDs+7WKq7AD6ePvSrq?= =?iso-8859-1?Q?vwmtY70lC7062YVaMppT3jqqz/2zJy08A8DfJdh89a7tGHXBGv/oCNPA1A?= =?iso-8859-1?Q?Oa4jdfQd8fMyY4j1xIOsQ5KyZ6qU26CkXf8sHOBDkHuZsVHqtUKgrQZTzs?= =?iso-8859-1?Q?PkAEKm0oNMqZff9xbhIe+4SeNPsZmhuDLd0Vpi2n74g8/u8dmK9MRzK6oI?= =?iso-8859-1?Q?0c+exhyZuL1xtdQSWdfZGXThHqoKFu32yoUiN4pfWfBJHWdUFs03FhJUlP?= =?iso-8859-1?Q?vH2ZQVyqmJFG+n6KE79SKa+RV7mFx5peHGPi48QeM0c8/hR2IIl2I91ECH?= =?iso-8859-1?Q?gZw94VWqYjRYp6Yg6s/vY1MQjn4Pyb/LWiw9zc3DiBC5wx9nYbIqdLSw?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: d2213a6d-543a-4163-380d-08dba54d0407 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 09:24:00.4991 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uhYt0Er58PR2juciWZwIqsjQq+NmgY4DXSnxMFujJ8+JCnwSQpp1wRdBUuwj0tfBbk8JK5Dkkphk7G8ldUZ448667O+TiKSTDD+IFR6uOOk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR11MB8484 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Aug 25, 2023 at 11:06:01AM +0200, Morten Brørup wrote: > +CC mempool maintainers > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Sent: Friday, 25 August 2023 10.23 > > > > On Fri, Aug 25, 2023 at 08:45:12AM +0200, Morten Brørup wrote: > > > Bruce, > > > > > > With this patch [1], it is noted that the ring producer and consumer data > > should not be on adjacent cache lines, for performance reasons. > > > > > > [1]: > > https://git.dpdk.org/dpdk/commit/lib/librte_ring/rte_ring.h?id=d9f0d3a1ffd4b66 > > e75485cc8b63b9aedfbdfe8b0 > > > > > > (It's obvious that they cannot share the same cache line, because they are > > accessed by two different threads.) > > > > > > Intuitively, I would think that having them on different cache lines would > > suffice. Why does having an empty cache line between them make a difference? > > > > > > And does it need to be an empty cache line? Or does it suffice having the > > second structure start at two cache lines after the start of the first > > structure (e.g. if the size of the first structure is two cache lines)? > > > > > > I'm asking because the same principle might apply to other code too. > > > > > Hi Morten, > > > > this was something we discovered when working on the distributor library. > > If we have cachelines per core where there is heavy access, having some > > cachelines as a gap between the content cachelines can help performance. We > > believe this helps due to avoiding issues with the HW prefetchers (e.g. > > adjacent cacheline prefetcher) bringing in the second cacheline > > speculatively when an operation is done on the first line. > > I guessed that it had something to do with speculative prefetching, but wasn't sure. Good to get confirmation, and that it has a measureable effect somewhere. Very interesting! > > NB: More comments in the ring lib about stuff like this would be nice. > > So, for the mempool lib, what do you think about applying the same technique to the rte_mempool_debug_stats structure (which is an array indexed per lcore)... Two adjacent lcores heavily accessing their local mempool caches seems likely to me. But how heavy does the access need to be for this technique to be relevant? > No idea how heavy the accesses need to be for this to have a noticable effect. For things like debug stats, I wonder how worthwhile making such a change would be, but then again, any change would have very low impact too in that case. /Bruce