From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 962D3430FE; Fri, 25 Aug 2023 10:23:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2800440A7A; Fri, 25 Aug 2023 10:23:10 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 8020D40695 for ; Fri, 25 Aug 2023 10:23:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692951788; x=1724487788; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=c+Hj1Aye4/KVPP+O+2BC6YrxolIyrcMd2Ce6p+YXCeQ=; b=iWYhzR+bYjhUFA12g3n8j9rv5jhSfuJA85R/FG7wERxNd9O3N16Zza30 ukPECMQ8HKsBXXibJKmHvseeAVe8PuoAc2dLoyTE+Y0s6TnWobYkTSEy7 oh3uNsDZ+f5PRamRvIpN3+vwu5ZIvGFsmx/u72yc7wWquyReyKnUvlcZ7 /xRTMiv5jSMHqNZfsomTdfdv1R1uPk7RUKqZlC6LWXFMYbdWf3vJ3DfVJ zUNJL8INFxMhqIGZagp+A6Px5+JQgvNwLQpElpCce54owSh7Aw+B0XTL+ 9fIIZw8sfEar3Fn9ZJT7fzuqT7JPi5GWrb2g0CFEQ/Xc8fgAI62RiMDDT A==; X-IronPort-AV: E=McAfee;i="6600,9927,10812"; a="377404088" X-IronPort-AV: E=Sophos;i="6.02,195,1688454000"; d="scan'208";a="377404088" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Aug 2023 01:23:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10812"; a="766881633" X-IronPort-AV: E=Sophos;i="6.02,195,1688454000"; d="scan'208";a="766881633" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga008.jf.intel.com with ESMTP; 25 Aug 2023 01:23:07 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 25 Aug 2023 01:23:06 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Fri, 25 Aug 2023 01:23:05 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Fri, 25 Aug 2023 01:23:05 -0700 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (104.47.73.48) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Fri, 25 Aug 2023 01:23:05 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lXbMmWZ60JPV+hQ1BGvIbz9bAKvDOzI/HUAlNAMjm4FLtaKKaW9tCqTIxzV1JbDbbGkh+1A0koXvqcEcvYBXut4y6F2OJCB1UsAqpvgWrCuw7VakL95/V8Sg9QTqqCA5mvC2IeV/Nia0uHd4Qc5xb3JZATeeOF7+KTaCUjpl7UjRUd339NKcrfYgA1oNz9k/50P/GNWwgmXlsgTq9URWdyxHgOG/35oVMGornct14ZG6zt8ZhYq6oyFguFSpYu1qCwUriBylJ2LtyuZSfthg8ZkIaog5LUN6y7O+ReERhZPmsx6/QV/l2qPQ4a6JLnu2ltHVfzww4iGJFtoawtmwoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c+TXESrjyC1XD9ttMucEes+hITnvhJIS/eGN1fYN5U4=; b=mGRAxPrCu4vFYg0++IUupH9M8A1Yx3WrUv+2FkXpjA5x9YH4sq1KiQWlUHgbPs50ZFEIr+HbN5NaC+r+JD61VaFlDGh/gW07oUTaptFKjlvq1dmqIMFphBhctgRqX+3q8/K01zUHW6y6zZHivMIR/4Ne/Er7ctmp4C7votJAchD6hCYAlpgKaYnbrqec4jhA1BWlgXGINDQUjnr7iZ+MTCv/O0OgfScUvU3ZR9XJhVoXLTeqpHy8Q1MfG4NJY7SLK035YWihECHCiHVcwG/iXxp+nRbdyovFcCFtuAFNc0yNi9PbPa1m8QT47BNqLkkxuANcYxMrdkc83CIucKlb3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by CY5PR11MB6137.namprd11.prod.outlook.com (2603:10b6:930:2b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.29; Fri, 25 Aug 2023 08:23:04 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::43d1:af60:464:347%5]) with mapi id 15.20.6699.027; Fri, 25 Aug 2023 08:23:03 +0000 Date: Fri, 25 Aug 2023 09:22:57 +0100 From: Bruce Richardson To: Morten =?iso-8859-1?Q?Br=F8rup?= CC: Subject: Re: cache thrashing question Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D87B39@smartserver.smartshare.dk> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D87B39@smartserver.smartshare.dk> X-ClientProxiedBy: DB8PR06CA0010.eurprd06.prod.outlook.com (2603:10a6:10:100::23) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|CY5PR11MB6137:EE_ X-MS-Office365-Filtering-Correlation-Id: 83a24c46-0c5c-46de-d052-08dba5448034 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AxPw63HoMxiJ195WxJoRAghw4h8eojhnTgbUCIdPMta7ZerjGY1/nabEiJ4wPtCOZEafy7IGSQMv4og2d4qxcpDDhIF9tOgTWxM+eHyd7Bn5owxfcMvjWQkzY6G357ouZk7ezUJ7Qj+7QPWv070+9uoUhh2xWSBo1mlZNDS8IXzQiSbckDs5juB+Oc1gi006MdbHrJ8WP0npuMkAWj+DONCMtO6ofhOrG+Db9H3V7KqrJLnQhTROqk0Ftv9B3Eweo7B85D5RFO7cOwY18l/K2l8D1P9P3moLtM76DK6wU+v26OBfEmpFytxWj4wOPbEH5K02findTDsXjqRJXK3cP0QKIRkNNt0RCXGZV0mQCpdHaYhM/oWfpLOQTT9lumTZpk0baZ0pt1hAlFyfulrD+VCz2bYG2b70zjRD5hegUlxECUzOwLvk96G1eGgfgS2KMF9jRKAdzFrbAWVBCO2wQFUUpla56i7PyZZDfB39J88UJhRuLnpdhseYPHWvEm0aPuCJ4nsOkuFprngrrs9umFtvRfUm/MQkSRfOAFywtUw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(346002)(396003)(366004)(376002)(136003)(39860400002)(1800799009)(451199024)(186009)(44832011)(83380400001)(6512007)(26005)(66574015)(478600001)(7116003)(5660300002)(3480700007)(966005)(2906002)(8676002)(8936002)(4326008)(82960400001)(38100700002)(66476007)(66556008)(66946007)(6916009)(86362001)(41300700001)(6506007)(6486002)(6666004)(316002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?5L2OmKQNjG1p7PgmjbN4xNPfgCwdRC5sTlSCSz2aGdul49qiPeol8KLaTe?= =?iso-8859-1?Q?sOI9Wrbp6UajyrV8tn3jOFkg1m0EArFAknIHArTF688RB+gqw2pyOjK1M7?= =?iso-8859-1?Q?VapXuJEwki7ehMZq8upB7+bSjA9JvOl2uDUpqqK+c97Jn2dTDlHQxe7WHi?= =?iso-8859-1?Q?d6Hm07S1mXgDA1Nrv3cqmSVRpS2RUqyx45Th1cpkCJO+CG9PL/BmSXhLMQ?= =?iso-8859-1?Q?v2mMoELtcCbRWqVbtqwuP5UBSC5xLCBOXeeWWyuTT743L0y1cTn3rsALfC?= =?iso-8859-1?Q?G8CJy4xo6DQFyNnb/aVNml/DR0tshiUTSFDpSqape1S0d+PIxb1zY4YyP0?= =?iso-8859-1?Q?vLyHPjQlYUYmVglcHp/3oYLUJeAUvjSy10qTXF3SQvZZclwFaFfU45WFQm?= =?iso-8859-1?Q?yx4N6qU0C1JkObp6eBQVex18L4+FesFC8mzz0qiFzdFJklAWWjXqdotBlo?= =?iso-8859-1?Q?tH4MM8dGEnbn4M6e1+ayjpOQ+qb8JNXZiRTn3BO0Zx1d3BJBZXl4bc9D0Q?= =?iso-8859-1?Q?ELsPi9b6C5WR5iCD5SieaTZB7eyeF/3sTWxuDo50M5h2o8BwJtNumMdctq?= =?iso-8859-1?Q?stb0GWk4442u6JYKeLFCaCuQmfArn45BPUJW9yBW4fdFksU6UOFADU6xka?= =?iso-8859-1?Q?SXZFAqlXQoaeo4adpo6NN0AsMPfKsEa+3SVon8ir6f8oaKE5pT+qS+4aIf?= =?iso-8859-1?Q?MBhhE4m58DzIEiCcq+lQWJbgy4R2ccSQ18Yscnc30EPgkQwrV/vsGCkj2R?= =?iso-8859-1?Q?pzREMCUEmmRAzkJ8ok4q2cw8Tk9EglzV5/OwbxVvX5TDSKNWHUH9jsDfVn?= =?iso-8859-1?Q?A37Yc2ZQoMBBpXqBZ3UMurq6SquvieveJlwGrVfrJl9tZMYKbq+dKKmS3s?= =?iso-8859-1?Q?SdDx+DjRfvQhB/KchPyPQWpb0QPDrNBZz8O8X4StdBf1rEj8EQRM8rMqyF?= =?iso-8859-1?Q?GjoFwireaq5nT4hmXiWqmiS2q5JH++bmK/L2Du8qdeRg5W8N10hemiO/35?= =?iso-8859-1?Q?XJB6CZTPdUi1YyGk7SjbbY7JklAWGxZ8oxulmLMxNDQ/WvRXrs9K2mrtgk?= =?iso-8859-1?Q?CkdON3Lm4WnbhCQ0z/UtfZ0oGrPEk6NG2wdENbYYtAZA/nN/mzD9xk8uZA?= =?iso-8859-1?Q?cJhUJen0ngBmLmKwvIWjnq8T4cgDJq1eOmRKjjL2BfHUkFTRlIDItSZ4d3?= =?iso-8859-1?Q?mv+IggjhFTEPR9klIQBpXopm0cMjzarriP7nUDMSanu7dsNM/o6kNV+T3c?= =?iso-8859-1?Q?4OuXBX2EMyImASKYGKSHYXoMWlWe3gdXKa22fHh6IGKwD3+K777Wr225Dk?= =?iso-8859-1?Q?baIYadafCs7dmZJT4OHSwrOkU5CM8ZDroj0SKBR5g7zIbHxuGV3hkfV8ph?= =?iso-8859-1?Q?wiHf3pCapOqJTgCr5cenO6WQGGv9v92gjz0T+IoaIQJ3G10wY/UL7YjkW1?= =?iso-8859-1?Q?wDSTNW79d9AN7kQ5FGCPU2oV0enIDE1Iej1pvAb0AWDex8wTUUpzoVqm66?= =?iso-8859-1?Q?FDks8DZBmkEHNPQBwpVMARslemuiBlUGwv/XlrzzzVg1L4LVYfk5kKkJ6/?= =?iso-8859-1?Q?p5H8PYu1Vjb70oCIfLH6VLMwhydT/D34YPcI3Go/2/mj8WsyXdZ2o5OvZL?= =?iso-8859-1?Q?GPCHS3UbvOef+P9bdzUgAcndNM/fCfEq5p4UXUqqTJLxJL2H1/s0YarQ?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 83a24c46-0c5c-46de-d052-08dba5448034 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 08:23:03.2071 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cBXsP88qpeOY22dkH8IJzltIe2hCXA2TF3hrltKgj0dedO1xoKp7bdAPyNWFREQkdIMbHs8hvFckcEpLSBqa+8Md7a3rVohMVVZ4bpo/Qao= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6137 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Aug 25, 2023 at 08:45:12AM +0200, Morten Brørup wrote: > Bruce, > > With this patch [1], it is noted that the ring producer and consumer data should not be on adjacent cache lines, for performance reasons. > > [1]: https://git.dpdk.org/dpdk/commit/lib/librte_ring/rte_ring.h?id=d9f0d3a1ffd4b66e75485cc8b63b9aedfbdfe8b0 > > (It's obvious that they cannot share the same cache line, because they are accessed by two different threads.) > > Intuitively, I would think that having them on different cache lines would suffice. Why does having an empty cache line between them make a difference? > > And does it need to be an empty cache line? Or does it suffice having the second structure start at two cache lines after the start of the first structure (e.g. if the size of the first structure is two cache lines)? > > I'm asking because the same principle might apply to other code too. > Hi Morten, this was something we discovered when working on the distributor library. If we have cachelines per core where there is heavy access, having some cachelines as a gap between the content cachelines can help performance. We believe this helps due to avoiding issues with the HW prefetchers (e.g. adjacent cacheline prefetcher) bringing in the second cacheline speculatively when an operation is done on the first line. /Bruce