From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6C92E45962; Wed, 11 Sep 2024 18:03:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 53C674029E; Wed, 11 Sep 2024 18:03:46 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) by mails.dpdk.org (Postfix) with ESMTP id EA4F240267 for ; Wed, 11 Sep 2024 18:03:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726070625; x=1757606625; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=0QpWcZc399+qZvj2sAnUVZkQVufJfM9F6wmqJ3Ztz3E=; b=az96S94HiBpCmkE4xy0UGd724kya4QN0igVvTPx76eIcgFTjM0DthHlF rCW7NFP/IO7tI2pT1NcAo3YH/aWBsGnU+2CPxT+Pc+EBNyVXrNbXloC6v F55pKQwaubmEs2H+WFBxo9GCaanm6ZzfPUYgaYOITcHvmFQbIzoTLly0B nPCK8jZ75eKgaIEo0hkJpI4NTNY4FO+zEQR1eL/N+N8ofz3B77BHFGcQ9 g8uYAy1sGOeeR20a+A2NuPYheqTjhNOKTvdKi3+UoGrYDlBjK5PNn1ZA/ XFY2l5EarKfaf1zO9dUJiF7nhUG9AlIfh8V7zDKhOp0GyLodBkc1NSOSu A==; X-CSE-ConnectionGUID: cKNP2LhrTq67lrorKVHF0w== X-CSE-MsgGUID: WOsSmVudRZKXvv6uAd4HbA== X-IronPort-AV: E=McAfee;i="6700,10204,11192"; a="25024358" X-IronPort-AV: E=Sophos;i="6.10,220,1719903600"; d="scan'208";a="25024358" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Sep 2024 09:03:44 -0700 X-CSE-ConnectionGUID: mi/u4tQhQTm8bDWPUeCAFg== X-CSE-MsgGUID: HcVF9CcXSZenj96CoCnWLw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,220,1719903600"; d="scan'208";a="72011051" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmviesa004.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 11 Sep 2024 09:03:37 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 11 Sep 2024 09:03:33 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 11 Sep 2024 09:03:05 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Wed, 11 Sep 2024 09:03:05 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.170) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Wed, 11 Sep 2024 09:02:06 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=m0OXsS6AdW+EZEMiFZ/t84y0P/hwwGgE1J19Hfw558je7IcIr0TkmKSZ6eW+mVHfxNLbTRAjqTlrp1ofmhWOXwykc1QENRmv+VOETRgToSyRzWG/7heU4076gEQ5itN+rXs3MK0jAVW1M2mL1REAAh1M28WEgTsmmxyp1zz8hA6GYlRE7azB4A+bCjzDzHfjmbucyQU29Iyos4xmqpLbjKas+m3QbqW2Gho9d/PRUGFJ1VIoJJjua4+/LaMBC51pnVtfM/EiIUS1pkvR4VH9ID4kddlBTESc0XqFy+BVkm28rf2vs8M1xWFjyluoMrUdeWZWy5a4/MR+hr9q/xJ/gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S0RZfAafHtoFHlcUDtajmLLG6GRAACuVYLezhZblLZ4=; b=S6WhOn2anTJcZsxdUF3Ofa/jcGklWFOeAViF1Nr+4lziNeYJAXNHSJCyuvOEdxKyWPKhKFXB8O5izFcQyKeEegqiWFK9taGw2zI4uBtXYlhHu33lUocUYJnFslKRArnfGuA2lX9aZ5FAXm8I8qI7p9625tuBjYf/1AsW/MNsN4eKLQDNNQkvhrxhcFc9IOvf+83Dvn5JQk7HKcj18Flccn2VQjyPvFp6xbtAql8JQOFBRzAX6hn/tlrIl7H7H2qlLLJM+4jNKS/y+J0ZRhb5ISgprzgcSshl8FtO4F58vxMgkpWMkXlf7VfKRVwKHyKoNElfpuzmFjLISZmTX/G1vw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH0PR11MB7423.namprd11.prod.outlook.com (2603:10b6:510:282::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.26; Wed, 11 Sep 2024 16:01:57 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b%2]) with mapi id 15.20.7939.022; Wed, 11 Sep 2024 16:01:57 +0000 Date: Wed, 11 Sep 2024 17:01:52 +0100 From: Bruce Richardson To: "Varghese, Vipin" CC: Mattias =?iso-8859-1?Q?R=F6nnblom?= , "Yigit, Ferruh" , "dev@dpdk.org" Subject: Re: [RFC 0/2] introduce LLC aware functions Message-ID: References: <20240827151014.201-1-vipin.varghese@amd.com> <45f26104-ad6c-4e42-8446-d8b51ac3f2dd@lysator.liu.se> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DUZPR01CA0119.eurprd01.prod.exchangelabs.com (2603:10a6:10:4bc::19) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH0PR11MB7423:EE_ X-MS-Office365-Filtering-Correlation-Id: dd70305b-798a-4fdd-5b65-08dcd27b101f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vwpPjlCDYq4cp4zo3pEGYvqPNUXeYK4t0OaiwtKCTnoFRo8kuPwfpjA+svoj?= =?us-ascii?Q?/GX4Ch5BNbjZu7Y3o/zIoqvuWac8HrqXj2ntAx5mP3usBKMtw9DILGgdvi5N?= =?us-ascii?Q?1Sufn6E9Bvlj4TQWTU3Yp+WegV5DkBtjIcbDbyD47t83+UPukG2E15Gfuh47?= =?us-ascii?Q?hoTgpQa8dZYMh80fuQsSBg58agiLZ3sQ92/DZAsdnlBxMeixttMsGxM9rS+8?= =?us-ascii?Q?rFaqrcLtIs2Y0w90bsY430ZxgYapSEqJx6wV63K1a9Q29FhVqRCzNZgq8f1O?= =?us-ascii?Q?mjnmkWtqgwgWet6yVk6M3ZbMSYWyL8OFx9vGEpBLIwvOE7qFPxyQFiQiLNqx?= =?us-ascii?Q?M9wS0+EM5MN/bB2OnSDe5ma4bmVGZhk0UCLeiyW9LTt08PIBIAKE/8X+HPp9?= =?us-ascii?Q?CjcBRd8Yzpa0xZ01Btc1XQcJv8mv6CpdYnnrYoD77UiV/wgRqf3j0xx4gE7h?= =?us-ascii?Q?DbIuCpy5xQs216WdUCy/B1vaKzrO9HAiN5Noe7G6J0C5GmuMCYFKl3b8HgRi?= =?us-ascii?Q?fq4trx64v4k8zOq13aVXzBj4jtKtKDeNvvRKWq9FAh30R79tdQgCrfDnLyzj?= =?us-ascii?Q?fb9iu6G14cq6nZDZpE5CIiU6btm/7LOptPp+LlU7JFH/w2+ThNxaxhnd/3QE?= =?us-ascii?Q?YuaN40svV/UeytWQmYEbInwGgx6BeyOjWax1lk0RSf+zupBijrzSrvrTQ6qB?= =?us-ascii?Q?/oKVxLEsTJUFlf/S5qYJrasOYk8if85f6gBO2Jujng/x9QPsxNv3jQjXJU7y?= =?us-ascii?Q?VVD4eFt8c3rAxbwvwqzZRLFJkuRjF3+cI1iNjlyiRvNmgMJMwDx0j+PU+n/h?= =?us-ascii?Q?4b3P3dWo8MIdSC2Fb6dqD3GPWEkgt4RUo6C87DSJJk+JTfQeqY9C8x5maijr?= =?us-ascii?Q?e/CkwQrb4rctEvb81Iu6ix2B8zFTX9SYAMFYgvlIigfbcdEYy6O6SwOhu2Qd?= =?us-ascii?Q?XlGkhGj6vcc6TwB3HdaoHmVduh69lte/sx9/og9OU9+lq8u2LSg4iRQkk0rP?= =?us-ascii?Q?dqBok7h5RwYFfpZ8Wg/FKabs7utxyC85EB533j3Cw9X0s2UOCmM0bBWbF+tG?= =?us-ascii?Q?RvMZlysAdlDHisA000U46arJza7i74JaK2/n93uNnGg1fwovFa0dJ0JdpKlh?= =?us-ascii?Q?C6en7Ts0Y7KYHXzkyXmfA17njRSgziUoSMEIgXKeLCNedN6976BwqtqY/YCG?= =?us-ascii?Q?Jxg4awdBTsb5/qtvG5nSENufpzT5p6JHiG8zSlNSzRK6JriA9Nxk6qyYMUpC?= =?us-ascii?Q?WbaRUaM20mkCQ9B3QpQfl/SDe9V/ccjGuhNRMCo1/Xj0Z1+OXbogpOdORty6?= =?us-ascii?Q?IR2Wo1H3gVtrwAFup94uwmSThjE/qV4Bljg4oQyupu6Wgw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UsRfGjeMJf2fLAuvLCd3ciwVb4xZGmeZCAdP1KFa6fteIYfOsyVobQZRle+8?= =?us-ascii?Q?PJzpPUxtpBSb99qcu07tEWszdPhUbHvEZvxRBJQuXiwd+sdJ7l5YPqtkEfFZ?= =?us-ascii?Q?nbrkTIAeb+E3RgqlhW0h+N/mAoptJw4thKdzJvqcF8yHjjMO/1aVAf73zpTn?= =?us-ascii?Q?lH2RH3sPjPoUuC75gepiAS3RTz3OogGWrtfbhymttMHVBP1BQIjej/6kthsJ?= =?us-ascii?Q?YRQtYwybvlJXvOjR8lwdj/qf3r6yzO7rpbolofCV55qCTTmapa4ZKjF0KU3g?= =?us-ascii?Q?clCG+/7qunM8kqR3t+QqlSTyRKWr9d4d6psdJul9c37lDY3sRgWS6XuLIz3f?= =?us-ascii?Q?OFLpWzUP7vuih0Q1RRSnrQADItu1VzCgUHTPk9hR++B34R/SNIsKIbU1Q9oT?= =?us-ascii?Q?x8HdinEAhdPOMZHOz6sMPRG3maKFdOsWrOOJpQ1x0Yqst35WqZ9lRrlRZ0J3?= =?us-ascii?Q?oNGqN9Btn3Ot298FEcLfRZaNrYjJnr2DGH4bT3TVud9LvKKDbOT+rOQYo6jJ?= =?us-ascii?Q?Br8TlyOQLHPqBeekwNPVeAS5uCC7+K/cc6j054o+db8ZpS9wtuflLtOkc5mo?= =?us-ascii?Q?5MuOmQBe7OMhIT9q84vrO90+RahaJoUzMTW4jp1IRFU/dw/O6RWvnV4oP5mU?= =?us-ascii?Q?K8ctDilB9uyO1yL8Gfvm3z21wpZql7/Vmg6H+l8Jfwd8zKpKA+eHe8Td7jQz?= =?us-ascii?Q?bzJZ2ioz/zCZTyfU79M5odWh2Uaryf08dy4+160N4RgEGVFEKquaKTfSexFQ?= =?us-ascii?Q?O/Hw8JSBQ9C/ZeHbFFk6RPqCK4JdYR2wUZ7kO+5WxRqOknInAwKZ3E6vCcgj?= =?us-ascii?Q?ZDf6irQvF8Vl9vDx6hDI/UOQcm7RUVPDZDOStWklp6EK78mvQuSflbrm4rFJ?= =?us-ascii?Q?gXiMnY8mndlhws+8u0zmZhbw811M3o4h70IFDWMjIiW3PRWY1Ux09/1s/w0E?= =?us-ascii?Q?FG+UCFxK/hg73dJtdCWw9xSUhSIabLSqegN6zWI8PigqODhiBbTV9qnAPHi9?= =?us-ascii?Q?Nq6DSQdkm9+JT8T13h22hbGWMx/wWLCfQ6+nE6CCyu0xLuqzMe1uVnhOCyNQ?= =?us-ascii?Q?Px7plsPcn1q3HT+BuXFmiU/fOLmjQnmi2JB13XfyC58YrpwvObXoV4HO5ppk?= =?us-ascii?Q?W+JO8D5bwaFWdkblSZphDolk/G1nQpchTga7Ip5p8GbnWG4dxX/fzy0wf5Bh?= =?us-ascii?Q?SqY4jeiptBpF9jftedql92Zht9wJK7CUHd/R+kQeuqfy0VXZWiVFFrJbDQD3?= =?us-ascii?Q?6oH2r1TZnI20TL2nv4AFgqh2w/ftgGemPnlF1x36FH85ga3/J1Co+W91pO6z?= =?us-ascii?Q?oXHL9mVogGQQnH/9hfbXQan/d8B+0OFs7u4r/0R+9fABxDhgOMmElvmCW8ho?= =?us-ascii?Q?fpBWR4P+ysiz39nRBIEfowOPjNEaT3dmklWYGWV7S8IQEFQK14JFmACGM5i6?= =?us-ascii?Q?4Qz+UsfejiggypvvIeXFzsp3VnaKO6q9D2aYoa/L3cPIBfUsBfHm9aKWR/Iu?= =?us-ascii?Q?hIu9Q6aP5bDj3HGJD/oYh8cSArF4FAK6TCwr0A6SvyWmeesKOCQb/456BXL5?= =?us-ascii?Q?osf60TYlZx/ssXws7lDQIFRECqU/DQvBPrNUEx3SI7CQKBg3N57TPmwhD2Z0?= =?us-ascii?Q?xw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: dd70305b-798a-4fdd-5b65-08dcd27b101f X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2024 16:01:57.5393 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MFNoFSLomhZx4xWMzXqVWupmeAts3aBhaxbwIq8mHMHtk9Zv8d6hY1xcLdVMPPAQLI5YP/RHfJ07DXdR1T7nYbzLGnNAo8Y1htXp19DOfIE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7423 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Sep 11, 2024 at 03:26:20AM +0000, Varghese, Vipin wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > > > > > > On 2024-09-09 16:22, Varghese, Vipin wrote: > > > [AMD Official Use Only - AMD Internal Distribution Only] > > > > > > > > > > > >>> > > >>> > > >>> Thank you Mattias for the comments and question, please let me try > > >>> to explain the same below > > >>> > > >>>> We shouldn't have a separate CPU/cache hierarchy API instead? > > >>> > > >>> Based on the intention to bring in CPU lcores which share same L3 > > >>> (for better cache hits and less noisy neighbor) current API focuses > > >>> on using > > >>> > > >>> Last Level Cache. But if the suggestion is `there are SoC where L2 > > >>> cache are also shared, and the new API should be provisioned`, I am > > >>> also > > >>> > > >>> comfortable with the thought. > > >>> > > >> > > >> Rather than some AMD special case API hacked into , I > > >> think we are better off with no DPDK API at all for this kind of functionality. > > > > > > Hi Mattias, as shared in the earlier email thread, this is not a AMD special > > case at all. Let me try to explain this one more time. One of techniques used to > > increase cores cost effective way to go for tiles of compute complexes. > > > This introduces a bunch of cores in sharing same Last Level Cache (namely > > L2, L3 or even L4) depending upon cache topology architecture. > > > > > > The API suggested in RFC is to help end users to selectively use cores under > > same Last Level Cache Hierarchy as advertised by OS (irrespective of the BIOS > > settings used). This is useful in both bare-metal and container environment. > > > > > > > I'm pretty familiar with AMD CPUs and the use of tiles (including the > > challenges these kinds of non-uniformities pose for work scheduling). > > > > To maximize performance, caring about core<->LLC relationship may well not > > be enough, and more HT/core/cache/memory topology information is > > required. That's what I meant by special case. A proper API should allow > > access to information about which lcores are SMT siblings, cores on the same > > L2, and cores on the same L3, to name a few things. Probably you want to fit > > NUMA into the same API as well, although that is available already in > > . > > Thank you Mattias for the information, as shared by in the reply with Anatoly we want expose a new API `rte_get_next_lcore_ex` which intakes a extra argument `u32 flags`. > The flags can be RTE_GET_LCORE_L1 (SMT), RTE_GET_LCORE_L2, RTE_GET_LCORE_L3, RTE_GET_LCORE_BOOST_ENABLED, RTE_GET_LCORE_BOOST_DISABLED. > For the naming, would "rte_get_next_sibling_core" (or lcore if you prefer) be a clearer name than just adding "ex" on to the end of the existing function? Looking logically, I'm not sure about the BOOST_ENABLED and BOOST_DISABLED flags you propose - in a system with multiple possible standard and boost frequencies what would those correspond to? What's also missing is a define for getting actual NUMA siblings i.e. those sharing common memory but not an L3 or anything else. My suggestion would be to have the function take just an integer-type e.g. uint16_t parameter which defines the memory/cache hierarchy level to use, 0 being lowest, 1 next, and so on. Different systems may have different numbers of cache levels so lets just make it a zero-based index of levels, rather than giving explicit defines (except for memory which should probably always be last). The zero-level will be for "closest neighbour" whatever that happens to be, with as many levels as is necessary to express the topology, e.g. without SMT, but with 3 cache levels, level 0 would be an L2 neighbour, level 1 an L3 neighbour. If the L3 was split within a memory NUMA node, then level 2 would give the NUMA siblings. We'd just need an API to return the max number of levels along with the iterator. Regards, /Bruce