From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6E5AF4596C; Thu, 12 Sep 2024 11:17:39 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 28A6A40E5A; Thu, 12 Sep 2024 11:17:39 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by mails.dpdk.org (Postfix) with ESMTP id BC893406BA for ; Thu, 12 Sep 2024 11:17:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726132657; x=1757668657; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=NUFvDHYGLewiiBiL4W0xzzPgq0mGgYcTGBTdgtwngXU=; b=N4VA3RMyK+LYwis8PLvF5qwRJ9hFsUNVOaRVCqKqM8BY2TTidm+ZePGw mk3KvaIG7Pr1I1wiEn5WgZuTn7eNTD/abV9+qqxept/FWBUtVbz6kUOQn DsckPndOBkqL3yI7hflkY7dZPDUrbsbXdmJVLoIPYofTIbnHoU/nsQAeX RF96vIAmXOXsjhK+0rpWYTF0AYSWALQ4hmlrpLx6z0zdt6PTyxVWg6jw5 adqiIFjR1mT7GOmW1xo+sr0cJjIKFAU3iSYTSQXIFfnxy9w7VpqXEVhHx BmNkThxmJsj1IvVvybRSSHZxsX8VoWsQMw4naSZDjEb4fA0hQKqB+KshF w==; X-CSE-ConnectionGUID: 0nun0TZUR8eRZZXpCIY77g== X-CSE-MsgGUID: JopWSR/ITHeIOgSlmKKRkw== X-IronPort-AV: E=McAfee;i="6700,10204,11192"; a="47486050" X-IronPort-AV: E=Sophos;i="6.10,222,1719903600"; d="scan'208";a="47486050" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Sep 2024 02:17:36 -0700 X-CSE-ConnectionGUID: 1M1/tDHhQAahfaxxA14E/g== X-CSE-MsgGUID: oL+XYHy7QvqnInBdL4KykQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,222,1719903600"; d="scan'208";a="71761018" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmviesa003.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 12 Sep 2024 02:17:35 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 12 Sep 2024 02:17:35 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 12 Sep 2024 02:17:34 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 12 Sep 2024 02:17:34 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.43) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 12 Sep 2024 02:17:34 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=irlFR0NnCGv9agqriNQgp5jyYmrqIPsIkt9qPTFDUxZsaGvisqM1ywiC9S3r5aARok31QJW9fSyFclZuEGtS3HzUx5ehMXlAc5ob6mQ+s1hOwOVWfHKz+9PcUppv2urzSBao91VP9GpUJrn84i3xO4mujWTfnIZxrk8v7SGOmzss1RZ26hLdKqK6/8Si3+fzxdSrHVMedQ7ZxYEdM118kcFDmyov8V/I76ORpYakC25gSD2wLabHWD+U6MhY0eAviFgZD6E0kEFzeOcGNEbgq4fGUrAO4ClpwHn+BSEIdP3LCAVZxkcBkWiGx/G65hhpW+7ucyEOV6EeJTuPERZqmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vKjGm3C/obp3dHqeuT1+aGQLcOhd0kK//4HgNH2MA+8=; b=kdVz7hkilM4jcK49QEYwVcDJJIVvF0gx08uVOOpdhecnC6EabElVo5aw6lyRF5me03pFAR0SlX4vM/181A5dbKRd/Ji0SIU48ciQggiNEPwEkt4Bh+d6aKv+xo89rbcMdbqdss7bqkQy391TAzwjChjd0zSPc36z6rTnU/6Nl09Yy07ZbbpDzOk4EG+c5cbG4mzAQ91AKScCY6svLz0ffNwz4+4gg6AcMTJqeHatpt9bYZs2ik4wwVPXTVquHd383mTuk6v12GwV8yM8VzhbetPI34T86H3mMFOfgb6YhsylAyLCzVbXvdidAyj96Jltk0F8D6uJ2eBSfXijhr8lIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH0PR11MB5063.namprd11.prod.outlook.com (2603:10b6:510:3d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.18; Thu, 12 Sep 2024 09:17:27 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b%2]) with mapi id 15.20.7939.022; Thu, 12 Sep 2024 09:17:26 +0000 Date: Thu, 12 Sep 2024 10:17:20 +0100 From: Bruce Richardson To: "Varghese, Vipin" CC: Mattias =?iso-8859-1?Q?R=F6nnblom?= , "Yigit, Ferruh" , "dev@dpdk.org" Subject: Re: [RFC 0/2] introduce LLC aware functions Message-ID: References: <20240827151014.201-1-vipin.varghese@amd.com> <45f26104-ad6c-4e42-8446-d8b51ac3f2dd@lysator.liu.se> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: DB9PR02CA0007.eurprd02.prod.outlook.com (2603:10a6:10:1d9::12) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH0PR11MB5063:EE_ X-MS-Office365-Filtering-Correlation-Id: 2cc5e6b4-7b18-4270-7409-08dcd30bb7a9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?B?a29WQktJUkZYdk9HaXBSa3dmNE1NRGVCKzVEZkVYa20zUEp2b1hUUzhsWVFo?= =?utf-8?B?MCtPMEx2R2N3ZXI2R3NVNU9JMUdIQ2kyY3R0eGZBWkYybUs0TDlieHZiTUtZ?= =?utf-8?B?eXlEM3lyRWR2cE9NWlVUWDhVbm9lVzg5MWk0bnY3VkhmQ3Rtb1paLzdWcUZV?= =?utf-8?B?eisxcnlyU0U0UTd2Q2JBZEYzemY0Qks1NEJ5aTlUT09OUW1rSzB4SGVyeEZo?= =?utf-8?B?aUZTUU1nV3hCT2xzZGhJM3Mvc0xMYlVGTGZSSHBBOXFEczJ4OTNJR3Z3RHpB?= =?utf-8?B?RGVZUnFMMGowN3k5bGJ4NnJJb1ZqTFNhbk9DWlVkN1RtN3JObXhjYWpaYU1k?= =?utf-8?B?dFgwZFZ3TUxpbStvTVJUSjM3TTM0cEh0RjFEc1BmUXFlNnBHNXBKVFBPYU1M?= =?utf-8?B?Nkl0RG5QY0NkSjdtZm54U3U3dTBLZWt1MGlaWHdFVDVtWDh0aVNDTHNsd0hO?= =?utf-8?B?ZFhSZkVFSXYrRk4wU0wxQXZGVjlZSHplVkpKY0pNc2lXMXM2cjBjQlhrM01Q?= =?utf-8?B?bE5PRHVsT2dnRTJISnZwVGoxcnZZNEJ1elpGejZ3Q0JkQnZFVkc2ZmVWTGZV?= =?utf-8?B?eTMwWDJacFBXeld6OTJzT0orN0d0NVVoU2dBRXkrZ25UcGVPc3N2NXVXMm1j?= =?utf-8?B?WnpHY1lDSC9ncHUxSk9Bc2ljSG1lMGdRZ1NneEhOVlowZFladDBEdVdPUzRI?= =?utf-8?B?WmJmU2pXN2ZQeUxsSzZNYUtSK2JvaCsrWnlheUlnZmk0SXZ3cUxrempYWk1S?= =?utf-8?B?dXlFRXFLd0k2TFpwQjFCMEE1UGpUNDZpVVNLMFpGR3VxWkpnQ3JnWFJxenZO?= =?utf-8?B?bXJDS1VjeTl5R1hwVzJNYnQzK0k5dGxrbWc3MU8xTEpXUVRBamRLZFlUdXo3?= =?utf-8?B?S0h6MU9UOGk3MnNzV3ZFMldTTHZFY1JlVkYxRkdsU0l1Qko4czZsSkJKMkw3?= =?utf-8?B?WUkrSnJUOVpOaFdSb2lrWUJxR2Z2T3pPQjhid1NVNUo2d3BqZzVLOXBLeDQy?= =?utf-8?B?NTF4NTQ4eUVYY21GTFJ5dE9JQzFDSWJwNUltWDZFc1RZajZNNGJTeERzMnN2?= =?utf-8?B?NDE2M28yTHhOUTkxQXVrUmJkOForY1hHWlViL1IzNFhObk1ZdjFHeXpBV2Zv?= =?utf-8?B?dlp0WEFtYWFFTjRRUG5NSU42bXc5Y3hweFVScmg4NWJUTmZZM1ZQcHVuL2lP?= =?utf-8?B?V3NnelRIcUhEU2pRb0VDbG02OUprODBOME80d3cwaDdsY1d4bEdoNXFSbzlS?= =?utf-8?B?VnBpeHByYkpBelJPeXk0Z21ZeG5MQy9TRERtS0RRQ0xIVXkxU3diTWNLTkll?= =?utf-8?B?TDVRQkFsWWNwU3hoSUZkUXF0WUtwV3FSOU1QanA2RDQ2ZE43MGs0ZU9kcU1p?= =?utf-8?B?Skllb3d0cHlDTUg3L09XVzZBZmltQzJoT1MvWmFBSnkvU09KbXQ2M0tyUkt0?= =?utf-8?B?N2dTdTFnVys0VEVnd01SV0tsREFGaGZzbnFMcUw3T3NrcW9pd2I2WVpwNXBj?= =?utf-8?B?SUNLSTYxRVdwNjFpdWZuREpFY1YvbzkxdS9uMkJwaDhFYlRVQ0sydzRibkxs?= =?utf-8?B?RjlmeE9hRkYvaUwrcVFiRk1UN0RnOThtZTRtcEdaQ1pnNG1qWUpPUU5BM05I?= =?utf-8?B?dXdYRmN4L0tSN29YdkFEVkxvNnBsV2hRVDJ1dUZnZFVJK2pPQzU2ejRKMXNE?= =?utf-8?B?L0F0QnFFSUxuMFZaTHBTbU8xTWZDMWN4eEpKVHhTZUhjeUZtejd6ZHRET21W?= =?utf-8?B?ZkJDeUxrSTcwV3FYeFkrN1p2Ui82aDEwWjJkUEtkWG42MGtVeGV0OE4zbEF2?= =?utf-8?B?V2s4Vys5d2tUaEx0d2RtQT09?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YlozTWh6YWZNS1VUMEtiaTRSb0pvRGRwMG94akxvZU0rNnNXcmt6S2lXOTcr?= =?utf-8?B?dmRLU1FVenF0Z3A0YVZnNUtSckE1YmtmVFZDMmlKY2VXM2lZS1lNeGk4Z3FC?= =?utf-8?B?Qis1bnZoMVVHTnV5VDgzOWJLZnZhSUk2S3h6WXpxQWp1aGNITXdVSVBVWWYw?= =?utf-8?B?clRXUmNqSDMzS2pma3E3bkRLUE9nNzNOR3Zja2ZidEhxTzkwbG5ZdUVYd29L?= =?utf-8?B?Y1JPNzR6K2hjQTVoTlpKSTNmbkhQNkN2ZTNPbWRjQ2NnVDVTejN6RVBTSHlD?= =?utf-8?B?eGg3MVhhUGZXTEhVMFJQbDZqWSs1SmlPeDJJT1p3K2FwdWNjL1JNWDcvanFy?= =?utf-8?B?MzJNWjV3TjJHRFNoQ2ROZDVKV04vREtraGNoSXk4ZGhCMnMrOUh4aHo3M2FD?= =?utf-8?B?Z0ozMFNnQ3ladDN4RG81Z21VQ25QdEt2UWtjbURPWit1dW9pZmEyNElORTN5?= =?utf-8?B?ZjJCNE0yZGZQREdKbS9yTFdDdHpWUzJpQnA0UHdEZEFmOGVnc0hwdHZLbnRT?= =?utf-8?B?SjdKMDNiV1hXaHJTUXlJUkdVNnloaVFYd2dZbDlneVNaVkNNYWlZQzdmK2Zi?= =?utf-8?B?RE9vaW5KRFI2MVVQY3N1K1VlWk8vZzBDN3BrOHUrMDZBV09sV1FZSVFGVnBp?= =?utf-8?B?NnN6azBWREhFVlhBaXZhNGwyZHhqSDBkbFhiMkZONUQyc2xUZDFZNTg2WEYy?= =?utf-8?B?clBGNjhiQndnVU9oNmc2dVVHOGt6V0s5bHFzWWRrT1NOazBnZVh3V052Tzh6?= =?utf-8?B?MlFCVC9tNkJYMkJVWEo0MVdTaG1nZHdpZXE0cFJjRzVZZ2ovRCs2M2Y3dXpr?= =?utf-8?B?T3hTcERmRityTys0aUgxWmRHVVJIbnBxM3k4SXU1Um52SVdDNHNtcnVvaWdo?= =?utf-8?B?NjhDK2Fjdjk1RjdsdVkyekZDRXZRMUJMazJKUk5lcDJhSURQK3hPUjU0N1Rj?= =?utf-8?B?MzB6S2xVRXFZdHV6L2pUUU1nS2hxUWZXYitnNjZCb1pLa3M4eEQ2QnZrVmRQ?= =?utf-8?B?RVVkLzlBZUpJRTdONjk2bVcvRkpwNkhaRDFmZngrMGFkSG5BUWNlMVBSc05Y?= =?utf-8?B?TEpkbEVQWmFDb01wSHh5cThEd0JLZ1pBVjhobVJXUVNZR2ZkOFJ0eHdqRjk1?= =?utf-8?B?c0FCYVVCT2ExSkpyQmwzRXB4NXlqRS9PRHYzdjgySFhjb3BFUlM1TnlqbHYy?= =?utf-8?B?STlWRU9WcnlOSTc2UFMvazlxZ0NLd0paRGVhd2VjM3J2YnpLSTluKzYya0F5?= =?utf-8?B?dmtqWVBxZ04vUTlvV2wvbS8zeHlnZjNoR0JkSWc1QmJpL2ZKSnc3L1ZMcHMz?= =?utf-8?B?aldpdUtWMk1VV2NqNjFNTzNDaVVqTWNLd1pJQWFJRlFmVUk4cVNGMUtnTDVN?= =?utf-8?B?dkFNZGY1VXFUNUIyU3czTzVjVWRRcWZxTjhUY1JiUGlYWCs5SkgyVmlHQkZE?= =?utf-8?B?WUs4Vi9ESGFaeFhLelBZdkpJeTQzdGRRVk5sUE03WEpFUnJvVHJlTHBoZjcx?= =?utf-8?B?VkJpZmF3bDh6Y0h0UG1WM2VTa3lGZDlpMC9UUnMwNFZ3eGZUVFQ1Z21Rcy81?= =?utf-8?B?MDVBMGlyOWpIWC9wcFR3bzBxUVhuQ2ExVFdrQmoxRmdaK2liU1BUVWVBUG1I?= =?utf-8?B?TnRVNXBWWkdYa3hXbU1VSXA2bkcyd3cwN1dNVnpuVmtGN0Z1Zm40djJtUmpY?= =?utf-8?B?WEZ5Zm1VZm44NlZSQW5qNjJIZnU1NnVlczluRjBwT3VZcmhWWmdBcHVpWno0?= =?utf-8?B?UmxCUkp5L1ZTL2g0UWUrUmtXenlMOHJ6cmxVN3oxMFVlbzJBWElGdnp6QmVB?= =?utf-8?B?WjQ2Q0F2dHd1Q2UxNWdxMzNJam4xY3VQdG1MVEN3S0w5SFJxb0I5SlZNelIy?= =?utf-8?B?S2MzOGZybk1IbEJHR1VKdU5ObVloRGZ5Ty95QXZXMGRIa1JsaHN0QitLWmFP?= =?utf-8?B?S0kzcG5pUmpSN0V6S3ZMSk9EamhXcEF5U2tKMENlcXZDdGllcnlac05lK2Zl?= =?utf-8?B?OGowUUlrNFdVdHc0L1k1VXEycko4aEwxRHRUNU1aWDlFdzduZ1g1QWdwczBp?= =?utf-8?B?aEZ5MnZlSjV3cmVvU3M3TzRMSzV0djJ4MThtcUNHUGdMM3JOajA0dkR5NFRr?= =?utf-8?B?dzQ0ZEZnYTl3cXZkS0Nkd0wxWnlxKzMyai93NVlocVE1WUFiL3B2Z2U2a2x4?= =?utf-8?B?ZGc9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 2cc5e6b4-7b18-4270-7409-08dcd30bb7a9 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2024 09:17:26.1209 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mk1uIv3DZ7alHkk+UAwzMQH+JTIjOf2JV6AvrugivTuCfNSThiVx0RWQPzEeue21+zQiYgz69KOJxHJORTKoFtaeo3dp0rTqkOwBkKuHIxk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5063 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Sep 12, 2024 at 02:19:07AM +0000, Varghese, Vipin wrote: > [Public] > > > > > > > > > > > > > > > > > > > > > >>> > > > > > >>> > > > > > >>> Thank you Mattias for the comments and question, please let > me > > > > > >>> try to explain the same below > > > > > >>> > > > > > >>>> We shouldn't have a separate CPU/cache hierarchy API > instead? > > > > > >>> > > > > > >>> Based on the intention to bring in CPU lcores which share > same > > > > > >>> L3 (for better cache hits and less noisy neighbor) current > API > > > > > >>> focuses on using > > > > > >>> > > > > > >>> Last Level Cache. But if the suggestion is `there are SoC > where > > > > > >>> L2 cache are also shared, and the new API should be > > > > > >>> provisioned`, I am also > > > > > >>> > > > > > >>> comfortable with the thought. > > > > > >>> > > > > > >> > > > > > >> Rather than some AMD special case API hacked into > , > > > > > >> I think we are better off with no DPDK API at all for this > kind of > > > functionality. > > > > > > > > > > > > Hi Mattias, as shared in the earlier email thread, this is not > a > > > > > > AMD special > > > > > case at all. Let me try to explain this one more time. One of > > > > > techniques used to increase cores cost effective way to go for > tiles of > > > compute complexes. > > > > > > This introduces a bunch of cores in sharing same Last Level > Cache > > > > > > (namely > > > > > L2, L3 or even L4) depending upon cache topology architecture. > > > > > > > > > > > > The API suggested in RFC is to help end users to selectively > use > > > > > > cores under > > > > > same Last Level Cache Hierarchy as advertised by OS (irrespective > of > > > > > the BIOS settings used). This is useful in both bare-metal and > container > > > environment. > > > > > > > > > > > > > > > > I'm pretty familiar with AMD CPUs and the use of tiles (including > > > > > the challenges these kinds of non-uniformities pose for work > scheduling). > > > > > > > > > > To maximize performance, caring about core<->LLC relationship may > > > > > well not be enough, and more HT/core/cache/memory topology > > > > > information is required. That's what I meant by special case. A > > > > > proper API should allow access to information about which lcores > are > > > > > SMT siblings, cores on the same L2, and cores on the same L3, to > > > > > name a few things. Probably you want to fit NUMA into the same > API > > > > > as well, although that is available already in . > > > > > > > > Thank you Mattias for the information, as shared by in the reply > with > > > Anatoly we want expose a new API `rte_get_next_lcore_ex` which > intakes a > > > extra argument `u32 flags`. > > > > The flags can be RTE_GET_LCORE_L1 (SMT), RTE_GET_LCORE_L2, > > > RTE_GET_LCORE_L3, RTE_GET_LCORE_BOOST_ENABLED, > > > RTE_GET_LCORE_BOOST_DISABLED. > > > > > > > > > > For the naming, would "rte_get_next_sibling_core" (or lcore if you > prefer) be a > > > clearer name than just adding "ex" on to the end of the existing > function? > > Thank you Bruce, Please find my answer below > > > > Functions shared as per the RFC were > > ``` > > - rte_get_llc_first_lcores: Retrieves all the first lcores in the > shared LLC. > > - rte_get_llc_lcore: Retrieves all lcores that share the LLC. > > - rte_get_llc_n_lcore: Retrieves the first n or skips the first n > lcores in the shared LLC. > > ``` > > > > MACRO’s extending the usability were > > ``` > > RTE_LCORE_FOREACH_LLC_FIRST: iterates through all first lcore from each > LLC. > > RTE_LCORE_FOREACH_LLC_FIRST_WORKER: iterates through all first worker > lcore from each LLC. > > RTE_LCORE_FOREACH_LLC_WORKER: iterates lcores from LLC based on hint > (lcore id). > > RTE_LCORE_FOREACH_LLC_SKIP_FIRST_WORKER: iterates lcores from LLC while > skipping first worker. > > RTE_LCORE_FOREACH_LLC_FIRST_N_WORKER: iterates through `n` lcores from > each LLC. > > RTE_LCORE_FOREACH_LLC_SKIP_N_WORKER: skip first `n` lcores, then > iterates through reaming lcores in each LLC. > > ``` > > > > Based on the discussions we agreed on sharing version-2 FRC for > extending API as `rte_get_next_lcore_extnd` with extra argument as > `flags`. > > As per my ideation, for the API ` rte_get_next_sibling_core`, the above > API can easily with flag ` RTE_GET_LCORE_L1 (SMT)`. Is this right > understanding? > > We can easily have simple MACROs like `RTE_LCORE_FOREACH_L1` which > allows to iterate SMT sibling threads. > > This seems like a lot of new macro and API additions! I'd really like to cut that back and simplify the amount of new things we are adding to DPDK for this. I tend to agree with others that external libs would be better for apps that really want to deal with all this. > > > > > > Looking logically, I'm not sure about the BOOST_ENABLED and > > > BOOST_DISABLED flags you propose > > The idea for the BOOST_ENABLED & BOOST_DISABLED is based on DPDK power > library which allows to enable boost. > > Allow user to select lcores where BOOST is enabled|disabled using MACRO > or API. > > > > - in a system with multiple possible > > > standard and boost frequencies what would those correspond to? > > I now understand the confusion, apologies for mixing the AMD EPYC SoC > boost with Intel Turbo. > > > > Thank you for pointing out, we will use the terminology ` > RTE_GET_LCORE_TURBO`. > > That still doesn't clarify it for me. If you start mixing in power management related functions in with topology ones things will turn into a real headache. What does boost or turbo correspond to? Is it for cores that have the feature enabled - whether or not it's currently in use - or is it for finding cores that are currently boosted? Do we need additions for cores that are boosted by 100Mhz vs say 300Mhz. What about cores that are in lower frequencies for power-saving. Do we add macros for finding those? > > What's also > > > missing is a define for getting actual NUMA siblings i.e. those > sharing common > > > memory but not an L3 or anything else. > > This can be extended into `rte_get_next_lcore_extnd` with flag ` > RTE_GET_LCORE_NUMA`. This will allow to grab all lcores under the same > sub-memory NUMA as shared by LCORE. > > If SMT sibling is enabled and DPDK Lcore mask covers the sibling > threads, then ` RTE_GET_LCORE_NUMA` get all lcore and sibling threads > under same memory NUMA of lcore shared. > > Yes. That can work. But it means we are basing the implementation on a fixed idea of what topologies there are or can exist. My suggestion below is just to ignore the whole idea of L1 vs L2 vs NUMA - just give the app a way to find it's nearest nodes. After all, the app doesn't want to know the topology just for the sake of knowing it - it wants it to ensure best placement of work on cores! To that end, it just needs to know what cores are near to each other and what are far away. > > > > > > My suggestion would be to have the function take just an integer-type > e.g. > > > uint16_t parameter which defines the memory/cache hierarchy level to > use, 0 > > > being lowest, 1 next, and so on. Different systems may have different > numbers > > > of cache levels so lets just make it a zero-based index of levels, > rather than > > > giving explicit defines (except for memory which should probably > always be > > > last). The zero-level will be for "closest neighbour" > > Good idea, we did prototype this internally. But issue it will keep on > adding the number of API into lcore library. > > To keep the API count less, we are using lcore id as hint to sub-NUMA. > I'm unclear about this keeping the API count down - you are proposing a lot of APIs and macros up above. My suggestion is basically to add two APIs and no macros: one API to get the max number of topology-nearness levels, and a second API to get the next sibling a given nearness level from 0(nearest)..N(furthest). If we want, we can also add a FOREACH macro too. Overall, though, as I say above, let's focus on the problem the app actually wants these APIs for, not how we think we should solve it. Apps don't want to know the topology for knowledge sake, they want to use that knowledge to improve performance by pinning tasks to cores. What is the minimum that we need to provide to enable the app to do that? For example, if there are no lcores that share an L1, then from an app topology viewpoint that L1 level may as well not exist, because it provides us no details on how to place our work. For the rare app that does have some esoteric use-case that does actually want to know some intricate details of the topology, then having that app use an external lib is probably a better solution than us trying to cover all possible options in DPDK. My 2c. on this at this stage anyway. /Bruce