From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 167BA4590E; Thu, 5 Sep 2024 16:45:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EA51240273; Thu, 5 Sep 2024 16:45:22 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by mails.dpdk.org (Postfix) with ESMTP id F1BBB4025C for ; Thu, 5 Sep 2024 16:45:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1725547522; x=1757083522; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=vVa32FD49KvtRjQkiSTT/fqqq2LjVt3sUfaPqgzaWcs=; b=nX5OW+ls/O37/4v/DL+nCq/NeRbG56bspIYUIrqEDbo1pOrpzRyvByoA qzm/M1L5FkkzUpQUW/ZiJ3++zqIFhFfdSNOYl+pfiRmfGJdzCMaMBO91a 5JssjcdDv9iyH0v4KVeinZGI78qLtld7J396NY9qYWrM3cC65JzqQE5xK GdcMb7yNVPtAYVYGWfbPiqjyhzHkG1U6iD3azIO9vsvWHgO6f7xchd/8O ovLmdBnfOBqA+0hKfXg4r7WSpEy5BRbohQDWi7IqNZJ/Xlq2tNZhTimiM sApEBtrK957i63sdhq0f47Iki2QQH6U5ADlKW+yg400pbApzaxAdUUTs7 A==; X-CSE-ConnectionGUID: FWuX0RTKTaKFEFEm2qCwBQ== X-CSE-MsgGUID: BCLTQRkdS/qBgB/jNdDjRw== X-IronPort-AV: E=McAfee;i="6700,10204,11186"; a="49680855" X-IronPort-AV: E=Sophos;i="6.10,205,1719903600"; d="scan'208";a="49680855" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Sep 2024 07:45:21 -0700 X-CSE-ConnectionGUID: zW121SW0Qj+ikmGxobO0Jw== X-CSE-MsgGUID: fGgpNebWSpaSsUKA6dfnsg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,205,1719903600"; d="scan'208";a="70442570" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by orviesa005.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 05 Sep 2024 07:45:21 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 5 Sep 2024 07:45:19 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 5 Sep 2024 07:45:19 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.171) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 5 Sep 2024 07:45:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KWMojFKrb5E/7ppHPjfUejkeMNtefGTwLvt1+zzOuvoqEu5hkdsZ/pGe6lCgPELhucRJN7/P6edibWt1OaFkIVMzsmJMUspxcIAOzMBn+DbrXwXGL5OMnmUXCH55OnbsqcJ9dnOduIHYmgYb2rrj1Dtn/9yJzHorNkdMNVxWJqfKI2W+Od90khE+ekOvb9PpOfoprOEmaSu4d3HACnD5AKjZWBS5XsfmtMCJMkBjfZigvxHT/1gbbsV4u5imQa45CCzS90iCAnjBiqv0PxpcBzXahm6BBa5cvw1BkzjR+GyzwVMHadQIsgQjOnMtw2O1MyoCLFKVXhsacqKlsG8mRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gBsX5SQOAjMuB2/ew9UEMIcU9mTiWJX5a7IS7K+v9hE=; b=R+WOqEfQtT5i8A7B7zL1w6+DJeSpLqQXdBTtNZtGzsD8kVLiKT8HB2Wm7FCLSyQWnY+FpE8r0M9POl/0Brt2U1/WMmJw4fWgrpsT15VJasex7wLVcACTkA/9JkrrnanTrSZNhietR7xKaz+AJHROpvH98mz9jWRSHe7IL+zGmwVySwi4qd4nyx3NxGEXM9hxpV8qdGbWarKEPMYDmG7jRl5E0T8Kkq08b5JIEHcKemiJUGoRDearz6laef4dOS4VtfHXB5R1tyJW0cVytO54vVY1I8aTDTlwYtX26wn1lQGNk5OdMQhSO7SRWumA1FHns/Qi6bcJ6jyckYo/F9TDSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) by SN7PR11MB8281.namprd11.prod.outlook.com (2603:10b6:806:26b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.25; Thu, 5 Sep 2024 14:45:17 +0000 Received: from DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::21e4:2d98:c498:2d7a]) by DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::21e4:2d98:c498:2d7a%4]) with mapi id 15.20.7918.024; Thu, 5 Sep 2024 14:45:17 +0000 Message-ID: Date: Thu, 5 Sep 2024 16:45:10 +0200 User-Agent: Mozilla Thunderbird Subject: Re: [RFC 0/2] introduce LLC aware functions To: Ferruh Yigit , "Varghese, Vipin" , CC: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= References: <20240827151014.201-1-vipin.varghese@amd.com> <288d9e9e-aaec-4dac-b969-54e01956ef4e@intel.com> <65f3dc80-2d07-4b8b-9a5c-197eb2b21180@amd.com> <8addd7f6-fac8-45ec-a44f-f81eb008cc36@intel.com> <3edc8a89-7d10-47f4-8f95-856c2a7fc7ba@intel.com> <3eae1577-f06f-48f2-863a-faf70b97bc72@amd.com> Content-Language: en-US From: "Burakov, Anatoly" In-Reply-To: <3eae1577-f06f-48f2-863a-faf70b97bc72@amd.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: DU2PR04CA0070.eurprd04.prod.outlook.com (2603:10a6:10:232::15) To DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB6502:EE_|SN7PR11MB8281:EE_ X-MS-Office365-Filtering-Correlation-Id: 1cc6e35b-7e5d-4eb8-eba3-08dccdb95bd3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?B?OWNlU0xhdkpleUI1TUx0Sko1b3hOYnZiOGR5dzZoYlRxMEJ6dkE5c3hOendz?= =?utf-8?B?ZmJaSWlLdkV6T2ZsQVpRTExjS2pseFFDbFRiL2ZtTUlqYTJIUy9zdUE4c09E?= =?utf-8?B?TkE4TC8wUE5WNDN2VlRiOUlTVDRUTGJUaC8wbGFXNWtSRzJhUTJsNE03c09u?= =?utf-8?B?Sk11eTBMUVJ0ZG1ObnZYNjlqNjVXTzMxTEt2SGtPaGVCd2xYSC9vN3lPS3RO?= =?utf-8?B?T0NMbGl4V2xldHAxbE5ZWDkrTDNkUXNnb1o0OFMxejJCZExuWHlaeWV0djhP?= =?utf-8?B?NEt0Z3k4VSs4L2REY3p1d2M3anBPbDFNNVp4MmdBb3AvbEdXOFF6cGU0dzBh?= =?utf-8?B?WnhoMUt1YTJPTmlEaEV4ZUdBbmR6Yk9TZU5UTUJSSzV2VURFR3Z5M0h5eDNH?= =?utf-8?B?cGFCbDBpUFJGMmZmekxneUNrYkduWE1MT2lvUEFsZG51SC91MExjSHdlQXVG?= =?utf-8?B?c0V4VDFBVzc2a3RYNUU1RUxhdW4xNzhOSXNKWStjNFJCQ2pnSkU5UGRSRDcv?= =?utf-8?B?eGVVNHR6Szd3MXBGRXVFSVQwYzUrL1c5UFpkOWNsU0ZsVjZ5aU51Ty9NTWxR?= =?utf-8?B?MlFvQlYrcU9ZTkRsNXdhcmVzZThOT3NKeHkvcUVKZGRKcXM0ZnRwR2p0dmJF?= =?utf-8?B?L0lMdjNtZVpwZ28zTXlpWlFKN3B6Uyt0V0lHcEVVVy91MkgwOG9vTkJxNENp?= =?utf-8?B?cnoxTEdUbEdVMVRvNlA3QlpXL0wrTWlQTmpDbXY3eHhmZ0Z5Ni9ydFcrcHRV?= =?utf-8?B?aXNNemR6azVYMEFRRUdqRWdMYjd2U0NqSjJ2MXN5RFFnMHNOL3ZUMGtxaDhI?= =?utf-8?B?d2M4dmcxVFBMUDhDRC9OK1ZTOVVYM3NENzQ5WTFERE5HeVpycGp0NFFiWWxU?= =?utf-8?B?M0w2ZXJ6cjliMGlNVHJFMWVUZFV2VmNoSVlGT3hoWlVpbWxvR1AvV3dtY3Uz?= =?utf-8?B?VHRhRUhDTEFrdFh3SVZYRzZBeDA5c2Nxb0RYV2FxT3Q4Nmt5K1NObGJUWDRw?= =?utf-8?B?Y05lRzY1aTQyTGF4MUU2STVvWkpsUWorTkp6Q1UwbW9sY0hKRHY2MVpkSHBY?= =?utf-8?B?OVRqRDE1dXhldDVkVHFCSGRLRDFFcFNsMk5ZbzZDWGJ1Y0RNOHZIZkZtSDFM?= =?utf-8?B?RVFHTkhLUVlrUHpKUTdJMmw0VmZGV3J0V2EyOVpwdDhTdDc4OHFZaGdiNTlH?= =?utf-8?B?anl3YXd6L2YwTW1nL2JXZEFoOEpIWmw3ZklTWkV5MTlIdVA0cHpRSHcydm1B?= =?utf-8?B?VENGZ2xSL2NWUy9wRlgzamJsWlZYMzBmWE5zb1JRdkltNkJuUUhvUXRmeUVU?= =?utf-8?B?VnZkemxqYWg3SWRJdGlNVUczL0hzN0lDVHRVblN0ZCt4clJ1ZVVzMzNlcm82?= =?utf-8?B?VlF6MEdibFR1SGRiUjlxdjZHSFYyaEFaOTJqTVRGb2k1U3JmMksvQUhCb3lX?= =?utf-8?B?dDdGQy9QUzVsR2lDWndtMnRhUWVuSW5lQ2NxVXhaZURyVFp5RWJMaldEdmph?= =?utf-8?B?UC94Y3FLWXRmWStleHQ2MTVQQVRTVE1ad1dwNkwyWUhPdTkyekJkc1R4dW5L?= =?utf-8?B?WkxETWkrZjhydUt5STdvbjlGSnloZWtDbzJ3ZEx1U1hCRGhoMVRuRFFGOUpO?= =?utf-8?B?cWRwaHdockljeThEbWI0dHdWazVTT0tKMGF6THFhQ2M5WHpBMFlCY2ZVRFcw?= =?utf-8?B?M3lUUlkvckRvellkcEF5akxiOGV1Z2NYNU1Yc0xtQnRWQ0Zuand6S3ovT1ZW?= =?utf-8?B?a3JpTkgwbGJPdE5KS2ZqYVhwSFVJTUJKUHp3enpmSStNWCt5ZG1RbUJldzdW?= =?utf-8?B?RXIzTVdPb0p1Q3dGODNlZz09?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB6502.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?b2E1U3E4QTJPWGlYeFZ1RW4ydUhEYkQyYUlyOU8xM0tucVp4cWhXTGpvQmo0?= =?utf-8?B?emtjRXZLaGZaMmQ2dU9HNllkc0dOVDUwb2hMQmFoa29CRHZXZXAxVWVBM3RG?= =?utf-8?B?UkRjdysyUzJBTmJnaWlBZm03SDA0ejBmdk51ZW1rTy9takEraFI0ZVQ1T0tF?= =?utf-8?B?Z2QvSG83b29sTWVyV0I2MitJV2xjZGpBYi84KzBLb2hVWUMxc2ZnZzE4blpr?= =?utf-8?B?QnMwUG1lMDNjNUYya0F1eWZLbHRVOGZYZUN3TnNHYzVoRTlLZkJPTmZwOEgr?= =?utf-8?B?VFFPTUdKOXphbWtKUlZGOWFBYUxDYzVscXRhWDl1a1R1RmZoUWFzaytzSVlW?= =?utf-8?B?MDB2cGpielg5M2J1bXpEamtBaU50c05RelZaNGdkL2wwS2ovckdTcG5KRXNF?= =?utf-8?B?WTFtb0VwNjVVbUJva2g1M1dxZlVmRWRNaCt5Mnp3MmE3Wm9tdys4ZWhPWkJ2?= =?utf-8?B?YXVDUElhcDE3R0xOMWQ3Z2pHcDFZbXJzbnhLdW1wSWlHdWlXM29vb3FGcE9j?= =?utf-8?B?QlkxYkt3bWlJelcyc2JlTWJGcTNNTWRIeVE2ZmFCTWgrQkxDZEcxcTZKQ0RM?= =?utf-8?B?OFlCZTVRRy95c0kwa3JFUmdqMnprTlVkNEdTRWFFaWUydHBNMWV6ZklYeWVr?= =?utf-8?B?Z3pPVmtUbjZablVDNUhqWGxZSUkwWjVReks1VnJ6d3dFbHo1MFJza1o3Z3NG?= =?utf-8?B?QnRVWElnOTV5Y3FtK3pndlFnTXlYNXBPY0laMEJEVXBFV290dnZjdFlmVWRH?= =?utf-8?B?YmIxd2FMZi8zUzNrU3h0SXA4dHlzaWM0TUd4UXEwcm9XSW1JRjJQeHUxbm5s?= =?utf-8?B?ZGNLZnFnVXNDYnlIQ0lidHBac24veExRcU55anpSN0RycnFKY1VJU3dGZTZI?= =?utf-8?B?SHFSVnBnWXBQdTZNbHZaNGYvRXBHdFZ2OEx4SmN5UDlzcmp0NWxRZnQyNXBp?= =?utf-8?B?Qjd0WHJ1a2FVaWZwcndhRXVXWGdWKy80S3FWZnc2eVN6a3hHSG5BcTlSM0VE?= =?utf-8?B?SGxqRmQ3bmthWVlzVjZyUDZYVWFmRkJDZ3hDN3VBT1FrcFJwMi9NaGRCendK?= =?utf-8?B?K0ZBNWhuNFVpMDZxQ25WVWxiTkVUdVNKVGl2aFdQWmdlRVpLQ3VOOGl6ZG1T?= =?utf-8?B?R1ZNL29jcER6NVREVFhzYnVuUjJVdnlxd0xUMVd0cU14eVAvcng1M3l0TkhQ?= =?utf-8?B?cWcyUlZJZmxaOE5RUnNXSS9lY0FpbndiWE1sNTEyZVRLM0N2WFRIRkVTZmFG?= =?utf-8?B?TDBwRWVjQUdqUzhjeU1kVDlzcjcya1R4K3F5V2VCTGNUVWwwbjNlWElMQ1Vw?= =?utf-8?B?UnhJVERwSkN4aGsxcUZPWE9vUnB5cU9rakExMTlzWjdha2puVEF4aWN6Umpa?= =?utf-8?B?RVZ2UU5kNTMwWXg3bThEZnROVVlweno5K2JuNWJhbGtYMnlpRGkvUHJuOWxR?= =?utf-8?B?dFArTkxRS3ViRVlkZDdtTmpiSG1aMm1Kc0RSR0xXakpxcEZlMjEzSFowREg4?= =?utf-8?B?dHJIR3ZYeFMrOXpjRkFHdW9xQzdPTkhEOHVmdnJNYnpEM1R3Sk5VVTcvQzd4?= =?utf-8?B?ZHNCcE1rQlVtZGFnaU5EWUhkM0Y5bDYrT2NrOHpoS0RuamNTL2RsWDh0ekYx?= =?utf-8?B?L2w3ako3d29UMGpuSWw4WVRlRTRJR1BjYys0OXFvazFwY2lNcTJWVDI0cS9Z?= =?utf-8?B?amd0SzRqTFVrUTV5K3dCTlVjQWg1QU5vQmNnL1NxWE9vYVg2Z0NaN3FhdXA4?= =?utf-8?B?NHRSUFBrZko2OG1rQW45U1pxckVuS09lc0dubkxaV1MyMi9mbEZEQnhaVnpq?= =?utf-8?B?TjA5Vi9wL1V1dEw3N3hZL0lRWjJGN1VydXlXVE1YcGNsQ2Y1V2JUTkh5Tjdj?= =?utf-8?B?ZGd6cWpIeHo4eWh3aVgxRXc5aEVIK1BabXIzcVltY2dyTVQ3YTJnelFIdW80?= =?utf-8?B?RmRTQlhrWHI5dzJaWFVLNGIxMkV1c1lXQjdUL2xUQUtvK0ZvWXh5ejF1VG9U?= =?utf-8?B?TnJRU2VXYmIveVJwZjNNZXpla0J3UEVYWDdqbWhZYVpYN01uNjJ4UzZxVWZz?= =?utf-8?B?UTAyelAwa2VkR28wZzR6cktOTHhmdTVZNUJXME5PY0ZYMHpsTGtxeFFtenB3?= =?utf-8?B?c1NkbWlhb3VsMmh0YTlJWTBPRVNpcUhON3FHcjQ4ZUJHakxVMkI2TDNqcldN?= =?utf-8?B?bmc9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 1cc6e35b-7e5d-4eb8-eba3-08dccdb95bd3 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6502.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2024 14:45:17.4434 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mXjTrKHGX4vKlBpdouQCYopgsMfmcfICO+QCKdZQcX0Pf5s7NozF1atXNGEi/1+RNquFMqszN2d5il6L1ZelOQ1AUtwJ+jQQVqS/bwNgizY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB8281 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 9/5/2024 3:05 PM, Ferruh Yigit wrote: > On 9/3/2024 9:50 AM, Burakov, Anatoly wrote: >> On 9/2/2024 5:33 PM, Varghese, Vipin wrote: >>> >>>>> Hi Ferruh, >> >> I feel like there's a disconnect between my understanding of the problem >> space, and yours, so I'm going to ask a very basic question: >> >> Assuming the user has configured their AMD system correctly (i.e. >> enabled L3 as NUMA), are there any problem to be solved by adding a new >> API? Does the system not report each L3 as a separate NUMA node? >> > > Hi Anatoly, > > Let me try to answer. > > To start with, Intel "Sub-NUMA Clustering" and AMD NUMA is different, as > far as I understand SNC is more similar to more classic physical socket > based NUMA. > > Following is the AMD CPU: > ┌─────┐┌─────┐┌──────────┐┌─────┐┌─────┐ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > │TILE1││TILE2││ ││TILE5││TILE6│ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > └─────┘└─────┘│ IO │└─────┘└─────┘ > ┌─────┐┌─────┐│ TILE │┌─────┐┌─────┐ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > │TILE3││TILE4││ ││TILE7││TILE8│ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > │ ││ ││ ││ ││ │ > └─────┘└─────┘└──────────┘└─────┘└─────┘ > > Each 'Tile' has multiple cores, and 'IO Tile' has memory controller, bus > controllers etc.. > > When NPS=x configured in bios, IO tile resources are split and each seen > as a NUMA node. > > Following is NPS=4 > ┌─────┐┌─────┐┌──────────┐┌─────┐┌─────┐ > │ ││ ││ . ││ ││ │ > │ ││ ││ . ││ ││ │ > │TILE1││TILE2││ . ││TILE5││TILE6│ > │ ││ ││NUMA .NUMA││ ││ │ > │ ││ ││ 0 . 1 ││ ││ │ > │ ││ ││ . ││ ││ │ > └─────┘└─────┘│ . │└─────┘└─────┘ > ┌─────┐┌─────┐│..........│┌─────┐┌─────┐ > │ ││ ││ . ││ ││ │ > │ ││ ││NUMA .NUMA││ ││ │ > │TILE3││TILE4││ 2 . 3 ││TILE7││TILE8│ > │ ││ ││ . ││ ││ │ > │ ││ ││ . ││ ││ │ > │ ││ ││ . ││ ││ │ > └─────┘└─────┘└─────.────┘└─────┘└─────┘ > > Benefit of this is approach is now all cores has to access all NUMA > without any penalty. Like a DPDK application can use cores from 'TILE1', > 'TILE4' & 'TILE7' to access to NUMA0 (or any NUMA) resources in high > performance. > This is different than SNC where cores access to cross NUMA resources > hit by performance penalty. > > Now, although which tile cores come from doesn't matter from NUMA > perspective, it may matter (based on workload) to have them under same LLC. > > One way to make sure all cores are under same LLC, is to enable "L3 as > NUMA" BIOS option, which will make each TILE shown as a different NUMA, > and user select cores from one NUMA. > This is sufficient up to some point, but not enough when application > needs number of cores that uses multiple tiles. > > Assume each tile has 8 cores, and application needs 24 cores, when user > provide all cores from TILE1, TILE2 & TILE3, in DPDK right now there is > now way for application to figure out how to group/select these cores to > use cores efficiently. > > Indeed this is what Vipin is enabling, from a core, he is finding list > of cores that will work efficiently with this core. In this perspective > this is nothing really related to NUMA configuration, and nothing really > specific to AMD, as defined Linux sysfs interface is used for this. > > There are other architectures around that has similar NUMA configuration > and they can also use same logic, at worst we can introduce an > architecture specific code that all architectures can have a way to find > other cores that works more efficient with given core. This is a useful > feature for DPDK. > > Lets looks into another example, application uses 24 cores in an graph > library like usage, that we want to group each three cores to process a > graph node. Application needs to a way to select which three cores works > most efficient with eachother, that is what this patch enables. In this > case enabling "L3 as NUMA" does not help at all. With this patch both > bios config works, but of course user should select cores to provide > application based on configuration. > > > And we can even improve this effective core selection, like as Mattias > suggested we can select cores that share L2 caches, with expansion of > this patch. This is unrelated to NUMA, and again it is not introducing > architecture details to DPDK as this implementation already relies on > Linux sysfs interface. > > I hope it clarifies a little more. > > > Thanks, > ferruh > Yes, this does help clarify things a lot as to why current NUMA support would be insufficient to express what you are describing. However, in that case I would echo sentiment others have expressed already as this kind of deep sysfs parsing doesn't seem like it would be in scope for EAL, it sounds more like something a sysadmin/orchestration (or the application itself) would do. I mean, in principle I'm not opposed to having such an API, it just seems like the abstraction would perhaps need to be a bit more robust than directly referencing cache structure? Maybe something that degenerates into NUMA nodes would be better, so that applications wouldn't have to *specifically* worry about cache locality but instead have a more generic API they can use to group cores together? -- Thanks, Anatoly