From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1E286458C8; Mon, 2 Sep 2024 03:08:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5A7840269; Mon, 2 Sep 2024 03:08:39 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2086.outbound.protection.outlook.com [40.107.94.86]) by mails.dpdk.org (Postfix) with ESMTP id 384D6400D7 for ; Mon, 2 Sep 2024 03:08:33 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=b1UxlkvdkzKF7aim2vXJdAaR7tKM2FwgKhFOK4wrfWH+Aekka+s9wI2AZgBZJF/UVKFoz57zWpWBKEowzNJ8FnUMaBo9dmIy8ONsmPOZIVUJjwaoJGZ7Q5m8ytlcg70sSM0cEv0kkYQxmXHhf4Qrn4BvuNwDYLOCcqTOWx3hi8KNnaJqjXVeNVKlAg7f4DESyffLcau8N7M03od9UW4ViDhm4sVCgZENamcZgKZjicsOpr5ogvwyn6usNEyLdok5SqP0bp1bzMYr35Fm6mPvUSurpPfZoxsIEeMkQK2V2nwBBmIATQw66S1+HFNAxoRX77TQ/I9P+YID02buV9gVew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=x2R/aDukjFRHH2WaNEEKSo8/H+J1KMWOrpVYlNkwiF8=; b=vHTD3IQO3+rkeLxdgti7MUsy1SnQ5vFrvDyHfPz/1nxaqM3i/VGLtL2m1JTKikLXwHKqI7hYOUqhzO4n5xcKi0oZZ5o3A5m0LIcZKSF4KzZ2vkknpbbX7xEnZOMgp8YEIGZPLyucQM1dc3xfoNl4/3vWAyzZVM1DU3D+mLVaEWLwv/kBfi4iGaB9fqVtV64OTyEs7Aq2MkCIar6GWGzQdx31npr4KPu1zf7kGajXEzVnjXu/xgwngpfIz1U1bXd40ttJcuCP8aka0Pwud3nVc6KED/iedmpa4E9KFPcdVlwz/map6+3JX9aVRPdfsNXO2+YurjgWV3lah0QEjggptg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x2R/aDukjFRHH2WaNEEKSo8/H+J1KMWOrpVYlNkwiF8=; b=IHToJEa6javqedYVfBMTBLwWM8aAmaDhsE60oyaIsUkywOvQd2Io7vkl54imLacuu3E9PJmur2J4L4u1oUuiAIdbqIffCXOr6kqOfLJ53C5bzb9IS/iULIq9fMXPEyQ0R2u3z/jadkELSEAhoHOu4sIuJDoR8y0qR/IWmNLzlF0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from PH7PR12MB8596.namprd12.prod.outlook.com (2603:10b6:510:1b7::6) by IA1PR12MB6556.namprd12.prod.outlook.com (2603:10b6:208:3a0::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.23; Mon, 2 Sep 2024 01:08:29 +0000 Received: from PH7PR12MB8596.namprd12.prod.outlook.com ([fe80::a011:943d:7291:8069]) by PH7PR12MB8596.namprd12.prod.outlook.com ([fe80::a011:943d:7291:8069%5]) with mapi id 15.20.7918.024; Mon, 2 Sep 2024 01:08:28 +0000 Content-Type: multipart/alternative; boundary="------------VITCIWUFQtMLPc17thORNsaM" Message-ID: <65f3dc80-2d07-4b8b-9a5c-197eb2b21180@amd.com> Date: Mon, 2 Sep 2024 06:38:20 +0530 User-Agent: Mozilla Thunderbird Subject: Re: [RFC 0/2] introduce LLC aware functions Content-Language: en-US To: "Burakov, Anatoly" , ferruh.yigit@amd.com, dev@dpdk.org References: <20240827151014.201-1-vipin.varghese@amd.com> <288d9e9e-aaec-4dac-b969-54e01956ef4e@intel.com> From: "Varghese, Vipin" In-Reply-To: <288d9e9e-aaec-4dac-b969-54e01956ef4e@intel.com> X-ClientProxiedBy: BM1PR01CA0161.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:68::31) To PH7PR12MB8596.namprd12.prod.outlook.com (2603:10b6:510:1b7::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH7PR12MB8596:EE_|IA1PR12MB6556:EE_ X-MS-Office365-Filtering-Correlation-Id: 87028420-c76f-44de-0719-08dccaebc12a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?B?dkR5WVg3Ri85MXRhSmFoRFFtd0NtelVHWGhXZmZrTCtla00yYzNxY1FPaWFq?= =?utf-8?B?c2QyTXhqaGU2VDA2bHNjc000UW1oZElza1gyQ0puZDRpaGZaMTlNL21OZ2VL?= =?utf-8?B?aVQraUNidUNxNDJPRlV2VjRRRU5kRjlKeVM4K25tdG9MK2VDRHVpL0M1MTVi?= =?utf-8?B?K1RySWgyUkRHYml6K1JPL3luc2dvTFgwdnY3c3ZPMTZWc05VcjQza3NCYlM1?= =?utf-8?B?ZFJZdmxlQVlsNmgrYXhHM2pBWEMrd2xBMVFHZ3Y0UzVvRHl1VHlueXdsaUg1?= =?utf-8?B?QlJWWWhobjl6Vlp0TWh6bXJzcHVPbVRla2tjUlFGRmRyUmpuUkdQazRMODlw?= =?utf-8?B?MFdHdGNNdXFnbHFOK1hjL0o1SXVhS3hqRUJ6MXJCSC9wWWxabW1UdCtyYTQy?= =?utf-8?B?ak5WWkE5RVdpMFE5T3dWcXpvWHNRdU1QYzJEUTk3N2wzZzcwRWNMRk1HQXEr?= =?utf-8?B?RWdsQlVVN1hmZFZZWTUzdEorczl2bHF0WmFKengzbEhoZUdLWnk2YlV3dzhO?= =?utf-8?B?WGtpcnVHMWVxNVAvVHBrelA0a1RzdGtmR3ErVDFJWlk3d1dkdkFoNUZzenBG?= =?utf-8?B?UVYzN3dHZVptN3JWL0tLbzJYTHVOYzRZeHRISENrOHVkeEcwcFlOLzZXNDRj?= =?utf-8?B?aG1kRzA5Rk1xSVVqVjM1aEFYc2ZGaHhYZEozYktBYml5RE9veGU5NTVTU25k?= =?utf-8?B?NW5uNy9mN05JR0NJamI1VklTd0w0SkxsenA3M1ZkdnRpc3BJczNhTHBZMVVZ?= =?utf-8?B?emdoVEo5d280V1gxMGQ5SVJSYmFnTXV1YzVFUFRWUXVFVjBZSEltYVVLdCti?= =?utf-8?B?aC9VUFAwdy9nQlFVWGQ3eDl1L0paOHEzQ0piWjVQVmp3T3YwU0w2dFRnS01C?= =?utf-8?B?OFhkK004TXZreENMMEttVnVVZDFuYUVFM29PSzR4MjJObkMxQStUaGZ2VFYv?= =?utf-8?B?NmxPYUtYSDhjQkZBbTdIU1pZZWNtcUVYelN5Z3FFY2g2Qm11b1BKN2hrUlZm?= =?utf-8?B?MVR3MzRFQ1dsdGJqMy9kNGVIK2QwdmcxdzBLY3EwMTBwSmdCeTYrZGtkdFEw?= =?utf-8?B?elNVWnh6MUxFZWNHMW8wMjVydlVGVTlDdEZqK2xTejU0YXBYNFh6NkxBajBH?= =?utf-8?B?b0NuZFdaUWNwRlZtQjNPaWFJWEV5eTFnSlNSR1dWNW1DbVFIVlhDMnA4VTkr?= =?utf-8?B?Y3BDMHBxK0FjVjJVeFljcHZsL1BVTENXOTcxSXNiYk1wdktQM2hPNFF0WWs1?= =?utf-8?B?cS9TRTEzT2FSUE5Rdk5aejFMMXBWN01IYXRjN3p6YVIrcy9URnd4Y1Ficklx?= =?utf-8?B?Z2FYUUdNNk9DSEJheFRSeGtRK1ZJbDlvdVpoT3B6akRYbnZQTkZIbXpaZEQv?= =?utf-8?B?YVlsS2ovc1hrSnI0YVZVdjBRZ0djWGpRaE1TOVB6ZElhdkEvNHdCV2lXZS9W?= =?utf-8?B?VEpyaktuckp0dURiU0wzZloySzZ6Q2pwVitNNFBzZ0hQU1BkUnBLVFF3bC9k?= =?utf-8?B?N3pxSGFwaGNkQTlPRDhGTHdxRitob0xiK3BJaENiWTdGOTFKczM5NVc5dFl0?= =?utf-8?B?KzFIelZSRVR4d1ZhM2VzUEpLN0pqTnZralVKeUxhdWxUYzNkcE51Z1dla0dw?= =?utf-8?B?bVZCZC9iNSs4Z2VuWGJya3Q0R1ZUL291clZtQ2N4RkdpOWNBR2w1eU5WZEN2?= =?utf-8?B?OStjZjgzOG5DNlViTWx4djJocHExNW9HTmhMM1pLcGJMa0ZqMmlNMjkxd3ls?= =?utf-8?Q?ypxha9qx86XQhN1WTA=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR12MB8596.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V2NpSm5DYVR3aTdKOWtrRGNDdDhGcVl6Y1huS2tiaEhwTk9CSlJDSHdIeDlm?= =?utf-8?B?em4vMjZ6TmVNNmV4YWhDWUdtb2Y0MG1UMU04aVFqK1pSbmp4amwwOTA4NGgw?= =?utf-8?B?YjlrWktuUVBKZXBKeEJnMkxMV3lQOFNNeW53ZDF3Mis5SFJiWXI3WHZRM0gz?= =?utf-8?B?TnVocGw5cnZFZEZXTTNaNXlVbGQzcFBEZnNSenhyTW1jM1FYNTdOYjZZMUUy?= =?utf-8?B?aXFtajZHcFltYXRibER3UEY2MXd5TXZBMTVhdGY1Mk9maU1xK2xPODlkbmln?= =?utf-8?B?eDZSUmxOa1pFVnlKc1BMa25PVDlLTGFSZzdTSE12NkFKV0doWmRON3kzcTh3?= =?utf-8?B?cGlaMzJvaVpRZTFPaUVCcG1vcVA3SkNMYTVxWDExR0JLMVNDS3hESlc2Nk5h?= =?utf-8?B?OEpGYnFXTWgxbll5WWVVS3RyU1ZDT1EzODVyNk9sZVcvWFZuQ29KWGhoRThu?= =?utf-8?B?QVdlcHhjbkcxY3dBOWtENDhmRjBkTkYxNXAyYTdPTlRiRkhFUlRETnc0QU5s?= =?utf-8?B?dnZSR0w3U2s5QU1CSmdXS3A4S3dWcUNaaUswWkNPdDJJRTJ1TWk0NFdhRDcx?= =?utf-8?B?cVdXaExpaGFNREZoOG9JSC80enhWVkE1WC9EK1BqV2lRdXZnSUlLS1pNaE1w?= =?utf-8?B?WmJGL2NMN282SWozT3hvRjlrenJoV29UM1B1UlBaMDV2azhiS0hlbmZsbmtQ?= =?utf-8?B?RlBlN1R4c3NnV2cyK3Rpa3luVzFlWUdxN2R3REZQTGdtb0lBVmxsODF0NVlU?= =?utf-8?B?ZDh1MUFjRnpKLzFGQU4wYVo2bERvakxkL3Q2Mmd6dVN0amtJNUdIa2gxQ095?= =?utf-8?B?N29CazVzRnc4UnYxY1FMSkxJTk1Lcm55ZnE3Zy9sODVBa1MzbXZNT2orM1lZ?= =?utf-8?B?emFLNFlRVWZ5ZkhFVmwyMHNOdm5yOVV3ZE4yN2s4RXgySjJIbXJ0a1V1eE9C?= =?utf-8?B?SVVHc2JIYk92YXd6MzhnSDRJcERJMEJLeHNzMTRIWkg1RFFsbSttOUpZMFEy?= =?utf-8?B?NTQ0bFVHSTJrWEhOY2VyVlRsakc3UVFZWnBCUGRpd25wTHBYeWF4NmlraG1C?= =?utf-8?B?Q0huZWgxUUN2NGpNQUMzaG83UXp4NnBKSmtadjBwNjFWTElJY1NvUVdtNklV?= =?utf-8?B?RUNwZGFsTEIxbVpCd1dvZG0vM3lHclpOTjliY2xBZ05jd3lGRFd3NFAySTF5?= =?utf-8?B?VjFXdER5WnZvT0g5OXByYjNQWTJ0aWtqMFFlSnpnU0ZnVjlPSzZCOEpoZCtQ?= =?utf-8?B?RkFjS0FKUXRJUVpqclEvRmozV1JwMjQrWHMrSFVYQWczeXp5cGtLVk9mZVpq?= =?utf-8?B?Y3ZPRWUweFVVU2J4T2MzbGJNYmtHRkdEMnVOck4wRXpJSFpzQm1hRnR5SEQ5?= =?utf-8?B?N0tRQTRrRU9HL0FSSEhNL25oQnRjZEVEQi9tdU9UNS9QQ2dUQTRRdFpyU1Nh?= =?utf-8?B?QThVTmJWc1JHRHpZU2k2SHZWLytIUXB6eFBjbS9SQUgwamhPMDBER1RZUWIx?= =?utf-8?B?UUZlUDRhZWVwMnZvbURPSlY5bmJ0M0JNM2xhM3Z3WGNxN294Nmw2U01DVUNq?= =?utf-8?B?bnZWK2VvZC9LdnF5cWN6QzVVZ2xyMWR0VXBaRHc0UktsVEpzZ2FJeithVmZ5?= =?utf-8?B?eDFocURXUkRmMWhlNDl2R0cyL3NjSlV3TTdhUTE0dHpubkk0aG1pVXgvZ29T?= =?utf-8?B?bGhOVVRjWnJtQmVueWtwUXN1anhnVTVodkt6bVN3Z1NPQ2xYUkUvQXBsdDFF?= =?utf-8?B?Y3hRZi9tMUV3dGRaR1dBc2pYYitTNjRqYmIyOExmc21DTmVkVHNkV0Z1Qm5l?= =?utf-8?B?SThERmNpb3JJR2p0ZEQ3RzZpRzRKd2loSmZ6ZmVCQ1hGZ3JkRWNXQnNFdU9D?= =?utf-8?B?RVg2WDBPS2pjOUI2Q2FnUnBOYWszNVcxR1p2amRCZjk0ZW5Pa2xIT0pJNENH?= =?utf-8?B?bzFNa2pDWVROTVJMM2JwUjRuN0Y4bXBDMnJPWjhiM3A3Q21DdVNzMFhTdUZ1?= =?utf-8?B?ZUFxQWd1NHo0djRTM2t2bFdLeHNtNS9nelJnT1JHbDRUSEVnSEsxUG54elNx?= =?utf-8?B?UlVuQW9RcWxKR3NSOVQrSm5oZGFOY0lSV2VzVWxZcE9Nd01kODVJS2RYcFNU?= =?utf-8?Q?gbEn6kL/jD/RYMxu7xcwcng60?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 87028420-c76f-44de-0719-08dccaebc12a X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB8596.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2024 01:08:28.7593 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WPmZ1CiDJnagQJmtC6zWKW9BPc/5DwnJJ6+CwHTkkJteuu1rWj+Cz9ZCG/o/3ctkfLztFe59f3yX5BLiBX9uvw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6556 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --------------VITCIWUFQtMLPc17thORNsaM Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Thank you Antaloy for the response. Let me try to share my understanding. > I recently looked into how Intel's Sub-NUMA Clustering would work within > DPDK, and found that I actually didn't have to do anything, because the > SNC "clusters" present themselves as NUMA nodes, which DPDK already > supports natively. yes, this is correct. In Intel Xeon Platinum BIOS one can enable `Cluster per NUMA` as `1,2 or4`. This divides the tiles into Sub-Numa parition, each having separate lcores,memory controllers, PCIe and accelerator. > > Does AMD's implementation of chiplets not report themselves as separate > NUMA nodes? In AMD EPYC Soc, this is different. There are 2 BIOS settings, namely 1. NPS: `Numa Per Socket` which allows the IO tile (memory, PCIe and Accelerator) to be partitioned as Numa 0, 1, 2 or 4. 2. L3 as NUMA: `L3 cache of CPU tiles as individual NUMA`. This allows all CPU tiles to be independent NUMA cores. The above settings are possible because CPU is independent from IO tile. Thus allowing 4 combinations be available for use. These are covered in the tuning gudie for the SoC in 12. How to get best performance on AMD platform — Data Plane Development Kit 24.07.0 documentation (dpdk.org) . > Because if it does, I don't really think any changes are > required because NUMA nodes would give you the same thing, would it not? I have a different opinion to this outlook. An end user can 1. Identify the lcores and it's NUMA user `usertools/cpu-layout.py` 2. But it is core mask in eal arguments which makes the threads available to be used in a process. 3. there are no API which distinguish L3 numa domain. Function `rte_socket_id ` for CPU tiles like AMD SoC will return physical socket. Example: In AMD EPYC Genoa, there are total of 13 tiles. 12 CPU tiles and 1 IO tile. Setting 1. NPS to 4 will divide the memory, PCIe and accelerator into 4 domain. While the all CPU will appear as single NUMA but each 12 tile having independent L3 caches. 2. Setting `L3 as NUMA` allows each tile to appear as separate L3 clusters. Hence, adding an API which allows to select available lcores based on Split L3 is essential irrespective of the BIOS setting. > > -- > Thanks, > Anatoly > --------------VITCIWUFQtMLPc17thORNsaM Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

<Snipped>

Thank you Antaloy for the response. Let me try to share my understanding.

I recently looked into how Intel's Sub-NUMA Clustering would work within
DPDK, and found that I actually didn't have to do anything, because the
SNC "clusters" present themselves as NUMA nodes, which DPDK already
supports natively.

yes, this is correct. In Intel Xeon Platinum BIOS one can enable `Cluster per NUMA` as `1,2 or4`.

This divides the tiles into Sub-Numa parition, each having separate lcores,memory controllers, PCIe

and accelerator. 


Does AMD's implementation of chiplets not report themselves as separate
NUMA nodes?

In AMD EPYC Soc, this is different. There are 2 BIOS settings, namely

1. NPS: `Numa Per Socket` which allows the IO tile (memory, PCIe and Accelerator) to be partitioned as Numa 0, 1, 2 or 4.

2. L3 as NUMA: `L3 cache of CPU tiles as individual NUMA`. This allows all CPU tiles to be independent NUMA cores.


The above settings are possible because CPU is independent from IO tile. Thus allowing 4 combinations be available for use.

These are covered in the tuning gudie for the SoC in 12. How to get best performance on AMD platform — Data Plane Development Kit 24.07.0 documentation (dpdk.org).


Because if it does, I don't really think any changes are
required because NUMA nodes would give you the same thing, would it not?

I have a different opinion to this outlook. An end user can

1. Identify the lcores and it's NUMA user `usertools/cpu-layout.py`

2. But it is core mask in eal arguments which makes the threads available to be used in a process.

3. there are no API which distinguish L3 numa domain. Function `rte_socket_id ` for CPU tiles like AMD SoC will return physical socket.


Example: In AMD EPYC Genoa, there are total of 13 tiles. 12 CPU tiles and 1 IO tile. Setting 

1. NPS to 4 will divide the memory, PCIe and accelerator into 4 domain. While the all CPU will appear as single NUMA but each 12 tile having independent L3 caches.

2. Setting `L3 as NUMA` allows each tile to appear as separate L3 clusters.


Hence, adding an API which allows to select available lcores based on Split L3 is essential irrespective of the BIOS setting.



--
Thanks,
Anatoly

--------------VITCIWUFQtMLPc17thORNsaM--