From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1170A458C8; Mon, 2 Sep 2024 02:39:39 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C441C40269; Mon, 2 Sep 2024 02:39:38 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2063.outbound.protection.outlook.com [40.107.244.63]) by mails.dpdk.org (Postfix) with ESMTP id 76F20400D7 for ; Mon, 2 Sep 2024 02:39:35 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PViTJ49JwmhOoSNwlVio8iSEQnAanO0krT88TP1wBrMw8V1jdhTnF1VrE7RFnII5lwxyrG7cD2Jqve8Nv1aWapLRzyTvKo5zv09wtPW390oqTZWyg97gt3nDby8wx/0t5NAoEs8c+tpXoIMYCMPn+pgTW28P7HTttWAyhhHc1BT3AdVywuqlA17kNjzQMWHshogz3RT15Y8sUQwfdqFCieP3jENr9nMRhMUVvuJqzTxupMIFEMqewE8jqICmV85rSFWuBmtLrbEXKP708Aa+0X4ESjLYcxV9VNWYi0tAO3bBrNSZXdQKPhijpFIyo3MImg5Y8UsYvdSAaBZOjFcSiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ds6yv/l9mxOykhZce6BLHhHvqxBeFisoGGHYqO6hKXw=; b=yWbXIvFQ2zN++8h8w1Dp66KnXRIaLiAF6i2GezXYzFrHl3sZmWasEd8rG9CVKHMivE4K7MfTsF5QtF6yYKZ/h6hX3Qpo8ieCZIyGYNEViqC+6D5wRNgzl6uxW64+dLqYhcq+GUtSErvMjxnIxlXfnr7hx4sAnBVSp5beJ3oa9fCvnU4IigL4/QBrUpuJ8iP38b6+08kgSROQbO7mJy6B7zwQE7M4NCldD7Zvt83GAIik6SmLdcHG/wovF+ZxHs1Het+urd+EmLlkcySQJg5qvKWUHG3719l7j78agSlVdQlz16D/9DXHRkfkJ7EFqS715Z5cZ9x1fE1ITs8M0kiWRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ds6yv/l9mxOykhZce6BLHhHvqxBeFisoGGHYqO6hKXw=; b=MOK9MFAuIHBy5CEB4CJGL+fCWYI1nC8uy6rqxw6pnpXh0YVPtWihA0pIp21oMgCQebCbXgNngCg7CZgTsg5gfpD7S3VnC3sRVI33X5LxDa94+HK3mQ2NBCbA3lD7uyxVXbZUPIPqhmO6rFILmuPvs01PKmQLEMGkkQb4Wd8zJLk= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from PH7PR12MB8596.namprd12.prod.outlook.com (2603:10b6:510:1b7::6) by CH2PR12MB4070.namprd12.prod.outlook.com (2603:10b6:610:ae::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.22; Mon, 2 Sep 2024 00:39:29 +0000 Received: from PH7PR12MB8596.namprd12.prod.outlook.com ([fe80::a011:943d:7291:8069]) by PH7PR12MB8596.namprd12.prod.outlook.com ([fe80::a011:943d:7291:8069%5]) with mapi id 15.20.7918.024; Mon, 2 Sep 2024 00:39:29 +0000 Message-ID: Date: Mon, 2 Sep 2024 06:09:21 +0530 User-Agent: Mozilla Thunderbird Subject: Re: [RFC 0/2] introduce LLC aware functions Content-Language: en-US To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= , ferruh.yigit@amd.com, dev@dpdk.org References: <20240827151014.201-1-vipin.varghese@amd.com> <45f26104-ad6c-4e42-8446-d8b51ac3f2dd@lysator.liu.se> From: "Varghese, Vipin" In-Reply-To: <45f26104-ad6c-4e42-8446-d8b51ac3f2dd@lysator.liu.se> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: PN3PR01CA0044.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:98::21) To PH7PR12MB8596.namprd12.prod.outlook.com (2603:10b6:510:1b7::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH7PR12MB8596:EE_|CH2PR12MB4070:EE_ X-MS-Office365-Filtering-Correlation-Id: e3f87949-53a9-46eb-da6c-08dccae7b44f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?B?bkdLVXV0bWdjdFRrUm5UaEx6N1BwRDROblhsdElkbkpnRnAwR2plVG5wa29w?= =?utf-8?B?WGFkSlVzbGh1YUEyTFlBUmQyRm9GYlZoeXZoT2tRTndxZW1VZ250ZTV4NTVC?= =?utf-8?B?Vmo4aGtsS0lKam54TmY0bXAydFBCZXBNU0tqVjE5WkdTektkMlk1OGI2MklH?= =?utf-8?B?WjJ3dFVQbDJoRHdJMHh6NlpXR0wrWURGajY2ZFI1QTBnUDIrZmNvZEJqZSt5?= =?utf-8?B?eHMyL0d3NUVyN2R0Z25mOFJLZDlOVzN2LzYzWG1Jc2tPb2RzLzV2Z3JMZW9S?= =?utf-8?B?elZ3V2dXc0hVcHVxODJYU2xLVENxWE92RlJjTkZTVEtZcTR4R3FWVnpnR2Fx?= =?utf-8?B?V2ZBU2gycEd1Z082My9PN2gwUlZ6bWRDWGkwWWtVb2ZIK0t6anR3WGU3eDVz?= =?utf-8?B?UG1MVnNwVVlhaTh5OWlLU2NGZmtLOHJxdWhmblBhZFZyTkhLOEFUaTRCenc5?= =?utf-8?B?U25XSkxOUmxVNjFHOUlCc1pSSG1wZ1pRT1pRUEVpU3YrOGhuYTl3cHU4ak5U?= =?utf-8?B?MWs2SFNQVkFKY0pudkZVSEk0dlE5MzJsQjQ2blNNeXhNUldyZWZoU1I2WVgy?= =?utf-8?B?cEhiQW1ZcEpKZGNNRFlsS254K3pnSjB4a0hYakhDaSt3UFc0NWRjVnpsMytP?= =?utf-8?B?UmFDNkV5R2JJY3RWOGdzcnEremY5M1dic0JydlJRL3BzVGM0cWxteHI4RUox?= =?utf-8?B?WFJyeU1RREQraFZLQTB0eENHUHMwUWpOU1ArVGplTDR5Z3NPczRDc05zVHdZ?= =?utf-8?B?YWRzbURGSW5ucm5sWkZOSHU4ZExBVTEzTzdFZU9uL1MranFXenQraVBIVDFt?= =?utf-8?B?c3V1VEVlZVF5a2JZUXFReXE3VEdpS1FLQmxMazV1dk9pRFZqM2ZOS2NEbWxT?= =?utf-8?B?MzdQWmFlakY2ZDkxVEQ5dE1QNlZuVjJrQjJneUFDQ0w0bmZBbDRpQy93eFJ5?= =?utf-8?B?VzcrK2tHUTFqdUVCaU0yRWpIdHZPZXFhQllmQlhCZjZoU2FodC9CMnZiallE?= =?utf-8?B?bEZhdU5HZklTdkNTbTFtVG94bmo3Z013Sy9OeVUyWkg2ZGZSQmw2TGkyMDFD?= =?utf-8?B?K2pUcEpaTkN6aWdldzBETHdFREoybUFpYWViVlB3YkowMnFoaFJUR2Fkblkz?= =?utf-8?B?TXVoRmw3bVIwNlJCb1JoWmZzYjBUVStUYzRRU1VHMTRldXFUcUpPYk1VZ29u?= =?utf-8?B?VTE2MG92RmRFeGJmQ1BIWFp1S1lrV05uTmpVMkdBdHI3QXMzcHYxWEtTSlor?= =?utf-8?B?S0FlUmtjT3JWK2NuZ3lmMkpkM093L0FhYU9LeDJOMzhId0lmOURLTHZ3M2Ur?= =?utf-8?B?ejMrM0hZOHhycGY0SkxOK2pUSkJKMm9PZmJGUXRaYW4zNjFKTThpMFRNNi9O?= =?utf-8?B?WHNpY2dqUkFackxBUlBXQlplNC9VRjUwWEE2bWw3RzhjTGtySGJXQ3BOblp0?= =?utf-8?B?RlRqaVp0QTYxSzFMUzBVbVNMbzltT0c4OTFjWnZycnZ2NGYzcGUzRFJuTkRX?= =?utf-8?B?WEF0RHRUbmFZdnVMWFJ6bWgzc0Z4Z2dxemRvL2ZWdXBBTktHTkJodnNzWHdu?= =?utf-8?B?bGhjQjAyVnZlTzZJT2xiN2VHd1FZVWdQQ1E1Y1R3T0FhZzk5U293L2xkOXFw?= =?utf-8?B?a1NxRFZ3N1N5bmFhdlF2OVJSMDMyczNLQldiWXpuVGc0SHVMaDFPemltR3l3?= =?utf-8?B?UFZDbWcwbWgwTFdqa3Z1cmlhT2E4WEdVUC9JU1hIUU8vVFVwQllFbzVOQUdE?= =?utf-8?B?TDBBeFVoWnRVRU9wVDlpZFJPS1pOaU1XZjh2WUxsMWRZaUhGTTk1QzFQY3Ft?= =?utf-8?B?V01tUUVBV2NqZ2hSRHF1UT09?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH7PR12MB8596.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(1800799024)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NGVQaHE4OFp4bHpSc0pUZnVDR1hlVjNUekIrWUNZL2ZIWjNodHBIc1dZS2JM?= =?utf-8?B?Rkx5S3V1Yit0Uk1Yam1CbkhZRFkxODgzYUFldUlxQ3BzYllhRjRjWGZvMVJy?= =?utf-8?B?aDBvZFNmMjRHZGd2RCs0aFVOYjA3dG1WdHpZNkRVVkV2UmgxSm1FbVZmdk02?= =?utf-8?B?Ky9TMEJVakhhTndTTG5zaUI3QW5RRHNGVjdyRDl2dHNtbkJ4T1lJeDRqRW9j?= =?utf-8?B?Q1RIcHBBU3VWTnlHTUVoZStteUNRNTczTjRCbEw5TGM3cTFIY2ljQ3h0YkZM?= =?utf-8?B?QWZGdloxQmpQVnlRRlZzSXRtYkkwRTA0SWlnbk5ZSGpvZDVvQTUwVlgxY28w?= =?utf-8?B?bzU4akJLTDB3SlNLN0NxeWdLazRmSTdmdTlyand6cE1YenBGckl3TjJub0hU?= =?utf-8?B?YWtOTUlrZnRwWTkybG5BcWJBaGVPMFMyMmJRdUEwaFZzcTc5ek0reVpCcUJR?= =?utf-8?B?WTBNL21hdVY2SDNETms5NzJad1gxazZiaVBlbVNURnBMRjhuVTFKS0dRNitX?= =?utf-8?B?K0xKUy9MWkthMWFacDVCWHRRU2EvM1V1cGdxUDhITmkrWEdnVS92dzlFUTZV?= =?utf-8?B?Z3ptbGYwSUtraWVkZ2UyV0pZU1lpNC9IUHJ0Y2pTWHFKZnBTeTVOeEZ2MWZZ?= =?utf-8?B?RytXTU9kZ2dFUlR1S0pqd2VXRXlMRU1XMVRwRzdoVzBuQkZjQ0hnZmgvN1cz?= =?utf-8?B?LzlTb0swSiszMWVYck5URWhaWFBHSkkxUk53MDhFZ2djbUdXR3hZaEFuRTFN?= =?utf-8?B?WVlFeDE4WkJyQmtjOUxNNHM0ZVhMcThrVFk1eTFLMlJZcWJWaVZhK0NUUkRW?= =?utf-8?B?T2lYV2xCTzViU05mR040R2N4aGZCL2xjVDRIOWQyQXcyQk5SakIvTjJ3SDVE?= =?utf-8?B?OVR3WE04eUc0T1N5VDFxRURiUUFWa3ljT3NwaEhzRnhUMFM5TTIvK0FXdndk?= =?utf-8?B?aWp2VThMMXpGd0dGMXBCNmpzNTZLZ1c1KzdLUkRPeUUvYWtkVmtBbnh2OWgr?= =?utf-8?B?WGdFMHNxVG1HWURoZUxXN2lCUzl2emtTSWNlcEdDQW1yckdyUlRvclRBMzE4?= =?utf-8?B?cVNPQjFFK05RVjN1dWoxNUUreTJtRC9LSUVneVhQNFhNSGdzTEkyS1lSUDZW?= =?utf-8?B?bW1tRkNVbzFneXdkaFJBd2EvSVJEejVEQWM0ZkU5ZitvcG13YnBaK3Q3ZFRk?= =?utf-8?B?K1VkcnBkOVdQTENrOEVXTVcyc2pGa0pSRFdKcVNRakRibHNxWTlycS9vU1NC?= =?utf-8?B?UVlIaC9PeDdxSlFCS1V0TVlQM0NCYjk2d0dnamFtczNNRFB2cmVVOVhHSFhK?= =?utf-8?B?YlRTekFNVmlUbXIvQzVmUmExYXQ4U1VPWEFUalpvTlEvREdKMHhpdVdIbW1Q?= =?utf-8?B?a0V3aFFOQldMQStTMzNkNjkwcDZDejQ2eXNVOU80OVFzV0J2Qm5JWU1xTVpv?= =?utf-8?B?VGR6bWU0dHp2N2F6dXhlT1p5ZkcvNTRMZ1pmd1djalFYMXk1OStrVGdIYkdk?= =?utf-8?B?cXdjdVBsbkYyNWNOVkhlbERwUGxNQzYzczJqcmtKRFNad3F5emtaSFVxRTJF?= =?utf-8?B?bXFHbXNUTG5EdnlpQVUrYTZvVUgveGEzM2dLcjVtUHo0NEprKzkvMDNmYlA5?= =?utf-8?B?SEhDVmlIUlZheHJDb1plekd3WkRKN2tDZ1E0QkI4bWduM1pPYVVFelF1VjZJ?= =?utf-8?B?WUVqd0NRZXBmTzlPNnR1WEEwdzcvTGgxNDBkeUxTNzducGFSWllrMEg5aWRL?= =?utf-8?B?LzIzSmgyUHN1VU1Tb3l2aDFPVnRlZkIxb3ozQk1xeXhLM0NScFRzOE5FaUdU?= =?utf-8?B?WDI0SEZuY2NVM0prYTgwcVNVUDdTWHl1SmF0N3c5S21xY1prMzBlT2JuV0Y4?= =?utf-8?B?ajZjUGlXb2xGKzhQbWtNR3Npb3Y5Q1puOG0zSTdKMmQrNUtRUzRnbWVQbDVK?= =?utf-8?B?UnJkSnd0YjVnaDlrdEhQVFN4M0V6TXUvWTdmRGxQTVVTYUdyVGM3ek9IdGZn?= =?utf-8?B?ZEkzVTB0Tlg1NTlGSE91OFg5M1FKQVMwLytUT2lvajBLTzkrZ25qcEk2bXhG?= =?utf-8?B?MUt6cktxMnZ0V1Zjb0lBMHpzRFVYeTdrdEV1UDcxSG5zcEhtV1VpaXRqcGs2?= =?utf-8?Q?Ek8mG1PtElUIhUTufxreBfvYO?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: e3f87949-53a9-46eb-da6c-08dccae7b44f X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB8596.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2024 00:39:29.4367 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AQMVV7EH+CWWhfCVu6k9ovplkEIdARwm2RMh8eja9Jtc0d8PkL7vqQdkyprztl11wu9FNJArcplnpbuEBm/vIQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4070 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Thank you Mattias for the comments and question, please let me try to explain the same below > We shouldn't have a separate CPU/cache hierarchy API instead? Based on the intention to bring in CPU lcores which share same L3 (for better cache hits and less noisy neighbor) current API focuses on using Last Level Cache. But if the suggestion is `there are SoC where L2 cache are also shared, and the new API should be provisioned`, I am also comfortable with the thought. > > Could potentially be built on the 'hwloc' library. There are 3 reason on AMD SoC we did not explore this path, reasons are 1. depending n hwloc version and kernel version certain SoC hierarchies are not available 2. CPU NUMA and IO (memory & PCIe) NUMA are independent on AMD Epyc Soc. 3. adds the extra dependency layer of library layer to be made available to work. hence we have tried to use Linux Documented generic layer of `sysfs CPU cache`. I will try to explore more on hwloc and check if other libraries within DPDK leverages the same. > > I much agree cache/core topology may be of interest of the application > (or a work scheduler, like a DPDK event device), but it's not limited to > LLC. It may well be worthwhile to care about which cores shares L2 > cache, for example. Not sure the RTE_LCORE_FOREACH_* approach scales. yes, totally understand as some SoC, multiple lcores shares same L2 cache. Can we rework the API to be rte_get_cache_ where user argument is desired lcore index. 1. index-1: SMT threads 2. index-2: threads sharing same L2 cache 3. index-3: threads sharing same L3 cache 4. index-MAX: identify the threads sharing last level cache. > >> < Function: Purpose > >> --------------------- >>   - rte_get_llc_first_lcores: Retrieves all the first lcores in the >> shared LLC. >>   - rte_get_llc_lcore: Retrieves all lcores that share the LLC. >>   - rte_get_llc_n_lcore: Retrieves the first n or skips the first n >> lcores in the shared LLC. >> >> < MACRO: Purpose > >> ------------------ >> RTE_LCORE_FOREACH_LLC_FIRST: iterates through all first lcore from >> each LLC. >> RTE_LCORE_FOREACH_LLC_FIRST_WORKER: iterates through all first worker >> lcore from each LLC. >> RTE_LCORE_FOREACH_LLC_WORKER: iterates lcores from LLC based on hint >> (lcore id). >> RTE_LCORE_FOREACH_LLC_SKIP_FIRST_WORKER: iterates lcores from LLC >> while skipping first worker. >> RTE_LCORE_FOREACH_LLC_FIRST_N_WORKER: iterates through `n` lcores >> from each LLC. >> RTE_LCORE_FOREACH_LLC_SKIP_N_WORKER: skip first `n` lcores, then >> iterates through reaming lcores in each LLC. >> While the MACRO are simple wrapper invoking appropriate API. can this be worked out in this fashion?