From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7CC7045C87; Tue, 5 Nov 2024 11:29:18 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 19DEE402B3; Tue, 5 Nov 2024 11:29:18 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2074.outbound.protection.outlook.com [40.107.237.74]) by mails.dpdk.org (Postfix) with ESMTP id 3421A40151 for ; Tue, 5 Nov 2024 11:29:16 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ME44f6a4UP5aL6i031aRElcvNHY2J9T/CIfeGuQuFqN3VgW6zyL/EIoQgbJ17IN48XO4IDxagmnKOwYMbWlV4SZZoJA9ACYTJTc5pOW7St5TU/BpPX599ULcQyja+N+YEskNkEHC8buyjBI/I/qknewqpCPfERb+2NaXSZdkIbpSNm2wMA9t8ji+6BJ0Ef9FVr0vIYtEUKh3mPcQ3JFs7RWLsTqp0JMHrtPy05X9R6eMI44vXVO1zKibmF6Z9I9AsocBjUdJBLvKrAB2GFAqjwWVmKkXs+kPMHfJTBSLGpcqAqr81gJrXNlWitIeC6N432jr6Ebvhw6fkvm3l2AW2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=057eWRqEAz8Wr6XL4kZUM4UBc+UCaGLa5QIxH6uBS0E=; b=az9dY3QTnNW6ZxX6R52cNzRTCACKwkC0B5oKh2yLaHVNa4ulQWkGoiHjjd7JR1TpYhR4hak/dH4EdRJbGqGHq215GuRKOahP+8ATJu8axVWbkEdEh+bFUWnucHNK61pUWLac9aHN7sQ6Y2X04VsHdWHW48Rs0eecsDMZr+o8a8C6C0Y53a+CbY4LbG+FaP1r7pwmxYAtfkFm995CqlnMXvhdrrBK5DZNXYjVn4SWJPDx9yhKUBdCDKpHqvrYF1Xy4eOBIepK8APnqbWRwdycv4ggGgVlqtU3v5D+fIybdlTnxjny5+vBOMGY1/76T7av77yIZ9+FqQN2tkOti0IH6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=057eWRqEAz8Wr6XL4kZUM4UBc+UCaGLa5QIxH6uBS0E=; b=Sv74Dup6qEVQ0f6ooewNyxYn9fn4ThoVwp4Sgqlyg+wrR8WPF91DuQ57xfpVXb6RADJAyVqDikTmmCC80DeJvMDDNXPRdRoatHvmGzC4F+X2OWhVHyeYGuQeJ5Ap3iMa9DefLyM1M26ZQGgxsWrJVT+VI40dJKP4/124R4oHgvE= Received: from SA9PR13CA0121.namprd13.prod.outlook.com (2603:10b6:806:27::6) by CH3PR12MB8755.namprd12.prod.outlook.com (2603:10b6:610:17e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.31; Tue, 5 Nov 2024 10:29:13 +0000 Received: from SN1PEPF000252A3.namprd05.prod.outlook.com (2603:10b6:806:27:cafe::b4) by SA9PR13CA0121.outlook.office365.com (2603:10b6:806:27::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.16 via Frontend Transport; Tue, 5 Nov 2024 10:29:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF000252A3.mail.protection.outlook.com (10.167.242.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8137.17 via Frontend Transport; Tue, 5 Nov 2024 10:29:13 +0000 Received: from BLRVIVARGHE.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 5 Nov 2024 04:29:08 -0600 From: Vipin Varghese To: , , , , CC: , , , , , , , , , , Subject: [PATCH v4 0/4] Introduce Topology NUMA grouping for lcores Date: Tue, 5 Nov 2024 15:58:45 +0530 Message-ID: <20241105102849.1947-1-vipin.varghese@amd.com> X-Mailer: git-send-email 2.47.0.windows.1 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A3:EE_|CH3PR12MB8755:EE_ X-MS-Office365-Filtering-Correlation-Id: 43704778-47f6-4687-74ac-08dcfd84b15f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|7416014|376014|82310400026|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?B?ZnhPVjBLekU3MGZQOEdNRDl4akV6Zm5rZSs2SlJuK0RaWHFDbmxkK0FRV0pp?= =?utf-8?B?TTFZRW14Q2xCakM0TFdMS25GMlM3cE93aXVoSXh5aHVBUERqWHRxeGE1N3J0?= =?utf-8?B?c3FseDQ0RTM5L3RoczQ1dDRIVElzSGtGZ0tpdFVib09yV0wxUTI1Y2I5RDNJ?= =?utf-8?B?bE16TVBFUUxGcFY5RlBHTjMxS3F6SktuTW15TXZ4RTNCU3BHc0djUHl6dXFa?= =?utf-8?B?dWZTanNlNERPeHlHTm1waDdWUGpQSndlNEVMWmJKdHpWQ3g1Y3ArbmREUHIr?= =?utf-8?B?a3FLRThSbzhLMmlVN1Jaak9uWDlWbEJtZzBzbGlsd3JQSzdtMmlpMzhXTUxy?= =?utf-8?B?SVZ1U2ZKRjR6UE9aV0liREZFSk54VlB5NGRCcjcveVoyeThtWGUrSTYzUVRu?= =?utf-8?B?SjZSQ2l1UEN5ZkE2R2hyTzBvNVlUamFtbEpVRmRjNU5KRXhVODhIM2lCSnlt?= =?utf-8?B?T2xsaFpZMGhYbXpZMjBDa1pLQVJpUkJranh4RkQxQ3FzWnN5WHNlT2U0Wkwr?= =?utf-8?B?MGFHUmRPanRGVzh0NnRjWGJoRlVGWWtwQnBTT2lqTFhpekhZR2VTeDdoOGZD?= =?utf-8?B?SHAwYlRoMHZpdllxaUVBb2I4cnAydUV1YThsRGhUbVlzbm1udlhkYkN5bm8r?= =?utf-8?B?UklPZ1lieko2bFc2Q2VQQWlUTWIzS1VlLytRZ2JKK0FnKzhNMC9FaEdGL2JB?= =?utf-8?B?cHdOUE05NHE4N3RuMDhmQzNFYW9pckNMb1pJbUJHUTFKK0x5SXZnUFpLSEFj?= =?utf-8?B?YmlnejRvWmxuNjg4dzJpSXJ4eHJQSWl4OUFWTi9vUkNtTWorVWpobWJCR285?= =?utf-8?B?V3o0bXR2TUtxMzd5NTFQbGZPdnhqalpSNDJnbWNabDI5Mk1SNDBZVG16RzJN?= =?utf-8?B?SUZQVU5ncVVlNUFkNmM0c0FHMERaZmo5Mmk1TE92K3o0dVJrZFlWZW1aZkc3?= =?utf-8?B?NEdnNnF2cERyb2FKN1hxQXlEeEdZa0IzaDNKUWVJNjF0SDRxdnhTaXJSbDdH?= =?utf-8?B?MFluT2lBZ2I5c2hVRkUwTWNHWHRXUXJFUlJBZVVKVmZJRXZkMEJ3aFZLM1lW?= =?utf-8?B?ejZmZExFWnJabzBpZnZmK08rMlBLaGdVdHRTRytybmRtRjQxck9FTkEwamdL?= =?utf-8?B?Z25uRVlrWFdudHo5TTJEaklzMEk2Rit1RGVMWm9PajhUY2hHWU5BWmZvS2V2?= =?utf-8?B?QVh6ZXhZMzdCaEFDMnhnQW9jU3ZLZ1p0dHhYMlB4b25yZFZNcTV3akNJaHN5?= =?utf-8?B?VXoyZ2g3V21GbEpDaVl1S3lhSi9uY2Y4YVZnLzJvTlNUOXJ5OUlmRm0zK1lr?= =?utf-8?B?RXMxSWp3aEx2a2hoQWdDU25GQk9oNzcxNUtudmJRWVNESDNpRHJybU8zZytj?= =?utf-8?B?UnNYL1JTcHNlMDB1alNoMFhKZjNYOXBTc2J2TFZpUTdOSVBhTE41Slk4UkV6?= =?utf-8?B?Y0ZYTUMrZHk2Yjh1UEgvREh3OHg2MzIrUDU1K2RRTGpHV0ZTNXlNQmxxaFls?= =?utf-8?B?WVo5ZzFDM1ZGcWRJTGNtODBDNFJzdmRCOXg0Wkt4TXk4b2VZUUFqUHE1VExB?= =?utf-8?B?cmxNMDdpazh2NTJyajFTK2t1Y2x4bHFPYTFLR1ZoNVdITk1hZTl5N015TCts?= =?utf-8?B?NmRhRzZyMy9iTUpHWTlJNUUwNUtGaWtKK0VtL2NiM0lXRDdrVzIwb2dzemYx?= =?utf-8?B?Y05OdXR3MlZoTGpOOVZHTzZDYXZYbVZ6OU1BN0dQZGNNK2dCbnFNWXRXUktZ?= =?utf-8?B?cHlWTzZ0V2Z5UE0wMzE0ekNhUzdQd1VuU0hkemRHUUE2R2Z6L2F2Z1NJaHB1?= =?utf-8?B?RlZJOVMvaFhrQVdXcnp6N292b2JWWXhpTWczWDREcnI2TUJPWTdSMXpGaG5C?= =?utf-8?B?bEYyWjg1cDdqWWtqd1hBOFNNWW1XTlByZkQrUTNCSE9JdUE9PQ==?= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(7416014)(376014)(82310400026)(1800799024)(36860700013); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Nov 2024 10:29:13.2163 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 43704778-47f6-4687-74ac-08dcfd84b15f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A3.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8755 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch introduces improvements for NUMA topology awareness in relation to DPDK logical cores. The goal is to expose API which allows users to select optimal logical cores for any application. These logical cores can be selected from various NUMA domains like CPU and I/O. Change Summary: - Introduces the concept of NUMA domain partitioning based on CPU and I/O topology. - Adds support for grouping DPDK logical cores within the same Cache and I/O domain for improved locality. - Implements topology detection and core grouping logic that distinguishes between the following NUMA configurations: * CPU topology & I/O topology (e.g., AMD SoC EPYC, Intel Xeon SPR) * CPU+I/O topology (e.g., Ampere One with SLC, Intel Xeon SPR with SNC) - Enhances performance by minimizing lcore dispersion across tiles|compute package with different L2/L3 cache or IO domains. Reason: - Applications using DPDK libraries relies on consistent memory access. - Lcores being closer to same NUMA domain as IO. - Lcores sharing same cache. Latency is minimized by using lcores that share the same NUMA topology. Memory access is optimized by utilizing cores within the same NUMA domain or tile. Cache coherence is preserved within the same shared cache domain, reducing the remote access from tile|compute package via snooping (local hit in either L2 or L3 within same NUMA domain). Library dependency: hwloc Topology Flags: --------------- - RTE_LCORE_DOMAIN_L1: to group cores sharing same L1 cache - RTE_LCORE_DOMAIN_SMT: same as RTE_LCORE_DOMAIN_L1 - RTE_LCORE_DOMAIN_L2: group cores sharing same L2 cache - RTE_LCORE_DOMAIN_L3: group cores sharing same L3 cache - RTE_LCORE_DOMAIN_L4: group cores sharing same L4 cache - RTE_LCORE_DOMAIN_IO: group cores sharing same IO < Function: Purpose > --------------------- - rte_get_domain_count: get domain count based on Topology Flag - rte_lcore_count_from_domain: get valid lcores count under each domain - rte_get_lcore_in_domain: valid lcore id based on index - rte_lcore_cpuset_in_domain: return valid cpuset based on index - rte_lcore_is_main_in_domain: return true|false if main lcore is present - rte_get_next_lcore_from_domain: next valid lcore within domain - rte_get_next_lcore_from_next_domain: next valid lcore from next domain Note: 1. Topology is NUMA grouping. 2. Domain is various sub-groups within a specific Topology. Topology example: L1, L2, L3, L4, IO Domian example: IO-A, IO-B < MACRO: Purpose > ------------------ - RTE_LCORE_FOREACH_DOMAIN: iterate lcores from all domains - RTE_LCORE_FOREACH_WORKER_DOMAIN: iterate worker lcores from all domains - RTE_LCORE_FORN_NEXT_DOMAIN: iterate domain select n'th lcore - RTE_LCORE_FORN_WORKER_NEXT_DOMAIN: iterate domain for worker n'th lcore. Future work (after merge): -------------------------- - dma-perf per IO NUMA - eventdev per L3 NUMA - pipeline per SMT|L3 NUMA - distributor per L3 for Port-Queue - l2fwd-power per SMT - testpmd option for IO NUMA per port Platform tested on: ------------------- - INTEL(R) XEON(R) PLATINUM 8562Y+ (support IO numa 1 & 2) - AMD EPYC 8534P (supports IO numa 1 & 2) - AMD EPYC 9554 (supports IO numa 1, 2, 4) Logs: ----- 1. INTEL(R) XEON(R) PLATINUM 8562Y+: - SNC=1 Domain (IO): at index (0) there are 48 core, with (0) at index 0 - SNC=2 Domain (IO): at index (0) there are 24 core, with (0) at index 0 Domain (IO): at index (1) there are 24 core, with (12) at index 0 2. AMD EPYC 8534P: - NPS=1: Domain (IO): at index (0) there are 128 core, with (0) at index 0 - NPS=2: Domain (IO): at index (0) there are 64 core, with (0) at index 0 Domain (IO): at index (1) there are 64 core, with (32) at index 0 Signed-off-by: Vipin Varghese Vipin Varghese (4): eal/lcore: add topology based functions test/lcore: enable tests for topology doc: add topology grouping details examples: update with lcore topology API app/test/test_lcores.c | 528 +++++++++++++ config/meson.build | 18 + .../prog_guide/env_abstraction_layer.rst | 22 + examples/helloworld/main.c | 154 +++- examples/l2fwd/main.c | 56 +- examples/skeleton/basicfwd.c | 22 + lib/eal/common/eal_common_lcore.c | 714 ++++++++++++++++++ lib/eal/common/eal_private.h | 58 ++ lib/eal/freebsd/eal.c | 10 + lib/eal/include/rte_lcore.h | 209 +++++ lib/eal/linux/eal.c | 11 + lib/eal/meson.build | 4 + lib/eal/version.map | 11 + lib/eal/windows/eal.c | 12 + 14 files changed, 1819 insertions(+), 10 deletions(-) -- 2.34.1