From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id A2E8C45BCD;
Wed, 30 Oct 2024 06:43:39 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 5D13D42FB9;
Wed, 30 Oct 2024 06:43:39 +0100 (CET)
Received: from NAM12-MW2-obe.outbound.protection.outlook.com
(mail-mw2nam12on2068.outbound.protection.outlook.com [40.107.244.68])
by mails.dpdk.org (Postfix) with ESMTP id 0582842EC5
for <dev@dpdk.org>; Wed, 30 Oct 2024 06:43:38 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
b=pmdOcViVFir+Rwo5eInnfMgcB4kZkr+UozyzF3B0nqv7lr0WHQTyVfATvnQjIFMioA8OmOgAe3AzRFTi2d8CeKq/OtDA6py0Yf91NCFVVg/UtMi3o4eUAQeTaOpUoZgfkdJ+ORfORQFqBIxxGUQQwjJfodkP3Q23C/O/qSv+HJIOtR3AHMaAI0JVCogDHV9DmeBRQUm/x5YbTPdVT9Vv+n/eI4HIMhlcJ+RQOT27LuOHfjHE2EG55WRIctmnsB7uYb/tu8yf0Fd6XSIKQHYhAg3qjTRnvTmmFjAebZjtK+owaqCbqsLOZLvW7o5tgwrSPbNxzoYZdYgvmzrEdn0TrQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector10001;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=c+TnBRnQ9y+48u+r1ISq6xiy/lGMa2W0rDSzCcMc1zU=;
b=ZhEqCOPxDcQ7PIF0rPCDOtx23DOy+rryte1HmHLyv0XmHoU5d6nQg5Y5KIkDUimtIuH24qlP+IY6WLujBr2F//S+qSh/nbiUYPCOkydLIZIwphCIZ1jLjfEdM2Zl0Vzmu1+On1YkilmRz2ZAqK0xh8yDVgnMEDCNKqxRrj2Mmqz7wqc50a1jzqKafdI6rrGYrCrx9QTxHVUxt98b+OaZMEmSDIhsQrGGW65I2HEphwyr5CcAAUL7tvRl9hvX6foef99xHE7K0CAr3k81DJ3aWD1/U+oRcFg/hd/wCnfROEFzdf9gP8lAFs8I/he07yy8zpXmg17AgRdcB4XzYnz99g==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass
(p=quarantine sp=quarantine pct=100) action=none header.from=amd.com;
dkim=none (message not signed); arc=none (0)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=c+TnBRnQ9y+48u+r1ISq6xiy/lGMa2W0rDSzCcMc1zU=;
b=C7angzITFrOFoKJKLSrgT/zFAxbDQh74KVZKBgWKA5DdHyRQ2ua/XbAx3QO73tol/QtKP+xu7Ps6FG8uspHre41ZlxBJFAdxqlxA1eua6g2PZzq2B8pNcfeL9Y4U+bKgYUDS8mRBP6iHFaaPdWsNfiRSJAvVXJCk8gB/VWfqtrs=
Received: from MN0P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:208:52e::28)
by CY8PR12MB8266.namprd12.prod.outlook.com (2603:10b6:930:79::5) with
Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.25; Wed, 30 Oct
2024 05:43:31 +0000
Received: from BL02EPF0002992E.namprd02.prod.outlook.com
(2603:10b6:208:52e:cafe::3b) by MN0P220CA0013.outlook.office365.com
(2603:10b6:208:52e::28) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.17 via Frontend
Transport; Wed, 30 Oct 2024 05:43:31 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17)
smtp.mailfrom=amd.com; dkim=none (message not signed)
header.d=none;dmarc=pass action=none header.from=amd.com;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
165.204.84.17 as permitted sender) receiver=protection.outlook.com;
client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
Received: from SATLEXMB04.amd.com (165.204.84.17) by
BL02EPF0002992E.mail.protection.outlook.com (10.167.249.59) with Microsoft
SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.20.8114.16 via Frontend Transport; Wed, 30 Oct 2024 05:43:30 +0000
Received: from BLRVIVARGHE.amd.com (10.180.168.240) by SATLEXMB04.amd.com
(10.181.40.145) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 30 Oct
2024 00:42:08 -0500
From: Vipin Varghese <vipin.varghese@amd.com>
To: <dev@dpdk.org>, <roretzla@linux.microsoft.com>,
<bruce.richardson@intel.com>, <john.mcnamara@intel.com>,
<dmitry.kozliuk@gmail.com>, <jerinj@marvell.com>
CC: <ruifeng.wang@arm.com>, <mattias.ronnblom@ericsson.com>,
<anatoly.burakov@intel.com>, <stephen@networkplumber.org>,
<ferruh.yigit@amd.com>, <honnappa.nagarahalli@arm.com>,
<wathsala.vithanage@arm.com>, <konstantin.ananyev@huawei.com>
Subject: [RFC v3 0/3] Introduce Topology NUMA grouping for lcores
Date: Wed, 30 Oct 2024 11:11:30 +0530
Message-ID: <20241030054133.520-1-vipin.varghese@amd.com>
X-Mailer: git-send-email 2.47.0.windows.1
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-Originating-IP: [10.180.168.240]
X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com
(10.181.40.145)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: BL02EPF0002992E:EE_|CY8PR12MB8266:EE_
X-MS-Office365-Filtering-Correlation-Id: 4fc64e17-0521-4a18-9c49-08dcf8a5c939
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
ARA:13230040|36860700013|82310400026|7416014|1800799024|376014;
X-Microsoft-Antispam-Message-Info: =?utf-8?B?VDdUZExMUWRCNzdnMGx5SGo2Qjdqa0hLMGpzOFU1M2pxMDNYR0Fjay9OaEpx?=
=?utf-8?B?UlFhdHFMSWFaY2UyaFlQa1Uwc24zM1Z5QVAxL0hsbjNxNkdzcW1PNlY3b3Bs?=
=?utf-8?B?dUhGYjJaWnFLanJvbytZRkt2TmYrYkJFUU1KeXF6K3FRVTAxQ0FvS0g2bWhF?=
=?utf-8?B?TUUySjF6M0NLeVZBOUdaVHZBcWUrNTZ4K0E0dDNnS3gvbERnRytWZDNHZlhs?=
=?utf-8?B?YTBDYnU1UnR6QTJCdjBHZ29wVllJb1IwWkJQYnlIcjMrVWJzTHhxWDgyOGRJ?=
=?utf-8?B?YWVJVkIvdHJaM2ZyeHRURjhrMjcvWTRHREx0aGJvOVhuWGFhV0VSUFA2KzRM?=
=?utf-8?B?UWZhTzlZVDJCMkZMSWFtemorejNVME5KYTErRmtzTUJFKzBGUW8vTC9kVGZ3?=
=?utf-8?B?dE1xNC9WUVh4SGRpQ0ZEbkpyQm0rYUVUNFczajU4Q1liVkU1Y2FMRTBQKysr?=
=?utf-8?B?TzBTTzc1TU1BdGhqVWVEdzRGREUwNDArV1hyZExpUHpadUUveVlKNnlBUGtj?=
=?utf-8?B?Vm9CcEFOS0orL1paS2g5THNIRUZ4TXpJTnBLdHhCakJzMEh0MDNXcGRMN3Ex?=
=?utf-8?B?azBoUFJmVVBUeXJSTEF1VnNlY3EzTHdSbW1QaWtkcGtja3YwaUJkcElEck5C?=
=?utf-8?B?OVBJdlFYNmZnZkZzeVZRbEFONkJydGk1dFM2d2xhOGMyWjlITnRlSW9HSlhG?=
=?utf-8?B?MElMeHhnSlhRWm5xL2owWVF3MFFmeXk3Y2dQdGd1NGl1ZE9jZ1dLdFVtTHBU?=
=?utf-8?B?aW9NMFhaaDVGOFNTWUpvbTJpVWh3T215UUNjb1h3L2FSamVlcG9hTjdYWk16?=
=?utf-8?B?WWtVTWtiRzF5QWNaTjZRSktvUldneXUwblpBTHJGbkJsY2V4SStYQWlLS2RJ?=
=?utf-8?B?Qkp2a2RBam9VSGZpZjNURjJxZ1ZSSzVBa0hkeXdhcnBHRnNRWWJPbTk4M1Vv?=
=?utf-8?B?YkhSamFETzNxaXdzUE1BZ28ybHIzTmVtT01PRWZ1dUhYck1sbFlZTGlDUGJT?=
=?utf-8?B?TnlNYkFBbndpRDJBUXREZWhLNEEyN0NMQ3RvNHhqWUxPTVlDTlpleGU3bE52?=
=?utf-8?B?bzNTU1hmTlNYUWpzNTdYZCtDbWowV0hXODc1Y1FlNE9hRE9CRnNuU0hSNjdM?=
=?utf-8?B?Q0FpSGlvSVh2eGhLaDRjSVpTUnhSZVJET2RSLzhIOTZjTjhnSS9QRU1VZDM2?=
=?utf-8?B?K2szekhmOFdFU055Zi9mVE9KU1loM2lCakl6Ui9xdkxoZkhIWTkwUE5rNzlN?=
=?utf-8?B?RUFud0hhUlRrK0REUEcvUzJQY0x3a05vUGJ0M1hjcWtYUklvSzAwZHFMN2xi?=
=?utf-8?B?ZC9mSjdyak9LbktuY2xHcjhnS3o5VjhjdU9YV1Z2VWFlV01pZVllaXQ4NllX?=
=?utf-8?B?WjNmLzE3TTRmajh4Z3ZQdUQ4RW9KRjU3WEFtY2gxRWlnR1dsajZCL1d4b3hy?=
=?utf-8?B?V3QrYUN3alNPZEJkNXBUZ2Q5b2FFTW02KzdGbXEyUGhTVytMQlF3V042SzB2?=
=?utf-8?B?TkwvcXphUXhyb29BdEEyOG56cDI5QWVEVzY1OS9NenlabWNIRXdObUZscWkx?=
=?utf-8?B?em10N1pPbE5pOXhRZ2FJK3JicmtRZ01wdTR1akJaVkhvbXQ0RU9tU1RQeCsr?=
=?utf-8?B?YzNXTnFsbE9jL0JxRFpQbmpvWUZGME1rTG12U3RaN0o3dC9NaDgzMUcwdGNG?=
=?utf-8?B?ZzhIL081Y01oTDFSZXJVK3U2WHlrR3JkZExNbTlMSWZ6YnV6M3krR05FT1ZR?=
=?utf-8?B?VXIrZ3J6N1VxV21qQjFzaXd3S29md25PaDJERU4vREd0Lyt6ODZ3ODcyZWlG?=
=?utf-8?B?Z0NqRXpESlN6VWk4b051c0w2Vm5Wamt2L1gzbGhnYzNLOEtTSjU2aW4xNkRL?=
=?utf-8?B?bWdsaHRtSDFWNzlvNzIxamNneWxpVkNGTFdjQW9LSDZxVWc9PQ==?=
X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:;
IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE;
SFS:(13230040)(36860700013)(82310400026)(7416014)(1800799024)(376014); DIR:OUT;
SFP:1101;
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2024 05:43:30.8431 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 4fc64e17-0521-4a18-9c49-08dcf8a5c939
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17];
Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992E.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8266
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
One of the way to increase core density in a given physical package is
to use easy to replicate tiles. These tiles are either core complexes or
core complexes with IO (memory and PCIe). This results to possibility of
having two types NUMA topology.
- CPU topology & IO topology
- CPU+IO topology
For platforms like
- AMD SoC EPYC, core complexes are in separate CPU domain and IO
are in different NUMA domain (except for zen1 Naples)
- Intel 4th Xeon (SPR) & above, the CPU+IO NUMA partitioning is
achieved by BIOS option `SNC as 1, 2 or 4`.
- Ampere One allow CPU NUMA paritioning by BIOS option `SLC`.
- while other platforms has 2 or 4 cores sharing same L2 cache.
Grouping DPDK logical cores within the same Cache and IO, helps to
leverage application same cache or IO locality. For applications to
leverage cache or IO locality, ones needs to use lcores sharing same
topology. This approach ensures more consistent latencies by
minimizing the dispersion of lcores across different tiles.
Using lcores in same NUMA domain shows imporvement for applications
- using pipline staging
- each lcore processing part of payload
- eventual hit in either L2 or L3
Library dependency: hwloc
Topology Flags:
---------------
- RTE_LCORE_DOMAIN_L1: to group cores sharing same L1 cache
- RTE_LCORE_DOMAIN_SMT: same as RTE_LCORE_DOMAIN_L1
- RTE_LCORE_DOMAIN_L2: group cores sharing same L2 cache
- RTE_LCORE_DOMAIN_L3: group cores sharing same L3 cache
- RTE_LCORE_DOMAIN_IO: group cores sharing same IO
< Function: Purpose >
---------------------
- rte_get_domain_count: get domain count based on Topology Flag
- rte_lcore_count_from_domain: get valid lcores count under each domain
- rte_get_lcore_in_domain: valid lcore id based on index
- rte_get_next_lcore_from_domain: next valid lcore within domain
- rte_get_next_lcore_from_next_domain: next valid lcore from next domain
Note:
1. Topology is NUMA grouping.
2. Domain is various sub-groups within a specific Topology.
Topology example: L1, L2, L3, IO
Domian example: IO-A, IO-B
< MACRO: Purpose >
------------------
- RTE_LCORE_FOREACH_DOMAIN: iterate lcores from all domains
- RTE_LCORE_FOREACH_WORKER_DOMAIN: iterate worker lcores from all domains
- RTE_LCORE_FORN_NEXT_DOMAIN: iterate domain select n'th lcore
- RTE_LCORE_FORN_WORKER_NEXT_DOMAIN: iterate domain for worker n'th lcore.
Future work (after merge):
--------------------------
- dma-perf per IO NUMA
- eventdev per L3 NUMA
- pipeline per SMT|L3 NUMA
- distributor per L3 for Port-Queue
- l2fwd-power per SMT
- testpmd option for IO NUMA per port
Platform tested on:
-------------------
- INTEL(R) XEON(R) PLATINUM 8562Y+ (support IO numa 1 & 2)
- AMD EPYC 8534P (supports IO numa 1 & 2)
- AMD EPYC 9554 (supports IO numa 1, 2, 4)
Logs:
-----
1. INTEL(R) XEON(R) PLATINUM 8562Y+:
- SNC=1
Domain (IO): at index (0) there are 48 core, with (0) at index 0
- SNC=2
Domain (IO): at index (0) there are 24 core, with (0) at index 0
Domain (IO): at index (1) there are 24 core, with (12) at index 0
2. AMD EPYC 8534P:
- NPS=1:
Domain (IO): at index (0) there are 128 core, with (0) at index 0
- NPS=2:
Domain (IO): at index (0) there are 64 core, with (0) at index 0
Domain (IO): at index (1) there are 64 core, with (32) at index 0
Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Vipin Varghese (3):
eal/lcore: add topology based functions
test/lcore: enable tests for topology
examples: add lcore topology API calls
app/test/test_lcores.c | 189 ++++++++++
config/meson.build | 18 +
examples/helloworld/main.c | 142 +++++++-
examples/l2fwd/main.c | 56 ++-
examples/skeleton/basicfwd.c | 22 ++
lib/eal/common/eal_common_lcore.c | 580 ++++++++++++++++++++++++++++++
lib/eal/common/eal_private.h | 48 +++
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_lcore.h | 168 +++++++++
lib/eal/linux/eal.c | 11 +
lib/eal/meson.build | 4 +
lib/eal/version.map | 9 +
lib/eal/windows/eal.c | 12 +
13 files changed, 1259 insertions(+), 10 deletions(-)
--
2.34.1