From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0EB9545BA0; Tue, 22 Oct 2024 14:07:11 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F1372402E7; Tue, 22 Oct 2024 14:07:10 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2080.outbound.protection.outlook.com [40.107.237.80]) by mails.dpdk.org (Postfix) with ESMTP id 35DC0402DA for ; Tue, 22 Oct 2024 14:07:09 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qt1HlIqdKTa93/mk4V7KaDOC9G62+AgaWElYMYk5L8poqVCmCPCygf73EgmJSCrK0WA5JarHCCtWPmzXG+j/B9GTPVQrWVxPVMLCVwzRsbuPAWrTvxaZ6V81IWJ4MT5kVNNpa+Krqx7A42kvqMinwzsT+uicClqTsLeOSmETAN0tjxF6AwiJ4DAAQvFBHRFimoCcDxgwPGTyYzXP/9zH99l5m1d9mqmA+Cn6byA/VmHkTL2LbWE1harfR1uNMC8gVP0r2lPqYM+Ldux6R73ghPzK5QKNy60Z/pr2e5Jn9UpBtCyjzXwAzkdjR/KIRo9Zk1QYrv6Iue5kWXP/o4f9Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MejQHc1PhFeM1jK59nLMWKT5VG17vn08PgcBNcIYlOQ=; b=hbKYf6GUeFH2CCch9PAYcyJLojoRRiMEaoWIPrV/r0u8PdY90DBU34MI1TMMT8eqXmXD0nB//5g8Mi3Bu7E7bP6RIfqd4ug54Ts21IL49+8R0bFm7/yHEpYfnd0tROpPhPNO4LKBTjG1TikD2LvS8Uor1NVOVBJ336Le1Ry/4E/8N0rk2rM5SdJH6AmIcWtaiLbCDOT7VQqhISWTu0rLJc8VDpvuoIS2urBdl2eK+BMZ2uoM2uZRNpyqZbAgI6QXrUeBHCBAEqFV7BF/WWiCwk8MPmusXvdN/hSkSE+N3q+V46P7v+TK6aJGM8zzdvAfTZIi8GOcjNr5k9Q0Mp96lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MejQHc1PhFeM1jK59nLMWKT5VG17vn08PgcBNcIYlOQ=; b=WkBbUub03F8dw8NnLNMG7LJHujaeGaLaJahs8hBM5S1O/upo64BssyK+NwZAO+gemHi34LTUeOqUn6uVLm9sgnUzEJS67tzT15bn4P36hom7yar5CKqbIYtZM0wDr2nOsQ3dLCl04l0mWphtY8P1jvHhAxMhyMYd8mkbJWOnohG08qulDBlgF01D+7TQqwzouiAqm4ACV6igAiLhtauWoOOPd2jA0ltyoWn0oArWff8VG44s5F0S+5OEgNNZ6lBXSkpoAJ3FT1vs4t5QQS1ysoduYzfutKy/N23lrFJNfoAZ3MVNh9wHZe1QR99lMbSCnOnpFE4MDQorPsQ3MjRPSQ== Received: from BY5PR16CA0025.namprd16.prod.outlook.com (2603:10b6:a03:1a0::38) by IA0PR12MB7579.namprd12.prod.outlook.com (2603:10b6:208:43c::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.16; Tue, 22 Oct 2024 12:07:05 +0000 Received: from SJ1PEPF000023DA.namprd21.prod.outlook.com (2603:10b6:a03:1a0:cafe::c8) by BY5PR16CA0025.outlook.office365.com (2603:10b6:a03:1a0::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.28 via Frontend Transport; Tue, 22 Oct 2024 12:07:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SJ1PEPF000023DA.mail.protection.outlook.com (10.167.244.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.2 via Frontend Transport; Tue, 22 Oct 2024 12:07:04 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 22 Oct 2024 05:06:52 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 22 Oct 2024 05:06:50 -0700 From: Dariusz Sosnowski To: Viacheslav Ovsiienko , Bing Zhao , Ori Kam , Suanming Mou , Matan Azrad CC: Subject: [PATCH v2 00/10] net/mlx5: improve MAC address and VLAN add latency Date: Tue, 22 Oct 2024 14:06:08 +0200 Message-ID: <20241022120618.512091-1-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241017075738.190064-1-dsosnowski@nvidia.com> References: <20241017075738.190064-1-dsosnowski@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF000023DA:EE_|IA0PR12MB7579:EE_ X-MS-Office365-Filtering-Correlation-Id: 0aa11bdb-d92f-402c-7edf-08dcf2920b17 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?tSd1Q9I8n7lrfXquN9KdByF5E6JW2IVGe8Lv40XXUGS/uVDJO8jD3Y2t1nJL?= =?us-ascii?Q?5sjUf2ekCinzCz+hG2pn0D8NWumhoDecghkz969wloJtrpaDqxD+Y8ymP3c4?= =?us-ascii?Q?GLUxUtqX9zKVextv2ba68pRqmGW0jReZbo0PLzoBqAJ4mbvxXx4ywDmLGGSo?= =?us-ascii?Q?jnfthBlxyz2/x8fJ3L8Xx4lN2fNfKYHvruHd4RStm1SXUOHHLoUlM6PqTKLL?= =?us-ascii?Q?GHsMk4A4O55L5BnTIPiBw8hjbEAiWpoWPqK/iW5z2aPccERnnr1LP4m+GDlm?= =?us-ascii?Q?xcZ1DL/mfTkeWPFtGRWGLxtnO2g5fmxi/j1brcsMm+ZQwbMoW5slcXfnrYj1?= =?us-ascii?Q?8NX3zY7N5D9pYd8FkML9Pda0P4n6ozbmOIqMVLSyG4Tky7PvcE0MIJ/64AaY?= =?us-ascii?Q?taQXhEkLrfk4ywSRLoXi/O1OyoR75qk2Feld0OVhylMO8VeOxRllOwo8PFbk?= =?us-ascii?Q?jKJtRkKEoT8cy2G0FOPu5UJuuHmjhkScWs3UUdLXMzrm6+SieyUedSdTQQxJ?= =?us-ascii?Q?nNip2jWEqncEhMm2HfGWwOIOp8WuGQLZttZPWg16c+TenmQDnELOkD3cTXFm?= =?us-ascii?Q?9YzoIYjIp9pqAKM90cVzqab8Tgrcchpq9AoRk9kZNb/BiHa3DjxJa52sR9RG?= =?us-ascii?Q?1Hgtflcel8/P7Vdl6ppdVKOM4W2C8vwn970FHUMhWOhd5of2RHzcvJB194UK?= =?us-ascii?Q?khxvXH9WZF+HiKNklBpBqBeP8HAZnaVYBvuH0qcKE8g6ZP+n/Hr4ynDNB1D3?= =?us-ascii?Q?5eL/YCTOaRdXnMoryDRtYKl5vpB9o1J7tB8cfUWohernxFnZnuDcwRWVj6tL?= =?us-ascii?Q?WIYgxgUaJaiouHz/SNjUWVUBJcAlHitxmeHpXEcUEkRghSHYm923Z3HKWn5U?= =?us-ascii?Q?AYb6JHm3vO3eIAyUHQuUglx3Nf/DgCL2/Xpk5UMNZLfWlG2gCapAAL9xUM2+?= =?us-ascii?Q?iCisLwNxZZdGxVBkMfzc8YK0juwnMJr/Vu+RlbwKDPKokkyx+nNMJrNlsDJr?= =?us-ascii?Q?j268GBxG1p+tW50erb/BQyd7i2mdXQfWmXGyb9w3eWpkliD0o0PvrkzMJEXG?= =?us-ascii?Q?6YPl8U1K5Z8/b+zFKLWb5Zcw6k2DhvZfAvmqoeooSDnvdqSAKRiJOpe2af40?= =?us-ascii?Q?LupeDuxMa97t/2gk6WUETVew83DrbxvNqrQ6d73Y+hgrk3FBZCU+NMgWhWC0?= =?us-ascii?Q?GBp58Rms7rOiaWcqaEYEJYsn5EX6OXzUkz3fZjOJxLv1manFeIn2Ft4D0tAv?= =?us-ascii?Q?PcVarXgkp0tt9BhmaQ2uekfAFlUO9BCJ8Mm38BhZ8eUsEavJ5F5ZDndjv0wG?= =?us-ascii?Q?IcoYSOkd/M/O4l3V5ueRorany8988LzVrYlKvMpnW73+MbsZ/YvAbeaXZmuA?= =?us-ascii?Q?U1ct2OGsyuJsNam3pk2XJ90UhHDTPIb72Jjl6oPi+FkYXtePBA=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(376014)(1800799024)(36860700013); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Oct 2024 12:07:04.3754 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0aa11bdb-d92f-402c-7edf-08dcf2920b17 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF000023DA.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7579 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Whenever a new MAC address is added to the port, mlx5 PMD will: - Add this address to `dev->data->mac_addrs[]`. - Destroy all control flow rules. - Recreate all control flow rules. Similar logic is also implemented for VLAN filters. Because of such logic, the latency of adding the new MAC address (i.e., latency of `rte_eth_dev_mac_addr_add()` function call) is actually linear to number of MAC addresses already configured. Since each operation of creating/destroying a control flow rule, involves an `ioctl()` syscall, on some setups the latency of adding a single MAC address can reach ~100ms, when port is operating with >= 100 MAC addresses. The same problem exists for VLAN filters (and even compounded by it). This patchset aims to resolve these issues, by reworking how mlx5 PMD handles adding/removing MAC addresses and VLAN filters. Instead of recreating all control flow rules, only necessary flow rules will be created/removed on each operation, thus minimizing number of syscalls triggered. Summary of patches: - Patch 1-2 - Extends existing `mlx5_hw_ctrl_flow_type` enum with special variants, which will be used for tracking MAC and VLAN control flow rules. - Patch 3-4 - Refactors HWS code for control flow rule creation to allow creation of specific control flow rules with unicast MAC/VLAN match. Also functions are added for deletion of specific rules. - Patch 5-6 - Prepares the control flow rules list, used by HWS flow engine, to be used by other flow engine. Goal is to reuse the similar logic in Verbs and DV flow engines. - Patch 7-8 - Adjusts legacy flow engines, so that unicast DMAC/VLAN control flow rules are added to the control flow rules list. Also exposes functions for creating/destroying specific ones. - Patch 9-10 - Extends `mlx5_traffic_*` interface with `mlx5_traffic_mac_add/remove` and `mlx5_traffic_vlan_add/remove` functions. They are used in implementations of DPDK APIs for adding/removing MAC addresses/VLAN filters and their goal is to update the set of control flow rules in a minimal number of steps possible, without recreating the rules. As a result of these patches the time to add 128th MAC address, after 127th was added drops **from ~72 ms to ~197 us** (at least on my setup). v2: - Rebase on top of 24.11-rc1. - Added Acked-by tags from v1. Dariusz Sosnowski (10): net/mlx5: track unicast DMAC control flow rules net/mlx5: add checking if unicast flow rule exists net/mlx5: rework creation of unicast flow rules net/mlx5: support destroying unicast flow rules net/mlx5: rename control flow rules types net/mlx5: shared init of control flow rules net/mlx5: add legacy unicast flow rules management net/mlx5: add legacy unicast flow rule registration net/mlx5: add dynamic unicast flow rule management net/mlx5: optimize MAC address and VLAN filter handling drivers/net/mlx5/linux/mlx5_os.c | 3 + drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5.h | 62 +++-- drivers/net/mlx5/mlx5_flow.c | 149 ++++++++++- drivers/net/mlx5/mlx5_flow.h | 36 +++ drivers/net/mlx5/mlx5_flow_hw.c | 349 ++++++++++++++++++++------ drivers/net/mlx5/mlx5_flow_hw_stubs.c | 68 +++++ drivers/net/mlx5/mlx5_mac.c | 41 ++- drivers/net/mlx5/mlx5_trigger.c | 262 ++++++++++++++++++- drivers/net/mlx5/mlx5_vlan.c | 9 +- drivers/net/mlx5/windows/mlx5_os.c | 3 + 11 files changed, 867 insertions(+), 116 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_flow_hw_stubs.c -- 2.39.5