From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BEC1C45E41 for ; Sat, 7 Dec 2024 09:02:33 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B075A4065A; Sat, 7 Dec 2024 09:02:33 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2085.outbound.protection.outlook.com [40.107.220.85]) by mails.dpdk.org (Postfix) with ESMTP id B871140E4C for ; Sat, 7 Dec 2024 09:02:31 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nrxTiF1dsrKTboqXyjuGxKyDcNbPxIfSL93Ow0S1VmP0tuf8/QqcftE7fMRxIKTaEHXAHYCewe0GS4R3GIDApSlstgcxu/oaSy2idywaOXWaNXyq6l06xkuDeJnLoiO3AAIJM++3yBIx/eWIhZf7jvxF5TVUw2Ibz210PzUVRch9EbIHibHMWCZ6+rkT64NcFqTxIGaecWgJi/I6YHQgj6FDDltPYQff4WccLZwowJWf0qcgxTXE3FjtSQ6ovPHUmF7vW9phdHtpIucUAG8Z/aJHeUC5GExb0cPUZxjPnfDOt42fW0k4PuDxRT+9H7DJY6uewTVtzVQ04YUte1YZlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YVfx7wjKetjvKFsa4RKzq7wb5seNj73EruXeheohMaE=; b=V68QAvQgVZHpbkp203av7g7n8bn80NkfDQXjPpx4ylXHh4/WOnf9NHxzT2YfGhBgFQqsm221KF/lhP7Jjxh8blO8cW3OPWNXIQNDBDqA4sYtQRfoqxxIXakMZengKOBQ847A0d8ki4V3EF7mbJ+gP+1f5arlo9R1iqaEpcsXmiT8YVQmx+58HoznyAXLWxiDTZrBCQvjMNy9u5QwaEXQnos3U31MV/tKtjhcaKlGdBwm0J0sEQ0yYJ055z8MPcvxv53J5UHTHmi9hhpObI0Dp2KEfHEleoNU/HmbcrQqz5iPqdDGoEjR1YWgvatszZnJR8kcpgpSNF3uNGVnQT9C7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YVfx7wjKetjvKFsa4RKzq7wb5seNj73EruXeheohMaE=; b=IpE8YlMNyjPsa+KZCHLZNMO61l8B3yOi3n3q8Qy9nJ4GLxIulNcSahxMe9cJ2UOltUPeNqv2hqbbazPKSqs7SZ1S+8zrk8u5F8CLfrI+EMpUsuK71/4Msl2wyTUkrw4nyA2OD7E/9BRfIVsfE7ujR9d0Y5fQlwpwDVAzj6oDQYlsgKncH6qCzL8Q7HrqHb+i9dNV2MI926tNORrVd7ZGShl/j8VJolrkbwmwt2l5ZPiJl/pXfG3zy6/r6y32fcUm3zZ+hiolBeGaOg+Yk4LLunwbLYjhYC6geuBZZjON0ygh3ZfmKaOiBNEwJMQuU+FfxpDtF2t+bqOJQntTDQjjqg== Received: from PH7P220CA0159.NAMP220.PROD.OUTLOOK.COM (2603:10b6:510:33b::28) by DS0PR12MB8042.namprd12.prod.outlook.com (2603:10b6:8:141::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.11; Sat, 7 Dec 2024 08:02:26 +0000 Received: from CY4PEPF0000E9DB.namprd05.prod.outlook.com (2603:10b6:510:33b:cafe::3d) by PH7P220CA0159.outlook.office365.com (2603:10b6:510:33b::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8230.13 via Frontend Transport; Sat, 7 Dec 2024 08:02:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9DB.mail.protection.outlook.com (10.167.241.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sat, 7 Dec 2024 08:02:26 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sat, 7 Dec 2024 00:02:15 -0800 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sat, 7 Dec 2024 00:02:13 -0800 From: Xueming Li To: David Marchand CC: Xueming Li , Luca Boccassi , Stephen Hemminger , Chengwen Feng , dpdk stable Subject: patch 'eal/unix: optimize thread creation' has been queued to stable release 23.11.3 Date: Sat, 7 Dec 2024 15:59:30 +0800 Message-ID: <20241207080055.488538-13-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241207080055.488538-1-xuemingl@nvidia.com> References: <20241111062847.216344-122-xuemingl@nvidia.com> <20241207080055.488538-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DB:EE_|DS0PR12MB8042:EE_ X-MS-Office365-Filtering-Correlation-Id: f6355559-26b6-4ff4-a285-08dd16957d2a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|82310400026|36860700013|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vhhCvPtbkvkbaawmm+ZqRTtDUdsXODsnSQAr2CdFJ9PdfpXwzujKzXLXL/zU?= =?us-ascii?Q?bW4fcV8g1gJWRNbexIPPQAoXRY6c6MB6+sqvQTjODE1gM4inJPfuNpopP2V1?= =?us-ascii?Q?3jIuvSQuOnoqUaYKlK+W8vnb52Lnc3hUpFiq5rp+UnEFUC4lK5dfzFkQzx+z?= =?us-ascii?Q?8yZnSVCEL7jzRqh6CXl4x5Z7nTU/rsNdvRptMd6KEDZ6QZwcbAopqDt9zgwz?= =?us-ascii?Q?LfMrvKbW03yMboG4ypxlKvnXg6ayoPfANRabGI3Ltiv/PMnrobW69/Dy1/9K?= =?us-ascii?Q?6AEblIHMAHRTd/4d6rrJjRGIlkKhwLGcBrdKKNsjAtNNYEbU+IRqpGo5j4tB?= =?us-ascii?Q?S6ukEWc7ogK4ACuGlkICi1KQQOCU9ECmaXiWFPJmUXlxR0Wg2PcPLTb1F2c7?= =?us-ascii?Q?LFt6yyEMEVyudiUgxNP9dG4T2IcHozuMt7ftAAKht9EhCkiH/JXbsCuBGzyH?= =?us-ascii?Q?7WoE4yg+2RFIZmR4NEwRAcVHaAyZTFHucpWuH3/GKroaEy5ypmXeLaqn7j+L?= =?us-ascii?Q?sCCtApiBn+yRdtyOd8cm9I0LsSSvKb0KTlKSvGvurFNI0bWHHdUnqKgRZvHx?= =?us-ascii?Q?d9ZlYa0f0FqyxpYCz3Awm/LjKs+2Iti5s8szOOogBZMwOJbjjOiG3ykLH8xP?= =?us-ascii?Q?6Ud39unAxsGZKLSJDVXB0+ZPsqcF2GGzE4Txr5unjTAG83dmzb9U4TLXHEBy?= =?us-ascii?Q?PIrZ54C71mwO0wjBU/17Jx3DXJS0A/DFenXF1AiBrrTF1smiBxHv6OS7SW8J?= =?us-ascii?Q?vDOOlCo7/2AKWU6ZGkW8P8js19H9ejp5er5eBZ7rjmbDTgehV1oeD9xh1qOH?= =?us-ascii?Q?S3AfswGM2CVa4LwNhwfHzENfFhlX5bWKGW1t4bkoJX7Vy2yf0mUrnZoJjQRF?= =?us-ascii?Q?mbnlTmlbd968h74GDHWXPSpoKpntwnLCUqywBxJqvlMohgUtwz8eCIPZuch8?= =?us-ascii?Q?FDOgnOy99QhY/kec2CkZTPKK5Hjm6ap4ci/aNnOJyPltWea0+qlxVwgxZ37J?= =?us-ascii?Q?p6NLTljC/Jghw9X+4Ip+UI07PetMGyAzP3HblXeAslhIS1FtwYPUtwtyvys9?= =?us-ascii?Q?bBMAaO9e8WsK+w47lf7v8lR6GGY1dgXyund430wFPnhYxIs9a5MzOKxI92wL?= =?us-ascii?Q?5kxP80kEQfGc5cKjY6ZRhR6HKsBk0A7SzVF9psnPedzXVc6Q8PD4fW+PHeaN?= =?us-ascii?Q?jTjnEC2Nh0yHyGFfNsq56V6veZLzoTVyaB4qy+aiUWitmkGnMfWy97pJMWlV?= =?us-ascii?Q?hCg+UPH9eBoWtq0UdC7YnNHpNow2seHqxjU5yhdmdXINDp78EJRjnqWfAhKI?= =?us-ascii?Q?N+GnYQ7H7NFr/D1EmSf/sf0FNu4FqfHkWWb30gkjuQqlGnIoAtpsvi6bH8B3?= =?us-ascii?Q?LnmS3CqtgW7Rba5l2Iw/rGnNjSu7Wf4TBodLWN0GTxMb5zHHi6kIzCd2q23P?= =?us-ascii?Q?xzpwJI8DqSW1CHdokFOmc3jEaQMKbp8b?= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024)(7053199007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Dec 2024 08:02:26.1112 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f6355559-26b6-4ff4-a285-08dd16957d2a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DB.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8042 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 23.11.3 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 12/10/24. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://git.dpdk.org/dpdk-stable/log/?h=23.11-staging This queued commit can be viewed at: https://git.dpdk.org/dpdk-stable/commit/?h=23.11-staging&id=64ea25a3a4316914d15d09ec9c695cf50f009ba5 Thanks. Xueming Li --- >From 64ea25a3a4316914d15d09ec9c695cf50f009ba5 Mon Sep 17 00:00:00 2001 From: David Marchand Date: Sat, 2 Nov 2024 10:38:16 +0100 Subject: [PATCH] eal/unix: optimize thread creation Cc: Xueming Li [ upstream commit 64f27886b8bf127cd365a8a3ed5c05852a5ae81d ] Setting the cpu affinity of the child thread from the parent thread is racy when using pthread_setaffinity_np, as the child thread may start running and initialize before affinity is set. On the other hand, setting the cpu affinity from the child thread itself may fail, so the parent thread waits for the child thread to report whether this call succeeded. This synchronisation point resulted in a significant slow down of rte_thread_create() (as seen in the lcores_autotest unit tests, in OBS for some ARM systems). Another option for setting cpu affinity is to use the not portable pthread_attr_setaffinity_np available in FreeBSD and glibc, but not available in musl. Fixes: b28c6196b132 ("eal/unix: fix thread creation") Signed-off-by: David Marchand Acked-by: Luca Boccassi Acked-by: Stephen Hemminger Acked-by: Chengwen Feng --- lib/eal/unix/meson.build | 5 +++++ lib/eal/unix/rte_thread.c | 25 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/lib/eal/unix/meson.build b/lib/eal/unix/meson.build index cc7d67dd32..f1eb82e16a 100644 --- a/lib/eal/unix/meson.build +++ b/lib/eal/unix/meson.build @@ -11,3 +11,8 @@ sources += files( 'eal_unix_timer.c', 'rte_thread.c', ) + +if is_freebsd or cc.has_function('pthread_attr_setaffinity_np', args: '-D_GNU_SOURCE', + prefix : '#include ') + cflags += '-DRTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP' +endif diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c index 36a21ab2f9..7eb9098254 100644 --- a/lib/eal/unix/rte_thread.c +++ b/lib/eal/unix/rte_thread.c @@ -17,6 +17,7 @@ struct eal_tls_key { pthread_key_t thread_index; }; +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP struct thread_start_context { rte_thread_func thread_func; void *thread_args; @@ -26,6 +27,7 @@ struct thread_start_context { int wrapper_ret; bool wrapper_done; }; +#endif static int thread_map_priority_to_os_value(enum rte_thread_priority eal_pri, int *os_pri, @@ -86,6 +88,7 @@ thread_map_os_priority_to_eal_priority(int policy, int os_pri, return 0; } +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP static void * thread_start_wrapper(void *arg) { @@ -111,6 +114,7 @@ thread_start_wrapper(void *arg) return (void *)(uintptr_t)thread_func(thread_args); } +#endif int rte_thread_create(rte_thread_t *thread_id, @@ -124,6 +128,7 @@ rte_thread_create(rte_thread_t *thread_id, .sched_priority = 0, }; int policy = SCHED_OTHER; +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP struct thread_start_context ctx = { .thread_func = thread_func, .thread_args = args, @@ -132,6 +137,7 @@ rte_thread_create(rte_thread_t *thread_id, .wrapper_mutex = PTHREAD_MUTEX_INITIALIZER, .wrapper_cond = PTHREAD_COND_INITIALIZER, }; +#endif if (thread_attr != NULL) { ret = pthread_attr_init(&attr); @@ -142,6 +148,16 @@ rte_thread_create(rte_thread_t *thread_id, attrp = &attr; +#ifdef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP + if (CPU_COUNT(&thread_attr->cpuset) > 0) { + ret = pthread_attr_setaffinity_np(attrp, sizeof(thread_attr->cpuset), + &thread_attr->cpuset); + if (ret != 0) { + RTE_LOG(DEBUG, EAL, "pthread_attr_setaffinity_np failed\n"); + goto cleanup; + } + } +#endif /* * Set the inherit scheduler parameter to explicit, * otherwise the priority attribute is ignored. @@ -176,6 +192,14 @@ rte_thread_create(rte_thread_t *thread_id, } } +#ifdef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP + ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp, + (void *)(void *)thread_func, args); + if (ret != 0) { + RTE_LOG(DEBUG, EAL, "pthread_create failed"); + goto cleanup; + } +#else /* !RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP */ ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp, thread_start_wrapper, &ctx); if (ret != 0) { @@ -191,6 +215,7 @@ rte_thread_create(rte_thread_t *thread_id, if (ret != 0) rte_thread_join(*thread_id, NULL); +#endif /* RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP */ cleanup: if (attrp != NULL) -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2024-12-06 23:26:44.552623621 +0800 +++ 0012-eal-unix-optimize-thread-creation.patch 2024-12-06 23:26:43.863044829 +0800 @@ -1 +1 @@ -From 64f27886b8bf127cd365a8a3ed5c05852a5ae81d Mon Sep 17 00:00:00 2001 +From 64ea25a3a4316914d15d09ec9c695cf50f009ba5 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 64f27886b8bf127cd365a8a3ed5c05852a5ae81d ] @@ -23 +25,0 @@ -Cc: stable@dpdk.org @@ -48 +50 @@ -index 1b4c73f58e..ea629c2065 100644 +index 36a21ab2f9..7eb9098254 100644 @@ -51 +53 @@ -@@ -19,6 +19,7 @@ struct eal_tls_key { +@@ -17,6 +17,7 @@ struct eal_tls_key { @@ -59 +61 @@ -@@ -28,6 +29,7 @@ struct thread_start_context { +@@ -26,6 +27,7 @@ struct thread_start_context { @@ -67 +69 @@ -@@ -88,6 +90,7 @@ thread_map_os_priority_to_eal_priority(int policy, int os_pri, +@@ -86,6 +88,7 @@ thread_map_os_priority_to_eal_priority(int policy, int os_pri, @@ -75 +77 @@ -@@ -113,6 +116,7 @@ thread_start_wrapper(void *arg) +@@ -111,6 +114,7 @@ thread_start_wrapper(void *arg) @@ -83 +85 @@ -@@ -126,6 +130,7 @@ rte_thread_create(rte_thread_t *thread_id, +@@ -124,6 +128,7 @@ rte_thread_create(rte_thread_t *thread_id, @@ -91 +93 @@ -@@ -134,6 +139,7 @@ rte_thread_create(rte_thread_t *thread_id, +@@ -132,6 +137,7 @@ rte_thread_create(rte_thread_t *thread_id, @@ -99 +101 @@ -@@ -144,6 +150,16 @@ rte_thread_create(rte_thread_t *thread_id, +@@ -142,6 +148,16 @@ rte_thread_create(rte_thread_t *thread_id, @@ -108 +110 @@ -+ EAL_LOG(DEBUG, "pthread_attr_setaffinity_np failed"); ++ RTE_LOG(DEBUG, EAL, "pthread_attr_setaffinity_np failed\n"); @@ -116 +118 @@ -@@ -178,6 +194,14 @@ rte_thread_create(rte_thread_t *thread_id, +@@ -176,6 +192,14 @@ rte_thread_create(rte_thread_t *thread_id, @@ -124 +126 @@ -+ EAL_LOG(DEBUG, "pthread_create failed"); ++ RTE_LOG(DEBUG, EAL, "pthread_create failed"); @@ -131 +133 @@ -@@ -193,6 +217,7 @@ rte_thread_create(rte_thread_t *thread_id, +@@ -191,6 +215,7 @@ rte_thread_create(rte_thread_t *thread_id,