From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3F59C459C3; Wed, 18 Sep 2024 08:54:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E7F974069F; Wed, 18 Sep 2024 08:54:41 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by mails.dpdk.org (Postfix) with ESMTP id 67A874029B for <dev@dpdk.org>; Wed, 18 Sep 2024 08:54:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726642481; x=1758178481; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=eS5GxkSaeWzqObKDuk2be3lnorTgD6wXVOf7PHL857A=; b=mTDi/fI4FOmujKsBgldRDCfDkFJFQ8QgBvnt71QbJ8ojxkRnOv+0ejG0 zhJGgd/XtETtfG6FXJOBSQPC7ibIApl2kLqU3Uqp7Dg1InCh9HP43pBql o/0TmpcRlR6OG7qXGghmX3jZ4+VKC15NDhhlACRy7Va2GeRX7JczFHTpR h8y2/BkH0BD6cNB4lp9M/E5nhOPTqK/aGrixbdaDqlBVQuEUoFwJ4KU07 rWDdZY6KN7SgjLDiJXQ7FWgY9mqYEiUwZv7BaCOUVYGoSlZ+9Q+LsCDn3 Dq6G9zXxTde9a47eIF3arFPYGtyo2fWuov7C0RbvVpcZK1V/Q/M3b2QYg A==; X-CSE-ConnectionGUID: hO5K0HkxTIOmO1E8grFFmw== X-CSE-MsgGUID: 7Xi8ZpWiSP+5HlVdsS4QPQ== X-IronPort-AV: E=McAfee;i="6700,10204,11198"; a="13570064" X-IronPort-AV: E=Sophos;i="6.10,235,1719903600"; d="scan'208";a="13570064" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Sep 2024 23:54:39 -0700 X-CSE-ConnectionGUID: ZAE34arjQu+l0DT0iBi27A== X-CSE-MsgGUID: pptWR96hTPGFzDi0/mz5sQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,235,1719903600"; d="scan'208";a="106914200" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orviesa001.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 17 Sep 2024 23:54:39 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 17 Sep 2024 23:54:38 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 17 Sep 2024 23:54:38 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Tue, 17 Sep 2024 23:54:38 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.100) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 17 Sep 2024 23:54:38 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=aNpy/9eKD1xzU2jjX+tEfeKOY0kvA4IdJ6QIJpmTuLHTE5c8xlbfkNt64flZg6BrqWGSNfeEsU9RHYjbudsjrALQywbxQJqKciQjPUD3rK4p4c2/VopRoJjsyxO644aTAUTmgMNFrxUJwRkgC0ZAS6X/ikIpbmAAW3nU9POSoYZpZPTWIaNlRPl7E7bNlCxc6857IadVX7jzn8NZsJEryKFeMaZ7RLAThQTy4OkPtIT+b+ScONGq0MKAuDtGhSJ1htXEmugtjhURh928NzLrpXoBO0bfKzmZ3w7B/UKmOKtjxucl/+xIgsYdrv6w9JPLXQHisJ7p2SWH3L7HhG4+vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=We446kBU2yJUKk45JTy7Q3ujIuH4KHsl9UdBoeKRnO4=; b=DBaHRvlyg6bCF8q7RRs/GGw8hIcDXbZU+yatXnYMlI8DNg3SXfOqNsRfM/IUvKlaAagNw+m5DpKWSvowsexMc2v6hEJ+AKMgOsOMTsmb4lGUQiVkog22tdRd6Gr6V7x/i1hz1u+S3qSnlOl9CO0BIN6Lda2Z8Q6adjzTAlGoNzqGRjdI54F9gS+uJ0kf+NY1yYRAIH6V2LZ21uANCt/KXwGCGDYAPXD7aYKK+DNYkv3gMmmp++i32M7t7rMHEFqsUa+7JpE7HP9g5i6WIquAnjs6zrFXWkzYBh9+7EEeY24MiSE/ZYvmCVy9H/UMGi48ETxo00jcFn1PUIqzrEFyFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM6PR11MB4514.namprd11.prod.outlook.com (2603:10b6:5:2a3::17) by CO1PR11MB4980.namprd11.prod.outlook.com (2603:10b6:303:98::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.25; Wed, 18 Sep 2024 06:54:36 +0000 Received: from DM6PR11MB4514.namprd11.prod.outlook.com ([fe80::c1a6:18dc:f83b:df71]) by DM6PR11MB4514.namprd11.prod.outlook.com ([fe80::c1a6:18dc:f83b:df71%4]) with mapi id 15.20.7982.012; Wed, 18 Sep 2024 06:54:35 +0000 From: Wojciech Panfil <wojciech.panfil@intel.com> To: <bruce.richardson@intel.com> CC: <pallavi.kadam@intel.com>, <dev@dpdk.org>, <jacek.kalwas@intel.com>, <konrad.sztyber@intel.com>, <dmitry.kozliuk@gmail.com>, <roretzla@linux.microsoft.com>, <wojciech.panfil@intel.com> Subject: [PATCH] eal/alarm_cancel: Fix thread starvation Date: Wed, 18 Sep 2024 09:02:12 +0200 Message-ID: <20240918070212.10339-1-wojciech.panfil@intel.com> X-Mailer: git-send-email 2.46.0 Organization: Intel Corporation Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: AS4P189CA0012.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:5d7::15) To DM6PR11MB4514.namprd11.prod.outlook.com (2603:10b6:5:2a3::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR11MB4514:EE_|CO1PR11MB4980:EE_ X-MS-Office365-Filtering-Correlation-Id: 7f09240f-ab4d-4278-97b8-08dcd7aec1e3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Fq1SOuTJMdRnm11XlUBlmbPbgKyXNNPzq7d+kLRNgIlZrH/5qZiu4DHv4d2Q?= =?us-ascii?Q?2/yUQEyNTJFGNJsJxlG8TKJgUdC+8UtRAWhSNPgVpFuEIAfIdxZR9Mj69uYO?= =?us-ascii?Q?7BGXNUZ60Sn09Lxd0Y3Ygc6PCqf0GB0w3zUEyQQVsQBzJpE0+QeCPGaczV4R?= =?us-ascii?Q?z22n+yyTW5iVqt1arNN1Eb4Nr2Arcopt333OMIwsDn3vUb/FSWAoC/8sRmkp?= =?us-ascii?Q?IJf5nF+1NIGKKWB25IqaZwWjjbF3qpZKMlNlrTKx2niRzJIhT0wFJKhvIDMg?= =?us-ascii?Q?v+cssVCFrLNyLVDSQ1oyCpdeZEHQo04DGPpY58iNKS8p7tvli5pRnfWmwF5X?= =?us-ascii?Q?jQ2KnFmEjFdzqIton0yHYhVpOIRXiczvkWRXIuyvlUFbDLetSIzKCPeRtdOq?= =?us-ascii?Q?Olo9wBAy3OQiJxGX+MEcEQtKgzpNoLYOlR3KaY+dWFUTmECWql1LM5pESU5l?= =?us-ascii?Q?ZLE/o+RKRbShhSl4jCKdbAKVLTA2/gEsk+5NXSWEUPVMvjmoB+jtAJSD+zYu?= =?us-ascii?Q?meFjl7zlWfHA2eKywKyYKEFgaWwCn1WNe/kaK8XIgQnA+11jMQ6snwcfxiUt?= =?us-ascii?Q?rTd+C5l3sNVl59x63HmQcn+R4xZ9wq679MQN6kRbObjykMS6LMw5LPmyeNVk?= =?us-ascii?Q?Hx8anFraJev3FNzBn6YJrbkTzIdI1cP4LlZBiYRzzJOTd1DsM0YI0LoXfBpT?= =?us-ascii?Q?UudSNXWC/KIrdcjIhff6X3878DNj3o3PAvZ82lwsOmSCMfWb4tqT1jiCs1Q7?= =?us-ascii?Q?a/NfCNWg+VTeVnjnhrP2PHJr8JjXGD5O7WBUHPiJz0Z31nrAkLBZxFmpCGGn?= =?us-ascii?Q?vHof9AVO9HBa6qGP3Z56m34ilsLd8dmrMICvQTTEpPX1dijAXjm9GWu9E+Ef?= =?us-ascii?Q?TfCVEWrtPFbLak83Eo5THxHVKJFGR95g2mMo2mlzf3yDBL64tWu9fpwFBJjM?= =?us-ascii?Q?fnx75ooFpEkkbphsroqUIly7+a/XYJ8gcO1HWffDrmvDxGrW5QNmDJL0Jttc?= =?us-ascii?Q?NpjMtIc66TgCuukQx9hddgqZH35dbkf+NBJDP4vrBD80aavEu1awZ7osC3LT?= =?us-ascii?Q?iAdbkiUbvI/PjmLpCnYn4pl7Ua8kBOKKBj2k+2nJ2TVN3SNWx8Ryft+6w84h?= =?us-ascii?Q?lAeKtIKKL2cg2f1Ui6Q/LvtZn2hzSioYPrydBNUAbWJZ9SMUs9tWaxvvf1vl?= =?us-ascii?Q?hdMh/DiFB6Ler8K+G/o3BK0+zqU/07DI8MajUwN0KZaypoiT7+ENkt/LnynX?= =?us-ascii?Q?5ND90NAQpjhReV1T66oCYG6a7fqcU2cJMWYAKhuSqZUFYFBR8rbncI/5TEgf?= =?us-ascii?Q?xcEF959I7yVAJb3+nkHKsUUFOW+xWv0lqWgjin3ok6bJig=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB4514.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(376014)(366016)(1800799024); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bmjbZBt9JhOCaYQiAWeyCj95ecQNO4a10tBOhCDF6tJu3MERKnqfQ8Gi3J6Y?= =?us-ascii?Q?EWFCFHFl7YjKsaLG5xxONt+B8hAh5aKhGiVarUzhViC7hGGbg/PkGWwN+ntH?= =?us-ascii?Q?BJaMFGyey5K1X7FRO0+tmuy5fEt9vP3u/MfW1XfmFNZyVIkfPi8ISlUo9zDQ?= =?us-ascii?Q?t9uJL6iw9IUyJvOT1Dn//jez68OdfxOE64cor1Eu1u1gn0W1cUleStANnBw7?= =?us-ascii?Q?l4an2tjrVKRdV28dRz8ky4TvVDSLgK0s5nIUguqrnZOD28eLa46KsYeTBR6b?= =?us-ascii?Q?73YLeNa70NgWPc/0giAY21PAjj0KyVeabBaczMDYWyItB7xmgG7rta7A5rjW?= =?us-ascii?Q?+8NI/UmdRnxcvPchO24TZCsPeQaaoIqFw6i4xxfCpPDec29SZhZkxPoKgu7i?= =?us-ascii?Q?R/F6Qt+A9Sv2NbDLTfVgD8h4yCIp0TnIdWj/Jl88oUZmKzWBvqU7779N/Dkk?= =?us-ascii?Q?i6zU6XA2jxP+WiAIYC6yfSUa/8SQyK7mS/yxGXttlmM6xE61B0SQ3rDVOByR?= =?us-ascii?Q?gGQNGjs/HJdQoeyqfVThPG9DQ9Ap70ZqNx66xU/ZYfoOTEK/63fmxzSWRv0o?= =?us-ascii?Q?oxi+RwCwKtDs0ZM8eoEAf9TmxeItOz8E242sSXl/E2INMEwNDgUQBI5aRVnD?= =?us-ascii?Q?PUzDy+Xk8zJjL5jizy3UPrimXU2gREQoFdZWWhzu3Hj77H8HH4zg3Ng36IL/?= =?us-ascii?Q?k6cXwC2+hXEeUkjOLWB2CrBv0BLLsm6YJYZmagX7ODWF2qOTqSJ79+nhZIQm?= =?us-ascii?Q?6J4tykq8A56emAx2HsGQsOIBSAoAHPYmWeM6Z9XzTpLRwIX17f2mf8lrmhz4?= =?us-ascii?Q?9ncx6eW5f38xQ+NTjFoswdEA4zJLYjB15Sv9AZTVqDd8mt4y4SUzUq+6Wc0M?= =?us-ascii?Q?Mr5gQXCeZlt4XKHxVHOdD8hJwI+hEFcNfjrUznYmYeRl9JXo2JZn/q3Bhg3r?= =?us-ascii?Q?0GdB30TTUrNnwTon/KlF92TEMiriqjAUFGhvAptBliQIGI+M2rufVummmNtX?= =?us-ascii?Q?W49JjRYN8nmI87/rpCDYyl44n6DwrzSmmghv+nugwdtWoXoTGJ7Wgq6DNHm9?= =?us-ascii?Q?1zKomymxQeFjrTGNTs0k8+qcNbWChQcF4DloII8ObrbaHnbtoO2gFxhxVQCU?= =?us-ascii?Q?ICAGw3tBHyCXn/3Ljxb3WPS4eL9ixnlJlz4AizRXR2EO1KkWMia0h2GrS3Kz?= =?us-ascii?Q?8gzNKWOYkPXdIOI/WbsZMW/zTNxyNy3yQrqZ3tpvXWevoPBf+wYd5U+Ar0bp?= =?us-ascii?Q?g/PRi9HIQmihfjKduLhApEE1tbohEis0ns1fb+prGneO91yhqpQhRZ/fouF3?= =?us-ascii?Q?ed7VBLMA/shgZWci+W0aaFsnU8L3uazUk37na2CR3LALKim7zNdnvOes1gHa?= =?us-ascii?Q?6QQtrSEtnDUbBH5gRgNBQ7mqDXTCY3SD1kgYM1t9SvmWYAhiNQxvNLyCJiOT?= =?us-ascii?Q?eTe7O3qVxdFZj5vSAjPVblWgY3IYrc6Vm7mqARbYHfb36LyBFtT559IEk1Rt?= =?us-ascii?Q?oGkojfQE3i7Lr6B47cdzeAmRSLh7a9fV2/JkcVOx3DY+OjoTs+hJCcP0pIhw?= =?us-ascii?Q?aW8b7+ysB0+k7oAsgy8OekNVid+K4oNFDhE8xsLgmE2kGV+46A8gV3Xpj8M3?= =?us-ascii?Q?Jg=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 7f09240f-ab4d-4278-97b8-08dcd7aec1e3 X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4514.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2024 06:54:35.8709 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nZrwo2lWs1jN4k1OTlNg57ruaYfrshqvHulnRsEFadRaj/5PknxP7BWlE8Gd47rGw0ts59qc+OiOT8/7xZflqt/rz9e35NuSapls+jGASGA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4980 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Issue: Two threads: - A, executing rte_eal_alarm_cancel, - B, executing eal_alarm_callback. Such case can cause starvation of thread B. Please see that there is a small time window between lock and unlock in thread A, so thread B must be switched to within a very small time window, so that it can obtain the lock. Solution to this problem is use sched_yield(), which puts current thread (A) at the end of thread execution priority queue and allows thread B to execute. The issue can be observed e.g. on hot-pluggable device detach path. On such path, rte_alarm can used to check if DPDK has completed the detachment. Waiting for completion, rte_eal_alarm_cancel is called, while another thread periodically calls eal_alarm_callback causing the issue to occur. Signed-off-by: Wojciech Panfil <wojciech.panfil@intel.com> --- lib/eal/freebsd/eal_alarm.c | 5 +++++ lib/eal/linux/eal_alarm.c | 5 +++++ lib/eal/windows/eal_alarm.c | 4 ++++ 3 files changed, 14 insertions(+) diff --git a/lib/eal/freebsd/eal_alarm.c b/lib/eal/freebsd/eal_alarm.c index 94cae5f4b6..8425b4f5a2 100644 --- a/lib/eal/freebsd/eal_alarm.c +++ b/lib/eal/freebsd/eal_alarm.c @@ -318,7 +318,12 @@ rte_eal_alarm_cancel(rte_eal_alarm_callback cb_fn, void *cb_arg) } ap_prev = ap; } + rte_spinlock_unlock(&alarm_list_lk); + + /* Yield control to a second thread executing eal_alarm_callback to avoid its starvation, + * as it is waiting for the lock we have just released. */ + sched_yield(); } while (executing != 0); if (count == 0 && err == 0) diff --git a/lib/eal/linux/eal_alarm.c b/lib/eal/linux/eal_alarm.c index eeb096213b..5326b1895f 100644 --- a/lib/eal/linux/eal_alarm.c +++ b/lib/eal/linux/eal_alarm.c @@ -248,7 +248,12 @@ rte_eal_alarm_cancel(rte_eal_alarm_callback cb_fn, void *cb_arg) } ap_prev = ap; } + rte_spinlock_unlock(&alarm_list_lk); + + /* Yield control to a second thread executing eal_alarm_callback to avoid its starvation, + * as it is waiting for the lock we have just released. */ + sched_yield(); } while (executing != 0); if (count == 0 && err == 0) diff --git a/lib/eal/windows/eal_alarm.c b/lib/eal/windows/eal_alarm.c index 052af4b21b..43e8d7881f 100644 --- a/lib/eal/windows/eal_alarm.c +++ b/lib/eal/windows/eal_alarm.c @@ -211,6 +211,10 @@ rte_eal_alarm_cancel(rte_eal_alarm_callback cb_fn, void *cb_arg) } rte_spinlock_unlock(&alarm_lock); + + /* Yield control to a second thread executing eal_alarm_callback to avoid its starvation, + * as it is waiting for the lock we have just released. */ + SwitchToThread(); } while (executing); rte_eal_trace_alarm_cancel(cb_fn, cb_arg, removed); -- 2.46.0