From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 31820A04E6; Wed, 9 Dec 2020 17:08:15 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 025C9BE79; Wed, 9 Dec 2020 17:08:14 +0100 (CET) Received: from mail1.bemta26.messagelabs.com (mail1.bemta26.messagelabs.com [85.158.142.115]) by dpdk.org (Postfix) with ESMTP id A461ABE75 for ; Wed, 9 Dec 2020 17:08:11 +0100 (CET) Received: from [100.113.7.143] (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256 bits)) by server-4.bemta.az-b.eu-central-1.aws.symcld.net id DE/D6-50748-A66F0DF5; Wed, 09 Dec 2020 16:08:10 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA1WSb0wTdxjH+fXacnYtO1ugzzpRcpGwKa0UmXY zJp3LMpSx7I2JYQ68ytHeVq54VzJgL0RwGjvNcAtxNkKxodtSh1HQwaRktDK2OQWsZpuAARTn +CNs/HHGEN1dD9327vN8v8/veb53eXBMe0ppwOlyN82xlJNUquSOdeosI3O/vzCzv3ejZXq2T WZFOQ8bA4q3Ub6CYW2u8l0Kx9V7HYrSFn25Z6pZWYVu6jxIhSMigMGZ4el4qeiRQzjQLpOKEI IbDw4oPWgZLifOY3D9R43IWqJeBqNhUmzSErcQhM//EC8aSiIbJjprYq8TiR4EkYEvFWKBiS/ uXa6JdemIfDjQ/I1M5ETiHThYPa+QOAtmPl/EPAgX1q2Gi90virKGoGC4eQqTNr8LAW84xsuI AvgkOhkbg4gUmN93KqZjhB4GxnwxHQgCmkJ9mMRJMH77kUL6tFoEndNdSDJehurP/PESp0DU9 /GSngcXOoeXBmXAYOh7ucSl0HrTKxNzgpDzUaRCktNh/18NColXQvDI6FL7Chj5rU0p7gViEY NgjQ9Jxd8YHPLeRbVonfc/wSXOgMaOWaXEa+GLk5OYN/YzlsNPx8fkjUgeRBYbx9gd7hKKcRr NmZlGs3m9cYNAJqrSaDPRZcbdNOvmKME0UR/wJr6iZLezyMTS7hYkXEzRHqWqHd2Z/NMUQc/h MjJJUxftL9Qm2FxFFQ6KdxRyZU6aj6AVOE6CxjIveMs52k6XFzNO4e6e2ICryUQNJlyeVsOXU iU8Y5esS2g9Xjte78fwGzNNfkwrZ10sbdBr6haEVkJsdZSxTwc9ueEoSjHoNCguLk6rLqW5Es b9f38C6XFE6jSUOEXNsO6n+yaEKDIhijraJ0ZxU/9ahirZm7lhb+Wr41eev8swc9uTB3RZ7q1 909xevXooOrJJ1/LY3vzg9dMJmwf1Xdnh6sj1wdnZ7v0//5p8coe9ns5id/S990J2ezR3ocO4 IXR/jrMeyd9c1+uzdp9zjV7qzXx2gXt4560G2vF1/K5FNonJ9agS0j51pa85PXRi59zKoWNrU /i9ztTvLm7ctrOpMY/Zkp0KYy+lFX/4lfvy2ZxnXuO/DfkLokdztiwe3nqtIfkNyxSZcahq2J rXuq/h91afzZAEmw7bCyyvBIL5XVe2d4/suaDyrDqb7v/lo/ra1ongTDGNM2G4lUZe1V5zkqs fu3oq37ceNw8dPPpH6jY69XYbKecdlHkNxvHUP9t4jD0+BAAA X-Env-Sender: John.Alexander@datapath.co.uk X-Msg-Ref: server-28.tower-246.messagelabs.com!1607530089!140087!1 X-Originating-IP: [104.47.12.50] X-SYMC-ESS-Client-Auth: mailfrom-relay-check=pass X-StarScan-Received: X-StarScan-Version: 9.60.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 12117 invoked from network); 9 Dec 2020 16:08:09 -0000 Received: from mail-db3eur04lp2050.outbound.protection.outlook.com (HELO EUR04-DB3-obe.outbound.protection.outlook.com) (104.47.12.50) by server-28.tower-246.messagelabs.com with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 9 Dec 2020 16:08:09 -0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ABq+ZPSKqx7wFi6Jzj8N4Z4VOnfl8G7KiJ173Xbl/3hfFp8QcRJ54kRMQ8bvigKeNkd+NdkSoB4y2HTRONXAIm5i96P4KHd5nY93k9jcrzMyasX7WdKZU7SNUBIj2oTeh+EYb7GMbUuIij/CjQersCr8q5w/9NA/wk99UsIf3UACa0Ulm4Rgj/mR59E0neUggZ15VM1UcwDbn5AQ4nco3Dn0YypnVtrH34lAInN7UQgzAU97Yc9h+XTwplAWuB2J+Y01YzJ6tKtoOlidqX8etjUD11AMm0gNKygD4C5EMwRi2mF5bYL9XAu90CYH4kauCp8eS5OC+O5X+6SvywXmsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gbX4SThj2TJaLDINfagmH8kDp5Bc9bcKQRbas6hzXD0=; b=fk9XOJykeuz7RELCZQXZ6IL5B5p7W/hegFjMWIN87c/09pzFQXfh1kwSBJzDdcXREVmEUfKEEC4tQgHdcTdeMwF/QTby+fh4lWwZRrpK8XrnoYCPBklVfZlK1XikTA/7BgtoTdQO0rR/NTFZtbcfOqqHcQb055viSm6dzXEKX4rolEl/TI0kjEVYhD+J4K58gws4ViopwgMPVZREU3yjhGb/NXRIHitcTbHXG60BXPvJModx9PCNBrwlggUVh1sMTT/Jn8ApFu33tmGsd2KBg2YMBld+fXOm9GzsRPV7mQyhZD7rANZdBXCv9t7HKEwkMHrmQDmQEFTmr/RZ0AJz9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=datapath.co.uk; dmarc=pass action=none header.from=datapath.co.uk; dkim=pass header.d=datapath.co.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datapathuk.onmicrosoft.com; s=selector2-datapathuk-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gbX4SThj2TJaLDINfagmH8kDp5Bc9bcKQRbas6hzXD0=; b=h/IUS0bpUO/5Awfmu5ALHCYcfxIbVb4hAdIaMN3vfBv5WWSoFK7uaZ7AcFH0uHLagdPSYry4YdF9DnpUHP71TeS99LpLmMYbgwHeE5/IB6t66OD5kdPoxdY5EX5UIrobYPCfUk6bdjJCdyESW1QjVSPAVMvS8iUVQaFAJXd8kr0= Received: from DB6PR0902MB2072.eurprd09.prod.outlook.com (2603:10a6:6:8::23) by DB6PR0901MB1654.eurprd09.prod.outlook.com (2603:10a6:4:79::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.21; Wed, 9 Dec 2020 16:08:08 +0000 Received: from DB6PR0902MB2072.eurprd09.prod.outlook.com ([fe80::98b5:621b:8f2e:7df4]) by DB6PR0902MB2072.eurprd09.prod.outlook.com ([fe80::98b5:621b:8f2e:7df4%3]) with mapi id 15.20.3632.023; Wed, 9 Dec 2020 16:08:08 +0000 From: John Alexander To: Tal Shnaiderman , Dmitry Kozlyuk , "Dmitry Malloy (MESHCHANINOV)" , Narcisa Ana Maria Vasile CC: Eilon Greenstein , Omar Cardona , Rani Sharoni , Odi Assli , Harini Ramakrishnan , NBU-Contact-Thomas Monjalon , "dev@dpdk.org" Thread-Topic: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation Thread-Index: AdbONYWvtK4WZhcIQAeTWkc2TKlT0AAD41WA Date: Wed, 9 Dec 2020 16:08:07 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2a00:23c4:403:e001:9d6c:dd9:6e2c:8c0e] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a8dc7b1e-419a-41cb-c1c2-08d89c5c9e54 x-ms-traffictypediagnostic: DB6PR0901MB1654: x-microsoft-antispam-prvs: x-ms-exchange-transport-forked: True x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cqjQHeEK4aoZtXK0VYnJ0VXsNdvngB8545pyB+FWlDsNzOaKRS8nmp1qjk121g7A0gJ05ATR2TST3bEn7yzCAKwEMK0MjTDik3rfbam6dtGYQEJKOE+Tl3CDVao17ZCMC6PkIIkQ/REzxMUwfW+hIf1dlIKa2cegQObK1+r50VkyQFTdyhX3t3RhNKFBM2x4LaC0PB4wAfmGdyV0fq3NQW7hfF8cSX3BKIt7d9ZrsPO8FtWa8EexoZX3MJMbdjataGmqNMkjNrTn89ByxWJWz5P+RvRVNmEzWhsdNahvmgwNNSDRdYN5ATt2LpyH/RzePVxhHhzBoDhUHWr9QTdwmofez/y7+ubCtfNhE+mF4np3vVLOYmFOPLLSrfvy/14jNFiYIBa4XeNBKKYRwzwBxAEC8pm8tC5eU5GFwy2N4PoH/kAzZyybfQEpaVYhEm9gJI8efvqzaVjDuSeTLsAgMw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB6PR0902MB2072.eurprd09.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(39840400004)(346002)(396003)(366004)(136003)(110136005)(8936002)(71200400001)(6506007)(186003)(9686003)(52536014)(33656002)(66946007)(64756008)(55016002)(54906003)(66476007)(478600001)(66446008)(2906002)(7696005)(83380400001)(66556008)(966005)(76116006)(5660300002)(86362001)(53546011)(316002)(4326008)(460985005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?zxpga+sTME5xTnMhdEgWJwTAf6bnGOX8iMfOawfoIoHPeePrf42bI7Ipxt4I?= =?us-ascii?Q?ok/7GcrnFNFlmVx4162MnViYBv5gCDD/hgOrFmcANMxyX+XRwYGnVtO4a90G?= =?us-ascii?Q?63RIyLcbCMiRmRtimrNFoCgnWU2GaAXYQl8UXjD9FqVCSZ3tNlODnnXj6Xbp?= =?us-ascii?Q?jI4ihlYlonuIhjHJ/aS0/6vTAnZLxS2H1L97rbiED1i8pGjO8KDjr44QFEXn?= =?us-ascii?Q?NIBvcP9hedcQSiWPzCsDuaxtRLXrBDtJBGUNWINfpmKK2qhJs7BxATS5Ar8/?= =?us-ascii?Q?aT3X4lbN3izOyoMoDGV7dQGpIoOSeE6IoU2JVGy1Da8a9QVRgqw0R+sgEyiz?= =?us-ascii?Q?fZoYCZ5UkF+JIdA+bsc1VB5oH2Hwts6FgQOqtnDMgbLVN8IpFJQPzQ1XSc+H?= =?us-ascii?Q?pSKY2L9YD1wySKu+UVN745s/W7m6TAURZeWSEOBpjQu401enu4Fy2ApYe9vX?= =?us-ascii?Q?bnUhka1lnzdmitzdo7fdU31lFqRnV+gyywKie2Q4MKhZsATt+HwOGipgoQci?= =?us-ascii?Q?mXPGiruIO0SBL+kpJAbhp7UfKVdfY9RzPVimfDyEJhDuMkgARSuiesmKjJZp?= =?us-ascii?Q?P5x8lBzAkmHKq3X3r8v6jbjxO8QYqu/CMB2DW6FckNA0vw3CclTQdId+whwV?= =?us-ascii?Q?tJ8c/ddt3KuR9iAn7Q4uCZvZuujpNHhbmcYyVYgfdI7lgIm5deGcT7tEAaIx?= =?us-ascii?Q?MKUAbvRRLlNC9ALM0zJK88UEnwNnJO+aqav7gDCIwfKaypnTSe3qf8aCrSQe?= =?us-ascii?Q?fsHxPRpKR3A1ZCcasI8WaeSoNO6uBKLwSdjmADCV6/qDREYfMLunUBf1Zq9/?= =?us-ascii?Q?c+NvkIaJ+nf3cyXxJsFS7uMvpZRHEfASOna4WBRp0cBICdrY1TmPT36r/zFF?= =?us-ascii?Q?3faFQIfA8Cb8bpK0pDJzvQPZ32uUoyhLGv3AXYtkdW3ezKZ7UhuyWAeiuysf?= =?us-ascii?Q?mi05jF98/a2LaEr+mDyOQCyti84TiwBo8wKAnPl/5r59MTp/4FDl7OrUdNCq?= =?us-ascii?Q?eqHKpfOCXRHsqW0DWuqCgO+bst5a2UXl1VfC3RLOQ68zdwI=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: datapath.co.uk X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DB6PR0902MB2072.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a8dc7b1e-419a-41cb-c1c2-08d89c5c9e54 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2020 16:08:07.9244 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 102e0f24-523c-4823-a9ce-5a8ebc4e32a7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: C4WXcmWdq25G12jOeLW498QwQRFQaSMcmHYDHZVQw1k/OlQ1puCdTRUjAzoqGJr15mLjRmlmoXa4HumNq927uhWveoC85DlExmTIWsOigw4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0901MB1654 Subject: Re: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, I tend to run with a winbdg kernel debugger (KDNET) connected to my debug t= arget machines. It quite often reports deadlock detection when we have suc= h "real-time" threads never yielding on a core. If we hog core-0 in partic= ular dwm.exe never gets a look in so the desktop stops being drawn too. John. > -----Original Message----- > From: dev On Behalf Of Tal Shnaiderman > Sent: 09 December 2020 14:16 > To: Dmitry Kozlyuk ; Dmitry Malloy > (MESHCHANINOV) ; Narcisa Ana Maria Vasile > > Cc: Eilon Greenstein ; Omar Cardona > ; Rani Sharoni ; Odi Assli > ; Harini Ramakrishnan > ; NBU-Contact-Thomas Monjalon > ; dev@dpdk.org > Subject: [dpdk-dev] Windows DPDK real-time priority threads causing threa= d > starvation >=20 > CAUTION: This email originated from outside of the organization. Do not c= lick > links or open attachments unless you recognize the sender and know the > content is safe. >=20 > Hi, >=20 > During our verification tests on Windows DPDK we've noticed that DPDK > polling threads, which run in REALTIME_PRIORITY_CLASS are causing > starvation to other threads from the OS which need to change affinity and > run in lower priority. >=20 > While running an application for a while we see the OS thread waits for 2= :30 > minutes and raises a bugcheck, see below example of such flow: >=20 > 1) DPDK thread running on core-0 in real-time high priority(24) polling m= ode. > 2) The thread is blocking the system function NtSetSystemInformation > (ExpUpdateTimerConfiguration) in another thread from > switching to core-0 via KeSetSystemGroupAffinityThread since the calli= ng > thread is priority 15. > 3) NtSetSystemInformation exclusively acquired system-wide lock > (ExpTimeRefreshLock) hence > it blocks other threads (e.g. calling NtQuerySystemInformation). >=20 > We've seen this behavior only while running on Windows 2019 VMs, maybe > on native machines OS scheduling of such flow is done differently? >=20 > Below is usage explanation from the documentation of SetPriorityClass [1]= : >=20 > - REALTIME_PRIORITY_CLASS > Process that has the highest possible priority. The threads of the proces= s > preempt the threads of all other processes, including operating system > processes performing important tasks. For example, a real-time process th= at > executes for more than a very brief interval can cause disk caches not to > flush or cause the mouse to be unresponsive. >=20 > So I assume using this kind of thread for a long period as we do can caus= e > unstable behavior. >=20 > How do you think we can resolve this? Are there such cases in Linux? >=20 > [1] - https://docs.microsoft.com/en- > us/windows/win32/api/processthreadsapi/nf-processthreadsapi- > setpriorityclass >=20 > Thanks, >=20 > Tal.