DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation
@ 2020-12-09 14:15 Tal Shnaiderman
  2020-12-09 16:08 ` John Alexander
  2020-12-09 16:08 ` Stephen Hemminger
  0 siblings, 2 replies; 5+ messages in thread
From: Tal Shnaiderman @ 2020-12-09 14:15 UTC (permalink / raw)
  To: Dmitry Kozlyuk, Dmitry Malloy (MESHCHANINOV), Narcisa Ana Maria Vasile
  Cc: Eilon Greenstein, Omar Cardona, Rani Sharoni, Odi Assli,
	Harini Ramakrishnan, NBU-Contact-Thomas Monjalon, dev

Hi,

During our verification tests on Windows DPDK we've noticed that DPDK polling threads, which run in REALTIME_PRIORITY_CLASS are causing starvation to other threads from the OS which need to change affinity and run in lower priority.

While running an application for a while we see the OS thread waits for 2:30 minutes and raises a bugcheck, see below example of such flow:

1) DPDK thread running on core-0 in real-time high priority(24) polling mode.
2) The thread is blocking the system function NtSetSystemInformation (ExpUpdateTimerConfiguration) in another thread from 
   switching to core-0 via KeSetSystemGroupAffinityThread since the calling thread is priority 15. 
3) NtSetSystemInformation exclusively acquired system-wide lock (ExpTimeRefreshLock) hence 
    it blocks other threads (e.g. calling NtQuerySystemInformation).

We've seen this behavior only while running on Windows 2019 VMs, maybe on native machines OS scheduling of such flow is done differently? 

Below is usage explanation from the documentation of SetPriorityClass [1]:

- REALTIME_PRIORITY_CLASS
Process that has the highest possible priority. The threads of the process preempt the threads of all other processes, including operating system processes performing important tasks. For example, a real-time process that executes for more than a very brief interval can cause disk caches not to flush or cause the mouse to be unresponsive. 

So I assume using this kind of thread for a long period as we do can cause unstable behavior.

How do you think we can resolve this? Are there such cases in Linux?

[1] - https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setpriorityclass

Thanks,

Tal.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation
  2020-12-09 14:15 [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation Tal Shnaiderman
@ 2020-12-09 16:08 ` John Alexander
  2020-12-09 16:08 ` Stephen Hemminger
  1 sibling, 0 replies; 5+ messages in thread
From: John Alexander @ 2020-12-09 16:08 UTC (permalink / raw)
  To: Tal Shnaiderman, Dmitry Kozlyuk, Dmitry Malloy (MESHCHANINOV),
	Narcisa Ana Maria Vasile
  Cc: Eilon Greenstein, Omar Cardona, Rani Sharoni, Odi Assli,
	Harini Ramakrishnan, NBU-Contact-Thomas Monjalon, dev

Hi,

I tend to run with a winbdg kernel debugger (KDNET) connected to my debug target machines.  It quite often reports deadlock detection when we have such "real-time" threads never yielding on a core.  If we hog core-0 in particular dwm.exe never gets a look in so the desktop stops being drawn too.

John.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Tal Shnaiderman
> Sent: 09 December 2020 14:16
> To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Dmitry Malloy
> (MESHCHANINOV) <dmitrym@microsoft.com>; Narcisa Ana Maria Vasile
> <Narcisa.Vasile@microsoft.com>
> Cc: Eilon Greenstein <eilong@nvidia.com>; Omar Cardona
> <ocardona@microsoft.com>; Rani Sharoni <ranish@nvidia.com>; Odi Assli
> <odia@nvidia.com>; Harini Ramakrishnan
> <Harini.Ramakrishnan@microsoft.com>; NBU-Contact-Thomas Monjalon
> <thomas@monjalon.net>; dev@dpdk.org
> Subject: [dpdk-dev] Windows DPDK real-time priority threads causing thread
> starvation
> 
> CAUTION: This email originated from outside of the organization. Do not click
> links or open attachments unless you recognize the sender and know the
> content is safe.
> 
> Hi,
> 
> During our verification tests on Windows DPDK we've noticed that DPDK
> polling threads, which run in REALTIME_PRIORITY_CLASS are causing
> starvation to other threads from the OS which need to change affinity and
> run in lower priority.
> 
> While running an application for a while we see the OS thread waits for 2:30
> minutes and raises a bugcheck, see below example of such flow:
> 
> 1) DPDK thread running on core-0 in real-time high priority(24) polling mode.
> 2) The thread is blocking the system function NtSetSystemInformation
> (ExpUpdateTimerConfiguration) in another thread from
>    switching to core-0 via KeSetSystemGroupAffinityThread since the calling
> thread is priority 15.
> 3) NtSetSystemInformation exclusively acquired system-wide lock
> (ExpTimeRefreshLock) hence
>     it blocks other threads (e.g. calling NtQuerySystemInformation).
> 
> We've seen this behavior only while running on Windows 2019 VMs, maybe
> on native machines OS scheduling of such flow is done differently?
> 
> Below is usage explanation from the documentation of SetPriorityClass [1]:
> 
> - REALTIME_PRIORITY_CLASS
> Process that has the highest possible priority. The threads of the process
> preempt the threads of all other processes, including operating system
> processes performing important tasks. For example, a real-time process that
> executes for more than a very brief interval can cause disk caches not to
> flush or cause the mouse to be unresponsive.
> 
> So I assume using this kind of thread for a long period as we do can cause
> unstable behavior.
> 
> How do you think we can resolve this? Are there such cases in Linux?
> 
> [1] - https://docs.microsoft.com/en-
> us/windows/win32/api/processthreadsapi/nf-processthreadsapi-
> setpriorityclass
> 
> Thanks,
> 
> Tal.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation
  2020-12-09 14:15 [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation Tal Shnaiderman
  2020-12-09 16:08 ` John Alexander
@ 2020-12-09 16:08 ` Stephen Hemminger
  2020-12-09 16:12   ` [dpdk-dev] [EXTERNAL] " Dmitry Malloy (MESHCHANINOV)
  1 sibling, 1 reply; 5+ messages in thread
From: Stephen Hemminger @ 2020-12-09 16:08 UTC (permalink / raw)
  To: Tal Shnaiderman
  Cc: Dmitry Kozlyuk, Dmitry Malloy (MESHCHANINOV),
	Narcisa Ana Maria Vasile, Eilon Greenstein, Omar Cardona,
	Rani Sharoni, Odi Assli, Harini Ramakrishnan,
	NBU-Contact-Thomas Monjalon, dev

On Wed, 9 Dec 2020 14:15:30 +0000
Tal Shnaiderman <talshn@nvidia.com> wrote:

> Hi,
> 
> During our verification tests on Windows DPDK we've noticed that DPDK polling threads, which run in REALTIME_PRIORITY_CLASS are causing starvation to other threads from the OS which need to change affinity and run in lower priority.
> 
> While running an application for a while we see the OS thread waits for 2:30 minutes and raises a bugcheck, see below example of such flow:
> 
> 1) DPDK thread running on core-0 in real-time high priority(24) polling mode.
> 2) The thread is blocking the system function NtSetSystemInformation (ExpUpdateTimerConfiguration) in another thread from 
>    switching to core-0 via KeSetSystemGroupAffinityThread since the calling thread is priority 15. 
> 3) NtSetSystemInformation exclusively acquired system-wide lock (ExpTimeRefreshLock) hence 
>     it blocks other threads (e.g. calling NtQuerySystemInformation).
> 
> We've seen this behavior only while running on Windows 2019 VMs, maybe on native machines OS scheduling of such flow is done differently? 
> 
> Below is usage explanation from the documentation of SetPriorityClass [1]:
> 
> - REALTIME_PRIORITY_CLASS
> Process that has the highest possible priority. The threads of the process preempt the threads of all other processes, including operating system processes performing important tasks. For example, a real-time process that executes for more than a very brief interval can cause disk caches not to flush or cause the mouse to be unresponsive. 
> 
> So I assume using this kind of thread for a long period as we do can cause unstable behavior.
> 
> How do you think we can resolve this? Are there such cases in Linux?
> 
> [1] - https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setpriorityclass
> 
> Thanks,
> 
> Tal.

This is not unique to Windows, Linux has same thing when using SCHED_FIFO.
Setting REALTIME is not a magic "go fast" flag it tells scheduler to "run this thread
at higher priority than kernel".  Setting real time is not compatible with applications
doing 100% polling. 

If you have to use REALTIME then application must change to doing sleep/wakeup type
architecture, not pure polling.

Typical DPDK style application is incompatible with SCHED_FIFO/SCHED_RR.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [dpdk-dev] [EXTERNAL] Re: Windows DPDK real-time priority threads causing thread starvation
  2020-12-09 16:08 ` Stephen Hemminger
@ 2020-12-09 16:12   ` Dmitry Malloy (MESHCHANINOV)
  2020-12-16 14:53     ` Tal Shnaiderman
  0 siblings, 1 reply; 5+ messages in thread
From: Dmitry Malloy (MESHCHANINOV) @ 2020-12-09 16:12 UTC (permalink / raw)
  To: Stephen Hemminger, Tal Shnaiderman
  Cc: Dmitry Kozlyuk, Narcisa Ana Maria Vasile, Eilon Greenstein,
	Omar Cardona, Rani Sharoni, Odi Assli, Harini Ramakrishnan,
	thomas, dev

There is a configuration in Windows similar to Linux isolcpus where scheduler tries not to run anything on such cores, and implementation is being enhanced for the next Windows release with User-mode vmswitch feedback.

I'll dig out the details.

Dmitry

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: Wednesday, December 9, 2020 8:09 AM
To: Tal Shnaiderman <talshn@nvidia.com>
Cc: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Dmitry Malloy (MESHCHANINOV) <dmitrym@microsoft.com>; Narcisa Ana Maria Vasile <Narcisa.Vasile@microsoft.com>; Eilon Greenstein <eilong@nvidia.com>; Omar Cardona <ocardona@microsoft.com>; Rani Sharoni <ranish@nvidia.com>; Odi Assli <odia@nvidia.com>; Harini Ramakrishnan <Harini.Ramakrishnan@microsoft.com>; thomas <thomas@monjalon.net>; dev@dpdk.org
Subject: [EXTERNAL] Re: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation

On Wed, 9 Dec 2020 14:15:30 +0000
Tal Shnaiderman <talshn@nvidia.com> wrote:

> Hi,
> 
> During our verification tests on Windows DPDK we've noticed that DPDK polling threads, which run in REALTIME_PRIORITY_CLASS are causing starvation to other threads from the OS which need to change affinity and run in lower priority.
> 
> While running an application for a while we see the OS thread waits for 2:30 minutes and raises a bugcheck, see below example of such flow:
> 
> 1) DPDK thread running on core-0 in real-time high priority(24) polling mode.
> 2) The thread is blocking the system function NtSetSystemInformation 
> (ExpUpdateTimerConfiguration) in another thread from
>    switching to core-0 via KeSetSystemGroupAffinityThread since the 
> calling thread is priority 15.
> 3) NtSetSystemInformation exclusively acquired system-wide lock 
> (ExpTimeRefreshLock) hence
>     it blocks other threads (e.g. calling NtQuerySystemInformation).
> 
> We've seen this behavior only while running on Windows 2019 VMs, maybe on native machines OS scheduling of such flow is done differently? 
> 
> Below is usage explanation from the documentation of SetPriorityClass [1]:
> 
> - REALTIME_PRIORITY_CLASS
> Process that has the highest possible priority. The threads of the process preempt the threads of all other processes, including operating system processes performing important tasks. For example, a real-time process that executes for more than a very brief interval can cause disk caches not to flush or cause the mouse to be unresponsive. 
> 
> So I assume using this kind of thread for a long period as we do can cause unstable behavior.
> 
> How do you think we can resolve this? Are there such cases in Linux?
> 
> [1] - 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs
> .microsoft.com%2Fen-us%2Fwindows%2Fwin32%2Fapi%2Fprocessthreadsapi%2Fn
> f-processthreadsapi-setpriorityclass&amp;data=04%7C01%7Cdmitrym%40micr
> osoft.com%7C623844c12bc2440d3bbd08d89c5cc0f1%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637431269479649074%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp
> ;sdata=LSv%2F%2BZMxUkfgcQVD778wL4JbVdl2qYV1tHdfVrEck4c%3D&amp;reserved
> =0
> 
> Thanks,
> 
> Tal.

This is not unique to Windows, Linux has same thing when using SCHED_FIFO.
Setting REALTIME is not a magic "go fast" flag it tells scheduler to "run this thread at higher priority than kernel".  Setting real time is not compatible with applications doing 100% polling. 

If you have to use REALTIME then application must change to doing sleep/wakeup type architecture, not pure polling.

Typical DPDK style application is incompatible with SCHED_FIFO/SCHED_RR.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [dpdk-dev] [EXTERNAL] Re: Windows DPDK real-time priority threads causing thread starvation
  2020-12-09 16:12   ` [dpdk-dev] [EXTERNAL] " Dmitry Malloy (MESHCHANINOV)
@ 2020-12-16 14:53     ` Tal Shnaiderman
  0 siblings, 0 replies; 5+ messages in thread
From: Tal Shnaiderman @ 2020-12-16 14:53 UTC (permalink / raw)
  To: Dmitry Malloy (MESHCHANINOV), Stephen Hemminger
  Cc: Dmitry Kozlyuk, Narcisa Ana Maria Vasile, Eilon Greenstein,
	Omar Cardona, Rani Sharoni, Odi Assli, Harini Ramakrishnan,
	NBU-Contact-Thomas Monjalon, dev

It was suggested by DmitryM in the last DPDK Windows community call to have the priority parameter configurable (at least for Windows) to allow user to reduce the thread priority and avoid the issues mentioned below.

> Subject: RE: [EXTERNAL] Re: [dpdk-dev] Windows DPDK real-time priority
> threads causing thread starvation
> 
> There is a configuration in Windows similar to Linux isolcpus where scheduler
> tries not to run anything on such cores, and implementation is being
> enhanced for the next Windows release with User-mode vmswitch
> feedback.
> 
> I'll dig out the details.
> 
> Dmitry
> 
> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Wednesday, December 9, 2020 8:09 AM
> To: Tal Shnaiderman <talshn@nvidia.com>
> Cc: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Dmitry Malloy
> (MESHCHANINOV) <dmitrym@microsoft.com>; Narcisa Ana Maria Vasile
> <Narcisa.Vasile@microsoft.com>; Eilon Greenstein <eilong@nvidia.com>;
> Omar Cardona <ocardona@microsoft.com>; Rani Sharoni
> <ranish@nvidia.com>; Odi Assli <odia@nvidia.com>; Harini Ramakrishnan
> <Harini.Ramakrishnan@microsoft.com>; thomas <thomas@monjalon.net>;
> dev@dpdk.org
> Subject: [EXTERNAL] Re: [dpdk-dev] Windows DPDK real-time priority
> threads causing thread starvation
> 
> On Wed, 9 Dec 2020 14:15:30 +0000
> Tal Shnaiderman <talshn@nvidia.com> wrote:
> 
> > Hi,
> >
> > During our verification tests on Windows DPDK we've noticed that DPDK
> polling threads, which run in REALTIME_PRIORITY_CLASS are causing
> starvation to other threads from the OS which need to change affinity and
> run in lower priority.
> >
> > While running an application for a while we see the OS thread waits for 2:30
> minutes and raises a bugcheck, see below example of such flow:
> >
> > 1) DPDK thread running on core-0 in real-time high priority(24) polling
> mode.
> > 2) The thread is blocking the system function NtSetSystemInformation
> > (ExpUpdateTimerConfiguration) in another thread from
> >    switching to core-0 via KeSetSystemGroupAffinityThread since the
> > calling thread is priority 15.
> > 3) NtSetSystemInformation exclusively acquired system-wide lock
> > (ExpTimeRefreshLock) hence
> >     it blocks other threads (e.g. calling NtQuerySystemInformation).
> >
> > We've seen this behavior only while running on Windows 2019 VMs, maybe
> on native machines OS scheduling of such flow is done differently?
> >
> > Below is usage explanation from the documentation of SetPriorityClass [1]:
> >
> > - REALTIME_PRIORITY_CLASS
> > Process that has the highest possible priority. The threads of the process
> preempt the threads of all other processes, including operating system
> processes performing important tasks. For example, a real-time process that
> executes for more than a very brief interval can cause disk caches not to
> flush or cause the mouse to be unresponsive.
> >
> > So I assume using this kind of thread for a long period as we do can cause
> unstable behavior.
> >
> > How do you think we can resolve this? Are there such cases in Linux?
> >
> > [1] -
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs
> > .microsoft.com%2Fen-
> us%2Fwindows%2Fwin32%2Fapi%2Fprocessthreadsapi%2Fn
> > f-processthreadsapi-
> setpriorityclass&amp;data=04%7C01%7Cdmitrym%40micr
> >
> osoft.com%7C623844c12bc2440d3bbd08d89c5cc0f1%7C72f988bf86f141af91a
> b2d7
> >
> cd011db47%7C1%7C0%7C637431269479649074%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjo
> >
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C10
> 00&amp
> >
> ;sdata=LSv%2F%2BZMxUkfgcQVD778wL4JbVdl2qYV1tHdfVrEck4c%3D&amp;r
> eserved
> > =0
> >
> > Thanks,
> >
> > Tal.
> 
> This is not unique to Windows, Linux has same thing when using
> SCHED_FIFO.
> Setting REALTIME is not a magic "go fast" flag it tells scheduler to "run this
> thread at higher priority than kernel".  Setting real time is not compatible
> with applications doing 100% polling.
> 
> If you have to use REALTIME then application must change to doing
> sleep/wakeup type architecture, not pure polling.
> 
> Typical DPDK style application is incompatible with SCHED_FIFO/SCHED_RR.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2020-12-16 14:53 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-09 14:15 [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation Tal Shnaiderman
2020-12-09 16:08 ` John Alexander
2020-12-09 16:08 ` Stephen Hemminger
2020-12-09 16:12   ` [dpdk-dev] [EXTERNAL] " Dmitry Malloy (MESHCHANINOV)
2020-12-16 14:53     ` Tal Shnaiderman

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git