From: Stephen Hemminger <email@example.com> To: Tal Shnaiderman <firstname.lastname@example.org> Cc: Dmitry Kozlyuk <email@example.com>, "Dmitry Malloy (MESHCHANINOV)" <firstname.lastname@example.org>, Narcisa Ana Maria Vasile <Narcisa.Vasile@microsoft.com>, Eilon Greenstein <email@example.com>, Omar Cardona <firstname.lastname@example.org>, Rani Sharoni <email@example.com>, Odi Assli <firstname.lastname@example.org>, Harini Ramakrishnan <Harini.Ramakrishnan@microsoft.com>, NBU-Contact-Thomas Monjalon <email@example.com>, "firstname.lastname@example.org" <email@example.com> Subject: Re: [dpdk-dev] Windows DPDK real-time priority threads causing thread starvation Date: Wed, 9 Dec 2020 08:08:58 -0800 Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <DM6PR12MB39455643CEE4FF76DCA6743BA4CC0@DM6PR12MB3945.namprd12.prod.outlook.com> On Wed, 9 Dec 2020 14:15:30 +0000 Tal Shnaiderman <email@example.com> wrote: > Hi, > > During our verification tests on Windows DPDK we've noticed that DPDK polling threads, which run in REALTIME_PRIORITY_CLASS are causing starvation to other threads from the OS which need to change affinity and run in lower priority. > > While running an application for a while we see the OS thread waits for 2:30 minutes and raises a bugcheck, see below example of such flow: > > 1) DPDK thread running on core-0 in real-time high priority(24) polling mode. > 2) The thread is blocking the system function NtSetSystemInformation (ExpUpdateTimerConfiguration) in another thread from > switching to core-0 via KeSetSystemGroupAffinityThread since the calling thread is priority 15. > 3) NtSetSystemInformation exclusively acquired system-wide lock (ExpTimeRefreshLock) hence > it blocks other threads (e.g. calling NtQuerySystemInformation). > > We've seen this behavior only while running on Windows 2019 VMs, maybe on native machines OS scheduling of such flow is done differently? > > Below is usage explanation from the documentation of SetPriorityClass : > > - REALTIME_PRIORITY_CLASS > Process that has the highest possible priority. The threads of the process preempt the threads of all other processes, including operating system processes performing important tasks. For example, a real-time process that executes for more than a very brief interval can cause disk caches not to flush or cause the mouse to be unresponsive. > > So I assume using this kind of thread for a long period as we do can cause unstable behavior. > > How do you think we can resolve this? Are there such cases in Linux? > >  - https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setpriorityclass > > Thanks, > > Tal. This is not unique to Windows, Linux has same thing when using SCHED_FIFO. Setting REALTIME is not a magic "go fast" flag it tells scheduler to "run this thread at higher priority than kernel". Setting real time is not compatible with applications doing 100% polling. If you have to use REALTIME then application must change to doing sleep/wakeup type architecture, not pure polling. Typical DPDK style application is incompatible with SCHED_FIFO/SCHED_RR.
next prev parent reply other threads:[~2020-12-09 16:09 UTC|newest] Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-12-09 14:15 Tal Shnaiderman 2020-12-09 16:08 ` John Alexander 2020-12-09 16:08 ` Stephen Hemminger [this message] 2020-12-09 16:12 ` [dpdk-dev] [EXTERNAL] " Dmitry Malloy (MESHCHANINOV) 2020-12-16 14:53 ` Tal Shnaiderman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --cc=Harini.Ramakrishnan@microsoft.com \ --cc=Narcisa.Vasile@microsoft.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
DPDK patches and discussions This inbox may be cloned and mirrored by anyone: git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \ email@example.com public-inbox-index dev Example config snippet for mirrors. Newsgroup available over NNTP: nntp://inbox.dpdk.org/inbox.dpdk.dev AGPL code for this site: git clone https://public-inbox.org/public-inbox.git