[dpdk-dev] [Bug 600] Windows DPDK real-time priority threads causing thread starvation

[dpdk-dev] [Bug 600] Windows DPDK real-time priority threads causing thread starvation

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=600

            Bug ID: 600
           Summary: Windows DPDK real-time priority threads causing thread
                    starvation
           Product: DPDK
           Version: 20.08
          Hardware: All
                OS: Windows
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: core
          Assignee: dev@dpdk.org
          Reporter: odia@nvidia.com
  Target Milestone: ---

During our verification tests on Windows DPDK we've noticed that DPDK polling
threads, which run in REALTIME_PRIORITY_CLASS are causing starvation to other
threads from the OS which need to change affinity and run in lower priority.


While running an application for a while we see the OS thread waits for 2:30
minutes and raises a bugcheck, see below example of such flow:

1) DPDK thread running on core-0 in real-time high priority(24) polling mode.
2) The thread is blocking the system function NtSetSystemInformation
(ExpUpdateTimerConfiguration) in another thread from 
   switching to core-0 via KeSetSystemGroupAffinityThread since the calling
thread is priority 15. 
3) NtSetSystemInformation exclusively acquired system-wide lock
(ExpTimeRefreshLock) hence 
    it blocks other threads (e.g. calling NtQuerySystemInformation).


Call stack of the issue (from windbg):
 # RetAddr           : Args to Child                                           
               : Call Site
00 fffff802`4f43abba : fffff802`4f80bd20 fffffe0d`9d18e838 fffffe0d`00010224
fffff802`4f564050 : nt!ExpWaitForResource+0x1576d2
01 fffff802`4fabbd66 : fffffe0d`9d18eb80 00000000`00000000 00000000`0000002c
00000040`738fc2f0 : nt!ExAcquireResourceExclusiveLite+0x18a
02 fffff802`4f9bc3e9 : 00000000`00000000 00000040`00028000 0000017e`dc3be000
00000000`00000000 : nt!ExAcquireTimeRefreshLock+0x22
03 fffff802`4f9bb83b : 00000000`00000000 00000000`00002001 0000017e`dc3a7d90
00000000`00000000 : nt!ExpQuerySystemInformation+0xa89
04 fffff802`4f5ce605 : ffffae0f`258382c0 0000017e`dc3699d0 fffffe0d`9d18eb80
ffffae0f`2c96c2e0 : nt!NtQuerySystemInformation+0x2b
05 00007ffa`b0befe24 : 00007ffa`acd300d1 00007ffa`9fa5d858 0000017e`dbb3e130
00000000`0000004a : nt!KiSystemServiceCopyEnd+0x25
06 00007ffa`acd300d1 : 00007ffa`9fa5d858 0000017e`dbb3e130 00000000`0000004a
00007ffa`9f9d8269 : ntdll!NtQuerySystemInformation+0x14
07 00007ffa`8a35dcc4 : 0000017e`dc5534d4 00000000`00000000 00000000`00000000
0000017e`dc3bd110 : KERNELBASE!GetTimeZoneInformation+0x51
08 00007ffa`9e019abc : 0000017e`dc3814f0 00000040`738fc670 00007ffa`9e209540
0000017e`dc3814f0 : framedynos!CInstance::SetDateTime+0x264
09 00007ffa`9e01a2f5 : 0000017e`dc3bd110 00000000`00000000 00007ffa`9e209540
0000017e`dc3814f0 : cimwin32!Process::LoadCheapPropertiesNT+0x45c
0a 00007ffa`8a3559a2 : 00000000`00000000 0000017e`dc380b70 00000000`00000000
0000017e`dc383410 : cimwin32!Process::Enumerate+0x155
0b 00007ffa`8a356f9d : 00000000`00000000 00000040`738fde09 0000017e`dc380b70
0000017e`00000000 : framedynos!Provider::ExecuteQuery+0x92
0c 00007ff6`337527f5 : 0000017e`dbb3a380 0000017e`dbad5690 0000017e`dba750f8
0000017e`dbb3a370 : framedynos!CWbemProviderGlue::ExecQueryAsync+0x21d
0d 00007ff6`33752473 : 00000000`00000000 00000000`00000000 0000017e`dba75010
00000000`00001000 :
wmiprvse!CInterceptor_IWbemSyncProvider::Helper_ExecQueryAsync+0x2b5
0e 00007ffa`aee07853 : 0000017e`00000000 0000017e`dbae5cc8 0000017e`dc5504c8
00000000`00000000 :
wmiprvse!CInterceptor_IWbemSyncProvider::ExecQueryAsync+0x143

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dpdk-dev] [Bug 600] Windows DPDK real-time priority threads causing thread starvation

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=600

Tal Shnaiderman (talshn@nvidia.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED
                 CC|                            |talshn@nvidia.com

--- Comment #1 from Tal Shnaiderman (talshn@nvidia.com) ---
This was fixed in
http://patchwork.dpdk.org/project/dpdk/patch/20210218114058.13248-1-talshn@nvidia.com/

By lowering default thread priority

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dpdk-dev] [Bug 600] Windows DPDK real-time priority threads causing thread starvation

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=600

odia@nvidia.com (odia@nvidia.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
             Status|RESOLVED                    |CONFIRMED
         Resolution|FIXED                       |---

--- Comment #2 from odia@nvidia.com (odia@nvidia.com) ---
Fix tested and approved

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dpdk-dev] [Bug 600] Windows DPDK real-time priority threads causing thread starvation

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=600

Thomas Monjalon (thomas@monjalon.net) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|CONFIRMED                   |RESOLVED
         Resolution|---                         |FIXED

--- Comment #3 from Thomas Monjalon (thomas@monjalon.net) ---
Resolved in http://git.dpdk.org/dpdk/commit/?id=16afcbfa30

-- 
You are receiving this mail because:
You are the assignee for the bug.