From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from eu-smtp-delivery-178.mimecast.com (eu-smtp-delivery-178.mimecast.com [146.101.78.178]) by dpdk.org (Postfix) with ESMTP id 71EFB7CBD for ; Fri, 20 Apr 2018 12:14:23 +0200 (CEST) Received: from ex-uknb-01.ad.s-a-m.com (83-244-203-18.cust-83.exponential-e.net [83.244.203.18]) (Using TLS) by eu-smtp-1.mimecast.com with ESMTP id uk-mta-59-bs5m__YnMbyWWZSgr_yqAQ-1; Fri, 20 Apr 2018 11:14:22 +0100 Received: from EX-UKHA-01.ad.s-a-m.com ([fe80::bd21:e36b:5bf6:8ca8]) by EX-UKHA-01.ad.s-a-m.com ([fe80::bd21:e36b:5bf6:8ca8%11]) with mapi id 14.03.0361.001; Fri, 20 Apr 2018 11:14:19 +0100 From: Richard Nutman To: "users@dpdk.org" , "terry.montague.1980@btinternet.com" Thread-Topic: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load Thread-Index: AQHT2I5b6LyPx92SAkyUGCUskhItaKQJbLCQ Date: Fri, 20 Apr 2018 10:14:18 +0000 Message-ID: <733AB18813E3864094592CC5191B172A235BD476@EX-UKHA-01.ad.s-a-m.com> References: <10337109.30346.1524152612506.JavaMail.defaultUser@defaultHost> <21519087207c434382a81b92a2a2ca31@advaoptical.com> In-Reply-To: <21519087207c434382a81b92a2a2ca31@advaoptical.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.19.77.57] x-esetresult: clean, is OK x-esetid: 37303A29F33CD061647564 MIME-Version: 1.0 X-MC-Unique: bs5m__YnMbyWWZSgr_yqAQ-1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 10:14:23 -0000 Hi Terry, Following on from what Stephen mentioned, when you hit an AVX2 instruction = there is a warmup latency while the CPU powers on the upper half of the 256= bit lanes. It's normally around 10usecs, so possibly not accounting for everything you= 're seeing; https://software.intel.com/en-us/forums/intel-isa-extensions/topic/710248 Also with RT threads that never yield you should add nosoftlockup to your b= ootline to prevent the kernel assuming your thread has locked up. Some things to look into; 1. Are you using no_hz mode on the kernel bootline ? 2. Have you disabled RCU callbacks from your cpu's with rcu_nocbs on kernel= bootline ? 3. Have you manually IRQbalanced to move IRQ's off your isolated cpu's ? The clear_page_erms suggests it could be memory housekeeping like zone recl= aiming or transparent_hugepages, have you disabled these ? -Richard. > -----Original Message----- > From: Tim Shearer [mailto:TShearer@advaoptical.com] > Sent: 20 April 2018 03:00 > To: users@dpdk.org; terry.montague.1980@btinternet.com > Subject: Re: [dpdk-users] Linux forcibly descheduling isolated thread > on isolated cpu running DPDK rx under load >=20 > Hi Terry, >=20 > Without digging into this too much, it looks like the kernel is context > switching out to do a clear_page call, so I wonder if one of your other > threads is doing something memory related that's triggering this > behaviour. >=20 > Tim > ________________________________ > From: users on behalf of > terry.montague.1980@btinternet.com > Sent: Thursday, April 19, 2018 11:43:32 AM > To: users@dpdk.org > Subject: [dpdk-users] Linux forcibly descheduling isolated thread on > isolated cpu running DPDK rx under load >=20 > Hi there, > I wondered if anyone had come across this particular problem regarding > linux scheduling, or rather what appears to be a forced descheduling > effect. > I'm running on standard vanilla Ubuntu 17-10 using kernel 4.13.0-36- > generic. > Local Timer interrupts are therefore enabled.... > I'm running a dual CPU Xeon E5-2623v4 system. I have cpu 2 on the first > NUMA node (CPU 0) isolated for DPDK receive. I have an Intel X550 card > attached to NUMA 0. > What I'm doing is running my DPDK receive thread on the isolated core > (2) and changing the scheduling for this thread to SCHED_FIFO and > priority 98. > Most of the time this works really well. However, I'm running this DPDK > thread inside a larger application - there are probably 40 threads > inside this process at default priority. > What I'm seeing is, when the application is under load, the DPDK > receive thread is forcibly descheduled (observed with pidstat -p > -w and seeing the non-voluntary counts spike ) and the core appears to > go idle, sometimes for up to 1400uS. > This is obviously a problem.... > Running "perf" to sample activity on this isolated core only, I see the > following entries. > 0.90% swapper [kernel.kallsyms] [k] cpu_idle_poll > 0.60% lcore-slave-2 [kernel.kallsyms] [k] clear_page_erms > i.e - it has gone idle and 1.5% of the processing time has gone > elsewhere - which ties in pretty well with my ~1400uS deschedule > observation. > In normal operation I do not see this effect. > I've checked the code - it appears to go idle in the middle of some > AVX2 data processing code - there are no system calls taken, it just > goes idle. > Does anyone have any ideas ? > Many thanks > Terry=0A-----------------------------------------------------------------= ----------------------=0AThis email has been scanned for email related thre= ats and delivered safely by Mimecast.=0AFor more information please visit h= ttp://www.mimecast.com=0A--------------------------------------------------= -------------------------------------=0A