From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail3.advaoptical.com (mail3.advaoptical.com [74.202.24.82]) by dpdk.org (Postfix) with ESMTP id 5FFAF8DA1 for ; Fri, 20 Apr 2018 03:59:34 +0200 (CEST) Received: from ATL-S-EX16B.advaoptical.com ([172.16.5.11]) by atl-vs-fsmail.advaoptical.com (8.16.0.22/8.16.0.22) with ESMTPS id w3K1xVee008045 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA256 bits=128 verify=FAIL); Thu, 19 Apr 2018 21:59:32 -0400 Received: from ATL-S-EX16B.advaoptical.com (172.16.5.11) by ATL-S-EX16B.advaoptical.com (172.16.5.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Thu, 19 Apr 2018 21:59:31 -0400 Received: from ATL-S-EX16B.advaoptical.com ([fe80::d04c:21c4:92cf:e1cd]) by ATL-S-EX16B.advaoptical.com ([fe80::d04c:21c4:92cf:e1cd%12]) with mapi id 15.01.1466.003; Thu, 19 Apr 2018 21:59:31 -0400 From: Tim Shearer To: "users@dpdk.org" , "terry.montague.1980@btinternet.com" Thread-Topic: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load Thread-Index: AQHT1/UuQ/SnPoQAfkq3/C8uFgx14KQI45Iz Date: Fri, 20 Apr 2018 01:59:31 +0000 Message-ID: <21519087207c434382a81b92a2a2ca31@advaoptical.com> References: <10337109.30346.1524152612506.JavaMail.defaultUser@defaultHost> In-Reply-To: <10337109.30346.1524152612506.JavaMail.defaultUser@defaultHost> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.16.5.40] MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-19_09:, , signatures=0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 01:59:34 -0000 Hi Terry, Without digging into this too much, it looks like the kernel is context swi= tching out to do a clear_page call, so I wonder if one of your other thread= s is doing something memory related that's triggering this behaviour. Tim ________________________________ From: users on behalf of terry.montague.1980@btint= ernet.com Sent: Thursday, April 19, 2018 11:43:32 AM To: users@dpdk.org Subject: [dpdk-users] Linux forcibly descheduling isolated thread on isolat= ed cpu running DPDK rx under load Hi there, I wondered if anyone had come across this particular problem regarding linu= x scheduling, or rather what appears to be a forced descheduling effect. I'm running on standard vanilla Ubuntu 17-10 using kernel 4.13.0-36-generic= . Local Timer interrupts are therefore enabled.... I'm running a dual CPU Xeon E5-2623v4 system. I have cpu 2 on the first NUM= A node (CPU 0) isolated for DPDK receive. I have an Intel X550 card attache= d to NUMA 0. What I'm doing is running my DPDK receive thread on the isolated core (2) a= nd changing the scheduling for this thread to SCHED_FIFO and priority 98. Most of the time this works really well. However, I'm running this DPDK thr= ead inside a larger application - there are probably 40 threads inside this= process at default priority. What I'm seeing is, when the application is under load, the DPDK receive th= read is forcibly descheduled (observed with pidstat -p -w and seeing = the non-voluntary counts spike ) and the core appears to go idle, sometimes= for up to 1400uS. This is obviously a problem.... Running "perf" to sample activity on this isolated core only, I see the fol= lowing entries. 0.90% swapper [kernel.kallsyms] [k] cpu_idle_poll 0.60% lcore-slave-2 [kernel.kallsyms] [k] clear_page_erms i.e - it has gone idle and 1.5% of the processing time has gone elsewhere = - which ties in pretty well with my ~1400uS deschedule observation. In normal operation I do not see this effect. I've checked the code - it appears to go idle in the middle of some AVX2 da= ta processing code - there are no system calls taken, it just goes idle. Does anyone have any ideas ? Many thanks Terry