From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rgout0805.bt.lon5.cpcloud.co.uk (rgout0805.bt.lon5.cpcloud.co.uk [65.20.0.152]) by dpdk.org (Postfix) with ESMTP id CD5047CC5 for ; Thu, 19 Apr 2018 17:43:31 +0200 (CEST) X-OWM-Source-IP: 10.110.12.2 () X-OWM-Env-Sender: terry.montague.1980@btinternet.com X-RazorGate-Vade-Classification: clean X-RazorGate-Vade-Verdict: clean 20 X-VadeSecure-score: verdict=clean score=20/300, class=clean X-SNCR-VADESECURE: CLEAN X-RazorGate-Vade-Verdict: clean 20 X-RazorGate-Vade-Classification: clean X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgedtgedrjeehgdelfecutefuodetggdotefrodftvfcurfhrohhfihhlvgemuceutffkvffkuffjvffgnffgvefqofdpqfgfvfenuceurghilhhouhhtmecufedttdenucgotffhvedqqdeurggukfhmphhorhhtrghntggvucdlvddtmdenucfjughrpeffhfhrvffkufggtggkrfesmhdttggstddtjeenucfhrhhomhepfdhtvghrrhihrdhmohhnthgrghhuvgdrudelkedtsegsthhinhhtvghrnhgvthdrtghomhdfuceothgvrhhrhidrmhhonhhtrghguhgvrdduleektdessghtihhnthgvrhhnvghtrdgtohhmqeenucfkphepuddtrdduuddtrdduvddrvdenucfrrghrrghmpehhvghlohepfigvsghmrghilhduiedrsghtrdgvgihtrdgtphgtlhhouhgurdgtohdruhhkpdhinhgvthepuddtrdduuddtrdduvddrvddpmhgrihhlfhhrohhmpeeothgvrhhrhidrmhhonhhtrghguhgvrdduleektdessghtihhnthgvrhhnvghtrdgtohhmqedprhgtphhtthhopeeouhhsvghrshesughpughkrdhorhhgqeenucevlhhushhtvghrufhiiigvpedt Received: from webmail16.bt.ext.cpcloud.co.uk (10.110.12.2) by rgout08.bt.lon5.cpcloud.co.uk (9.0.019.26-1) (authenticated as terry.montague.1980@btinternet.com) id 5ACB6EFA0129EC1D for users@dpdk.org; Thu, 19 Apr 2018 16:43:31 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btinternet.com; s=btcpcloud; t=1524152611; bh=avpKRxdxDKFoBrCKCicF+jUbhsJN/1CNM6JvAsv7LD8=; h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version; b=oWizUzcl2p0YIVsZ2bRq6gzr3SF+wE9AyZ82giPjIyCIpUPsMM16DwzckhXAEM+WlpkNnDlXvhUEtXZRHQNakmVYe821U5gNRcTdfQpaJc71MTRYRcbQTnhDg/SIHNem05WJKlfJs8b0LDyIKyjCAoZBz78UOmq7RFBX88YM2gY= Date: Thu, 19 Apr 2018 16:43:32 +0100 (BST) From: "terry.montague.1980@btinternet.com" To: users@dpdk.org Message-ID: <10337109.30346.1524152612506.JavaMail.defaultUser@defaultHost> MIME-Version: 1.0 Importance: 3 (Normal) X-Priority: 3 (Normal) X-Client-IP: IPv4[86.134.190.82] Epoch[1524152612467] Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: terry.montague.1980@btinternet.com List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2018 15:43:32 -0000 Hi there, I wondered if anyone had come across this particular problem regarding linux scheduling, or rather what appears to be a forced descheduling effect. I'm running on standard vanilla Ubuntu 17-10 using kernel 4.13.0-36-generic. Local Timer interrupts are therefore enabled.... I'm running a dual CPU Xeon E5-2623v4 system. I have cpu 2 on the first NUMA node (CPU 0) isolated for DPDK receive. I have an Intel X550 card attached to NUMA 0. What I'm doing is running my DPDK receive thread on the isolated core (2) and changing the scheduling for this thread to SCHED_FIFO and priority 98. Most of the time this works really well. However, I'm running this DPDK thread inside a larger application - there are probably 40 threads inside this process at default priority. What I'm seeing is, when the application is under load, the DPDK receive thread is forcibly descheduled (observed with pidstat -p -w and seeing the non-voluntary counts spike ) and the core appears to go idle, sometimes for up to 1400uS. This is obviously a problem.... Running "perf" to sample activity on this isolated core only, I see the following entries. 0.90% swapper [kernel.kallsyms] [k] cpu_idle_poll 0.60% lcore-slave-2 [kernel.kallsyms] [k] clear_page_erms i.e - it has gone idle and 1.5% of the processing time has gone elsewhere - which ties in pretty well with my ~1400uS deschedule observation. In normal operation I do not see this effect. I've checked the code - it appears to go idle in the middle of some AVX2 data processing code - there are no system calls taken, it just goes idle. Does anyone have any ideas ? Many thanks Terry