From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f45.google.com (mail-pg0-f45.google.com [74.125.83.45]) by dpdk.org (Postfix) with ESMTP id 549048DAB for ; Fri, 20 Apr 2018 03:44:30 +0200 (CEST) Received: by mail-pg0-f45.google.com with SMTP id i29so1820853pgn.12 for ; Thu, 19 Apr 2018 18:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e0VVfF7VJmxefRJzOoEfFXqazKaP4Jv9Bm9XwW3r3xs=; b=vg3Hi6Li3ldwCdlxJC6x46FYSv49xIjJ+uU/wFHxxlRDDL+5h6Wpi1nrTLGETyPSrm xpnJ6m2ACkk24aCBQkxUzyIpBJdfxSasXND0jjxH6iHYv0RUKCdVl4v8FwEApfeCJ6Pr H+71II2kdc8jO7SqaD9yD02TXCsX90penipkmo0yfX/dqyJedAzPIJA7ZF0aYHT+oHN3 lxTM+0Owo+F1c24zR37zaySXrRwyaX88TxvOuinwbiw5hW/dEz6aD3coIcuIHjNOrnTQ BhWVTqkA/s9hgDkYwpL0R8oiFxbfpVctk1BYu8n6PVZbrshe42BOf8nKto32Ekv3EPUi wxnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e0VVfF7VJmxefRJzOoEfFXqazKaP4Jv9Bm9XwW3r3xs=; b=HPPD/KGN8Hc9Du2iLBg83lpPEb0UvKMFoVjOK9on7kPhY+uBCujxv8Ks0XznXHU00V 02gwHLTvEW81C/82ExqFwRH9z9QHZwZQIEdnfzpRTTCU6SkFQuNADyTnTCsgAk31J4lo IlhFRHoA2aa9Xs3kSozZKyIZQN9UGxXQyHuT5eTMuUDz05CYZPNKfLDzjj2x8BHkbdZN XrHowBalnvSG8ABvbU2arXj1j6IiYo8JjyZbAgW6x9Q9EJ1uaCCAGQ2tRKE1mr0VqN89 nZwMnX+zLBmZAtdZD4dS6veE5/GzNkmDSJ1Mf78AGZRJsV5ecCjXSF4USSxkHz03x33a zDVg== X-Gm-Message-State: ALQs6tDCVkRIZnTgr93YKxeWib843aXCz3iuR/GK+mt9eAOQLQ57OlXV kL/nUgXMMWdxVgiKLUsdhPZlGQ== X-Google-Smtp-Source: AIpwx49gDrWhfzU95eKLLkV87ex0mVjQcjfSQOsDS/N9/YnQjiqAzkm5pvExS52mpMC4Io9vs+Maow== X-Received: by 10.98.96.135 with SMTP id u129mr3987725pfb.141.1524188669390; Thu, 19 Apr 2018 18:44:29 -0700 (PDT) Received: from xeon-e3 (204-195-71-95.wavecable.com. [204.195.71.95]) by smtp.gmail.com with ESMTPSA id t25sm7775108pfh.184.2018.04.19.18.44.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 19 Apr 2018 18:44:29 -0700 (PDT) Date: Thu, 19 Apr 2018 18:44:27 -0700 From: Stephen Hemminger To: "terry.montague.1980@btinternet.com" Cc: users@dpdk.org Message-ID: <20180419184427.755465d9@xeon-e3> In-Reply-To: <29021657.45860.1524168075315.JavaMail.defaultUser@defaultHost> References: <10337109.30346.1524152612506.JavaMail.defaultUser@defaultHost> <29021657.45860.1524168075315.JavaMail.defaultUser@defaultHost> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 01:44:30 -0000 On Thu, 19 Apr 2018 21:01:15 +0100 (BST) "terry.montague.1980@btinternet.com" wrote: > I should also say - I've disabled the kernel's 5% time reservation for SCHED_FIFO through setting /proc/sys/kernel/sched_rt_runtime_us to -1. > > Any help with this problem gratefully appreciated. > > Many thanks > > Terry. > > > ----Original message---- > From : terry.montague.1980@btinternet.com > Date : 19/04/18 - 16:43 (BST) > To : users@dpdk.org > Subject : [dpdk-users] Linux forcibly descheduling isolated thread on isolated cpu running DPDK rx under load > > Hi there, > I wondered if anyone had come across this particular problem regarding linux scheduling, or rather what appears to be a forced descheduling effect. > I'm running on standard vanilla Ubuntu 17-10 using kernel 4.13.0-36-generic. > Local Timer interrupts are therefore enabled.... > I'm running a dual CPU Xeon E5-2623v4 system. I have cpu 2 on the first NUMA node (CPU 0) isolated for DPDK receive. I have an Intel X550 card attached to NUMA 0. > What I'm doing is running my DPDK receive thread on the isolated core (2) and > changing the scheduling for this thread to SCHED_FIFO and priority 98. > Most of the time this works really well. However, I'm running this DPDK thread inside a larger application - there are probably 40 threads inside this process at default priority. > What I'm seeing is, when the application is under load, the DPDK receive thread is forcibly descheduled (observed with pidstat -p -w and seeing the non-voluntary counts spike ) and the core appears to go idle, sometimes for up to 1400uS. > This is obviously a problem.... > Running "perf" to sample activity on this isolated core only, I see the following entries. > 0.90% swapper [kernel.kallsyms] [k] cpu_idle_poll > 0.60% lcore-slave-2 [kernel.kallsyms] [k] clear_page_erms > i.e - it has gone idle and 1.5% of the processing time has gone elsewhere - which ties in pretty well with my ~1400uS deschedule observation. > In normal operation I do not see this effect. > I've checked the code - it appears to go idle in the middle of some AVX2 data processing code - there are no system calls taken, it just goes idle. > Does anyone have any ideas ? > Many thanks > Terry > AVX2 has issue that it uses more cpu power, and the CPU will sometimes go into power management (self preservation state). At my previous employer, we experimented with AVX2 for firewall matching and discovered that under benchmark load the overall performance was worse. Also, unless you isolate cpu's from scheduler via kernel cmdline or offline/online with sysfs. you can run into SCHED_FIFO processes that never yield starving ksoftirqd. That means of softirq ever happens on that cpu, it will never get serviced leading to hangs etc.