From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f45.google.com (mail-pg0-f45.google.com [74.125.83.45]) by dpdk.org (Postfix) with ESMTP id 56A333798 for ; Wed, 6 Sep 2017 17:17:07 +0200 (CEST) Received: by mail-pg0-f45.google.com with SMTP id 63so6676220pgc.1 for ; Wed, 06 Sep 2017 08:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nDtNizGvTMk27C/Yr9C+Yr0ZFfd3JAJ2jcNDQdTSm3U=; b=1Ie3FHM0f835PyvW9AWG6hlIvgHUFpMhmRmoOim7NmVgPQH/gOu8gvMpkYq7LwMh2i g8GYd+LZGvzqMB4hKGx7NMOn1OzbawvCzgakK1pHaMuZJ1JZDF1ijjf35suXfxwyMTvt aJT0ZRSbRtOv6pMlNl1ZSW4h/RTCxVg3tyjwRlrMnWPGbQwq6gToA8mvb4j8Q2zNPbbS YBPDQ2xtMRihDxY2zO4qopbfrTtMDCfRYvK0LiAL+fuzEYY6uvGC8A8fCWanb69g46lN w0rCk5jbIXGamxWrtqzOcGP2s2u1JYOc1d88PAgN4dFOCzMbcmKop76uN4/r4nx9fz+E GSkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nDtNizGvTMk27C/Yr9C+Yr0ZFfd3JAJ2jcNDQdTSm3U=; b=I2CdRC/bKNjZgzZDAeEoeDtzmFBFgpDs/QdRjEdxgecAdhXwENZSpT1a8u6Yvm4TZh XsbBgRV0ML6jBTxVcBRZLVTioQsL4d0dg2IlQdg+2O/fc6M/y9gp2QxPBItnLkHZlvDm SPpwvk3iKS9WgipSf/5GDivM8T+3WQz7W8hRgpcyd5m+hI+4Dh+J3K8VdWjrvDWH/YAO j7yUCUUWtzyxB3KTSRQHTaOYzSWkW7FSSH/70rfCr4j0QpucTL8UW64v90cUXikFbMaS pGl7SUE0R1b0/sUAUbD1DtD7yIQFNjVPhlE/1G6m+Zl4dNXYRzrHnmnGVNAm1y8q9YbW vGTw== X-Gm-Message-State: AHPjjUiFpjFBrcf2hS6SXDmeqzpl9/PyrWVdZFffifkDnsJDWm16lnxC t28ki2PSjNqGrHop X-Google-Smtp-Source: ADKCNb4VOaSSkke7xifGQO6kYzPJsBF7MSuj8bVpvs4Ay6epXTxsgCyLUaLBEj9zX1dV42O3Vwii+A== X-Received: by 10.84.238.198 with SMTP id l6mr8579666pln.152.1504711026468; Wed, 06 Sep 2017 08:17:06 -0700 (PDT) Received: from xeon-e3 (76-14-207-240.or.wavecable.com. [76.14.207.240]) by smtp.gmail.com with ESMTPSA id 75sm49171pfx.184.2017.09.06.08.17.06 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Sep 2017 08:17:06 -0700 (PDT) Date: Wed, 6 Sep 2017 08:16:56 -0700 From: Stephen Hemminger To: "terry.montague.1980@btinternet.com" Cc: users@dpdk.org Message-ID: <20170906081508.4883c1c8@xeon-e3> In-Reply-To: <31543919.4352.1504684138463.JavaMail.defaultUser@defaultHost> References: <23108673.51650.1504629316520.JavaMail.defaultUser@defaultHost> <20170905095552.53cfcd17@xeon-e3> <31543919.4352.1504684138463.JavaMail.defaultUser@defaultHost> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Linux descheduling DPDK transmit thread in SMP system? Any help greatly appreciated... X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Sep 2017 15:17:07 -0000 On Wed, 6 Sep 2017 08:48:58 +0100 (BST) "terry.montague.1980@btinternet.com" wrote: > Stephen - thank-you for your input. > > Tens of milliseconds seems an awful long time for some kernel-side resource lock, although its obviously doing something with the time. There is only one user thread running on this core (my transmit thread). > > Does anyone have an approach to suggest to narrow this down a bit? Would I expect this to be worse on an SMP system? > > Many thanks > > Terry. > > > ----Original message---- > From : stephen@networkplumber.org > Date : 05/09/17 - 17:55 (BST) > To : terry.montague.1980@btinternet.com > Cc : users@dpdk.org > Subject : Re: [dpdk-users] Linux descheduling DPDK transmit thread in SMP system? Any help greatly appreciated... > > On Tue, 5 Sep 2017 17:35:16 +0100 (BST) > "terry.montague.1980@btinternet.com" wrote: > > > Hi all, > > I'm running a DPDK transmit thread with the ixgbe PMD on Ubuntu 17.04, with dual Xeon E5-2623 v4 CPUs on an Intel S2600 motherboard. The cores in use by DPDK are isolated in the boot cmdline with "isolcpus" > > What I'm observing is I believe, on occasion, the DPDK transmit thread being descheduled by the system for between 20 and 60 milliseconds. > > I've tried running as Real time with max priority, and tweaking the scheduling through /proc/sys/kernel/sched_rt_runtime_us to be 99.999% available for real time thread, but so far no improvement > > I notice there are various kernel threads resident on the isolated cpus, cpuhp/watchdog/migration/ksoftirqd +multiple kworker threads. > > I can only assume the system is occasionally running these kernel side instead, although the CPU time taken (tens of milliseconds) seems very high. Note please that I amd NOT running with a modified kernel (nohz_full is not set). > > Does anyone have any advice to keep the DPDK transmit thread running for more of the time on this SMP system ? > > Many thanks > > Terry. > > Probably not a Linux scheduler issue. More likely some system hardware thing like SMT or ME. > Maybe power management. If you have any BIOS settings that have balanced or power aware they can slow the CPU. Also, the hidden BIOS SMI can cause latency spikes. It may have nothing to do with your application or kernel. https://stackoverflow.com/questions/25399405/evaluating-smi-system-management-interrupt-latency-on-linux-centos-intel-machi Also this may provide some clues, although it is not DPDK related. https://access.redhat.com/sites/default/files/attachments/201501-perf-brief-low-latency-tuning-rhel7-v1.1.pdf