From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f46.google.com (mail-pg0-f46.google.com [74.125.83.46]) by dpdk.org (Postfix) with ESMTP id 8C3361C0B for ; Tue, 5 Sep 2017 18:56:00 +0200 (CEST) Received: by mail-pg0-f46.google.com with SMTP id 63so1382418pgc.1 for ; Tue, 05 Sep 2017 09:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=c4DsJPpOCMPd5H+jhQYAstX0CAUuLwKD5onn7ADoznU=; b=yO5leHwu1QKjPGgOr5YxzN+5dCmLqtlgj/AihS5ZkrF9tT63zOV7oIEdZY7o4MIcp8 gqYsmXQe6j8mGO4jWLEF3d0gEFIKwaTbG2adSY3+a3MQc+YMcgMv/AQuV9P9MkF1h7UM 9L5W9t/g71AzWLC/RIx2KVbYEFyxb0ey6m3jWY2C1Ca1hCjgbCkkMjSyJH+zRKm2ZbzU utlfQ+vTM8X3Wo5KTt7RuUT2IcbMOwJq/wRha+vrAJEAgETZaI+RNTIEX3vmvoLWYOrR iTh5HeJjOeaiQUiT7XJFHz09gvinCFhKph/eGOD8WNwmpT0DL8oKeIbak1Cs4B/0uARW jtxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=c4DsJPpOCMPd5H+jhQYAstX0CAUuLwKD5onn7ADoznU=; b=kRHCm73Bf5PPbI6RF+tvs0L9zioYDfLxgH6+t6Msk6HmARsHqyvQvG6ex/mXTJL/po Xg5jHyKJgp7fBIaIbuYyD4Q2wWDlkYi40IURj/ANe6gNI8YHHuz29xKGF+CHbFBoxjmZ 0HheOcRcNQAk6iwnBSck3Wn1XZ1O8hf5wnn/iQXFnOlO2JsiYAoVnXx69pIgPh1HI977 7yLWm6DUyC2SYR58qgYRjKwewkd5Y9e9LRwT5BLOJ1aksPfa8r4I8qrOwQL/nV5/7yFB aS08tCaPnTKbEz2DXrZjogpHh/+ScVPddAOnBPLWKoERSy52omAbU9JFQ7QRd2X2rjmG jFYg== X-Gm-Message-State: AHPjjUhstoW3bbT1P8E06m0WMxeiyAwevgxYm4Pj1Vaf3lRmgzC57t0/ B3meXvP4CpNDOi91Xea23w== X-Google-Smtp-Source: ADKCNb6aWYxXsx1aGxhUOnr6ijG309c5Ka9zJZGdmqYXM0MXf8YixVdugWrftuZrULq1F8GO+VNxPA== X-Received: by 10.84.130.39 with SMTP id 36mr5054621plc.287.1504630559707; Tue, 05 Sep 2017 09:55:59 -0700 (PDT) Received: from xeon-e3 (76-14-207-240.or.wavecable.com. [76.14.207.240]) by smtp.gmail.com with ESMTPSA id x4sm1713325pfb.101.2017.09.05.09.55.59 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 05 Sep 2017 09:55:59 -0700 (PDT) Date: Tue, 5 Sep 2017 09:55:52 -0700 From: Stephen Hemminger To: "terry.montague.1980@btinternet.com" Cc: users@dpdk.org Message-ID: <20170905095552.53cfcd17@xeon-e3> In-Reply-To: <23108673.51650.1504629316520.JavaMail.defaultUser@defaultHost> References: <23108673.51650.1504629316520.JavaMail.defaultUser@defaultHost> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Linux descheduling DPDK transmit thread in SMP system? Any help greatly appreciated... X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Sep 2017 16:56:00 -0000 On Tue, 5 Sep 2017 17:35:16 +0100 (BST) "terry.montague.1980@btinternet.com" wrote: > Hi all, > I'm running a DPDK transmit thread with the ixgbe PMD on Ubuntu 17.04, with dual Xeon E5-2623 v4 CPUs on an Intel S2600 motherboard. The cores in use by DPDK are isolated in the boot cmdline with "isolcpus" > What I'm observing is I believe, on occasion, the DPDK transmit thread being descheduled by the system for between 20 and 60 milliseconds. > I've tried running as Real time with max priority, and tweaking the scheduling through /proc/sys/kernel/sched_rt_runtime_us to be 99.999% available for real time thread, but so far no improvement > I notice there are various kernel threads resident on the isolated cpus, cpuhp/watchdog/migration/ksoftirqd +multiple kworker threads. > I can only assume the system is occasionally running these kernel side instead, although the CPU time taken (tens of milliseconds) seems very high. Note please that I amd NOT running with a modified kernel (nohz_full is not set). > Does anyone have any advice to keep the DPDK transmit thread running for more of the time on this SMP system ? > Many thanks > Terry. Probably not a Linux scheduler issue. More likely some system hardware thing like SMT or ME.