From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rgout04.bt.lon5.cpcloud.co.uk (rgout0407.bt.lon5.cpcloud.co.uk [65.20.0.220]) by dpdk.org (Postfix) with ESMTP id EB37BFFA for ; Wed, 6 Sep 2017 09:48:58 +0200 (CEST) X-OWM-Source-IP: 10.110.13.1 () X-OWM-Env-Sender: terry.montague.1980@btinternet.com X-Junkmail-Premium-Raw: score=17/50, refid=2.7.2:2017.9.6.72716:17:17.264, ip=, rules=__HAS_FROM, __FRAUD_WEBMAIL_FROM, __HAS_REPLYTO, __FRAUD_WEBMAIL_REPLYTO, __TO_MALFORMED_2, __TO_NO_NAME, __HAS_CC_HDR, __HAS_MSGID, __SANE_MSGID, INVALID_MSGID_NO_FQDN, __IN_REP_TO, __REFERENCES, __BOUNCE_CHALLENGE_SUBJ, __BOUNCE_NDR_SUBJ_EXEMPT, __MIME_VERSION, __CT, __CT_TEXT_PLAIN, CT_TEXT_PLAIN_UTF8_CAPS, __CTE, __HAS_X_PRIORITY, __REPLYTO_SAMEAS_FROM_ADDY, __REPLYTO_SAMEAS_FROM_ACC, __REPLYTO_SAMEAS_FROM_DOMAIN, __TO_IN_SUBJECT, __ANY_URI, __FRAUD_BODY_WEBMAIL, __URI_NO_WWW, __STOCK_PHRASE_7, __SUBJ_ALPHA_NEGATE, SUPERLONG_LINE, __FORWARDED_MSG, __NO_HTML_TAG_RAW, BODY_SIZE_1900_1999, BODYTEXTP_SIZE_3000_LESS, __MIME_TEXT_P1, __MIME_TEXT_ONLY, __URI_NS, URI_SNOWNS_NS_NAME[[networkplumber.org]=>[dns4.name-services.com]], HTML_00_01, HTML_00_10, BODY_SIZE_5000_LESS, __FRAUD_WEBMAIL, IN_REP_TO, MSG_THREAD, LEGITIMATE_SIGNS, __PHISH_SPEAR_STRUCTURE_1, BODY_SIZE_2000_LESS, PRIORITY_NO_NAME, TO_IN_SUBJECT, __MIME_TEXT_P, REPLYTO_SAMEAS_FROM, REFERENCES, __DQ_HEUR_1, NO_URI_HTTPS, BODY_SIZE_7000_LESS, DQ_SUSP_1 Received: from webmail10.bt.ext.cpcloud.co.uk (10.110.13.1) by rgout04.bt.lon5.cpcloud.co.uk (9.0.019.13-1) (authenticated as terry.montague.1980@btinternet.com) id 58482DA21C3E9C92 for users@dpdk.org; Wed, 6 Sep 2017 08:48:58 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btinternet.com; s=btcpcloud; t=1504684138; bh=bhzTa5XlGcmkiZVAVG4fFOtItQNjzggqxCFSrX8lVSo=; h=Date:From:Reply-To:To:Cc:Message-ID:In-Reply-To:References:Subject:MIME-Version; b=ESn/AH3Cy30rEpalIigLHqLM9Aeiu+ClNXh3pIvr3DKuybMlhYSRMPVjU6EuYSLfHuyMRSlb+RSokjK/m+cZdeYHEgFFe5lif48s8/tNo8/zFGWJbYUMdxlDnsG2l24n8HWFmdO/MZcssvt7GMjDlxB6eF2oitWGvK9Jdx/zSNU= Date: Wed, 6 Sep 2017 08:48:58 +0100 (BST) From: "terry.montague.1980@btinternet.com" To: users@dpdk.org Cc: users@dpdk.org Message-ID: <31543919.4352.1504684138463.JavaMail.defaultUser@defaultHost> In-Reply-To: <20170905095552.53cfcd17@xeon-e3> References: <23108673.51650.1504629316520.JavaMail.defaultUser@defaultHost> <20170905095552.53cfcd17@xeon-e3> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-CP-REPLY-ALL-UID: 310 X-CP-REPLY-ALL-UID: 310 X-CP-REPLY-ALL-PATH: INBOX X-CP-REPLY-ALL-PATH: INBOX Importance: 3 (Normal) X-Priority: 3 (Normal) X-Client-IP: IPv4[31.53.103.251] Epoch[1504684138422] Subject: Re: [dpdk-users] Linux descheduling DPDK transmit thread in SMP system? Any help greatly appreciated... X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: terry.montague.1980@btinternet.com List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Sep 2017 07:48:59 -0000 Stephen - thank-you for your input. Tens of milliseconds seems an awful long time for some kernel-side resource lock, although its obviously doing something with the time. There is only one user thread running on this core (my transmit thread). Does anyone have an approach to suggest to narrow this down a bit? Would I expect this to be worse on an SMP system? Many thanks Terry. ----Original message---- >>From : stephen@networkplumber.org Date : 05/09/17 - 17:55 (BST) To : terry.montague.1980@btinternet.com Cc : users@dpdk.org Subject : Re: [dpdk-users] Linux descheduling DPDK transmit thread in SMP system? Any help greatly appreciated... On Tue, 5 Sep 2017 17:35:16 +0100 (BST) "terry.montague.1980@btinternet.com" wrote: > Hi all, > I'm running a DPDK transmit thread with the ixgbe PMD on Ubuntu 17.04, with dual Xeon E5-2623 v4 CPUs on an Intel S2600 motherboard. The cores in use by DPDK are isolated in the boot cmdline with "isolcpus" > What I'm observing is I believe, on occasion, the DPDK transmit thread being descheduled by the system for between 20 and 60 milliseconds. > I've tried running as Real time with max priority, and tweaking the scheduling through /proc/sys/kernel/sched_rt_runtime_us to be 99.999% available for real time thread, but so far no improvement > I notice there are various kernel threads resident on the isolated cpus, cpuhp/watchdog/migration/ksoftirqd +multiple kworker threads. > I can only assume the system is occasionally running these kernel side instead, although the CPU time taken (tens of milliseconds) seems very high. Note please that I amd NOT running with a modified kernel (nohz_full is not set). > Does anyone have any advice to keep the DPDK transmit thread running for more of the time on this SMP system ? > Many thanks > Terry. Probably not a Linux scheduler issue. More likely some system hardware thing like SMT or ME.