From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 23154A0613 for ; Tue, 30 Jul 2019 17:05:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8706A1C198; Tue, 30 Jul 2019 17:05:38 +0200 (CEST) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 81EAF1BE77; Tue, 30 Jul 2019 17:05:35 +0200 (CEST) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A2467C049E17; Tue, 30 Jul 2019 15:05:34 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-235.brq.redhat.com [10.40.204.235]) by smtp.corp.redhat.com (Postfix) with ESMTP id CCBB9600CC; Tue, 30 Jul 2019 15:05:32 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: johan.kallstrom@ericsson.com, anatoly.burakov@intel.com, olivier.matz@6wind.com, jerinj@marvell.com, stable@dpdk.org Date: Tue, 30 Jul 2019 17:05:22 +0200 Message-Id: <1564499122-5014-1-git-send-email-david.marchand@redhat.com> In-Reply-To: <1564479354-11192-1-git-send-email-david.marchand@redhat.com> References: <1564479354-11192-1-git-send-email-david.marchand@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Tue, 30 Jul 2019 15:05:34 +0000 (UTC) Subject: [dpdk-dev] [PATCH v2] eal: fix ctrl thread affinity with --lcores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The ctrl thread cpu affinity setting has been broken when using --lcores. Using -l/-c options makes each lcore associated to a physical cpu in a 1:1 fashion. On the contrary, when using --lcores, each lcore cpu affinity can be set to a list of any online cpu on the system. To handle both cases, each lcore cpu affinity is considered and removed from the process startup cpu affinity. Introduced macros to manipulate dpdk cpu sets in both Linux and FreeBSD. Examples on a 8 cores Linux system: $ cd /sys/fs/cgroup/cpuset/ $ mkdir dpdk $ cd dpdk $ echo 4-7 > cpuset.cpus $ echo 0 > cpuset.mems $ echo $$ > tasks Before the fix: $ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 8427 cpu_list=4-5,7 testpmd 8428 cpu_list=4-6 eal-intr-thread 8429 cpu_list=4-6 rte_mp_handle 8430 cpu_list=4-5,7 lcore-slave-7 $ taskset -c 7 \ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Failed to create thread for interrupt handling EAL: FATAL: Cannot init interrupt-handling thread EAL: Cannot init interrupt-handling thread PANIC in main(): Cannot init EAL After the fix: $ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 15214 cpu_list=4-5,7 testpmd 15215 cpu_list=6 eal-intr-thread 15216 cpu_list=6 rte_mp_handle 15217 cpu_list=4-5,7 lcore-slave-7 $ taskset -c 7 \ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 15297 cpu_list=4-5,7 testpmd 15298 cpu_list=4-5,7 eal-intr-thread 15299 cpu_list=4-5,7 rte_mp_handle 15300 cpu_list=4-5,7 lcore-slave-7 Bugzilla ID: 322 Fixes: c3568ea37670 ("eal: restrict control threads to startup CPU affinity") Cc: stable@dpdk.org Reported-by: Johan Källström Signed-off-by: David Marchand --- Changelog since v1: - fixed build issue with icc - rewrote commit log - added examples of what was broken --- lib/librte_eal/common/eal_common_options.c | 16 +++++++++------- lib/librte_eal/common/include/rte_lcore.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c index 512d508..7b182b8 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -1452,11 +1452,11 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) unsigned int lcore_id; for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - if (eal_cpu_detected(lcore_id) && - rte_lcore_has_role(lcore_id, ROLE_OFF)) { - CPU_SET(lcore_id, cpuset); - } + if (rte_lcore_has_role(lcore_id, ROLE_OFF)) + continue; + RTE_CPU_OR(cpuset, cpuset, &lcore_config[lcore_id].cpuset); } + RTE_CPU_NOT(cpuset, cpuset); if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), &default_set)) @@ -1464,9 +1464,11 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) RTE_CPU_AND(cpuset, cpuset, &default_set); - /* if no detected CPU is off, use master core */ - if (!CPU_COUNT(cpuset)) - CPU_SET(rte_get_master_lcore(), cpuset); + /* if no remaining cpu, use master lcore cpu affinity */ + if (!CPU_COUNT(cpuset)) { + memcpy(cpuset, &lcore_config[rte_get_master_lcore()].cpuset, + sizeof(*cpuset)); + } } int diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h index 411df30..c86f72e 100644 --- a/lib/librte_eal/common/include/rte_lcore.h +++ b/lib/librte_eal/common/include/rte_lcore.h @@ -25,6 +25,20 @@ extern "C" { #if defined(__linux__) typedef cpu_set_t rte_cpuset_t; #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2) +#define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2) +#define RTE_CPU_FILL(set) do \ +{ \ + unsigned int i; \ + CPU_ZERO(set); \ + for (i = 0; i < CPU_SETSIZE; i++) \ + CPU_SET(i, set); \ +} while (0) +#define RTE_CPU_NOT(dst, src) do \ +{ \ + cpu_set_t tmp; \ + RTE_CPU_FILL(&tmp); \ + CPU_XOR(dst, &tmp, src); \ +} while (0) #elif defined(__FreeBSD__) #include typedef cpuset_t rte_cpuset_t; @@ -35,6 +49,21 @@ typedef cpuset_t rte_cpuset_t; CPU_AND(&tmp, src2); \ CPU_COPY(&tmp, dst); \ } while (0) +#define RTE_CPU_OR(dst, src1, src2) do \ +{ \ + cpuset_t tmp; \ + CPU_COPY(src1, &tmp); \ + CPU_OR(&tmp, src2); \ + CPU_COPY(&tmp, dst); \ +} while (0) +#define RTE_CPU_FILL(set) CPU_FILL(set) +#define RTE_CPU_NOT(dst, src) do \ +{ \ + cpuset_t tmp; \ + CPU_FILL(&tmp); \ + CPU_NAND(&tmp, src); \ + CPU_COPY(&tmp, dst); \ +} while (0) #endif /** -- 1.8.3.1