From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 20D2F5F17; Wed, 13 Feb 2019 17:13:48 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4B37681F18; Wed, 13 Feb 2019 16:13:47 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-16.brq.redhat.com [10.40.204.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 843D65C219; Wed, 13 Feb 2019 16:13:44 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: olivier.matz@6wind.com, anatoly.burakov@intel.com, ktraynor@redhat.com, stable@dpdk.org Date: Wed, 13 Feb 2019 17:13:32 +0100 Message-Id: <1550074412-31285-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 13 Feb 2019 16:13:47 +0000 (UTC) Subject: [dpdk-dev] [PATCH] eal: restrict ctrl threads to startup cpu affinity X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Feb 2019 16:13:48 -0000 Spawning the ctrl threads on anything that is not part of the eal coremask is not that polite to the rest of the system. Rather than introduce yet another eal options for this, let's take the startup cpu affinity as a reference and remove the eal coremask from it. If no cpu is left, then we default to the master core. The cpuset is computed once at init before the original cpu affinity. Fixes: d651ee4919cd ("eal: set affinity for control threads") Signed-off-by: David Marchand --- lib/librte_eal/common/eal_common_options.c | 28 ++++++++++++++++++++++++++++ lib/librte_eal/common/eal_common_thread.c | 21 ++++----------------- lib/librte_eal/common/eal_internal_cfg.h | 3 +++ 3 files changed, 35 insertions(+), 17 deletions(-) diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c index 6c96f45..b766252 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -217,6 +217,7 @@ struct device_option { internal_cfg->create_uio_dev = 0; internal_cfg->iova_mode = RTE_IOVA_DC; internal_cfg->user_mbuf_pool_ops_name = NULL; + CPU_ZERO(&internal_cfg->ctrl_cpuset); internal_cfg->init_complete = 0; } @@ -1360,6 +1361,31 @@ static int xdigit2val(unsigned char c) cfg->lcore_count -= removed; } +static void +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) +{ + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset; + rte_cpuset_t default_set; + unsigned int lcore_id; + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + if (eal_cpu_detected(lcore_id) && + rte_lcore_has_role(lcore_id, ROLE_OFF)) { + CPU_SET(lcore_id, cpuset); + } + } + + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), + &default_set) < 0) + CPU_ZERO(&default_set); + + CPU_AND(cpuset, cpuset, &default_set); + + /* if no detected cpu is off, use master core */ + if (!CPU_COUNT(cpuset)) + CPU_SET(rte_get_master_lcore(), cpuset); +} + int eal_cleanup_config(struct internal_config *internal_cfg) { @@ -1393,6 +1419,8 @@ static int xdigit2val(unsigned char c) lcore_config[cfg->master_lcore].core_role = ROLE_RTE; } + compute_ctrl_threads_cpuset(internal_cfg); + /* if no memory amounts were requested, this will result in 0 and * will be overridden later, right after eal_hugepage_info_init() */ for (i = 0; i < RTE_MAX_NUMA_NODES; i++) diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c index 48ef4d6..893b0c1 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -16,6 +16,7 @@ #include #include +#include "eal_internal_cfg.h" #include "eal_private.h" #include "eal_thread.h" @@ -168,10 +169,9 @@ static void *rte_thread_init(void *arg) const pthread_attr_t *attr, void *(*start_routine)(void *), void *arg) { + rte_cpuset_t *cpuset = &internal_config.ctrl_cpuset; struct rte_thread_ctrl_params *params; - unsigned int lcore_id; - rte_cpuset_t cpuset; - int cpu_found, ret; + int ret; params = malloc(sizeof(*params)); if (!params) @@ -195,20 +195,7 @@ static void *rte_thread_init(void *arg) "Cannot set name for ctrl thread\n"); } - cpu_found = 0; - CPU_ZERO(&cpuset); - for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - if (eal_cpu_detected(lcore_id) && - rte_lcore_has_role(lcore_id, ROLE_OFF)) { - CPU_SET(lcore_id, &cpuset); - cpu_found = 1; - } - } - /* if no detected cpu is off, use master core */ - if (!cpu_found) - CPU_SET(rte_get_master_lcore(), &cpuset); - - ret = pthread_setaffinity_np(*thread, sizeof(cpuset), &cpuset); + ret = pthread_setaffinity_np(*thread, sizeof(*cpuset), cpuset); if (ret < 0) goto fail; diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h index 60eaead..edff09d 100644 --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -13,6 +13,8 @@ #include #include +#include "eal_thread.h" + #define MAX_HUGEPAGE_SIZES 3 /**< support up to 3 page sizes */ /* @@ -73,6 +75,7 @@ struct internal_config { unsigned num_hugepage_sizes; /**< how many sizes on this system */ struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; enum rte_iova_mode iova_mode ; /**< Set IOVA mode on this system */ + rte_cpuset_t ctrl_cpuset; /**< cpuset for ctrl threads */ volatile unsigned int init_complete; /**< indicates whether EAL has completed initialization */ }; -- 1.8.3.1