From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id CBA2FA0096 for ; Wed, 10 Apr 2019 18:45:52 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B913B1B11F; Wed, 10 Apr 2019 18:45:52 +0200 (CEST) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 578331B105 for ; Wed, 10 Apr 2019 18:45:51 +0200 (CEST) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BDF0430A220C; Wed, 10 Apr 2019 16:45:50 +0000 (UTC) Received: from rh.redhat.com (ovpn-117-94.ams2.redhat.com [10.36.117.94]) by smtp.corp.redhat.com (Postfix) with ESMTP id 760D95D964; Wed, 10 Apr 2019 16:45:47 +0000 (UTC) From: Kevin Traynor To: David Marchand Cc: Anatoly Burakov , Olivier Matz , dpdk stable Date: Wed, 10 Apr 2019 17:43:58 +0100 Message-Id: <20190410164411.10546-50-ktraynor@redhat.com> In-Reply-To: <20190410164411.10546-1-ktraynor@redhat.com> References: <20190410164411.10546-1-ktraynor@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Wed, 10 Apr 2019 16:45:50 +0000 (UTC) Subject: [dpdk-stable] patch 'eal: restrict control threads to startup CPU affinity' has been queued to LTS release 18.11.2 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to LTS release 18.11.2 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 04/16/19. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Thanks. Kevin Traynor --- >From 23a9f69ed3d9c98b1912df305552ed6dbc639e8f Mon Sep 17 00:00:00 2001 From: David Marchand Date: Tue, 19 Feb 2019 21:41:11 +0100 Subject: [PATCH] eal: restrict control threads to startup CPU affinity [ upstream commit c3568ea376700df061abcbeabc40ddaed7841e1a ] Spawning the ctrl threads on anything that is not part of the eal coremask is not that polite to the rest of the system, especially when you took good care to pin your processes on cpu resources with tools like taskset (linux) / cpuset (freebsd). Rather than introduce yet another eal options to control on which cpu those ctrl threads are created, let's take the startup cpu affinity as a reference and remove the eal coremask from it. If no cpu is left, then we default to the master core. The cpuset is computed once at init before the original cpu affinity is lost. Introduced a RTE_CPU_AND macro to abstract the differences between linux and freebsd respective macros. Examples in a 4 cores FreeBSD vm: $ ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1057 PID TID COMM TDNAME CPU CSID CPU MASK 1057 100131 testpmd - 2 1 2 1057 100140 testpmd eal-intr-thread 1 1 0-1 1057 100141 testpmd rte_mp_handle 1 1 0-1 1057 100142 testpmd lcore-slave-3 3 1 3 $ cpuset -l 1,2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1061 PID TID COMM TDNAME CPU CSID CPU MASK 1061 100131 testpmd - 2 2 2 1061 100144 testpmd eal-intr-thread 1 2 1 1061 100145 testpmd rte_mp_handle 1 2 1 1061 100147 testpmd lcore-slave-3 3 2 3 $ cpuset -l 2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1065 PID TID COMM TDNAME CPU CSID CPU MASK 1065 100131 testpmd - 2 2 2 1065 100148 testpmd eal-intr-thread 2 2 2 1065 100149 testpmd rte_mp_handle 2 2 2 1065 100150 testpmd lcore-slave-3 3 2 3 Fixes: d651ee4919cd ("eal: set affinity for control threads") Signed-off-by: David Marchand Reviewed-by: Anatoly Burakov Reviewed-by: Olivier Matz --- .../prog_guide/env_abstraction_layer.rst | 22 +++++++++++++++ lib/librte_eal/common/eal_common_options.c | 28 +++++++++++++++++++ lib/librte_eal/common/eal_common_thread.c | 21 +++----------- lib/librte_eal/common/eal_internal_cfg.h | 3 ++ lib/librte_eal/common/include/rte_lcore.h | 17 ++++++++--- 5 files changed, 70 insertions(+), 21 deletions(-) diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst index 426acfc28..888ce7801 100644 --- a/doc/guides/prog_guide/env_abstraction_layer.rst +++ b/doc/guides/prog_guide/env_abstraction_layer.rst @@ -442,4 +442,26 @@ Those TLS include *_cpuset* and *_socket_id*: +Control Thread API +~~~~~~~~~~~~~~~~~~ + +It is possible to create Control Threads using the public API +``rte_ctrl_thread_create()``. +Those threads can be used for management/infrastructure tasks and are used +internally by DPDK for multi process support and interrupt handling. + +Those threads will be scheduled on CPUs part of the original process CPU +affinity from which the dataplane and service lcores are excluded. + +For example, on a 8 CPUs system, starting a dpdk application with -l 2,3 +(dataplane cores), then depending on the affinity configuration which can be +controlled with tools like taskset (Linux) or cpuset (FreeBSD), + +- with no affinity configuration, the Control Threads will end up on + 0-1,4-7 CPUs. +- with affinity restricted to 2-4, the Control Threads will end up on + CPU 4. +- with affinity restricted to 2-3, the Control Threads will end up on + CPU 2 (master lcore, which is the default when no CPU is available). + .. _known_issue_label: diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c index fb966780f..7ba1c449b 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -217,4 +217,5 @@ eal_reset_internal_config(struct internal_config *internal_cfg) internal_cfg->iova_mode = RTE_IOVA_DC; internal_cfg->user_mbuf_pool_ops_name = NULL; + CPU_ZERO(&internal_cfg->ctrl_cpuset); internal_cfg->init_complete = 0; } @@ -1359,4 +1360,29 @@ eal_auto_detect_cores(struct rte_config *cfg) } +static void +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) +{ + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset; + rte_cpuset_t default_set; + unsigned int lcore_id; + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + if (eal_cpu_detected(lcore_id) && + rte_lcore_has_role(lcore_id, ROLE_OFF)) { + CPU_SET(lcore_id, cpuset); + } + } + + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), + &default_set)) + CPU_ZERO(&default_set); + + RTE_CPU_AND(cpuset, cpuset, &default_set); + + /* if no detected CPU is off, use master core */ + if (!CPU_COUNT(cpuset)) + CPU_SET(rte_get_master_lcore(), cpuset); +} + int eal_cleanup_config(struct internal_config *internal_cfg) @@ -1392,4 +1418,6 @@ eal_adjust_config(struct internal_config *internal_cfg) } + compute_ctrl_threads_cpuset(internal_cfg); + /* if no memory amounts were requested, this will result in 0 and * will be overridden later, right after eal_hugepage_info_init() */ diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c index a3985ce86..14f206c04 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -17,4 +17,5 @@ #include +#include "eal_internal_cfg.h" #include "eal_private.h" #include "eal_thread.h" @@ -169,8 +170,7 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name, void *(*start_routine)(void *), void *arg) { + rte_cpuset_t *cpuset = &internal_config.ctrl_cpuset; struct rte_thread_ctrl_params *params; - unsigned int lcore_id; - rte_cpuset_t cpuset; - int cpu_found, ret; + int ret; params = malloc(sizeof(*params)); @@ -196,18 +196,5 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name, } - cpu_found = 0; - CPU_ZERO(&cpuset); - for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - if (eal_cpu_detected(lcore_id) && - rte_lcore_has_role(lcore_id, ROLE_OFF)) { - CPU_SET(lcore_id, &cpuset); - cpu_found = 1; - } - } - /* if no detected cpu is off, use master core */ - if (!cpu_found) - CPU_SET(rte_get_master_lcore(), &cpuset); - - ret = pthread_setaffinity_np(*thread, sizeof(cpuset), &cpuset); + ret = pthread_setaffinity_np(*thread, sizeof(*cpuset), cpuset); if (ret) goto fail; diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h index 783ce7de8..189d4f5b2 100644 --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -14,4 +14,6 @@ #include +#include "eal_thread.h" + #define MAX_HUGEPAGE_SIZES 3 /**< support up to 3 page sizes */ @@ -72,4 +74,5 @@ struct internal_config { struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; enum rte_iova_mode iova_mode ; /**< Set IOVA mode on this system */ + rte_cpuset_t ctrl_cpuset; /**< cpuset for ctrl threads */ volatile unsigned int init_complete; /**< indicates whether EAL has completed initialization */ diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h index 6e09d9181..dea17f500 100644 --- a/lib/librte_eal/common/include/rte_lcore.h +++ b/lib/librte_eal/common/include/rte_lcore.h @@ -24,8 +24,16 @@ extern "C" { #if defined(__linux__) - typedef cpu_set_t rte_cpuset_t; +typedef cpu_set_t rte_cpuset_t; +#define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2) #elif defined(__FreeBSD__) #include - typedef cpuset_t rte_cpuset_t; +typedef cpuset_t rte_cpuset_t; +#define RTE_CPU_AND(dst, src1, src2) do \ +{ \ + cpuset_t tmp; \ + CPU_COPY(src1, &tmp); \ + CPU_AND(&tmp, src2); \ + CPU_COPY(&tmp, dst); \ +} while (0) #endif @@ -281,6 +289,7 @@ int rte_thread_setname(pthread_t id, const char *name); * * Wrapper to pthread_create(), pthread_setname_np() and - * pthread_setaffinity_np(). The dataplane and service lcores are - * excluded from the affinity of the new thread. + * pthread_setaffinity_np(). The affinity of the new thread is based + * on the CPU affinity retrieved at the time rte_eal_init() was called, + * the dataplane and service lcores are then excluded. * * @param thread -- 2.20.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2019-04-10 14:06:11.726379211 +0100 +++ 0050-eal-restrict-control-threads-to-startup-CPU-affinity.patch 2019-04-10 14:06:07.997291198 +0100 @@ -1,8 +1,10 @@ -From c3568ea376700df061abcbeabc40ddaed7841e1a Mon Sep 17 00:00:00 2001 +From 23a9f69ed3d9c98b1912df305552ed6dbc639e8f Mon Sep 17 00:00:00 2001 From: David Marchand Date: Tue, 19 Feb 2019 21:41:11 +0100 Subject: [PATCH] eal: restrict control threads to startup CPU affinity +[ upstream commit c3568ea376700df061abcbeabc40ddaed7841e1a ] + Spawning the ctrl threads on anything that is not part of the eal coremask is not that polite to the rest of the system, especially when you took good care to pin your processes on cpu resources with @@ -52,7 +54,6 @@ 1065 100150 testpmd lcore-slave-3 3 2 3 Fixes: d651ee4919cd ("eal: set affinity for control threads") -Cc: stable@dpdk.org Signed-off-by: David Marchand Reviewed-by: Anatoly Burakov @@ -66,10 +67,10 @@ 5 files changed, 70 insertions(+), 21 deletions(-) diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst -index 929d76dba..67b45ba7d 100644 +index 426acfc28..888ce7801 100644 --- a/doc/guides/prog_guide/env_abstraction_layer.rst +++ b/doc/guides/prog_guide/env_abstraction_layer.rst -@@ -499,4 +499,26 @@ Those TLS include *_cpuset* and *_socket_id*: +@@ -442,4 +442,26 @@ Those TLS include *_cpuset* and *_socket_id*: +Control Thread API @@ -97,16 +98,16 @@ .. _known_issue_label: diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c -index 1f45f82f3..9e61ee436 100644 +index fb966780f..7ba1c449b 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c -@@ -218,4 +218,5 @@ eal_reset_internal_config(struct internal_config *internal_cfg) +@@ -217,4 +217,5 @@ eal_reset_internal_config(struct internal_config *internal_cfg) internal_cfg->iova_mode = RTE_IOVA_DC; internal_cfg->user_mbuf_pool_ops_name = NULL; + CPU_ZERO(&internal_cfg->ctrl_cpuset); internal_cfg->init_complete = 0; } -@@ -1360,4 +1361,29 @@ eal_auto_detect_cores(struct rte_config *cfg) +@@ -1359,4 +1360,29 @@ eal_auto_detect_cores(struct rte_config *cfg) } +static void @@ -136,7 +137,7 @@ + int eal_cleanup_config(struct internal_config *internal_cfg) -@@ -1393,4 +1419,6 @@ eal_adjust_config(struct internal_config *internal_cfg) +@@ -1392,4 +1418,6 @@ eal_adjust_config(struct internal_config *internal_cfg) } + compute_ctrl_threads_cpuset(internal_cfg); @@ -185,7 +186,7 @@ if (ret) goto fail; diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h -index 60eaead8f..edff09d07 100644 +index 783ce7de8..189d4f5b2 100644 --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -14,4 +14,6 @@ @@ -195,7 +196,7 @@ + #define MAX_HUGEPAGE_SIZES 3 /**< support up to 3 page sizes */ -@@ -74,4 +76,5 @@ struct internal_config { +@@ -72,4 +74,5 @@ struct internal_config { struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; enum rte_iova_mode iova_mode ; /**< Set IOVA mode on this system */ + rte_cpuset_t ctrl_cpuset; /**< cpuset for ctrl threads */