From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 203C45699; Tue, 19 Feb 2019 12:38:34 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Feb 2019 03:38:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,388,1544515200"; d="scan'208";a="300851583" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.110]) ([10.237.220.110]) by orsmga005.jf.intel.com with ESMTP; 19 Feb 2019 03:38:32 -0800 To: David Marchand , dev@dpdk.org Cc: olivier.matz@6wind.com, ktraynor@redhat.com, stable@dpdk.org References: <1550074412-31285-1-git-send-email-david.marchand@redhat.com> <1550151042-9764-1-git-send-email-david.marchand@redhat.com> <1550151042-9764-2-git-send-email-david.marchand@redhat.com> From: "Burakov, Anatoly" Message-ID: <6d3960bc-9798-fa79-2538-339d63caf81b@intel.com> Date: Tue, 19 Feb 2019 11:38:29 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.0 MIME-Version: 1.0 In-Reply-To: <1550151042-9764-2-git-send-email-david.marchand@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2 2/2] eal: restrict ctrl threads to startup cpu affinity X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Feb 2019 11:38:35 -0000 On 14-Feb-19 1:30 PM, David Marchand wrote: > Spawning the ctrl threads on anything that is not part of the eal > coremask is not that polite to the rest of the system, especially > when you took good care to pin your processes on cpu resources with > tools like taskset (linux) / cpuset (freebsd). > > Rather than introduce yet another eal options to control on which cpu > those ctrl threads are created, let's take the startup cpu affinity > as a reference and remove the eal coremask from it. > If no cpu is left, then we default to the master core. > > The cpuset is computed once at init before the original cpu affinity > is lost. > > Introduced a RTE_CPU_AND macro to abstract the differences between linux > and freebsd respective macros. > > Examples in a 4 cores FreeBSD vm: > > $ ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ > -- -i --total-num-mbufs=2048 > > $ procstat -S 1057 > PID TID COMM TDNAME CPU CSID CPU MASK > 1057 100131 testpmd - 2 1 2 > 1057 100140 testpmd eal-intr-thread 1 1 0-1 > 1057 100141 testpmd rte_mp_handle 1 1 0-1 > 1057 100142 testpmd lcore-slave-3 3 1 3 > > $ cpuset -l 1,2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ > -- -i --total-num-mbufs=2048 > > $ procstat -S 1061 > PID TID COMM TDNAME CPU CSID CPU MASK > 1061 100131 testpmd - 2 2 2 > 1061 100144 testpmd eal-intr-thread 1 2 1 > 1061 100145 testpmd rte_mp_handle 1 2 1 > 1061 100147 testpmd lcore-slave-3 3 2 3 > > $ cpuset -l 2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ > -- -i --total-num-mbufs=2048 > > $ procstat -S 1065 > PID TID COMM TDNAME CPU CSID CPU MASK > 1065 100131 testpmd - 2 2 2 > 1065 100148 testpmd eal-intr-thread 2 2 2 > 1065 100149 testpmd rte_mp_handle 2 2 2 > 1065 100150 testpmd lcore-slave-3 3 2 3 > > Fixes: d651ee4919cd ("eal: set affinity for control threads") > Signed-off-by: David Marchand > --- > > Changes since v1: > - added some description in the prog guide > - fixed FreeBSD build > > --- > doc/guides/prog_guide/env_abstraction_layer.rst | 14 +++++++++++++ > lib/librte_eal/common/eal_common_options.c | 28 +++++++++++++++++++++++++ > lib/librte_eal/common/eal_common_thread.c | 21 ++++--------------- > lib/librte_eal/common/eal_internal_cfg.h | 3 +++ > lib/librte_eal/common/include/rte_lcore.h | 17 +++++++++++---- > 5 files changed, 62 insertions(+), 21 deletions(-) > > diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst > index 929d76d..bfc66d9 100644 > --- a/doc/guides/prog_guide/env_abstraction_layer.rst > +++ b/doc/guides/prog_guide/env_abstraction_layer.rst > @@ -498,6 +498,20 @@ Those TLS include *_cpuset* and *_socket_id*: > * *_socket_id* stores the NUMA node of the CPU set. If the CPUs in CPU set belong to different NUMA node, the *_socket_id* will be set to SOCKET_ID_ANY. > > > +Control Thread API > +~~~~~~~~~~~~~~~~~~ > + > +It is possible to create Control Threads using the public API ``rte_ctrl_thread_create()``. > +Those threads can be used for management/infrastructure tasks and are used internally by DPDK for multi process support and interrupt handling. > + > +Those threads will be scheduled on cpus part of the original process cpu affinity from which the dataplane and service lcores are excluded. > + > +For example, on a 8 cpus system, starting a dpdk application with -l 2,3 (dataplane cores), then depending on the affinity configuration which can be controlled with tools like taskset (Linux) or cpuset (FreeBSD), > + > +- with no affinity configuration, the Control Threads will end up on 0-1,4-7 cpus. > +- with affinity restricted to 2-4, the Control Threads will end up on cpu 4. > +- with affinity restricted to 2-3, the Control Threads will end up on cpu 2 (master lcore, which is the default when no cpu is available). You're not winning anything by foregoing the 80 char limit on documentation (doxygen will still generate this correctly), but you're losing in readability when working in terminal. I would prefer if you didn't do those long lines :) Thomas, do we want checkpatch to warn about this? > + > .. _known_issue_label: > > Known Issues > diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c > index 1f45f82..fca3f83 100644 > --- a/lib/librte_eal/common/eal_common_options.c > +++ b/lib/librte_eal/common/eal_common_options.c > @@ -217,6 +217,7 @@ struct device_option { > internal_cfg->create_uio_dev = 0; > internal_cfg->iova_mode = RTE_IOVA_DC; > internal_cfg->user_mbuf_pool_ops_name = NULL; > + CPU_ZERO(&internal_cfg->ctrl_cpuset); > internal_cfg->init_complete = 0; > } > > @@ -1359,6 +1360,31 @@ static int xdigit2val(unsigned char c) > cfg->lcore_count -= removed; > } > > +static void > +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) > +{ > + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset; > + rte_cpuset_t default_set; > + unsigned int lcore_id; > + > + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { > + if (eal_cpu_detected(lcore_id) && > + rte_lcore_has_role(lcore_id, ROLE_OFF)) { > + CPU_SET(lcore_id, cpuset); > + } > + } > + > + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), > + &default_set) < 0) Shouldn't this be != 0? Manpage doesn't say the error values will be negative. -- Thanks, Anatoly