From: Kevin Traynor <ktraynor@redhat.com> To: David Marchand <david.marchand@redhat.com> Cc: Johan Källström <johan.kallstrom@ericsson.com>, dpdk stable <stable@dpdk.org> Subject: [dpdk-stable] patch 'eal: fix control thread affinity with --lcores' has been queued to LTS release 18.11.3 Date: Thu, 5 Sep 2019 11:17:26 +0100 Message-ID: <20190905101754.21933-26-ktraynor@redhat.com> (raw) In-Reply-To: <20190905101754.21933-1-ktraynor@redhat.com> Hi, FYI, your patch has been queued to LTS release 18.11.3 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 09/12/19. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/kevintraynor/dpdk-stable-queue This queued commit can be viewed at: https://github.com/kevintraynor/dpdk-stable-queue/commit/c0238120c05529f0f26505b6196edd53ee37e050 Thanks. Kevin Traynor --- From c0238120c05529f0f26505b6196edd53ee37e050 Mon Sep 17 00:00:00 2001 From: David Marchand <david.marchand@redhat.com> Date: Tue, 30 Jul 2019 17:05:22 +0200 Subject: [PATCH] eal: fix control thread affinity with --lcores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [ upstream commit b6be16acfeb1cc58c7f37638f007e302527657cd ] The ctrl thread cpu affinity setting has been broken when using --lcores. Using -l/-c options makes each lcore associated to a physical cpu in a 1:1 fashion. On the contrary, when using --lcores, each lcore cpu affinity can be set to a list of any online cpu on the system. To handle both cases, each lcore cpu affinity is considered and removed from the process startup cpu affinity. Introduced macros to manipulate dpdk cpu sets in both Linux and FreeBSD. Examples on a 8 cores Linux system: $ cd /sys/fs/cgroup/cpuset/ $ mkdir dpdk $ cd dpdk $ echo 4-7 > cpuset.cpus $ echo 0 > cpuset.mems $ echo $$ > tasks Before the fix: $ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 8427 cpu_list=4-5,7 testpmd 8428 cpu_list=4-6 eal-intr-thread 8429 cpu_list=4-6 rte_mp_handle 8430 cpu_list=4-5,7 lcore-slave-7 $ taskset -c 7 \ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Failed to create thread for interrupt handling EAL: FATAL: Cannot init interrupt-handling thread EAL: Cannot init interrupt-handling thread PANIC in main(): Cannot init EAL After the fix: $ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 15214 cpu_list=4-5,7 testpmd 15215 cpu_list=6 eal-intr-thread 15216 cpu_list=6 rte_mp_handle 15217 cpu_list=4-5,7 lcore-slave-7 $ taskset -c 7 \ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \ --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048 15297 cpu_list=4-5,7 testpmd 15298 cpu_list=4-5,7 eal-intr-thread 15299 cpu_list=4-5,7 rte_mp_handle 15300 cpu_list=4-5,7 lcore-slave-7 Bugzilla ID: 322 Fixes: c3568ea37670 ("eal: restrict control threads to startup CPU affinity") Reported-by: Johan Källström <johan.kallstrom@ericsson.com> Signed-off-by: David Marchand <david.marchand@redhat.com> --- lib/librte_eal/common/eal_common_options.c | 16 ++++++------ lib/librte_eal/common/include/rte_lcore.h | 29 ++++++++++++++++++++++ 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c index 79efb15d0..f742d4d38 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -1451,9 +1451,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - if (eal_cpu_detected(lcore_id) && - rte_lcore_has_role(lcore_id, ROLE_OFF)) { - CPU_SET(lcore_id, cpuset); - } + if (rte_lcore_has_role(lcore_id, ROLE_OFF)) + continue; + RTE_CPU_OR(cpuset, cpuset, &lcore_config[lcore_id].cpuset); } + RTE_CPU_NOT(cpuset, cpuset); if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), @@ -1463,7 +1463,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) RTE_CPU_AND(cpuset, cpuset, &default_set); - /* if no detected CPU is off, use master core */ - if (!CPU_COUNT(cpuset)) - CPU_SET(rte_get_master_lcore(), cpuset); + /* if no remaining cpu, use master lcore cpu affinity */ + if (!CPU_COUNT(cpuset)) { + memcpy(cpuset, &lcore_config[rte_get_master_lcore()].cpuset, + sizeof(*cpuset)); + } } diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h index dea17f500..31af0e501 100644 --- a/lib/librte_eal/common/include/rte_lcore.h +++ b/lib/librte_eal/common/include/rte_lcore.h @@ -26,4 +26,18 @@ extern "C" { typedef cpu_set_t rte_cpuset_t; #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2) +#define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2) +#define RTE_CPU_FILL(set) do \ +{ \ + unsigned int i; \ + CPU_ZERO(set); \ + for (i = 0; i < CPU_SETSIZE; i++) \ + CPU_SET(i, set); \ +} while (0) +#define RTE_CPU_NOT(dst, src) do \ +{ \ + cpu_set_t tmp; \ + RTE_CPU_FILL(&tmp); \ + CPU_XOR(dst, &tmp, src); \ +} while (0) #elif defined(__FreeBSD__) #include <pthread_np.h> @@ -36,4 +50,19 @@ typedef cpuset_t rte_cpuset_t; CPU_COPY(&tmp, dst); \ } while (0) +#define RTE_CPU_OR(dst, src1, src2) do \ +{ \ + cpuset_t tmp; \ + CPU_COPY(src1, &tmp); \ + CPU_OR(&tmp, src2); \ + CPU_COPY(&tmp, dst); \ +} while (0) +#define RTE_CPU_FILL(set) CPU_FILL(set) +#define RTE_CPU_NOT(dst, src) do \ +{ \ + cpuset_t tmp; \ + CPU_FILL(&tmp); \ + CPU_NAND(&tmp, src); \ + CPU_COPY(&tmp, dst); \ +} while (0) #endif -- 2.20.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2019-09-05 10:36:49.105515044 +0100 +++ 0026-eal-fix-control-thread-affinity-with-lcores.patch 2019-09-05 10:36:47.522700235 +0100 @@ -1 +1 @@ -From b6be16acfeb1cc58c7f37638f007e302527657cd Mon Sep 17 00:00:00 2001 +From c0238120c05529f0f26505b6196edd53ee37e050 Mon Sep 17 00:00:00 2001 @@ -8,0 +9,2 @@ +[ upstream commit b6be16acfeb1cc58c7f37638f007e302527657cd ] + @@ -71 +72,0 @@ -Cc: stable@dpdk.org @@ -81 +82 @@ -index 512d5088e..7b182b82d 100644 +index 79efb15d0..f742d4d38 100644 @@ -84 +85 @@ -@@ -1453,9 +1453,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) +@@ -1451,9 +1451,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) @@ -98 +99 @@ -@@ -1465,7 +1465,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) +@@ -1463,7 +1463,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) @@ -112 +113 @@ -index 411df30d5..c86f72eb1 100644 +index dea17f500..31af0e501 100644
next prev parent reply index Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-09-05 10:17 [dpdk-stable] patch 'ethdev: avoid getting uninitialized info for bad port' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'vfio: use contiguous mapping for IOVA as VA mode' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'compress/zlib: fix error handling' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'cryptodev: fix typo in comment' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'security: remove duplicated symbols from map file' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/i40e: fix RSS hash update for X722 VF' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: fix power of 2 round up when align has smaller type' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: fix align to power of 2 " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: unify power of 2 alignment check macro' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/bnxt: fix extended port counter statistics' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/bnxt: reduce verbosity of a message' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/pcap: fix Rx with small buffers' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/pcap: fix Tx return count in error conditions' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/avf: fix endless loop' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe: fix RETA size for VF' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'igb_uio: fix build on Linux 5.3 for fall through' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'distributor: fix livelock on flush' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: fix eth packet dump for small buffers' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: rename ambiguous VF config variable' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'examples/ip_frag: remove Tx fast free offload flag' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'examples/ptpclient: fix delay request message' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'test: enable installing app with meson' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'test/eal: fix --socket-mem option' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/virtio: fix build' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'examples/l3fwd: fix unaligned memory access on x86' " Kevin Traynor 2019-09-05 10:17 ` Kevin Traynor [this message] 2019-09-05 10:17 ` [dpdk-stable] patch 'doc: update features supported by mlx' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'usertools: replace unsafe input function' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'devtools: fix building kernel component tags' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'test/flow_classify: fix undefined behavior' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'acl: fix undefined behavior of bit shifts' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'test: remove link to ixgbe/i40e with meson' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'eal: hide internal function' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'bus/pci: remove unused x86 Linux constant' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/dpaa: fix build with 0 headroom' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'bus/fslmc: " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/virtio: " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/ena: fix L4 checksum Tx offload' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'doc: add co-existence consideration for bnx2x' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'doc: add co-existence consideration for qede' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/thunderx: fix crash on detach' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'examples/bpf: fix build' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe: fix address of first segment' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/i40e: " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/avf: " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/fm10k: " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx4: fix crash on info query in secondary process' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix validation of VLAN PCP item' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix VLAN inner type matching on DR/DV' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix link speed info when link is down' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: fix latency stats deinit on signal' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe/base: fix product version check' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'eal: increase maximum different hugepage sizes on Arm' " Kevin Traynor 2019-09-05 10:17 ` [dpdk-stable] patch 'usertools: fix input handling in telemetry script' " Kevin Traynor
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190905101754.21933-26-ktraynor@redhat.com \ --to=ktraynor@redhat.com \ --cc=david.marchand@redhat.com \ --cc=johan.kallstrom@ericsson.com \ --cc=stable@dpdk.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
patches for DPDK stable branches Archives are clonable: git clone --mirror http://inbox.dpdk.org/stable/0 stable/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 stable stable/ http://inbox.dpdk.org/stable \ stable@dpdk.org public-inbox-index stable Newsgroup available over NNTP: nntp://inbox.dpdk.org/inbox.dpdk.stable AGPL code for this site: git clone https://public-inbox.org/ public-inbox