patches for DPDK stable branches
 help / color / Atom feed
From: Kevin Traynor <ktraynor@redhat.com>
To: David Marchand <david.marchand@redhat.com>
Cc: Johan Källström <johan.kallstrom@ericsson.com>,
	dpdk stable <stable@dpdk.org>
Subject: [dpdk-stable] patch 'eal: fix control thread affinity with --lcores' has been queued to LTS release 18.11.3
Date: Thu,  5 Sep 2019 11:17:26 +0100
Message-ID: <20190905101754.21933-26-ktraynor@redhat.com> (raw)
In-Reply-To: <20190905101754.21933-1-ktraynor@redhat.com>

Hi,

FYI, your patch has been queued to LTS release 18.11.3

Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet.
It will be pushed if I get no objections before 09/12/19. So please
shout if anyone has objections.

Also note that after the patch there's a diff of the upstream commit vs the
patch applied to the branch. This will indicate if there was any rebasing
needed to apply to the stable branch. If there were code changes for rebasing
(ie: not only metadata diffs), please double check that the rebase was
correctly done.

Queued patches are on a temporary branch at:
https://github.com/kevintraynor/dpdk-stable-queue

This queued commit can be viewed at:
https://github.com/kevintraynor/dpdk-stable-queue/commit/c0238120c05529f0f26505b6196edd53ee37e050

Thanks.

Kevin Traynor

---
From c0238120c05529f0f26505b6196edd53ee37e050 Mon Sep 17 00:00:00 2001
From: David Marchand <david.marchand@redhat.com>
Date: Tue, 30 Jul 2019 17:05:22 +0200
Subject: [PATCH] eal: fix control thread affinity with --lcores
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

[ upstream commit b6be16acfeb1cc58c7f37638f007e302527657cd ]

The ctrl thread cpu affinity setting has been broken when using --lcores.

Using -l/-c options makes each lcore associated to a physical cpu in a 1:1
fashion.
On the contrary, when using --lcores, each lcore cpu affinity can be set
to a list of any online cpu on the system.

To handle both cases, each lcore cpu affinity is considered and removed
from the process startup cpu affinity.

Introduced macros to manipulate dpdk cpu sets in both Linux and FreeBSD.

Examples on a 8 cores Linux system:

$ cd /sys/fs/cgroup/cpuset/
$ mkdir dpdk
$ cd dpdk
$ echo 4-7 > cpuset.cpus
$ echo 0 > cpuset.mems
$ echo $$ > tasks

Before the fix:
$ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \
 --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048

8427	cpu_list=4-5,7	testpmd
8428	cpu_list=4-6	eal-intr-thread
8429	cpu_list=4-6	rte_mp_handle
8430	cpu_list=4-5,7	lcore-slave-7

$ taskset -c 7 \
 ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \
 --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048

EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Failed to create thread for interrupt handling
EAL: FATAL: Cannot init interrupt-handling thread
EAL: Cannot init interrupt-handling thread
PANIC in main():
Cannot init EAL

After the fix:
$ ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \
 --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048

15214	cpu_list=4-5,7	testpmd
15215	cpu_list=6	eal-intr-thread
15216	cpu_list=6	rte_mp_handle
15217	cpu_list=4-5,7	lcore-slave-7

$ taskset -c 7 \
 ./master/app/testpmd --master-lcore 0 --lcores '(0,7)@(7,4,5)' \
 --no-huge --no-pci -m 512 -- -i --total-num-mbufs=2048

15297	cpu_list=4-5,7	testpmd
15298	cpu_list=4-5,7	eal-intr-thread
15299	cpu_list=4-5,7	rte_mp_handle
15300	cpu_list=4-5,7	lcore-slave-7

Bugzilla ID: 322
Fixes: c3568ea37670 ("eal: restrict control threads to startup CPU affinity")

Reported-by: Johan Källström <johan.kallstrom@ericsson.com>
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/librte_eal/common/eal_common_options.c | 16 ++++++------
 lib/librte_eal/common/include/rte_lcore.h  | 29 ++++++++++++++++++++++
 2 files changed, 38 insertions(+), 7 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 79efb15d0..f742d4d38 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -1451,9 +1451,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
 
 	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
-		if (eal_cpu_detected(lcore_id) &&
-				rte_lcore_has_role(lcore_id, ROLE_OFF)) {
-			CPU_SET(lcore_id, cpuset);
-		}
+		if (rte_lcore_has_role(lcore_id, ROLE_OFF))
+			continue;
+		RTE_CPU_OR(cpuset, cpuset, &lcore_config[lcore_id].cpuset);
 	}
+	RTE_CPU_NOT(cpuset, cpuset);
 
 	if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t),
@@ -1463,7 +1463,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
 	RTE_CPU_AND(cpuset, cpuset, &default_set);
 
-	/* if no detected CPU is off, use master core */
-	if (!CPU_COUNT(cpuset))
-		CPU_SET(rte_get_master_lcore(), cpuset);
+	/* if no remaining cpu, use master lcore cpu affinity */
+	if (!CPU_COUNT(cpuset)) {
+		memcpy(cpuset, &lcore_config[rte_get_master_lcore()].cpuset,
+			sizeof(*cpuset));
+	}
 }
 
diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h
index dea17f500..31af0e501 100644
--- a/lib/librte_eal/common/include/rte_lcore.h
+++ b/lib/librte_eal/common/include/rte_lcore.h
@@ -26,4 +26,18 @@ extern "C" {
 typedef	cpu_set_t rte_cpuset_t;
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
+#define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
+#define RTE_CPU_FILL(set) do \
+{ \
+	unsigned int i; \
+	CPU_ZERO(set); \
+	for (i = 0; i < CPU_SETSIZE; i++) \
+		CPU_SET(i, set); \
+} while (0)
+#define RTE_CPU_NOT(dst, src) do \
+{ \
+	cpu_set_t tmp; \
+	RTE_CPU_FILL(&tmp); \
+	CPU_XOR(dst, &tmp, src); \
+} while (0)
 #elif defined(__FreeBSD__)
 #include <pthread_np.h>
@@ -36,4 +50,19 @@ typedef cpuset_t rte_cpuset_t;
 	CPU_COPY(&tmp, dst); \
 } while (0)
+#define RTE_CPU_OR(dst, src1, src2) do \
+{ \
+	cpuset_t tmp; \
+	CPU_COPY(src1, &tmp); \
+	CPU_OR(&tmp, src2); \
+	CPU_COPY(&tmp, dst); \
+} while (0)
+#define RTE_CPU_FILL(set) CPU_FILL(set)
+#define RTE_CPU_NOT(dst, src) do \
+{ \
+	cpuset_t tmp; \
+	CPU_FILL(&tmp); \
+	CPU_NAND(&tmp, src); \
+	CPU_COPY(&tmp, dst); \
+} while (0)
 #endif
 
-- 
2.20.1

---
  Diff of the applied patch vs upstream commit (please double-check if non-empty:
---
--- -	2019-09-05 10:36:49.105515044 +0100
+++ 0026-eal-fix-control-thread-affinity-with-lcores.patch	2019-09-05 10:36:47.522700235 +0100
@@ -1 +1 @@
-From b6be16acfeb1cc58c7f37638f007e302527657cd Mon Sep 17 00:00:00 2001
+From c0238120c05529f0f26505b6196edd53ee37e050 Mon Sep 17 00:00:00 2001
@@ -8,0 +9,2 @@
+[ upstream commit b6be16acfeb1cc58c7f37638f007e302527657cd ]
+
@@ -71 +72,0 @@
-Cc: stable@dpdk.org
@@ -81 +82 @@
-index 512d5088e..7b182b82d 100644
+index 79efb15d0..f742d4d38 100644
@@ -84 +85 @@
-@@ -1453,9 +1453,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
+@@ -1451,9 +1451,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
@@ -98 +99 @@
-@@ -1465,7 +1465,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
+@@ -1463,7 +1463,9 @@ compute_ctrl_threads_cpuset(struct internal_config *internal_cfg)
@@ -112 +113 @@
-index 411df30d5..c86f72eb1 100644
+index dea17f500..31af0e501 100644

  parent reply index

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-05 10:17 [dpdk-stable] patch 'ethdev: avoid getting uninitialized info for bad port' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'vfio: use contiguous mapping for IOVA as VA mode' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'compress/zlib: fix error handling' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'cryptodev: fix typo in comment' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'security: remove duplicated symbols from map file' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/i40e: fix RSS hash update for X722 VF' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: fix power of 2 round up when align has smaller type' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: fix align to power of 2 " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/sfc: unify power of 2 alignment check macro' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/bnxt: fix extended port counter statistics' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/bnxt: reduce verbosity of a message' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/pcap: fix Rx with small buffers' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/pcap: fix Tx return count in error conditions' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/avf: fix endless loop' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe: fix RETA size for VF' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'igb_uio: fix build on Linux 5.3 for fall through' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'distributor: fix livelock on flush' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: fix eth packet dump for small buffers' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: rename ambiguous VF config variable' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'examples/ip_frag: remove Tx fast free offload flag' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'examples/ptpclient: fix delay request message' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'test: enable installing app with meson' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'test/eal: fix --socket-mem option' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/virtio: fix build' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'examples/l3fwd: fix unaligned memory access on x86' " Kevin Traynor
2019-09-05 10:17 ` Kevin Traynor [this message]
2019-09-05 10:17 ` [dpdk-stable] patch 'doc: update features supported by mlx' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'usertools: replace unsafe input function' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'devtools: fix building kernel component tags' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'test/flow_classify: fix undefined behavior' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'acl: fix undefined behavior of bit shifts' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'test: remove link to ixgbe/i40e with meson' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'eal: hide internal function' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'bus/pci: remove unused x86 Linux constant' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/dpaa: fix build with 0 headroom' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'bus/fslmc: " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/virtio: " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/ena: fix L4 checksum Tx offload' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'doc: add co-existence consideration for bnx2x' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'doc: add co-existence consideration for qede' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/thunderx: fix crash on detach' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'examples/bpf: fix build' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe: fix address of first segment' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/i40e: " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/avf: " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/fm10k: " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx4: fix crash on info query in secondary process' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix validation of VLAN PCP item' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix VLAN inner type matching on DR/DV' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/mlx5: fix link speed info when link is down' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'app/testpmd: fix latency stats deinit on signal' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'net/ixgbe/base: fix product version check' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'eal: increase maximum different hugepage sizes on Arm' " Kevin Traynor
2019-09-05 10:17 ` [dpdk-stable] patch 'usertools: fix input handling in telemetry script' " Kevin Traynor

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190905101754.21933-26-ktraynor@redhat.com \
    --to=ktraynor@redhat.com \
    --cc=david.marchand@redhat.com \
    --cc=johan.kallstrom@ericsson.com \
    --cc=stable@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

patches for DPDK stable branches

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ http://inbox.dpdk.org/stable \
		stable@dpdk.org
	public-inbox-index stable


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.stable


AGPL code for this site: git clone https://public-inbox.org/ public-inbox