[PATCH 0/3] add eal functions for thread affinity

[PATCH 0/3] add eal functions for thread affinity

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes basic error handling, initial get/set thread
affinity functions and minimal unit test.

Tyler Retzlaff (3):
  eal/windows: translate Windows errors to errno-style errors
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  86 +++++++++++++++++++
 lib/eal/include/rte_thread.h     |  45 ++++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   4 +
 lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 179 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 472 insertions(+), 45 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH 1/3] eal/windows: translate Windows errors to errno-style errors

[Tyler Retzlaff]

Add function to translate Windows error codes to errno-style error
codes. The possible return values are chosen so that we have as
much semantical compatibility between platforms as possible.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/windows/rte_thread.c | 48 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..b3b5362 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -11,6 +11,54 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH 2/3] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h     |  45 ++++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   4 +
 lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 131 ++++++++++++++++++++++++++++-
 7 files changed, 336 insertions(+), 45 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..4eb113f 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -2,6 +2,8 @@
  * Copyright(c) 2021 Mellanox Technologies, Ltd
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,6 +23,13 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct rte_thread_tag {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
@@ -28,6 +37,42 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..3bb721d 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -89,3 +89,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..c8d5177 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,10 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_get_affinity_by_id;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..4ef5920 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -2,7 +2,6 @@
  * Copyright(c) 2019 Intel Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +26,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +48,116 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			log_early("Cannot get group information size, "
+				"error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		log_early("Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		log_early("Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/* NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++) {
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+	}
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -83,52 +188,24 @@ struct cpu_map {
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -164,3 +241,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index 245aa60..c96eca5 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -56,6 +56,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index b3b5362..b4fbcc8 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -5,7 +5,8 @@
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
@@ -135,3 +136,131 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			ret = EINVAL;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = rte_convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1..5a9d69b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -238,6 +239,7 @@ fast_tests = [
         ['reorder_autotest', true],
         ['service_autotest', true],
         ['thash_autotest', true],
+        ['threads_autotest', true],
         ['trace_autotest', true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..61b97af
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,83 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021-2022 Microsoft.
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static void *
+thread_main(void *arg)
+{
+	(void)arg;
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, NULL) == 0,
+		"Failed to create thread");
+	thread_id.opaque_id = id;
+
+	rte_cpuset_t cpuset0;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+
+	rte_cpuset_t cpuset1;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(0 == memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)),
+		"Affinity should be stable");
+
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(0 == memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)),
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(0 == memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)),
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH 0/3] add eal functions for thread affinity

[David Marchand]

Hello Tyler,

On Fri, Apr 1, 2022 at 3:30 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> this series provides basic dependencies for additional eal thread api
> additions. series includes basic error handling, initial get/set thread
> affinity functions and minimal unit test.
>
> Tyler Retzlaff (3):
>   eal/windows: translate Windows errors to errno-style errors
>   eal: implement functions for get/set thread affinity
>   test/threads: add unit test for thread API
>
>  app/test/meson.build             |   2 +
>  app/test/test_threads.c          |  86 +++++++++++++++++++
>  lib/eal/include/rte_thread.h     |  45 ++++++++++
>  lib/eal/unix/rte_thread.c        |  16 ++++
>  lib/eal/version.map              |   4 +
>  lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++----------
>  lib/eal/windows/eal_windows.h    |  10 +++
>  lib/eal/windows/include/rte_os.h |   2 +
>  lib/eal/windows/rte_thread.c     | 179 ++++++++++++++++++++++++++++++++++++++-
>  9 files changed, 472 insertions(+), 45 deletions(-)
>  create mode 100644 app/test/test_threads.c

We have two concurrent series, can you clarify what are the intentions
on this work?
Is this series superseding Narcisa series?

Thanks!

-- 
David Marchand

Re: [PATCH 0/3] add eal functions for thread affinity

[Tyler Retzlaff]

On Fri, Apr 08, 2022 at 10:57:55AM +0200, David Marchand wrote:
> Hello Tyler,
> 
> On Fri, Apr 1, 2022 at 3:30 PM Tyler Retzlaff
> <roretzla@linux.microsoft.com> wrote:
> >
> > this series provides basic dependencies for additional eal thread api
> > additions. series includes basic error handling, initial get/set thread
> > affinity functions and minimal unit test.
> >
> > Tyler Retzlaff (3):
> >   eal/windows: translate Windows errors to errno-style errors
> >   eal: implement functions for get/set thread affinity
> >   test/threads: add unit test for thread API
> >
> >  app/test/meson.build             |   2 +
> >  app/test/test_threads.c          |  86 +++++++++++++++++++
> >  lib/eal/include/rte_thread.h     |  45 ++++++++++
> >  lib/eal/unix/rte_thread.c        |  16 ++++
> >  lib/eal/version.map              |   4 +
> >  lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++----------
> >  lib/eal/windows/eal_windows.h    |  10 +++
> >  lib/eal/windows/include/rte_os.h |   2 +
> >  lib/eal/windows/rte_thread.c     | 179 ++++++++++++++++++++++++++++++++++++++-
> >  9 files changed, 472 insertions(+), 45 deletions(-)
> >  create mode 100644 app/test/test_threads.c
> 
> We have two concurrent series, can you clarify what are the intentions
> on this work?

yes, i should have clarified this up front sorry.

> Is this series superseding Narcisa series?

this series supersedes the series from Narcisa. it was resolved through
discussion that the current series should be abandoned as it is too
large and not making progress.

we've elected to submit a series of smaller patchsets that incorporate
the feedback received to date and build up the api surface for
threading. the patches are still the work of Narcisa but she is
overscheduled so i will assist in upstreaming and addressing feedback.

additionally, rather than port the tree to the new __experimental api as
they are added we will prefer to add unit tests that provide validation
of the api and example usage.

our hope is the smaller scoped series will attract more attention and
have better acknowledgement velocity.

i will have Narcisa mark the monolithic series as superseded on
patchwork.

ty

Re: [PATCH 3/3] test/threads: add unit test for thread API

[Dmitry Kozlyuk]

2022-04-01 06:29 (UTC-0700), Tyler Retzlaff:
[...]
> +static int
> +test_thread_affinity(void)
> +{
> +	pthread_t id;
> +	rte_thread_t thread_id;
> +
> +	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, NULL) == 0,
> +		"Failed to create thread");
> +	thread_id.opaque_id = id;

The need for this hack means that the new API is unusable in practice.
I think functions to get the current thread ID and to compare IDs
must be the part of this series to accept this unit test patch.

Re: [PATCH 2/3] eal: implement functions for get/set thread affinity

[Dmitry Kozlyuk]

2022-04-01 06:29 (UTC-0700), Tyler Retzlaff:
> Implement functions for getting/setting thread affinity.
> Threads can be pinned to specific cores by setting their
> affinity attribute.
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Please see some small comments below.

> ---
>  lib/eal/include/rte_thread.h     |  45 ++++++++++
>  lib/eal/unix/rte_thread.c        |  16 ++++
>  lib/eal/version.map              |   4 +
>  lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++++----------
>  lib/eal/windows/eal_windows.h    |  10 +++
>  lib/eal/windows/include/rte_os.h |   2 +
>  lib/eal/windows/rte_thread.c     | 131 ++++++++++++++++++++++++++++-
>  7 files changed, 336 insertions(+), 45 deletions(-)
> 
> diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> index 8be8ed8..4eb113f 100644
> --- a/lib/eal/include/rte_thread.h
> +++ b/lib/eal/include/rte_thread.h
> @@ -2,6 +2,8 @@
>   * Copyright(c) 2021 Mellanox Technologies, Ltd
>   */
>  
> +#include <stdint.h>
> +
>  #include <rte_os.h>
>  #include <rte_compat.h>
>  
> @@ -21,6 +23,13 @@
>  #endif
>  
>  /**
> + * Thread id descriptor.
> + */
> +typedef struct rte_thread_tag {
> +	uintptr_t opaque_id; /**< thread identifier */
> +} rte_thread_t;
> +
> +/**
>   * TLS key type, an opaque pointer.
>   */
>  typedef struct eal_tls_key *rte_thread_key;
> @@ -28,6 +37,42 @@
>  #ifdef RTE_HAS_CPUSET
>  
>  /**

Missing a common part for experimental functions:

 * @warning
 * @b EXPERIMENTAL: this API may change without prior notice.

> + * Set the affinity of thread 'thread_id' to the cpu set
> + * specified by 'cpuset'.
> + *
> + * @param thread_id
> + *    Id of the thread for which to set the affinity.
> + *
> + * @param cpuset
> + *   Pointer to CPU affinity to set.
> + *
> + * @return
> + *   On success, return 0.
> + *   On failure, return a positive errno-style error number.
> + */
> +__rte_experimental
> +int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
> +		const rte_cpuset_t *cpuset);
> +
> +/**

Same here.

> + * Get the affinity of thread 'thread_id' and store it
> + * in 'cpuset'.
> + *
> + * @param thread_id
> + *    Id of the thread for which to get the affinity.
> + *
> + * @param cpuset
> + *   Pointer for storing the affinity value.
> + *
> + * @return
> + *   On success, return 0.
> + *   On failure, return a positive errno-style error number.
> + */
> +__rte_experimental
> +int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
> +		rte_cpuset_t *cpuset);
> +
> +/**
>   * Set core affinity of the current thread.
>   * Support both EAL and non-EAL thread and update TLS.
>   *

[...]

> +static int
> +eal_query_group_affinity(void)
> +{
> +	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
> +	unsigned int *cpu_count = &cpu_map.cpu_count;
> +	DWORD infos_size = 0;
> +	int ret = 0;
> +	USHORT group_count;
> +	KAFFINITY affinity;
> +	USHORT group_no;
> +	unsigned int i;
> +
> +	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
> +			&infos_size)) {
> +		DWORD error = GetLastError();
> +		if (error != ERROR_INSUFFICIENT_BUFFER) {
> +			log_early("Cannot get group information size, "
> +				"error %lu\n", error);

Please don't break string constants for easy search.

> +			rte_errno = EINVAL;
> +			ret = -1;
> +			goto cleanup;
> +		}
> +	}
[...]

> +static bool
> +eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
> +{
> +	const unsigned int node_id = info->NumaNode.NodeNumber;
> +	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
> +	struct lcore_map *lcore;
> +	unsigned int socket_id;
> +	unsigned int i;
> +
> +	/* NUMA node may be reported multiple times if it includes
> +	 * cores from different processor groups, e. g. 80 cores
> +	 * of a physical processor comprise one NUMA node, but two
> +	 * processor groups, because group size is limited by 32/64.
> +	 */
> +	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++) {
> +		if (cpu_map.sockets[socket_id].node_id == node_id)
> +			break;
> +	}

Nit: multi-line comments should start with a line containing just "/*",
and {} are no needed here.

[...]
> +static int
> +rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
> +		PGROUP_AFFINITY affinity)
> +{
> +	int ret = 0;
> +	PGROUP_AFFINITY cpu_affinity = NULL;
> +	unsigned int cpu_idx;
> +
> +	memset(affinity, 0, sizeof(GROUP_AFFINITY));
> +	affinity->Group = (USHORT)-1;
> +
> +	/* Check that all cpus of the set belong to the same processor group and
> +	 * accumulate thread affinity to be applied.
> +	 */
> +	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
> +		if (!CPU_ISSET(cpu_idx, cpuset))
> +			continue;
> +
> +		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
> +
> +		if (affinity->Group == (USHORT)-1) {
> +			affinity->Group = cpu_affinity->Group;
> +		} else if (affinity->Group != cpu_affinity->Group) {
> +			ret = EINVAL;

Should it be ENOTSUP, because we don't support this case?
This limitation is also worth documenting in the commit log.

> +			goto cleanup;
> +		}
> +
> +		affinity->Mask |= cpu_affinity->Mask;
> +	}
> +
> +	if (affinity->Mask == 0) {
> +		ret = EINVAL;
> +		goto cleanup;
> +	}
> +
> +cleanup:
> +	return ret;
> +}
[...]

Re: [PATCH 2/3] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

On Fri, Apr 08, 2022 at 05:01:50PM +0300, Dmitry Kozlyuk wrote:
> 2022-04-01 06:29 (UTC-0700), Tyler Retzlaff:
> > Implement functions for getting/setting thread affinity.
> > Threads can be pinned to specific cores by setting their
> > affinity attribute.
> > 
> > Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> 
> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

thanks, will address all items mentioned in v2 in addition to some minor
issues detected with CI pipeline.

> 
> Please see some small comments below.
> 
> > ---
> >  lib/eal/include/rte_thread.h     |  45 ++++++++++
> >  lib/eal/unix/rte_thread.c        |  16 ++++
> >  lib/eal/version.map              |   4 +
> >  lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++++----------
> >  lib/eal/windows/eal_windows.h    |  10 +++
> >  lib/eal/windows/include/rte_os.h |   2 +
> >  lib/eal/windows/rte_thread.c     | 131 ++++++++++++++++++++++++++++-
> >  7 files changed, 336 insertions(+), 45 deletions(-)
> > 
> > diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> > index 8be8ed8..4eb113f 100644
> > --- a/lib/eal/include/rte_thread.h
> > +++ b/lib/eal/include/rte_thread.h
> > @@ -2,6 +2,8 @@
> >   * Copyright(c) 2021 Mellanox Technologies, Ltd
> >   */
> >  
> > +#include <stdint.h>
> > +
> >  #include <rte_os.h>
> >  #include <rte_compat.h>
> >  
> > @@ -21,6 +23,13 @@
> >  #endif
> >  
> >  /**
> > + * Thread id descriptor.
> > + */
> > +typedef struct rte_thread_tag {
> > +	uintptr_t opaque_id; /**< thread identifier */
> > +} rte_thread_t;
> > +
> > +/**
> >   * TLS key type, an opaque pointer.
> >   */
> >  typedef struct eal_tls_key *rte_thread_key;
> > @@ -28,6 +37,42 @@
> >  #ifdef RTE_HAS_CPUSET
> >  
> >  /**
> 
> Missing a common part for experimental functions:
> 
>  * @warning
>  * @b EXPERIMENTAL: this API may change without prior notice.
> 
> > + * Set the affinity of thread 'thread_id' to the cpu set
> > + * specified by 'cpuset'.
> > + *
> > + * @param thread_id
> > + *    Id of the thread for which to set the affinity.
> > + *
> > + * @param cpuset
> > + *   Pointer to CPU affinity to set.
> > + *
> > + * @return
> > + *   On success, return 0.
> > + *   On failure, return a positive errno-style error number.
> > + */
> > +__rte_experimental
> > +int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
> > +		const rte_cpuset_t *cpuset);
> > +
> > +/**
> 
> Same here.
> 
> > + * Get the affinity of thread 'thread_id' and store it
> > + * in 'cpuset'.
> > + *
> > + * @param thread_id
> > + *    Id of the thread for which to get the affinity.
> > + *
> > + * @param cpuset
> > + *   Pointer for storing the affinity value.
> > + *
> > + * @return
> > + *   On success, return 0.
> > + *   On failure, return a positive errno-style error number.
> > + */
> > +__rte_experimental
> > +int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
> > +		rte_cpuset_t *cpuset);
> > +
> > +/**
> >   * Set core affinity of the current thread.
> >   * Support both EAL and non-EAL thread and update TLS.
> >   *
> 
> [...]
> 
> > +static int
> > +eal_query_group_affinity(void)
> > +{
> > +	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
> > +	unsigned int *cpu_count = &cpu_map.cpu_count;
> > +	DWORD infos_size = 0;
> > +	int ret = 0;
> > +	USHORT group_count;
> > +	KAFFINITY affinity;
> > +	USHORT group_no;
> > +	unsigned int i;
> > +
> > +	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
> > +			&infos_size)) {
> > +		DWORD error = GetLastError();
> > +		if (error != ERROR_INSUFFICIENT_BUFFER) {
> > +			log_early("Cannot get group information size, "
> > +				"error %lu\n", error);
> 
> Please don't break string constants for easy search.
> 
> > +			rte_errno = EINVAL;
> > +			ret = -1;
> > +			goto cleanup;
> > +		}
> > +	}
> [...]
> 
> > +static bool
> > +eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
> > +{
> > +	const unsigned int node_id = info->NumaNode.NodeNumber;
> > +	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
> > +	struct lcore_map *lcore;
> > +	unsigned int socket_id;
> > +	unsigned int i;
> > +
> > +	/* NUMA node may be reported multiple times if it includes
> > +	 * cores from different processor groups, e. g. 80 cores
> > +	 * of a physical processor comprise one NUMA node, but two
> > +	 * processor groups, because group size is limited by 32/64.
> > +	 */
> > +	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++) {
> > +		if (cpu_map.sockets[socket_id].node_id == node_id)
> > +			break;
> > +	}
> 
> Nit: multi-line comments should start with a line containing just "/*",
> and {} are no needed here.
> 
> [...]
> > +static int
> > +rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
> > +		PGROUP_AFFINITY affinity)
> > +{
> > +	int ret = 0;
> > +	PGROUP_AFFINITY cpu_affinity = NULL;
> > +	unsigned int cpu_idx;
> > +
> > +	memset(affinity, 0, sizeof(GROUP_AFFINITY));
> > +	affinity->Group = (USHORT)-1;
> > +
> > +	/* Check that all cpus of the set belong to the same processor group and
> > +	 * accumulate thread affinity to be applied.
> > +	 */
> > +	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
> > +		if (!CPU_ISSET(cpu_idx, cpuset))
> > +			continue;
> > +
> > +		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
> > +
> > +		if (affinity->Group == (USHORT)-1) {
> > +			affinity->Group = cpu_affinity->Group;
> > +		} else if (affinity->Group != cpu_affinity->Group) {
> > +			ret = EINVAL;
> 
> Should it be ENOTSUP, because we don't support this case?
> This limitation is also worth documenting in the commit log.
> 
> > +			goto cleanup;
> > +		}
> > +
> > +		affinity->Mask |= cpu_affinity->Mask;
> > +	}
> > +
> > +	if (affinity->Mask == 0) {
> > +		ret = EINVAL;
> > +		goto cleanup;
> > +	}
> > +
> > +cleanup:
> > +	return ret;
> > +}
> [...]

Re: [PATCH 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

On Fri, Apr 08, 2022 at 05:01:45PM +0300, Dmitry Kozlyuk wrote:
> 2022-04-01 06:29 (UTC-0700), Tyler Retzlaff:
> [...]
> > +static int
> > +test_thread_affinity(void)
> > +{
> > +	pthread_t id;
> > +	rte_thread_t thread_id;
> > +
> > +	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, NULL) == 0,
> > +		"Failed to create thread");
> > +	thread_id.opaque_id = id;
> 
> The need for this hack means that the new API is unusable in practice.
> I think functions to get the current thread ID and to compare IDs
> must be the part of this series to accept this unit test patch.

are you proposing adding rte_thread_self and rte_thread_equals to this
series? i'm not sure i understand how that helps? while maybe the unit
test could be re-written in some fashion to use rte_thread_self it still
wouldn't do much for practically using the api outside of the test.

keep in mind i have no intention of combining api introduction and
integration into the same series so the hack is limited to the unit test
for now. it is my expectation that as subsequent series are merged the
hacks will be replaced with new api additions. i'm also not entertaining
the idea of growing this series further as we will just end up back
where we started with a series that is too large and nobody will
properly review or be bold enough to ack.

the quickest path to eliminate the hack is to improve the velocity of
getting the smaller series reviewed and merged. this is where i would
prefer to focus our effort.

if there is an absolute need to start using the api introduced in this
series outside of the unit test we could as a part of the __experimental
apis offer a function that formalizes the conversion to rte_thread_t from
pthread_t. the function would only be kept as long as needed and removed
when no longer necessary.

thanks

Re: [PATCH 0/3] add eal functions for thread affinity

[David Marchand]

On Fri, Apr 8, 2022 at 3:46 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> On Fri, Apr 08, 2022 at 10:57:55AM +0200, David Marchand wrote:
> > Hello Tyler,
> >
> > On Fri, Apr 1, 2022 at 3:30 PM Tyler Retzlaff
> > <roretzla@linux.microsoft.com> wrote:
> > >
> > > this series provides basic dependencies for additional eal thread api
> > > additions. series includes basic error handling, initial get/set thread
> > > affinity functions and minimal unit test.
> > >
> > > Tyler Retzlaff (3):
> > >   eal/windows: translate Windows errors to errno-style errors
> > >   eal: implement functions for get/set thread affinity
> > >   test/threads: add unit test for thread API
> > >
> > >  app/test/meson.build             |   2 +
> > >  app/test/test_threads.c          |  86 +++++++++++++++++++
> > >  lib/eal/include/rte_thread.h     |  45 ++++++++++
> > >  lib/eal/unix/rte_thread.c        |  16 ++++
> > >  lib/eal/version.map              |   4 +
> > >  lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++----------
> > >  lib/eal/windows/eal_windows.h    |  10 +++
> > >  lib/eal/windows/include/rte_os.h |   2 +
> > >  lib/eal/windows/rte_thread.c     | 179 ++++++++++++++++++++++++++++++++++++++-
> > >  9 files changed, 472 insertions(+), 45 deletions(-)
> > >  create mode 100644 app/test/test_threads.c
> >
> > We have two concurrent series, can you clarify what are the intentions
> > on this work?
>
> yes, i should have clarified this up front sorry.
>
> > Is this series superseding Narcisa series?
>
> this series supersedes the series from Narcisa. it was resolved through
> discussion that the current series should be abandoned as it is too
> large and not making progress.
>
> we've elected to submit a series of smaller patchsets that incorporate
> the feedback received to date and build up the api surface for
> threading. the patches are still the work of Narcisa but she is
> overscheduled so i will assist in upstreaming and addressing feedback.
>
> additionally, rather than port the tree to the new __experimental api as
> they are added we will prefer to add unit tests that provide validation
> of the api and example usage.
>
> our hope is the smaller scoped series will attract more attention and
> have better acknowledgement velocity.
>
> i will have Narcisa mark the monolithic series as superseded on
> patchwork.

Ok, thanks Tyler.


-- 
David Marchand

Re: [PATCH 3/3] test/threads: add unit test for thread API

[Dmitry Kozlyuk]

2022-04-09 01:56 (UTC-0700), Tyler Retzlaff:
> On Fri, Apr 08, 2022 at 05:01:45PM +0300, Dmitry Kozlyuk wrote:
> > 2022-04-01 06:29 (UTC-0700), Tyler Retzlaff:
> > [...]  
> > > +static int
> > > +test_thread_affinity(void)
> > > +{
> > > +	pthread_t id;
> > > +	rte_thread_t thread_id;
> > > +
> > > +	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, NULL) == 0,
> > > +		"Failed to create thread");
> > > +	thread_id.opaque_id = id;  
> > 
> > The need for this hack means that the new API is unusable in practice.
> > I think functions to get the current thread ID and to compare IDs
> > must be the part of this series to accept this unit test patch.  
> 
> are you proposing adding rte_thread_self and rte_thread_equals to this
> series? i'm not sure i understand how that helps? while maybe the unit
> test could be re-written in some fashion to use rte_thread_self it still
> wouldn't do much for practically using the api outside of the test.

Just rte_thread_self() should be enough; see below.
How this helps: the series becomes complete, because there is enough API to
verify it works without assumptions about implementation.

[...]
> if there is an absolute need to start using the api introduced in this
> series outside of the unit test we could as a part of the __experimental
> apis offer a function that formalizes the conversion to rte_thread_t from
> pthread_t. the function would only be kept as long as needed and removed
> when no longer necessary.

I rather meant this:

main thread:
	init barrier
	create thread
	enter barrier (wait)
	use shared thread ID

thread:
	init shared thread ID from rte_thread_self()
	enter barrier (will unblock)

After you will add rte_thread_create() in the subsequent series,
the same logic will suit for checking that rte_thread_self()
returns the proper ID.

[PATCH v2 0/4] add eal functions for thread affinity and self

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes

* basic platform error number conversion.
* function to get current thread identifier.
* functions to get and set affinity with platform agnostic thread
  identifier.
* minimal unit test of get and set affinity demonstrating usage.

note: previous series introducing these functions is now superseded by
this series.
http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*

Tyler Retzlaff (4):
  eal/windows: translate Windows errors to errno-style errors
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

v2:
* add missing boilerplate comments warning of experimental api
  for rte_thread_{set,get}_affinity_by_id().
* don't break literal format string to log_early to improve
  searchability.
* fix multi-line comment style to match file.
* return ENOTSUP instead of EINVAL from rte_convert_cpuset_to_affinity()
  if cpus in set are not part of the same processor group and note
  limitation in commit message.
* expand series to include rte_thread_self().
* modify unit test to remove use of implementation detail and
  get thread identifier use added rte_thread_self().
* move literal value to rhs when using memcmp in RTE_TEST_ASSERT

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  89 ++++++++++++++++++
 lib/eal/include/rte_thread.h     |  64 +++++++++++++
 lib/eal/unix/rte_thread.c        |  27 ++++++
 lib/eal/version.map              |   5 ++
 lib/eal/windows/eal_lcore.c      | 173 ++++++++++++++++++++++++++---------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 190 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 517 insertions(+), 45 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH v2 1/4] eal/windows: translate Windows errors to errno-style errors

[Tyler Retzlaff]

Add function to translate Windows error codes to errno-style error
codes. The possible return values are chosen so that we have as
much semantical compatibility between platforms as possible.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/windows/rte_thread.c | 49 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..c272018 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <rte_common.h>
@@ -11,6 +12,54 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v2 2/4] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

Provide a portable type-safe thread identifier.
Provide rte_thread_self for obtaining current thread identifier.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
 lib/eal/unix/rte_thread.c    | 11 +++++++++++
 lib/eal/version.map          |  3 +++
 lib/eal/windows/rte_thread.c | 10 ++++++++++
 4 files changed, 46 insertions(+)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..fb66d9a 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -1,7 +1,10 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,10 +24,29 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct rte_thread_tag {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the id of the calling thread.
+ *
+ * @return
+ *   Return the thread id of the calling thread.
+ */
+__rte_experimental
+rte_thread_t rte_thread_self(void);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..82e008f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <errno.h>
@@ -15,6 +16,16 @@ struct eal_tls_key {
 	pthread_key_t thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = (uintptr_t)pthread_self();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
 {
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..05ce8f9 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,9 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_self;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index c272018..d730bb4 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -60,6 +60,16 @@ struct eal_tls_key {
 	return thread_translate_win32_error(error);
 }
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = GetCurrentThreadId();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v2 3/4] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

note: rte_convert_cpuset_to_affinity has a limitation that all cpus of
the set belong to the same processor group.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h     |  42 ++++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal_lcore.c      | 173 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 131 ++++++++++++++++++++++++++++-
 7 files changed, 331 insertions(+), 45 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index fb66d9a..c26ca1d 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -50,6 +50,48 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index 82e008f..c64198f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -100,3 +100,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 05ce8f9..d49e30b 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -422,7 +422,9 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_get_affinity_by_id;
 	rte_thread_self;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..4f2224e 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -1,8 +1,8 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2019 Intel Corporation
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +27,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +49,115 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			log_early("Cannot get group information size, error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		log_early("Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		log_early("Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/*
+	 * NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++)
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -83,52 +188,24 @@ struct cpu_map {
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -164,3 +241,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index 245aa60..c96eca5 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -56,6 +56,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index d730bb4..032b09f 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -6,7 +6,8 @@
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
@@ -146,3 +147,131 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = rte_convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH v2 4/4] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1..5a9d69b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -238,6 +239,7 @@ fast_tests = [
         ['reorder_autotest', true],
         ['service_autotest', true],
         ['thash_autotest', true],
+        ['threads_autotest', true],
         ['trace_autotest', true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..0ca6745
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022 Microsoft Corporation
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static uint32_t thread_id_ready;
+
+static void *
+thread_main(void *arg)
+{
+	*(rte_thread_t *)arg = rte_thread_self();
+	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
+
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+		"Failed to create thread");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	rte_cpuset_t cpuset0;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+
+	rte_cpuset_t cpuset1;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH v2 1/4] eal/windows: translate Windows errors to errno-style errors

[Menon, Ranjit]

Hi, Tyler

On 4/12/2022 3:43 AM, Tyler Retzlaff wrote:
> Add function to translate Windows error codes to errno-style error
> codes. The possible return values are chosen so that we have as
> much semantical compatibility between platforms as possible.
>
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> ---
>   lib/eal/windows/rte_thread.c | 49 ++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 49 insertions(+)
>
> diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> index 667287c..c272018 100644
> --- a/lib/eal/windows/rte_thread.c
> +++ b/lib/eal/windows/rte_thread.c
> @@ -1,5 +1,6 @@
>   /* SPDX-License-Identifier: BSD-3-Clause
>    * Copyright 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>    */
>   
>   #include <rte_common.h>
> @@ -11,6 +12,54 @@ struct eal_tls_key {
>   	DWORD thread_index;
>   };
>   
> +/* Translates the most common error codes related to threads */
> +static int
> +thread_translate_win32_error(DWORD error)
> +{
> +	switch (error) {
> +	case ERROR_SUCCESS:
> +		return 0;
> +
> +	case ERROR_INVALID_PARAMETER:
> +		return EINVAL;
> +
> +	case ERROR_INVALID_HANDLE:
> +		return EFAULT;
> +
> +	case ERROR_NOT_ENOUGH_MEMORY:
> +		/* FALLTHROUGH */
> +	case ERROR_NO_SYSTEM_RESOURCES:
> +		return ENOMEM;
> +
> +	case ERROR_PRIVILEGE_NOT_HELD:
> +		/* FALLTHROUGH */
> +	case ERROR_ACCESS_DENIED:
> +		return EACCES;
> +
> +	case ERROR_ALREADY_EXISTS:
> +		return EEXIST;
> +
> +	case ERROR_POSSIBLE_DEADLOCK:
> +		return EDEADLK;
> +
> +	case ERROR_INVALID_FUNCTION:
> +		/* FALLTHROUGH */
> +	case ERROR_CALL_NOT_IMPLEMENTED:
> +		return ENOSYS;
> +	}
> +
> +	return EINVAL;
> +}
> +

Shouldn't we return all these error values as negative... as in -EINVAL, 
-EFAULT etc. ?


ranjit m.

Re: [PATCH v2 1/4] eal/windows: translate Windows errors to errno-style errors

[Tyler Retzlaff]

On Tue, Apr 12, 2022 at 10:26:27AM -0700, Menon, Ranjit wrote:
> Hi, Tyler
> 
> On 4/12/2022 3:43 AM, Tyler Retzlaff wrote:
> >Add function to translate Windows error codes to errno-style error
> >codes. The possible return values are chosen so that we have as
> >much semantical compatibility between platforms as possible.
> >
> >Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >---
> >  lib/eal/windows/rte_thread.c | 49 ++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 49 insertions(+)
> >
> >diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> >index 667287c..c272018 100644
> >--- a/lib/eal/windows/rte_thread.c
> >+++ b/lib/eal/windows/rte_thread.c
> >@@ -1,5 +1,6 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright 2021 Mellanox Technologies, Ltd
> >+ * Copyright (C) 2022 Microsoft Corporation
> >   */
> >  #include <rte_common.h>
> >@@ -11,6 +12,54 @@ struct eal_tls_key {
> >  	DWORD thread_index;
> >  };
> >+/* Translates the most common error codes related to threads */
> >+static int
> >+thread_translate_win32_error(DWORD error)
> >+{
> >+	switch (error) {
> >+	case ERROR_SUCCESS:
> >+		return 0;
> >+
> >+	case ERROR_INVALID_PARAMETER:
> >+		return EINVAL;
> >+
> >+	case ERROR_INVALID_HANDLE:
> >+		return EFAULT;
> >+
> >+	case ERROR_NOT_ENOUGH_MEMORY:
> >+		/* FALLTHROUGH */
> >+	case ERROR_NO_SYSTEM_RESOURCES:
> >+		return ENOMEM;
> >+
> >+	case ERROR_PRIVILEGE_NOT_HELD:
> >+		/* FALLTHROUGH */
> >+	case ERROR_ACCESS_DENIED:
> >+		return EACCES;
> >+
> >+	case ERROR_ALREADY_EXISTS:
> >+		return EEXIST;
> >+
> >+	case ERROR_POSSIBLE_DEADLOCK:
> >+		return EDEADLK;
> >+
> >+	case ERROR_INVALID_FUNCTION:
> >+		/* FALLTHROUGH */
> >+	case ERROR_CALL_NOT_IMPLEMENTED:
> >+		return ENOSYS;
> >+	}
> >+
> >+	return EINVAL;
> >+}
> >+
> 
> Shouldn't we return all these error values as negative... as in
> -EINVAL, -EFAULT etc. ?

no. since this is just a translation function it is up to the caller
what it will do with the translated value and any conversion to -errno.

since it is also used to store into rte_errno it would have been
confusing to see something like the following to reverse the sign back.

err = thread_translate_win32_error(win32err);
rte_errno = -err; // confusing

personally i don't like the -errno returns at all and this is just
another reason contributing to that opinion. you can refer to my
previous mailing list posts on the subject.

> 
> 
> ranjit m.

Re: [PATCH v2 3/4] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

On Tue, Apr 12, 2022 at 03:43:39AM -0700, Tyler Retzlaff wrote:

> diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
> index 476c2d2..4f2224e 100644
> --- a/lib/eal/windows/eal_lcore.c
> +++ b/lib/eal/windows/eal_lcore.c

... snip ...

>  int
>  eal_create_cpu_map(void)
>  {
>  	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
>  	DWORD infos_size;
>  	bool full = false;
> +	int ret = 0;
>  
> +	infos = NULL;
>  	infos_size = 0;
>  	if (!GetLogicalProcessorInformationEx(
>  			RelationNumaNode, NULL, &infos_size)) {
> @@ -62,7 +166,8 @@ struct cpu_map {
>  			log_early("Cannot get NUMA node info size, error %lu\n",
>  				GetLastError());
>  			rte_errno = ENOMEM;
> -			return -1;
> +			ret = -1;
> +			goto exit;
>  		}
>  	}
>  

it can't be seen in the context of this diff but there is a leak of
infos on an error path. a v3 of the series will be submitted to address
the half-conversion to ret = -1 goto exit pattern.

> @@ -83,52 +188,24 @@ struct cpu_map {
>  
>  	info = infos;
>  	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
> -		unsigned int node_id = info->NumaNode.NodeNumber;
> -		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
> -		struct lcore_map *lcore;
> -		unsigned int i, socket_id;
> -
> -		/* NUMA node may be reported multiple times if it includes
> -		 * cores from different processor groups, e. g. 80 cores
> -		 * of a physical processor comprise one NUMA node, but two
> -		 * processor groups, because group size is limited by 32/64.
> -		 */
> -		for (socket_id = 0; socket_id < cpu_map.socket_count;
> -		    socket_id++) {
> -			if (cpu_map.sockets[socket_id].node_id == node_id)
> -				break;
> -		}
> -
> -		if (socket_id == cpu_map.socket_count) {
> -			if (socket_id == RTE_DIM(cpu_map.sockets)) {
> -				full = true;
> -				goto exit;
> -			}
> -
> -			cpu_map.sockets[socket_id].node_id = node_id;
> -			cpu_map.socket_count++;
> -		}
> -
> -		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
> -			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
> -				continue;
> -
> -			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
> -				full = true;
> -				goto exit;
> -			}
> -
> -			lcore = &cpu_map.lcores[cpu_map.lcore_count];
> -			lcore->socket_id = socket_id;
> -			lcore->core_id =
> -				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
> -			cpu_map.lcore_count++;
> +		if (eal_create_lcore_map(info)) {
> +			full = true;
> +			break;
>  		}
>  
>  		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
>  			(uint8_t *)info + info->Size);
>  	}
>  
> +	if (eal_query_group_affinity()) {
> +		/*
> +		 * No need to set rte_errno here.
> +		 * It is set by eal_query_group_affinity().
> +		 */
> +		ret = -1;
> +		goto exit;
> +	}
> +
>  exit:
>  	if (full) {
>  		/* Not a fatal error, but important for troubleshooting. */

[PATCH v3 0/4] add eal functions for thread affinity and self

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes

* basic platform error number conversion.
* function to get current thread identifier.
* functions to get and set affinity with platform agnostic thread
  identifier.
* minimal unit test of get and set affinity demonstrating usage.

note: previous series introducing these functions is now superseded by
this series.
http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*

Tyler Retzlaff (4):
  eal/windows: translate Windows errors to errno-style errors
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

v3:
* fix memory leak on eal_create_cpu_map error paths.

v2:
* add missing boilerplate comments warning of experimental api
  for rte_thread_{set,get}_affinity_by_id().
* don't break literal format string to log_early to improve
  searchability.
* fix multi-line comment style to match file.
* return ENOTSUP instead of EINVAL from rte_convert_cpuset_to_affinity()
  if cpus in set are not part of the same processor group and note
  limitation in commit message.
* expand series to include rte_thread_self().
* modify unit test to remove use of implementation detail and
  get thread identifier use added rte_thread_self().
* move literal value to rhs when using memcmp in RTE_TEST_ASSERT

Tyler Retzlaff (4):
  eal/windows: translate Windows errors to errno-style errors
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  89 ++++++++++++++++++
 lib/eal/include/rte_thread.h     |  64 +++++++++++++
 lib/eal/unix/rte_thread.c        |  27 ++++++
 lib/eal/version.map              |   5 ++
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 190 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 522 insertions(+), 48 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH v3 1/4] eal/windows: translate Windows errors to errno-style errors

[Tyler Retzlaff]

Add function to translate Windows error codes to errno-style error
codes. The possible return values are chosen so that we have as
much semantical compatibility between platforms as possible.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/windows/rte_thread.c | 49 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..c272018 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <rte_common.h>
@@ -11,6 +12,54 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v3 2/4] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

Provide a portable type-safe thread identifier.
Provide rte_thread_self for obtaining current thread identifier.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
 lib/eal/unix/rte_thread.c    | 11 +++++++++++
 lib/eal/version.map          |  3 +++
 lib/eal/windows/rte_thread.c | 10 ++++++++++
 4 files changed, 46 insertions(+)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..fb66d9a 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -1,7 +1,10 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,10 +24,29 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct rte_thread_tag {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the id of the calling thread.
+ *
+ * @return
+ *   Return the thread id of the calling thread.
+ */
+__rte_experimental
+rte_thread_t rte_thread_self(void);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..82e008f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <errno.h>
@@ -15,6 +16,16 @@ struct eal_tls_key {
 	pthread_key_t thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = (uintptr_t)pthread_self();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
 {
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..05ce8f9 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,9 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_self;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index c272018..d730bb4 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -60,6 +60,16 @@ struct eal_tls_key {
 	return thread_translate_win32_error(error);
 }
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = GetCurrentThreadId();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v3 3/4] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

note: rte_convert_cpuset_to_affinity has a limitation that all cpus of
the set belong to the same processor group.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h     |  42 +++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 131 +++++++++++++++++++++++++++-
 7 files changed, 336 insertions(+), 48 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index fb66d9a..c26ca1d 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -50,6 +50,48 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index 82e008f..c64198f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -100,3 +100,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 05ce8f9..d49e30b 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -422,7 +422,9 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_get_affinity_by_id;
 	rte_thread_self;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..aa2fad9 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -1,8 +1,8 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2019 Intel Corporation
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +27,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +49,115 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			log_early("Cannot get group information size, error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		log_early("Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		log_early("Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/*
+	 * NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++)
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -70,7 +175,8 @@ struct cpu_map {
 	if (infos == NULL) {
 		log_early("Cannot allocate memory for NUMA node information\n");
 		rte_errno = ENOMEM;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	if (!GetLogicalProcessorInformationEx(
@@ -78,57 +184,30 @@ struct cpu_map {
 		log_early("Cannot get NUMA node information, error %lu\n",
 			GetLastError());
 		rte_errno = EINVAL;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -138,7 +217,7 @@ struct cpu_map {
 
 	free(infos);
 
-	return 0;
+	return ret;
 }
 
 int
@@ -164,3 +243,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index 245aa60..c96eca5 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -56,6 +56,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index d730bb4..032b09f 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -6,7 +6,8 @@
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
@@ -146,3 +147,131 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = rte_convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH v3 4/4] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1..5a9d69b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -238,6 +239,7 @@ fast_tests = [
         ['reorder_autotest', true],
         ['service_autotest', true],
         ['thash_autotest', true],
+        ['threads_autotest', true],
         ['trace_autotest', true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..0ca6745
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022 Microsoft Corporation
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static uint32_t thread_id_ready;
+
+static void *
+thread_main(void *arg)
+{
+	*(rte_thread_t *)arg = rte_thread_self();
+	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
+
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+		"Failed to create thread");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	rte_cpuset_t cpuset0;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+
+	rte_cpuset_t cpuset1;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH v2 1/4] eal/windows: translate Windows errors to errno-style errors

[David Marchand]

On Tue, Apr 12, 2022 at 12:44 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> Add function to translate Windows error codes to errno-style error
> codes. The possible return values are chosen so that we have as
> much semantical compatibility between platforms as possible.
>
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

This patch alone does not make sense, and I guess it breaks
compilation on Windows if we stop at it during a bisect (because it
adds two unused static symbols).
It can be squashed in patch 3 where thread_log_last_error is first used.


-- 
David Marchand

Re: [PATCH v3 2/4] eal: add basic thread ID and current thread identifier API

[David Marchand]

On Wed, Apr 13, 2022 at 9:43 AM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> Provide a portable type-safe thread identifier.
> Provide rte_thread_self for obtaining current thread identifier.
>
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> ---
>  lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>  lib/eal/unix/rte_thread.c    | 11 +++++++++++
>  lib/eal/version.map          |  3 +++
>  lib/eal/windows/rte_thread.c | 10 ++++++++++
>  4 files changed, 46 insertions(+)
>
> diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> index 8be8ed8..fb66d9a 100644
> --- a/lib/eal/include/rte_thread.h
> +++ b/lib/eal/include/rte_thread.h
> @@ -1,7 +1,10 @@
>  /* SPDX-License-Identifier: BSD-3-Clause
>   * Copyright(c) 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>   */
>
> +#include <stdint.h>
> +
>  #include <rte_os.h>
>  #include <rte_compat.h>
>
> @@ -21,10 +24,29 @@
>  #endif
>
>  /**
> + * Thread id descriptor.
> + */
> +typedef struct rte_thread_tag {

Is there a need for rte_thread_tag.
I don't see it used.


-- 
David Marchand

Re: [PATCH v3 3/4] eal: implement functions for get/set thread affinity

[David Marchand]

On Wed, Apr 13, 2022 at 9:43 AM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> Implement functions for getting/setting thread affinity.
> Threads can be pinned to specific cores by setting their
> affinity attribute.
>
> note: rte_convert_cpuset_to_affinity has a limitation that all cpus of
> the set belong to the same processor group.
>
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

[snip]

> diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> index d730bb4..032b09f 100644
> --- a/lib/eal/windows/rte_thread.c
> +++ b/lib/eal/windows/rte_thread.c
> @@ -6,7 +6,8 @@
>  #include <rte_common.h>
>  #include <rte_errno.h>
>  #include <rte_thread.h>
> -#include <rte_windows.h>
> +
> +#include "eal_windows.h"
>
>  struct eal_tls_key {
>         DWORD thread_index;
> @@ -146,3 +147,131 @@ struct eal_tls_key {
>         }
>         return output;
>  }
> +
> +static int
> +rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
> +               PGROUP_AFFINITY affinity)

This is Windows only, static code.
Please don't use rte_ prefix.

For static code, I see no need for a prefix at all.


-- 
David Marchand

Re: [PATCH v2 1/4] eal/windows: translate Windows errors to errno-style errors

[Tyler Retzlaff]

On Mon, Apr 25, 2022 at 10:25:30AM +0200, David Marchand wrote:
> On Tue, Apr 12, 2022 at 12:44 PM Tyler Retzlaff
> <roretzla@linux.microsoft.com> wrote:
> >
> > Add function to translate Windows error codes to errno-style error
> > codes. The possible return values are chosen so that we have as
> > much semantical compatibility between platforms as possible.
> >
> > Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> 
> This patch alone does not make sense, and I guess it breaks
> compilation on Windows if we stop at it during a bisect (because it
> adds two unused static symbols).

interesting, didn't happen locally but i can see that it should have.

> It can be squashed in patch 3 where thread_log_last_error is first used.

yep, makes sense to me. i'll push a new version.

thanks

Re: [PATCH v3 2/4] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

On Mon, Apr 25, 2022 at 10:26:42AM +0200, David Marchand wrote:
> On Wed, Apr 13, 2022 at 9:43 AM Tyler Retzlaff
> <roretzla@linux.microsoft.com> wrote:
> >
> > Provide a portable type-safe thread identifier.
> > Provide rte_thread_self for obtaining current thread identifier.
> >
> > Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> > ---
> >  lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >  lib/eal/unix/rte_thread.c    | 11 +++++++++++
> >  lib/eal/version.map          |  3 +++
> >  lib/eal/windows/rte_thread.c | 10 ++++++++++
> >  4 files changed, 46 insertions(+)
> >
> > diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> > index 8be8ed8..fb66d9a 100644
> > --- a/lib/eal/include/rte_thread.h
> > +++ b/lib/eal/include/rte_thread.h
> > @@ -1,7 +1,10 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright(c) 2021 Mellanox Technologies, Ltd
> > + * Copyright (C) 2022 Microsoft Corporation
> >   */
> >
> > +#include <stdint.h>
> > +
> >  #include <rte_os.h>
> >  #include <rte_compat.h>
> >
> > @@ -21,10 +24,29 @@
> >  #endif
> >
> >  /**
> > + * Thread id descriptor.
> > + */
> > +typedef struct rte_thread_tag {
> 
> Is there a need for rte_thread_tag.
> I don't see it used.

i don't think so. it was part of the original series i'll remove it.

> 
> 
> -- 
> David Marchand

Re: [PATCH v3 3/4] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

On Mon, Apr 25, 2022 at 10:26:58AM +0200, David Marchand wrote:
> On Wed, Apr 13, 2022 at 9:43 AM Tyler Retzlaff
> <roretzla@linux.microsoft.com> wrote:
> >
> > Implement functions for getting/setting thread affinity.
> > Threads can be pinned to specific cores by setting their
> > affinity attribute.
> >
> > note: rte_convert_cpuset_to_affinity has a limitation that all cpus of
> > the set belong to the same processor group.
> >
> > Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> 
> [snip]
> 
> > diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> > index d730bb4..032b09f 100644
> > --- a/lib/eal/windows/rte_thread.c
> > +++ b/lib/eal/windows/rte_thread.c
> > @@ -6,7 +6,8 @@
> >  #include <rte_common.h>
> >  #include <rte_errno.h>
> >  #include <rte_thread.h>
> > -#include <rte_windows.h>
> > +
> > +#include "eal_windows.h"
> >
> >  struct eal_tls_key {
> >         DWORD thread_index;
> > @@ -146,3 +147,131 @@ struct eal_tls_key {
> >         }
> >         return output;
> >  }
> > +
> > +static int
> > +rte_convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
> > +               PGROUP_AFFINITY affinity)
> 
> This is Windows only, static code.
> Please don't use rte_ prefix.
> 
> For static code, I see no need for a prefix at all.

agreed. will fix in next revision.

> 
> 
> -- 
> David Marchand

[PATCH v4 0/3] add eal functions for thread affinity and self

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes

* basic platform error number conversion.
* function to get current thread identifier.
* functions to get and set affinity with platform agnostic thread
  identifier.
* minimal unit test of get and set affinity demonstrating usage.

note: previous series introducing these functions is now superseded by
this series.
http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*

v4:
* combine patch eal/windows: translate Windows errors to errno-style
  errors into eal: implement functions for get/set thread affinity
  patch. the former introduced static functions that were not used
  without eal: implement functions for get/set thread affinity which
  would cause a build break when applied standalone.
* remove struct tag from rte_thread_t struct typedef.
* remove rte_ prefix from rte_convert_cpuset_to_affinity static
  function.

v3:
* fix memory leak on eal_create_cpu_map error paths.

v2:
* add missing boilerplate comments warning of experimental api
  for rte_thread_{set,get}_affinity_by_id().
* don't break literal format string to log_early to improve
  searchability.
* fix multi-line comment style to match file.
* return ENOTSUP instead of EINVAL from rte_convert_cpuset_to_affinity()
  if cpus in set are not part of the same processor group and note
  limitation in commit message.
* expand series to include rte_thread_self().
* modify unit test to remove use of implementation detail and
  get thread identifier use added rte_thread_self().
* move literal value to rhs when using memcmp in RTE_TEST_ASSERT

Tyler Retzlaff (3):
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  89 ++++++++++++++++++
 lib/eal/include/rte_thread.h     |  64 +++++++++++++
 lib/eal/unix/rte_thread.c        |  27 ++++++
 lib/eal/version.map              |   5 ++
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 190 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 522 insertions(+), 48 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH v4 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

Provide a portable type-safe thread identifier.
Provide rte_thread_self for obtaining current thread identifier.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
 lib/eal/unix/rte_thread.c    | 11 +++++++++++
 lib/eal/version.map          |  3 +++
 lib/eal/windows/rte_thread.c | 10 ++++++++++
 4 files changed, 46 insertions(+)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..14478ba 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -1,7 +1,10 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,10 +24,29 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the id of the calling thread.
+ *
+ * @return
+ *   Return the thread id of the calling thread.
+ */
+__rte_experimental
+rte_thread_t rte_thread_self(void);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..82e008f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <errno.h>
@@ -15,6 +16,16 @@ struct eal_tls_key {
 	pthread_key_t thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = (uintptr_t)pthread_self();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
 {
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..05ce8f9 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,9 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_self;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..59fed3c 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -11,6 +11,16 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = GetCurrentThreadId();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v4 2/3] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

Windows error codes are translated to errno-style error codes.
The possible return values are chosen so that we have as
much semantic compatibility between platforms as possible.

note: convert_cpuset_to_affinity has a limitation that all cpus of
the set belong to the same processor group.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 lib/eal/include/rte_thread.h     |  42 +++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 180 +++++++++++++++++++++++++++++++++++++-
 7 files changed, 385 insertions(+), 48 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 14478ba..7888f7a 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -50,6 +50,48 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index 82e008f..c64198f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -100,3 +100,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 05ce8f9..d49e30b 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -422,7 +422,9 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_get_affinity_by_id;
 	rte_thread_self;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..aa2fad9 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -1,8 +1,8 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2019 Intel Corporation
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +27,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +49,115 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			log_early("Cannot get group information size, error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		log_early("Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		log_early("Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/*
+	 * NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++)
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -70,7 +175,8 @@ struct cpu_map {
 	if (infos == NULL) {
 		log_early("Cannot allocate memory for NUMA node information\n");
 		rte_errno = ENOMEM;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	if (!GetLogicalProcessorInformationEx(
@@ -78,57 +184,30 @@ struct cpu_map {
 		log_early("Cannot get NUMA node information, error %lu\n",
 			GetLastError());
 		rte_errno = EINVAL;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -138,7 +217,7 @@ struct cpu_map {
 
 	free(infos);
 
-	return 0;
+	return ret;
 }
 
 int
@@ -164,3 +243,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index e4c4670..ab25814 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -58,6 +58,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 59fed3c..30dd6b7 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -1,16 +1,66 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
@@ -97,3 +147,131 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH v4 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1..5a9d69b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -238,6 +239,7 @@ fast_tests = [
         ['reorder_autotest', true],
         ['service_autotest', true],
         ['thash_autotest', true],
+        ['threads_autotest', true],
         ['trace_autotest', true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..0ca6745
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022 Microsoft Corporation
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static uint32_t thread_id_ready;
+
+static void *
+thread_main(void *arg)
+{
+	*(rte_thread_t *)arg = rte_thread_self();
+	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
+
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+		"Failed to create thread");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	rte_cpuset_t cpuset0;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+
+	rte_cpuset_t cpuset1;
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH v4 1/3] eal: add basic thread ID and current thread identifier API

[Dmitry Kozlyuk]

2022-04-26 00:50 (UTC-0700), Tyler Retzlaff:
> Provide a portable type-safe thread identifier.
> Provide rte_thread_self for obtaining current thread identifier.
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Re: [PATCH v4 2/3] eal: implement functions for get/set thread affinity

[Dmitry Kozlyuk]

2022-04-26 00:50 (UTC-0700), Tyler Retzlaff:
> Implement functions for getting/setting thread affinity.
> Threads can be pinned to specific cores by setting their
> affinity attribute.
> 
> Windows error codes are translated to errno-style error codes.
> The possible return values are chosen so that we have as
> much semantic compatibility between platforms as possible.
> 
> note: convert_cpuset_to_affinity has a limitation that all cpus of
> the set belong to the same processor group.
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
with two minor issues below.

> +static int
> +eal_query_group_affinity(void)
> +{
> +	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
> +	unsigned int *cpu_count = &cpu_map.cpu_count;
> +	DWORD infos_size = 0;
> +	int ret = 0;
> +	USHORT group_count;
> +	KAFFINITY affinity;
> +	USHORT group_no;
> +	unsigned int i;
> +
> +	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
> +			&infos_size)) {
> +		DWORD error = GetLastError();
> +		if (error != ERROR_INSUFFICIENT_BUFFER) {
> +			log_early("Cannot get group information size, error %lu\n", error);

This code is now called not only at initialization,
but also after the logging is set up,
in which case this line will not go into a redirected destination.

I think there should not be log_early() at all in EAL,
rte_log() could work from the very beginning.
We're not in the kernel, stderr is available at once.

So I would work around with a static flag maybe.

[...]
> +static int
> +convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
> +		PGROUP_AFFINITY affinity)
> +{
> +	int ret = 0;
> +	PGROUP_AFFINITY cpu_affinity = NULL;
> +	unsigned int cpu_idx;
> +
> +	memset(affinity, 0, sizeof(GROUP_AFFINITY));
> +	affinity->Group = (USHORT)-1;
> +
> +	/* Check that all cpus of the set belong to the same processor group and
> +	 * accumulate thread affinity to be applied.
> +	 */
> +	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
> +		if (!CPU_ISSET(cpu_idx, cpuset))
> +			continue;
> +
> +		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
> +
> +		if (affinity->Group == (USHORT)-1) {
> +			affinity->Group = cpu_affinity->Group;
> +		} else if (affinity->Group != cpu_affinity->Group) {
> +			ret = ENOTSUP;

Maybe worth a dedug log because the limitation is non-obvious.

Re: [PATCH v4 3/3] test/threads: add unit test for thread API

[Dmitry Kozlyuk]

2022-04-26 00:50 (UTC-0700), Tyler Retzlaff:
> Establish unit test for testing thread api. Initial unit tests
> for rte_thread_{get,set}_affinity_by_id().
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> ---
>  app/test/meson.build    |  2 ++
>  app/test/test_threads.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 91 insertions(+)
>  create mode 100644 app/test/test_threads.c
> 
> diff --git a/app/test/meson.build b/app/test/meson.build
> index 5fc1dd1..5a9d69b 100644
> --- a/app/test/meson.build
> +++ b/app/test/meson.build
> @@ -133,6 +133,7 @@ test_sources = files(
>          'test_tailq.c',
>          'test_thash.c',
>          'test_thash_perf.c',
> +        'test_threads.c',
>          'test_timer.c',
>          'test_timer_perf.c',
>          'test_timer_racecond.c',
> @@ -238,6 +239,7 @@ fast_tests = [
>          ['reorder_autotest', true],
>          ['service_autotest', true],
>          ['thash_autotest', true],
> +        ['threads_autotest', true],
>          ['trace_autotest', true],
>  ]
>  
> diff --git a/app/test/test_threads.c b/app/test/test_threads.c
> new file mode 100644
> index 0000000..0ca6745
> --- /dev/null
> +++ b/app/test/test_threads.c
> @@ -0,0 +1,89 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright (C) 2022 Microsoft Corporation
> + */
> +
> +#include <string.h>
> +#include <pthread.h>
> +
> +#include <rte_thread.h>
> +#include <rte_debug.h>
> +
> +#include "test.h"
> +
> +RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
> +
> +static uint32_t thread_id_ready;
> +
> +static void *
> +thread_main(void *arg)
> +{
> +	*(rte_thread_t *)arg = rte_thread_self();
> +	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
> +
> +	return NULL;
> +}
> +
> +static int
> +test_thread_affinity(void)
> +{
> +	pthread_t id;
> +	rte_thread_t thread_id;
> +
> +	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
> +		"Failed to create thread");
> +
> +	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
> +		;
> +
> +	rte_cpuset_t cpuset0;

Variables should be declared at the beginning of the block.

> +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
> +		"Failed to get thread affinity");
> +
> +	rte_cpuset_t cpuset1;
> +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
> +		"Failed to get thread affinity");
> +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> +		"Affinity should be stable");
> +
> +	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
> +		"Failed to set thread affinity");
> +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
> +		"Failed to get thread affinity");
> +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> +		"Affinity should be stable");
> +
> +	size_t i;
> +	for (i = 1; i < CPU_SETSIZE; i++)
> +		if (CPU_ISSET(i, &cpuset0)) {
> +			CPU_ZERO(&cpuset0);
> +			CPU_SET(i, &cpuset0);
> +
> +			break;
> +		}
> +	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
> +		"Failed to set thread affinity");
> +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
> +		"Failed to get thread affinity");
> +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> +		"Affinity should be stable");

The message is not really relevant to the check done.
"Retrieved affinity differs from requested"?

I think this is the only check worth keeping in this test.
The fist one is speculative: if we expect that a wrong implementation may
change affinity sporadically, the test can't prove it doesn't.
The second one isn't stronger than this one.

> +
> +	return 0;
> +}
> +
> +static struct unit_test_suite threads_test_suite = {
> +	.suite_name = "threads autotest",
> +	.setup = NULL,
> +	.teardown = NULL,
> +	.unit_test_cases = {
> +		TEST_CASE(test_thread_affinity),
> +		TEST_CASES_END()
> +	}
> +};
> +
> +static int
> +test_threads(void)
> +{
> +	return unit_test_suite_runner(&threads_test_suite);
> +}
> +
> +REGISTER_TEST_COMMAND(threads_autotest, test_threads);

Re: [PATCH v4 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

On Sun, May 01, 2022 at 12:18:36PM +0300, Dmitry Kozlyuk wrote:
> 2022-04-26 00:50 (UTC-0700), Tyler Retzlaff:
> > Establish unit test for testing thread api. Initial unit tests
> > for rte_thread_{get,set}_affinity_by_id().
> > 
> > Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> > ---
> >  app/test/meson.build    |  2 ++
> >  app/test/test_threads.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 91 insertions(+)
> >  create mode 100644 app/test/test_threads.c
> > 
> > diff --git a/app/test/meson.build b/app/test/meson.build
> > index 5fc1dd1..5a9d69b 100644
> > --- a/app/test/meson.build
> > +++ b/app/test/meson.build
> > @@ -133,6 +133,7 @@ test_sources = files(
> >          'test_tailq.c',
> >          'test_thash.c',
> >          'test_thash_perf.c',
> > +        'test_threads.c',
> >          'test_timer.c',
> >          'test_timer_perf.c',
> >          'test_timer_racecond.c',
> > @@ -238,6 +239,7 @@ fast_tests = [
> >          ['reorder_autotest', true],
> >          ['service_autotest', true],
> >          ['thash_autotest', true],
> > +        ['threads_autotest', true],
> >          ['trace_autotest', true],
> >  ]
> >  
> > diff --git a/app/test/test_threads.c b/app/test/test_threads.c
> > new file mode 100644
> > index 0000000..0ca6745
> > --- /dev/null
> > +++ b/app/test/test_threads.c
> > @@ -0,0 +1,89 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright (C) 2022 Microsoft Corporation
> > + */
> > +
> > +#include <string.h>
> > +#include <pthread.h>
> > +
> > +#include <rte_thread.h>
> > +#include <rte_debug.h>
> > +
> > +#include "test.h"
> > +
> > +RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
> > +
> > +static uint32_t thread_id_ready;
> > +
> > +static void *
> > +thread_main(void *arg)
> > +{
> > +	*(rte_thread_t *)arg = rte_thread_self();
> > +	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
> > +
> > +	return NULL;
> > +}
> > +
> > +static int
> > +test_thread_affinity(void)
> > +{
> > +	pthread_t id;
> > +	rte_thread_t thread_id;
> > +
> > +	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
> > +		"Failed to create thread");
> > +
> > +	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
> > +		;
> > +
> > +	rte_cpuset_t cpuset0;
> 
> Variables should be declared at the beginning of the block.

just curious because we don't require c99 compiler or because it is
not compliant with dpdk style/convention?

regardless, will fix just curious.

> 
> > +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
> > +		"Failed to get thread affinity");
> > +
> > +	rte_cpuset_t cpuset1;
> > +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
> > +		"Failed to get thread affinity");
> > +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> > +		"Affinity should be stable");
> > +
> > +	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset1) == 0,
> > +		"Failed to set thread affinity");
> > +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
> > +		"Failed to get thread affinity");
> > +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> > +		"Affinity should be stable");
> > +
> > +	size_t i;
> > +	for (i = 1; i < CPU_SETSIZE; i++)
> > +		if (CPU_ISSET(i, &cpuset0)) {
> > +			CPU_ZERO(&cpuset0);
> > +			CPU_SET(i, &cpuset0);
> > +
> > +			break;
> > +		}
> > +	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
> > +		"Failed to set thread affinity");
> > +	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
> > +		"Failed to get thread affinity");
> > +	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
> > +		"Affinity should be stable");
> 
> The message is not really relevant to the check done.
> "Retrieved affinity differs from requested"?

will fix.

> 
> I think this is the only check worth keeping in this test.
> The fist one is speculative: if we expect that a wrong implementation may
> change affinity sporadically, the test can't prove it doesn't.

if you're saying it isn't deterministic i partially agree. regardless
many bugs are not deterministic. e.g. composite/split initialization bugs
as a class, there could be side-effects get/get since the function is
not pure.

enforcing a rule that tests only ever fail for deterministic reasons
would eliminate opportunity to hint at various classes of initialization
bugs. as an example right now pthread_create shim has one of these class
of bugs and there is no deterministic test that guarantees 100% failure
though a non-deterministic test enabled today would show a high rate of
failure warranting investigation before now.

the only downside is that if a test like this does fail it is possible
to be unrelated to the implementation of the feature being tested and
that can be misleading and i admit frustrating.

> The second one isn't stronger than this one.

i agree that the set/get/compare is duplicated i'll remove it the
first occurence but the get/get/compare i think should stay.

> 
> > +
> > +	return 0;
> > +}
> > +
> > +static struct unit_test_suite threads_test_suite = {
> > +	.suite_name = "threads autotest",
> > +	.setup = NULL,
> > +	.teardown = NULL,
> > +	.unit_test_cases = {
> > +		TEST_CASE(test_thread_affinity),
> > +		TEST_CASES_END()
> > +	}
> > +};
> > +
> > +static int
> > +test_threads(void)
> > +{
> > +	return unit_test_suite_runner(&threads_test_suite);
> > +}
> > +
> > +REGISTER_TEST_COMMAND(threads_autotest, test_threads);

[PATCH v5 0/3] add eal functions for thread affinity and self

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes

* basic platform error number conversion.
* function to get current thread identifier.
* functions to get and set affinity with platform agnostic thread
  identifier.
* minimal unit test of get and set affinity demonstrating usage.

note: previous series introducing these functions is now superseded by
this series.
http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*

v5:
* use rte_log instead of log_early. it appears that logging is
  now initialized before the call to eal_query_group_affinity.
  note: removal of log_early is off-topic for this series and
        will be done separately.
* rte_log DEBUG for failure related to limitation where all
  processors must be in the same processor group.
* remove redundant tests. get/get/compare, set/get/compare
  are both retained.

v4:
* combine patch eal/windows: translate Windows errors to errno-style
  errors into eal: implement functions for get/set thread affinity
  patch. the former introduced static functions that were not used
  without eal: implement functions for get/set thread affinity which
  would cause a build break when applied standalone.
* remove struct tag from rte_thread_t struct typedef.
* remove rte_ prefix from rte_convert_cpuset_to_affinity static
  function.

v3:
* fix memory leak on eal_create_cpu_map error paths.

v2:
* add missing boilerplate comments warning of experimental api
  for rte_thread_{set,get}_affinity_by_id().
* don't break literal format string to log_early to improve
  searchability.
* fix multi-line comment style to match file.
* return ENOTSUP instead of EINVAL from rte_convert_cpuset_to_affinity()
  if cpus in set are not part of the same processor group and note
  limitation in commit message.
* expand series to include rte_thread_self().
* modify unit test to remove use of implementation detail and
  get thread identifier use added rte_thread_self().
* move literal value to rhs when using memcmp in RTE_TEST_ASSERT

Tyler Retzlaff (3):
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  81 +++++++++++++++++
 lib/eal/include/rte_thread.h     |  64 +++++++++++++
 lib/eal/unix/rte_thread.c        |  27 ++++++
 lib/eal/version.map              |   5 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 ++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 191 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 515 insertions(+), 48 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

Provide a portable type-safe thread identifier.
Provide rte_thread_self for obtaining current thread identifier.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
 lib/eal/unix/rte_thread.c    | 11 +++++++++++
 lib/eal/version.map          |  3 +++
 lib/eal/windows/rte_thread.c | 10 ++++++++++
 4 files changed, 46 insertions(+)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..14478ba 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -1,7 +1,10 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,10 +24,29 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the id of the calling thread.
+ *
+ * @return
+ *   Return the thread id of the calling thread.
+ */
+__rte_experimental
+rte_thread_t rte_thread_self(void);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..82e008f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <errno.h>
@@ -15,6 +16,16 @@ struct eal_tls_key {
 	pthread_key_t thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = (uintptr_t)pthread_self();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
 {
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..05ce8f9 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,9 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_self;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..59fed3c 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -11,6 +11,16 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = GetCurrentThreadId();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v5 2/3] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

Windows error codes are translated to errno-style error codes.
The possible return values are chosen so that we have as
much semantic compatibility between platforms as possible.

note: convert_cpuset_to_affinity has a limitation that all cpus of
the set belong to the same processor group.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/eal/include/rte_thread.h     |  42 +++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 181 ++++++++++++++++++++++++++++++++++++++-
 7 files changed, 386 insertions(+), 48 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 14478ba..7888f7a 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -50,6 +50,48 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index 82e008f..c64198f 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -100,3 +100,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 05ce8f9..d49e30b 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -422,7 +422,9 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_get_affinity_by_id;
 	rte_thread_self;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..286fe24 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -1,8 +1,8 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2019 Intel Corporation
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +27,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +49,115 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			RTE_LOG(ERR, EAL, "Cannot get group information size, error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		RTE_LOG(ERR, EAL, "Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/*
+	 * NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++)
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -70,7 +175,8 @@ struct cpu_map {
 	if (infos == NULL) {
 		log_early("Cannot allocate memory for NUMA node information\n");
 		rte_errno = ENOMEM;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	if (!GetLogicalProcessorInformationEx(
@@ -78,57 +184,30 @@ struct cpu_map {
 		log_early("Cannot get NUMA node information, error %lu\n",
 			GetLastError());
 		rte_errno = EINVAL;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -138,7 +217,7 @@ struct cpu_map {
 
 	free(infos);
 
-	return 0;
+	return ret;
 }
 
 int
@@ -164,3 +243,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index e4c4670..ab25814 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -58,6 +58,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 59fed3c..a616703 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -1,16 +1,66 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
@@ -97,3 +147,132 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			RTE_LOG(DEBUG, EAL, "All processors must belong to the same processor group\n");
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH v5 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1..5a9d69b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -238,6 +239,7 @@ fast_tests = [
         ['reorder_autotest', true],
         ['service_autotest', true],
         ['thash_autotest', true],
+        ['threads_autotest', true],
         ['trace_autotest', true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..e1a2ea5
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022 Microsoft Corporation
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static uint32_t thread_id_ready;
+
+static void *
+thread_main(void *arg)
+{
+	*(rte_thread_t *)arg = rte_thread_self();
+	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
+
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+		"Failed to create thread");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

04/05/2022 16:46, Tyler Retzlaff пишет:
> Provide a portable type-safe thread identifier.
> Provide rte_thread_self for obtaining current thread identifier.
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>   lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>   lib/eal/unix/rte_thread.c    | 11 +++++++++++
>   lib/eal/version.map          |  3 +++
>   lib/eal/windows/rte_thread.c | 10 ++++++++++
>   4 files changed, 46 insertions(+)
> 
> diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> index 8be8ed8..14478ba 100644
> --- a/lib/eal/include/rte_thread.h
> +++ b/lib/eal/include/rte_thread.h
> @@ -1,7 +1,10 @@
>   /* SPDX-License-Identifier: BSD-3-Clause
>    * Copyright(c) 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>    */
>   
> +#include <stdint.h>
> +
>   #include <rte_os.h>
>   #include <rte_compat.h>
>   
> @@ -21,10 +24,29 @@
>   #endif
>   
>   /**
> + * Thread id descriptor.
> + */
> +typedef struct {
> +	uintptr_t opaque_id; /**< thread identifier */


I know that currently on linux typeof(pthread_id) == unsigned long int.
Though wouldn't it be safer and cleaner to use pthread_t explicitly on 
posix-like systems?
Something like:
typedef struct {
#ifdef WINDOWS
	uintptr_t opaque_id;
#else
	pthread_t opaque_id;
#endif
};
AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic type'.


> +} rte_thread_t;
> +
> +/**
>    * TLS key type, an opaque pointer.
>    */
>   typedef struct eal_tls_key *rte_thread_key;
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Get the id of the calling thread.
> + *
> + * @return
> + *   Return the thread id of the calling thread.
> + */
> +__rte_experimental
> +rte_thread_t rte_thread_self(void);
> +
>   #ifdef RTE_HAS_CPUSET
>   
>   /**
> diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
> index c34ede9..82e008f 100644
> --- a/lib/eal/unix/rte_thread.c
> +++ b/lib/eal/unix/rte_thread.c
> @@ -1,5 +1,6 @@
>   /* SPDX-License-Identifier: BSD-3-Clause
>    * Copyright 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>    */
>   
>   #include <errno.h>
> @@ -15,6 +16,16 @@ struct eal_tls_key {
>   	pthread_key_t thread_index;
>   };
>   
> +rte_thread_t
> +rte_thread_self(void)
> +{
> +	rte_thread_t thread_id;
> +
> +	thread_id.opaque_id = (uintptr_t)pthread_self();
> +
> +	return thread_id;
> +}
> +
>   int
>   rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
>   {
> diff --git a/lib/eal/version.map b/lib/eal/version.map
> index b53eeb3..05ce8f9 100644
> --- a/lib/eal/version.map
> +++ b/lib/eal/version.map
> @@ -420,6 +420,9 @@ EXPERIMENTAL {
>   	rte_intr_instance_free;
>   	rte_intr_type_get;
>   	rte_intr_type_set;
> +
> +	# added in 22.07
> +	rte_thread_self;
>   };
>   
>   INTERNAL {
> diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> index 667287c..59fed3c 100644
> --- a/lib/eal/windows/rte_thread.c
> +++ b/lib/eal/windows/rte_thread.c
> @@ -11,6 +11,16 @@ struct eal_tls_key {
>   	DWORD thread_index;
>   };
>   
> +rte_thread_t
> +rte_thread_self(void)
> +{
> +	rte_thread_t thread_id;
> +
> +	thread_id.opaque_id = GetCurrentThreadId();
> +
> +	return thread_id;
> +}
> +
>   int
>   rte_thread_key_create(rte_thread_key *key,
>   		__rte_unused void (*destructor)(void *))

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
> 04/05/2022 16:46, Tyler Retzlaff пишет:
> >Provide a portable type-safe thread identifier.
> >Provide rte_thread_self for obtaining current thread identifier.
> >
> >Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> >---
> >  lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >  lib/eal/unix/rte_thread.c    | 11 +++++++++++
> >  lib/eal/version.map          |  3 +++
> >  lib/eal/windows/rte_thread.c | 10 ++++++++++
> >  4 files changed, 46 insertions(+)
> >
> >diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> >index 8be8ed8..14478ba 100644
> >--- a/lib/eal/include/rte_thread.h
> >+++ b/lib/eal/include/rte_thread.h
> >@@ -1,7 +1,10 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright(c) 2021 Mellanox Technologies, Ltd
> >+ * Copyright (C) 2022 Microsoft Corporation
> >   */
> >+#include <stdint.h>
> >+
> >  #include <rte_os.h>
> >  #include <rte_compat.h>
> >@@ -21,10 +24,29 @@
> >  #endif
> >  /**
> >+ * Thread id descriptor.
> >+ */
> >+typedef struct {
> >+	uintptr_t opaque_id; /**< thread identifier */
> 
> 
> I know that currently on linux typeof(pthread_id) == unsigned long int.
> Though wouldn't it be safer and cleaner to use pthread_t explicitly
> on posix-like systems?

i believe the previous discussions are.

* preference for reduced or no conditional compilation.
* preference for sizeof(type) to be `the same' on all platforms.
* preference for platform agnostic headers. i.e. don't drag
  platform specific headers into the application namespace when
  including rte_xxx.h headers.

> Something like:
> typedef struct {
> #ifdef WINDOWS
> 	uintptr_t opaque_id;
> #else
> 	pthread_t opaque_id;
> #endif
> };
> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic type'.

yes, this is correct. newer posix introduced this to allow the use of
structs. i assume prior reviewers are aware of the recent posix
standard (or should be).

this type makes no attempt to be usable on platforms that use
a handle > sizeof(uintptr_t). though any platform that does is free
to shove a pointer to struct into the handle at the cost of a
dereference if that is their implementation.

> 
> 
> >+} rte_thread_t;
> >+
> >+/**
> >   * TLS key type, an opaque pointer.
> >   */
> >  typedef struct eal_tls_key *rte_thread_key;
> >+/**
> >+ * @warning
> >+ * @b EXPERIMENTAL: this API may change without prior notice.
> >+ *
> >+ * Get the id of the calling thread.
> >+ *
> >+ * @return
> >+ *   Return the thread id of the calling thread.
> >+ */
> >+__rte_experimental
> >+rte_thread_t rte_thread_self(void);
> >+
> >  #ifdef RTE_HAS_CPUSET
> >  /**
> >diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
> >index c34ede9..82e008f 100644
> >--- a/lib/eal/unix/rte_thread.c
> >+++ b/lib/eal/unix/rte_thread.c
> >@@ -1,5 +1,6 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright 2021 Mellanox Technologies, Ltd
> >+ * Copyright (C) 2022 Microsoft Corporation
> >   */
> >  #include <errno.h>
> >@@ -15,6 +16,16 @@ struct eal_tls_key {
> >  	pthread_key_t thread_index;
> >  };
> >+rte_thread_t
> >+rte_thread_self(void)
> >+{
> >+	rte_thread_t thread_id;
> >+
> >+	thread_id.opaque_id = (uintptr_t)pthread_self();
> >+
> >+	return thread_id;
> >+}
> >+
> >  int
> >  rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
> >  {
> >diff --git a/lib/eal/version.map b/lib/eal/version.map
> >index b53eeb3..05ce8f9 100644
> >--- a/lib/eal/version.map
> >+++ b/lib/eal/version.map
> >@@ -420,6 +420,9 @@ EXPERIMENTAL {
> >  	rte_intr_instance_free;
> >  	rte_intr_type_get;
> >  	rte_intr_type_set;
> >+
> >+	# added in 22.07
> >+	rte_thread_self;
> >  };
> >  INTERNAL {
> >diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> >index 667287c..59fed3c 100644
> >--- a/lib/eal/windows/rte_thread.c
> >+++ b/lib/eal/windows/rte_thread.c
> >@@ -11,6 +11,16 @@ struct eal_tls_key {
> >  	DWORD thread_index;
> >  };
> >+rte_thread_t
> >+rte_thread_self(void)
> >+{
> >+	rte_thread_t thread_id;
> >+
> >+	thread_id.opaque_id = GetCurrentThreadId();
> >+
> >+	return thread_id;
> >+}
> >+
> >  int
> >  rte_thread_key_create(rte_thread_key *key,
> >  		__rte_unused void (*destructor)(void *))

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

05/05/2022 08:11, Tyler Retzlaff пишет:
> On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
>> 04/05/2022 16:46, Tyler Retzlaff пишет:
>>> Provide a portable type-safe thread identifier.
>>> Provide rte_thread_self for obtaining current thread identifier.
>>>
>>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
>>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
>>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
>>> ---
>>>   lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>>>   lib/eal/unix/rte_thread.c    | 11 +++++++++++
>>>   lib/eal/version.map          |  3 +++
>>>   lib/eal/windows/rte_thread.c | 10 ++++++++++
>>>   4 files changed, 46 insertions(+)
>>>
>>> diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
>>> index 8be8ed8..14478ba 100644
>>> --- a/lib/eal/include/rte_thread.h
>>> +++ b/lib/eal/include/rte_thread.h
>>> @@ -1,7 +1,10 @@
>>>   /* SPDX-License-Identifier: BSD-3-Clause
>>>    * Copyright(c) 2021 Mellanox Technologies, Ltd
>>> + * Copyright (C) 2022 Microsoft Corporation
>>>    */
>>> +#include <stdint.h>
>>> +
>>>   #include <rte_os.h>
>>>   #include <rte_compat.h>
>>> @@ -21,10 +24,29 @@
>>>   #endif
>>>   /**
>>> + * Thread id descriptor.
>>> + */
>>> +typedef struct {
>>> +	uintptr_t opaque_id; /**< thread identifier */
>>
>>
>> I know that currently on linux typeof(pthread_id) == unsigned long int.
>> Though wouldn't it be safer and cleaner to use pthread_t explicitly
>> on posix-like systems?
> 
> i believe the previous discussions are.
> 
> * preference for reduced or no conditional compilation.
> * preference for sizeof(type) to be `the same' on all platforms.


It would be the same as long as sizes of pthread_t uintptr_t are equal.

> * preference for platform agnostic headers. i.e. don't drag
>    platform specific headers into the application namespace when
>    including rte_xxx.h headers.
>> Something like:
>> typedef struct {
>> #ifdef WINDOWS
>> 	uintptr_t opaque_id;
>> #else
>> 	pthread_t opaque_id;
>> #endif
>> };
>> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic type'.
> 
> yes, this is correct. newer posix introduced this to allow the use of
> structs. i assume prior reviewers are aware of the recent posix
> standard (or should be).
> 
> this type makes no attempt to be usable on platforms that use
> a handle > sizeof(uintptr_t). though any platform that does is free
> to shove a pointer to struct into the handle at the cost of a
> dereference if that is their implementation.

Using pthread_t directly still seems like a safest bet to me.
Then we can avoid doing these explicit type conversions before/after 
each pthread_xxx() call and wouldn't need to worry if we'll ever have 
platform with bigger pthread_t (though yes, I admit it is very unlikely).
But, if we still prefer to go ahead with 'arch-neutral' approach,
then I think we need to have compilation time check that opaque_id
is big enough to hold pthread_t value:
RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.


> 
>>
>>
>>> +} rte_thread_t;
>>> +
>>> +/**
>>>    * TLS key type, an opaque pointer.
>>>    */
>>>   typedef struct eal_tls_key *rte_thread_key;
>>> +/**
>>> + * @warning
>>> + * @b EXPERIMENTAL: this API may change without prior notice.
>>> + *
>>> + * Get the id of the calling thread.
>>> + *
>>> + * @return
>>> + *   Return the thread id of the calling thread.
>>> + */
>>> +__rte_experimental
>>> +rte_thread_t rte_thread_self(void);
>>> +
>>>   #ifdef RTE_HAS_CPUSET
>>>   /**
>>> diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
>>> index c34ede9..82e008f 100644
>>> --- a/lib/eal/unix/rte_thread.c
>>> +++ b/lib/eal/unix/rte_thread.c
>>> @@ -1,5 +1,6 @@
>>>   /* SPDX-License-Identifier: BSD-3-Clause
>>>    * Copyright 2021 Mellanox Technologies, Ltd
>>> + * Copyright (C) 2022 Microsoft Corporation
>>>    */
>>>   #include <errno.h>
>>> @@ -15,6 +16,16 @@ struct eal_tls_key {
>>>   	pthread_key_t thread_index;
>>>   };
>>> +rte_thread_t
>>> +rte_thread_self(void)
>>> +{
>>> +	rte_thread_t thread_id;
>>> +
>>> +	thread_id.opaque_id = (uintptr_t)pthread_self();
>>> +
>>> +	return thread_id;
>>> +}
>>> +
>>>   int
>>>   rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
>>>   {
>>> diff --git a/lib/eal/version.map b/lib/eal/version.map
>>> index b53eeb3..05ce8f9 100644
>>> --- a/lib/eal/version.map
>>> +++ b/lib/eal/version.map
>>> @@ -420,6 +420,9 @@ EXPERIMENTAL {
>>>   	rte_intr_instance_free;
>>>   	rte_intr_type_get;
>>>   	rte_intr_type_set;
>>> +
>>> +	# added in 22.07
>>> +	rte_thread_self;
>>>   };
>>>   INTERNAL {
>>> diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
>>> index 667287c..59fed3c 100644
>>> --- a/lib/eal/windows/rte_thread.c
>>> +++ b/lib/eal/windows/rte_thread.c
>>> @@ -11,6 +11,16 @@ struct eal_tls_key {
>>>   	DWORD thread_index;
>>>   };
>>> +rte_thread_t
>>> +rte_thread_self(void)
>>> +{
>>> +	rte_thread_t thread_id;
>>> +
>>> +	thread_id.opaque_id = GetCurrentThreadId();
>>> +
>>> +	return thread_id;
>>> +}
>>> +
>>>   int
>>>   rte_thread_key_create(rte_thread_key *key,
>>>   		__rte_unused void (*destructor)(void *))

RE: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Morten Brørup]

> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> Sent: Friday, 6 May 2022 21.38
> 
> 05/05/2022 08:11, Tyler Retzlaff пишет:
> > On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
> >> 04/05/2022 16:46, Tyler Retzlaff пишет:
> >>> Provide a portable type-safe thread identifier.
> >>> Provide rte_thread_self for obtaining current thread identifier.
> >>>
> >>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> >>> ---
> >>>   lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >>>   lib/eal/unix/rte_thread.c    | 11 +++++++++++
> >>>   lib/eal/version.map          |  3 +++
> >>>   lib/eal/windows/rte_thread.c | 10 ++++++++++
> >>>   4 files changed, 46 insertions(+)
> >>>
> >>> diff --git a/lib/eal/include/rte_thread.h
> b/lib/eal/include/rte_thread.h
> >>> index 8be8ed8..14478ba 100644
> >>> --- a/lib/eal/include/rte_thread.h
> >>> +++ b/lib/eal/include/rte_thread.h
> >>> @@ -1,7 +1,10 @@
> >>>   /* SPDX-License-Identifier: BSD-3-Clause
> >>>    * Copyright(c) 2021 Mellanox Technologies, Ltd
> >>> + * Copyright (C) 2022 Microsoft Corporation
> >>>    */
> >>> +#include <stdint.h>
> >>> +
> >>>   #include <rte_os.h>
> >>>   #include <rte_compat.h>
> >>> @@ -21,10 +24,29 @@
> >>>   #endif
> >>>   /**
> >>> + * Thread id descriptor.
> >>> + */
> >>> +typedef struct {
> >>> +	uintptr_t opaque_id; /**< thread identifier */
> >>
> >>
> >> I know that currently on linux typeof(pthread_id) == unsigned long
> int.
> >> Though wouldn't it be safer and cleaner to use pthread_t explicitly
> >> on posix-like systems?
> >
> > i believe the previous discussions are.
> >
> > * preference for reduced or no conditional compilation.
> > * preference for sizeof(type) to be `the same' on all platforms.
> 
> 
> It would be the same as long as sizes of pthread_t uintptr_t are equal.

They are not. pthread_t (Linux thread ID) and DWORD (Windows thread ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer size (32 or 64 bit CPU).

> 
> > * preference for platform agnostic headers. i.e. don't drag
> >    platform specific headers into the application namespace when
> >    including rte_xxx.h headers.
> >> Something like:
> >> typedef struct {
> >> #ifdef WINDOWS
> >> 	uintptr_t opaque_id;
> >> #else
> >> 	pthread_t opaque_id;
> >> #endif
> >> };
> >> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
> type'.
> >
> > yes, this is correct. newer posix introduced this to allow the use of
> > structs. i assume prior reviewers are aware of the recent posix
> > standard (or should be).
> >
> > this type makes no attempt to be usable on platforms that use
> > a handle > sizeof(uintptr_t). though any platform that does is free
> > to shove a pointer to struct into the handle at the cost of a
> > dereference if that is their implementation.
> 
> Using pthread_t directly still seems like a safest bet to me.
> Then we can avoid doing these explicit type conversions before/after
> each pthread_xxx() call and wouldn't need to worry if we'll ever have
> platform with bigger pthread_t (though yes, I admit it is very
> unlikely).
> But, if we still prefer to go ahead with 'arch-neutral' approach,
> then I think we need to have compilation time check that opaque_id
> is big enough to hold pthread_t value:
> RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
> 

I agree with Konstantin's concerns. All this type casting increases the risk for bugs. Also, I don't understand the need to use a 64 bit value when thread IDs are 32 bit in both Linux and Windows.

The thread handling is O/S specific code, so #ifdef is unavoidable.

Furthermore, I don't think we should wrap O/S specific integer types into structs - it adds unnecessary complexity.

I would prefer:

#ifdef WINDOWS
typedef DWORD rte_thread_t;
#else
typedef pthread_t rte_thread_t;
#endif

> 
> >
> >>
> >>
> >>> +} rte_thread_t;
> >>> +
> >>> +/**
> >>>    * TLS key type, an opaque pointer.
> >>>    */
> >>>   typedef struct eal_tls_key *rte_thread_key;
> >>> +/**
> >>> + * @warning
> >>> + * @b EXPERIMENTAL: this API may change without prior notice.
> >>> + *
> >>> + * Get the id of the calling thread.
> >>> + *
> >>> + * @return
> >>> + *   Return the thread id of the calling thread.
> >>> + */
> >>> +__rte_experimental
> >>> +rte_thread_t rte_thread_self(void);
> >>> +
> >>>   #ifdef RTE_HAS_CPUSET
> >>>   /**
> >>> diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
> >>> index c34ede9..82e008f 100644
> >>> --- a/lib/eal/unix/rte_thread.c
> >>> +++ b/lib/eal/unix/rte_thread.c
> >>> @@ -1,5 +1,6 @@
> >>>   /* SPDX-License-Identifier: BSD-3-Clause
> >>>    * Copyright 2021 Mellanox Technologies, Ltd
> >>> + * Copyright (C) 2022 Microsoft Corporation
> >>>    */
> >>>   #include <errno.h>
> >>> @@ -15,6 +16,16 @@ struct eal_tls_key {
> >>>   	pthread_key_t thread_index;
> >>>   };
> >>> +rte_thread_t
> >>> +rte_thread_self(void)
> >>> +{
> >>> +	rte_thread_t thread_id;
> >>> +
> >>> +	thread_id.opaque_id = (uintptr_t)pthread_self();
> >>> +
> >>> +	return thread_id;
> >>> +}
> >>> +
> >>>   int
> >>>   rte_thread_key_create(rte_thread_key *key, void
> (*destructor)(void *))
> >>>   {
> >>> diff --git a/lib/eal/version.map b/lib/eal/version.map
> >>> index b53eeb3..05ce8f9 100644
> >>> --- a/lib/eal/version.map
> >>> +++ b/lib/eal/version.map
> >>> @@ -420,6 +420,9 @@ EXPERIMENTAL {
> >>>   	rte_intr_instance_free;
> >>>   	rte_intr_type_get;
> >>>   	rte_intr_type_set;
> >>> +
> >>> +	# added in 22.07
> >>> +	rte_thread_self;
> >>>   };
> >>>   INTERNAL {
> >>> diff --git a/lib/eal/windows/rte_thread.c
> b/lib/eal/windows/rte_thread.c
> >>> index 667287c..59fed3c 100644
> >>> --- a/lib/eal/windows/rte_thread.c
> >>> +++ b/lib/eal/windows/rte_thread.c
> >>> @@ -11,6 +11,16 @@ struct eal_tls_key {
> >>>   	DWORD thread_index;
> >>>   };
> >>> +rte_thread_t
> >>> +rte_thread_self(void)
> >>> +{
> >>> +	rte_thread_t thread_id;
> >>> +
> >>> +	thread_id.opaque_id = GetCurrentThreadId();
> >>> +
> >>> +	return thread_id;
> >>> +}
> >>> +
> >>>   int
> >>>   rte_thread_key_create(rte_thread_key *key,
> >>>   		__rte_unused void (*destructor)(void *))
> 

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

Hi Morten,

>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
>> Sent: Friday, 6 May 2022 21.38
>>
>> 05/05/2022 08:11, Tyler Retzlaff пишет:
>>> On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
>>>> 04/05/2022 16:46, Tyler Retzlaff пишет:
>>>>> Provide a portable type-safe thread identifier.
>>>>> Provide rte_thread_self for obtaining current thread identifier.
>>>>>
>>>>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
>>>>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
>>>>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
>>>>> ---
>>>>>    lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>>>>>    lib/eal/unix/rte_thread.c    | 11 +++++++++++
>>>>>    lib/eal/version.map          |  3 +++
>>>>>    lib/eal/windows/rte_thread.c | 10 ++++++++++
>>>>>    4 files changed, 46 insertions(+)
>>>>>
>>>>> diff --git a/lib/eal/include/rte_thread.h
>> b/lib/eal/include/rte_thread.h
>>>>> index 8be8ed8..14478ba 100644
>>>>> --- a/lib/eal/include/rte_thread.h
>>>>> +++ b/lib/eal/include/rte_thread.h
>>>>> @@ -1,7 +1,10 @@
>>>>>    /* SPDX-License-Identifier: BSD-3-Clause
>>>>>     * Copyright(c) 2021 Mellanox Technologies, Ltd
>>>>> + * Copyright (C) 2022 Microsoft Corporation
>>>>>     */
>>>>> +#include <stdint.h>
>>>>> +
>>>>>    #include <rte_os.h>
>>>>>    #include <rte_compat.h>
>>>>> @@ -21,10 +24,29 @@
>>>>>    #endif
>>>>>    /**
>>>>> + * Thread id descriptor.
>>>>> + */
>>>>> +typedef struct {
>>>>> +	uintptr_t opaque_id; /**< thread identifier */
>>>>
>>>>
>>>> I know that currently on linux typeof(pthread_id) == unsigned long
>> int.
>>>> Though wouldn't it be safer and cleaner to use pthread_t explicitly
>>>> on posix-like systems?
>>>
>>> i believe the previous discussions are.
>>>
>>> * preference for reduced or no conditional compilation.
>>> * preference for sizeof(type) to be `the same' on all platforms.
>>
>>
>> It would be the same as long as sizes of pthread_t uintptr_t are equal.
> 
> They are not. pthread_t (Linux thread ID) and DWORD (Windows thread ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer size (32 or 64 bit CPU).

What make you think pthread_t is 32-bit on linux?
 From <pthread.h> on my box:
typedef unsigned long int pthread_t;
So it is either 64-bit or 32-bit depending on arch.
Same as uintptr_t.

> 
>>
>>> * preference for platform agnostic headers. i.e. don't drag
>>>     platform specific headers into the application namespace when
>>>     including rte_xxx.h headers.
>>>> Something like:
>>>> typedef struct {
>>>> #ifdef WINDOWS
>>>> 	uintptr_t opaque_id;
>>>> #else
>>>> 	pthread_t opaque_id;
>>>> #endif
>>>> };
>>>> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
>> type'.
>>>
>>> yes, this is correct. newer posix introduced this to allow the use of
>>> structs. i assume prior reviewers are aware of the recent posix
>>> standard (or should be).
>>>
>>> this type makes no attempt to be usable on platforms that use
>>> a handle > sizeof(uintptr_t). though any platform that does is free
>>> to shove a pointer to struct into the handle at the cost of a
>>> dereference if that is their implementation.
>>
>> Using pthread_t directly still seems like a safest bet to me.
>> Then we can avoid doing these explicit type conversions before/after
>> each pthread_xxx() call and wouldn't need to worry if we'll ever have
>> platform with bigger pthread_t (though yes, I admit it is very
>> unlikely).
>> But, if we still prefer to go ahead with 'arch-neutral' approach,
>> then I think we need to have compilation time check that opaque_id
>> is big enough to hold pthread_t value:
>> RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
>>
> 
> I agree with Konstantin's concerns. All this type casting increases the risk for bugs. Also, I don't understand the need to use a 64 bit value when thread IDs are 32 bit in both Linux and Windows.
> 
> The thread handling is O/S specific code, so #ifdef is unavoidable.
> 
> Furthermore, I don't think we should wrap O/S specific integer types into structs - it adds unnecessary complexity.
> 
> I would prefer:
> 
> #ifdef WINDOWS
> typedef DWORD rte_thread_t;
> #else
> typedef pthread_t rte_thread_t;
> #endif
> 
>>
>>>
>>>>
>>>>
>>>>> +} rte_thread_t;
>>>>> +
>>>>> +/**
>>>>>     * TLS key type, an opaque pointer.
>>>>>     */
>>>>>    typedef struct eal_tls_key *rte_thread_key;
>>>>> +/**
>>>>> + * @warning
>>>>> + * @b EXPERIMENTAL: this API may change without prior notice.
>>>>> + *
>>>>> + * Get the id of the calling thread.
>>>>> + *
>>>>> + * @return
>>>>> + *   Return the thread id of the calling thread.
>>>>> + */
>>>>> +__rte_experimental
>>>>> +rte_thread_t rte_thread_self(void);
>>>>> +
>>>>>    #ifdef RTE_HAS_CPUSET
>>>>>    /**
>>>>> diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
>>>>> index c34ede9..82e008f 100644
>>>>> --- a/lib/eal/unix/rte_thread.c
>>>>> +++ b/lib/eal/unix/rte_thread.c
>>>>> @@ -1,5 +1,6 @@
>>>>>    /* SPDX-License-Identifier: BSD-3-Clause
>>>>>     * Copyright 2021 Mellanox Technologies, Ltd
>>>>> + * Copyright (C) 2022 Microsoft Corporation
>>>>>     */
>>>>>    #include <errno.h>
>>>>> @@ -15,6 +16,16 @@ struct eal_tls_key {
>>>>>    	pthread_key_t thread_index;
>>>>>    };
>>>>> +rte_thread_t
>>>>> +rte_thread_self(void)
>>>>> +{
>>>>> +	rte_thread_t thread_id;
>>>>> +
>>>>> +	thread_id.opaque_id = (uintptr_t)pthread_self();
>>>>> +
>>>>> +	return thread_id;
>>>>> +}
>>>>> +
>>>>>    int
>>>>>    rte_thread_key_create(rte_thread_key *key, void
>> (*destructor)(void *))
>>>>>    {
>>>>> diff --git a/lib/eal/version.map b/lib/eal/version.map
>>>>> index b53eeb3..05ce8f9 100644
>>>>> --- a/lib/eal/version.map
>>>>> +++ b/lib/eal/version.map
>>>>> @@ -420,6 +420,9 @@ EXPERIMENTAL {
>>>>>    	rte_intr_instance_free;
>>>>>    	rte_intr_type_get;
>>>>>    	rte_intr_type_set;
>>>>> +
>>>>> +	# added in 22.07
>>>>> +	rte_thread_self;
>>>>>    };
>>>>>    INTERNAL {
>>>>> diff --git a/lib/eal/windows/rte_thread.c
>> b/lib/eal/windows/rte_thread.c
>>>>> index 667287c..59fed3c 100644
>>>>> --- a/lib/eal/windows/rte_thread.c
>>>>> +++ b/lib/eal/windows/rte_thread.c
>>>>> @@ -11,6 +11,16 @@ struct eal_tls_key {
>>>>>    	DWORD thread_index;
>>>>>    };
>>>>> +rte_thread_t
>>>>> +rte_thread_self(void)
>>>>> +{
>>>>> +	rte_thread_t thread_id;
>>>>> +
>>>>> +	thread_id.opaque_id = GetCurrentThreadId();
>>>>> +
>>>>> +	return thread_id;
>>>>> +}
>>>>> +
>>>>>    int
>>>>>    rte_thread_key_create(rte_thread_key *key,
>>>>>    		__rte_unused void (*destructor)(void *))
>>
> 

RE: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Morten Brørup]

> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> Sent: Saturday, 7 May 2022 15.58
> 
> Hi Morten,
> 
> >> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> >> Sent: Friday, 6 May 2022 21.38
> >>
> >> 05/05/2022 08:11, Tyler Retzlaff пишет:
> >>> On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
> >>>> 04/05/2022 16:46, Tyler Retzlaff пишет:
> >>>>> Provide a portable type-safe thread identifier.
> >>>>> Provide rte_thread_self for obtaining current thread identifier.
> >>>>>
> >>>>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >>>>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >>>>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> >>>>> ---
> >>>>>    lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >>>>>    lib/eal/unix/rte_thread.c    | 11 +++++++++++
> >>>>>    lib/eal/version.map          |  3 +++
> >>>>>    lib/eal/windows/rte_thread.c | 10 ++++++++++
> >>>>>    4 files changed, 46 insertions(+)
> >>>>>
> >>>>> diff --git a/lib/eal/include/rte_thread.h
> >> b/lib/eal/include/rte_thread.h
> >>>>> index 8be8ed8..14478ba 100644
> >>>>> --- a/lib/eal/include/rte_thread.h
> >>>>> +++ b/lib/eal/include/rte_thread.h
> >>>>> @@ -1,7 +1,10 @@
> >>>>>    /* SPDX-License-Identifier: BSD-3-Clause
> >>>>>     * Copyright(c) 2021 Mellanox Technologies, Ltd
> >>>>> + * Copyright (C) 2022 Microsoft Corporation
> >>>>>     */
> >>>>> +#include <stdint.h>
> >>>>> +
> >>>>>    #include <rte_os.h>
> >>>>>    #include <rte_compat.h>
> >>>>> @@ -21,10 +24,29 @@
> >>>>>    #endif
> >>>>>    /**
> >>>>> + * Thread id descriptor.
> >>>>> + */
> >>>>> +typedef struct {
> >>>>> +	uintptr_t opaque_id; /**< thread identifier */
> >>>>
> >>>>
> >>>> I know that currently on linux typeof(pthread_id) == unsigned long
> >> int.
> >>>> Though wouldn't it be safer and cleaner to use pthread_t
> explicitly
> >>>> on posix-like systems?
> >>>
> >>> i believe the previous discussions are.
> >>>
> >>> * preference for reduced or no conditional compilation.
> >>> * preference for sizeof(type) to be `the same' on all platforms.
> >>
> >>
> >> It would be the same as long as sizes of pthread_t uintptr_t are
> equal.
> >
> > They are not. pthread_t (Linux thread ID) and DWORD (Windows thread
> ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer
> size (32 or 64 bit CPU).
> 
> What make you think pthread_t is 32-bit on linux?
>  From <pthread.h> on my box:
> typedef unsigned long int pthread_t;
> So it is either 64-bit or 32-bit depending on arch.
> Same as uintptr_t.

You are right, Konstantin. I had overlooked the "long" in there.

> 
> >
> >>
> >>> * preference for platform agnostic headers. i.e. don't drag
> >>>     platform specific headers into the application namespace when
> >>>     including rte_xxx.h headers.

So this is an exception from the "don't hide the types" rule that DPDK inherited from the Kernel. In theory, this makes really good sense for an EAL that really is what it says (i.e. an abstraction layer). In reality, though, this requires that the EAL offers all the features of the underlying O/S that the application wants to use - otherwise, the application will have to access private members of the EAL structures.

> >>>> Something like:
> >>>> typedef struct {
> >>>> #ifdef WINDOWS
> >>>> 	uintptr_t opaque_id;
> >>>> #else
> >>>> 	pthread_t opaque_id;
> >>>> #endif
> >>>> };
> >>>> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
> >> type'.
> >>>
> >>> yes, this is correct. newer posix introduced this to allow the use
> of
> >>> structs. i assume prior reviewers are aware of the recent posix
> >>> standard (or should be).
> >>>
> >>> this type makes no attempt to be usable on platforms that use
> >>> a handle > sizeof(uintptr_t). though any platform that does is free
> >>> to shove a pointer to struct into the handle at the cost of a
> >>> dereference if that is their implementation.
> >>
> >> Using pthread_t directly still seems like a safest bet to me.
> >> Then we can avoid doing these explicit type conversions before/after
> >> each pthread_xxx() call and wouldn't need to worry if we'll ever
> have
> >> platform with bigger pthread_t (though yes, I admit it is very
> >> unlikely).
> >> But, if we still prefer to go ahead with 'arch-neutral' approach,
> >> then I think we need to have compilation time check that opaque_id
> >> is big enough to hold pthread_t value:
> >> RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.

Yes, this should be in the O/S specific c files:

RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(pthread_t))

RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(DWORD))

> >>
> >
> > I agree with Konstantin's concerns. All this type casting increases
> the risk for bugs. Also, I don't understand the need to use a 64 bit
> value when thread IDs are 32 bit in both Linux and Windows.
> >
> > The thread handling is O/S specific code, so #ifdef is unavoidable.
> >
> > Furthermore, I don't think we should wrap O/S specific integer types
> into structs - it adds unnecessary complexity.
> >
> > I would prefer:
> >
> > #ifdef WINDOWS
> > typedef DWORD rte_thread_t;
> > #else
> > typedef pthread_t rte_thread_t;
> > #endif

OK, I was totally wrong here. But I still don't think we need to wrap the value into a structure, but can just use:

typedef uintptr_t rte_thread_t; /* And add comments about the underlying types, i.e. DWORD on Windows, pthread_t (not tid_t) on Linux/BSD. */

> >
> >>
> >>>
> >>>>
> >>>>
> >>>>> +} rte_thread_t;
> >>>>> +
> >>>>> +/**
> >>>>>     * TLS key type, an opaque pointer.
> >>>>>     */
> >>>>>    typedef struct eal_tls_key *rte_thread_key;
> >>>>> +/**
> >>>>> + * @warning
> >>>>> + * @b EXPERIMENTAL: this API may change without prior notice.
> >>>>> + *
> >>>>> + * Get the id of the calling thread.
> >>>>> + *
> >>>>> + * @return
> >>>>> + *   Return the thread id of the calling thread.
> >>>>> + */
> >>>>> +__rte_experimental
> >>>>> +rte_thread_t rte_thread_self(void);
> >>>>> +
> >>>>>    #ifdef RTE_HAS_CPUSET
> >>>>>    /**
> >>>>> diff --git a/lib/eal/unix/rte_thread.c
> b/lib/eal/unix/rte_thread.c
> >>>>> index c34ede9..82e008f 100644
> >>>>> --- a/lib/eal/unix/rte_thread.c
> >>>>> +++ b/lib/eal/unix/rte_thread.c
> >>>>> @@ -1,5 +1,6 @@
> >>>>>    /* SPDX-License-Identifier: BSD-3-Clause
> >>>>>     * Copyright 2021 Mellanox Technologies, Ltd
> >>>>> + * Copyright (C) 2022 Microsoft Corporation
> >>>>>     */
> >>>>>    #include <errno.h>
> >>>>> @@ -15,6 +16,16 @@ struct eal_tls_key {
> >>>>>    	pthread_key_t thread_index;
> >>>>>    };
> >>>>> +rte_thread_t
> >>>>> +rte_thread_self(void)
> >>>>> +{
> >>>>> +	rte_thread_t thread_id;
> >>>>> +
> >>>>> +	thread_id.opaque_id = (uintptr_t)pthread_self();
> >>>>> +
> >>>>> +	return thread_id;
> >>>>> +}
> >>>>> +
> >>>>>    int
> >>>>>    rte_thread_key_create(rte_thread_key *key, void
> >> (*destructor)(void *))
> >>>>>    {
> >>>>> diff --git a/lib/eal/version.map b/lib/eal/version.map
> >>>>> index b53eeb3..05ce8f9 100644
> >>>>> --- a/lib/eal/version.map
> >>>>> +++ b/lib/eal/version.map
> >>>>> @@ -420,6 +420,9 @@ EXPERIMENTAL {
> >>>>>    	rte_intr_instance_free;
> >>>>>    	rte_intr_type_get;
> >>>>>    	rte_intr_type_set;
> >>>>> +
> >>>>> +	# added in 22.07
> >>>>> +	rte_thread_self;
> >>>>>    };
> >>>>>    INTERNAL {
> >>>>> diff --git a/lib/eal/windows/rte_thread.c
> >> b/lib/eal/windows/rte_thread.c
> >>>>> index 667287c..59fed3c 100644
> >>>>> --- a/lib/eal/windows/rte_thread.c
> >>>>> +++ b/lib/eal/windows/rte_thread.c
> >>>>> @@ -11,6 +11,16 @@ struct eal_tls_key {
> >>>>>    	DWORD thread_index;
> >>>>>    };
> >>>>> +rte_thread_t
> >>>>> +rte_thread_self(void)
> >>>>> +{
> >>>>> +	rte_thread_t thread_id;
> >>>>> +
> >>>>> +	thread_id.opaque_id = GetCurrentThreadId();
> >>>>> +
> >>>>> +	return thread_id;
> >>>>> +}
> >>>>> +
> >>>>>    int
> >>>>>    rte_thread_key_create(rte_thread_key *key,
> >>>>>    		__rte_unused void (*destructor)(void *))
> >>
> >
> 

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

07/05/2022 20:47, Morten Brørup пишет:
>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
>> Sent: Saturday, 7 May 2022 15.58
>>
>> Hi Morten,
>>
>>>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
>>>> Sent: Friday, 6 May 2022 21.38
>>>>
>>>> 05/05/2022 08:11, Tyler Retzlaff пишет:
>>>>> On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
>>>>>> 04/05/2022 16:46, Tyler Retzlaff пишет:
>>>>>>> Provide a portable type-safe thread identifier.
>>>>>>> Provide rte_thread_self for obtaining current thread identifier.
>>>>>>>
>>>>>>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
>>>>>>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
>>>>>>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
>>>>>>> ---
>>>>>>>     lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>>>>>>>     lib/eal/unix/rte_thread.c    | 11 +++++++++++
>>>>>>>     lib/eal/version.map          |  3 +++
>>>>>>>     lib/eal/windows/rte_thread.c | 10 ++++++++++
>>>>>>>     4 files changed, 46 insertions(+)
>>>>>>>
>>>>>>> diff --git a/lib/eal/include/rte_thread.h
>>>> b/lib/eal/include/rte_thread.h
>>>>>>> index 8be8ed8..14478ba 100644
>>>>>>> --- a/lib/eal/include/rte_thread.h
>>>>>>> +++ b/lib/eal/include/rte_thread.h
>>>>>>> @@ -1,7 +1,10 @@
>>>>>>>     /* SPDX-License-Identifier: BSD-3-Clause
>>>>>>>      * Copyright(c) 2021 Mellanox Technologies, Ltd
>>>>>>> + * Copyright (C) 2022 Microsoft Corporation
>>>>>>>      */
>>>>>>> +#include <stdint.h>
>>>>>>> +
>>>>>>>     #include <rte_os.h>
>>>>>>>     #include <rte_compat.h>
>>>>>>> @@ -21,10 +24,29 @@
>>>>>>>     #endif
>>>>>>>     /**
>>>>>>> + * Thread id descriptor.
>>>>>>> + */
>>>>>>> +typedef struct {
>>>>>>> +	uintptr_t opaque_id; /**< thread identifier */
>>>>>>
>>>>>>
>>>>>> I know that currently on linux typeof(pthread_id) == unsigned long
>>>> int.
>>>>>> Though wouldn't it be safer and cleaner to use pthread_t
>> explicitly
>>>>>> on posix-like systems?
>>>>>
>>>>> i believe the previous discussions are.
>>>>>
>>>>> * preference for reduced or no conditional compilation.
>>>>> * preference for sizeof(type) to be `the same' on all platforms.
>>>>
>>>>
>>>> It would be the same as long as sizes of pthread_t uintptr_t are
>> equal.
>>>
>>> They are not. pthread_t (Linux thread ID) and DWORD (Windows thread
>> ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer
>> size (32 or 64 bit CPU).
>>
>> What make you think pthread_t is 32-bit on linux?
>>   From <pthread.h> on my box:
>> typedef unsigned long int pthread_t;
>> So it is either 64-bit or 32-bit depending on arch.
>> Same as uintptr_t.
> 
> You are right, Konstantin. I had overlooked the "long" in there.
> 
>>
>>>
>>>>
>>>>> * preference for platform agnostic headers. i.e. don't drag
>>>>>      platform specific headers into the application namespace when
>>>>>      including rte_xxx.h headers.
> 
> So this is an exception from the "don't hide the types" rule that DPDK inherited from the Kernel. In theory, this makes really good sense for an EAL that really is what it says (i.e. an abstraction layer). In reality, though, this requires that the EAL offers all the features of the underlying O/S that the application wants to use - otherwise, the application will have to access private members of the EAL structures.
> 
>>>>>> Something like:
>>>>>> typedef struct {
>>>>>> #ifdef WINDOWS
>>>>>> 	uintptr_t opaque_id;
>>>>>> #else
>>>>>> 	pthread_t opaque_id;
>>>>>> #endif
>>>>>> };
>>>>>> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
>>>> type'.
>>>>>
>>>>> yes, this is correct. newer posix introduced this to allow the use
>> of
>>>>> structs. i assume prior reviewers are aware of the recent posix
>>>>> standard (or should be).
>>>>>
>>>>> this type makes no attempt to be usable on platforms that use
>>>>> a handle > sizeof(uintptr_t). though any platform that does is free
>>>>> to shove a pointer to struct into the handle at the cost of a
>>>>> dereference if that is their implementation.
>>>>
>>>> Using pthread_t directly still seems like a safest bet to me.
>>>> Then we can avoid doing these explicit type conversions before/after
>>>> each pthread_xxx() call and wouldn't need to worry if we'll ever
>> have
>>>> platform with bigger pthread_t (though yes, I admit it is very
>>>> unlikely).
>>>> But, if we still prefer to go ahead with 'arch-neutral' approach,
>>>> then I think we need to have compilation time check that opaque_id
>>>> is big enough to hold pthread_t value:
>>>> RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
> 
> Yes, this should be in the O/S specific c files:
> 
> RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(pthread_t))
> 
> RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(DWORD))
> 
>>>>
>>>
>>> I agree with Konstantin's concerns. All this type casting increases
>> the risk for bugs. Also, I don't understand the need to use a 64 bit
>> value when thread IDs are 32 bit in both Linux and Windows.
>>>
>>> The thread handling is O/S specific code, so #ifdef is unavoidable.
>>>
>>> Furthermore, I don't think we should wrap O/S specific integer types
>> into structs - it adds unnecessary complexity.
>>>
>>> I would prefer:
>>>
>>> #ifdef WINDOWS
>>> typedef DWORD rte_thread_t;
>>> #else
>>> typedef pthread_t rte_thread_t;
>>> #endif
> 
> OK, I was totally wrong here. But I still don't think we need to wrap the value into a structure, but can just use:
> 
> typedef uintptr_t rte_thread_t; /* And add comments about the underlying types, i.e. DWORD on Windows, pthread_t (not tid_t) on Linux/BSD. */


I think we probably can have what you suggested above.
Though it might be better to move rte_thread_t typedef in OS-specific
header (rte_os.h).

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

On Tue, May 10, 2022 at 10:52:34PM +0100, Konstantin Ananyev wrote:
> 07/05/2022 20:47, Morten Brørup пишет:
> >>From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> >>Sent: Saturday, 7 May 2022 15.58
> >>
> >>Hi Morten,
> >>
> >>>>From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> >>>>Sent: Friday, 6 May 2022 21.38
> >>>>
> >>>>05/05/2022 08:11, Tyler Retzlaff пишет:
> >>>>>On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
> >>>>>>04/05/2022 16:46, Tyler Retzlaff пишет:
> >>>>>>>Provide a portable type-safe thread identifier.
> >>>>>>>Provide rte_thread_self for obtaining current thread identifier.
> >>>>>>>
> >>>>>>>Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >>>>>>>Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >>>>>>>Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> >>>>>>>---
> >>>>>>>    lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >>>>>>>    lib/eal/unix/rte_thread.c    | 11 +++++++++++
> >>>>>>>    lib/eal/version.map          |  3 +++
> >>>>>>>    lib/eal/windows/rte_thread.c | 10 ++++++++++
> >>>>>>>    4 files changed, 46 insertions(+)
> >>>>>>>
> >>>>>>>diff --git a/lib/eal/include/rte_thread.h
> >>>>b/lib/eal/include/rte_thread.h
> >>>>>>>index 8be8ed8..14478ba 100644
> >>>>>>>--- a/lib/eal/include/rte_thread.h
> >>>>>>>+++ b/lib/eal/include/rte_thread.h
> >>>>>>>@@ -1,7 +1,10 @@
> >>>>>>>    /* SPDX-License-Identifier: BSD-3-Clause
> >>>>>>>     * Copyright(c) 2021 Mellanox Technologies, Ltd
> >>>>>>>+ * Copyright (C) 2022 Microsoft Corporation
> >>>>>>>     */
> >>>>>>>+#include <stdint.h>
> >>>>>>>+
> >>>>>>>    #include <rte_os.h>
> >>>>>>>    #include <rte_compat.h>
> >>>>>>>@@ -21,10 +24,29 @@
> >>>>>>>    #endif
> >>>>>>>    /**
> >>>>>>>+ * Thread id descriptor.
> >>>>>>>+ */
> >>>>>>>+typedef struct {
> >>>>>>>+	uintptr_t opaque_id; /**< thread identifier */
> >>>>>>
> >>>>>>
> >>>>>>I know that currently on linux typeof(pthread_id) == unsigned long
> >>>>int.
> >>>>>>Though wouldn't it be safer and cleaner to use pthread_t
> >>explicitly
> >>>>>>on posix-like systems?
> >>>>>
> >>>>>i believe the previous discussions are.
> >>>>>
> >>>>>* preference for reduced or no conditional compilation.
> >>>>>* preference for sizeof(type) to be `the same' on all platforms.
> >>>>
> >>>>
> >>>>It would be the same as long as sizes of pthread_t uintptr_t are
> >>equal.
> >>>
> >>>They are not. pthread_t (Linux thread ID) and DWORD (Windows thread
> >>ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer
> >>size (32 or 64 bit CPU).
> >>
> >>What make you think pthread_t is 32-bit on linux?
> >>  From <pthread.h> on my box:
> >>typedef unsigned long int pthread_t;
> >>So it is either 64-bit or 32-bit depending on arch.
> >>Same as uintptr_t.
> >
> >You are right, Konstantin. I had overlooked the "long" in there.
> >
> >>
> >>>
> >>>>
> >>>>>* preference for platform agnostic headers. i.e. don't drag
> >>>>>     platform specific headers into the application namespace when
> >>>>>     including rte_xxx.h headers.
> >
> >So this is an exception from the "don't hide the types" rule that DPDK inherited from the Kernel. In theory, this makes really good sense for an EAL that really is what it says (i.e. an abstraction layer). In reality, though, this requires that the EAL offers all the features of the underlying O/S that the application wants to use - otherwise, the application will have to access private members of the EAL structures.
> >
> >>>>>>Something like:
> >>>>>>typedef struct {
> >>>>>>#ifdef WINDOWS
> >>>>>>	uintptr_t opaque_id;
> >>>>>>#else
> >>>>>>	pthread_t opaque_id;
> >>>>>>#endif
> >>>>>>};
> >>>>>>AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
> >>>>type'.
> >>>>>
> >>>>>yes, this is correct. newer posix introduced this to allow the use
> >>of
> >>>>>structs. i assume prior reviewers are aware of the recent posix
> >>>>>standard (or should be).
> >>>>>
> >>>>>this type makes no attempt to be usable on platforms that use
> >>>>>a handle > sizeof(uintptr_t). though any platform that does is free
> >>>>>to shove a pointer to struct into the handle at the cost of a
> >>>>>dereference if that is their implementation.
> >>>>
> >>>>Using pthread_t directly still seems like a safest bet to me.
> >>>>Then we can avoid doing these explicit type conversions before/after
> >>>>each pthread_xxx() call and wouldn't need to worry if we'll ever
> >>have
> >>>>platform with bigger pthread_t (though yes, I admit it is very
> >>>>unlikely).
> >>>>But, if we still prefer to go ahead with 'arch-neutral' approach,
> >>>>then I think we need to have compilation time check that opaque_id
> >>>>is big enough to hold pthread_t value:
> >>>>RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
> >
> >Yes, this should be in the O/S specific c files:
> >
> >RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(pthread_t))
> >
> >RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(DWORD))
> >
> >>>>
> >>>
> >>>I agree with Konstantin's concerns. All this type casting increases
> >>the risk for bugs. Also, I don't understand the need to use a 64 bit
> >>value when thread IDs are 32 bit in both Linux and Windows.
> >>>
> >>>The thread handling is O/S specific code, so #ifdef is unavoidable.
> >>>
> >>>Furthermore, I don't think we should wrap O/S specific integer types
> >>into structs - it adds unnecessary complexity.
> >>>
> >>>I would prefer:
> >>>
> >>>#ifdef WINDOWS
> >>>typedef DWORD rte_thread_t;
> >>>#else
> >>>typedef pthread_t rte_thread_t;
> >>>#endif
> >
> >OK, I was totally wrong here. But I still don't think we need to wrap the value into a structure, but can just use:

> >
> >typedef uintptr_t rte_thread_t; /* And add comments about the underlying types, i.e. DWORD on Windows, pthread_t (not tid_t) on Linux/BSD. */

a struct will guarantee no implicit conversion since it is a real type.

i am uncertain about why you feel it makes it more complex? it improves
safety by enforcing the semantics intended i.e. that it is an opaque type.

as a side-nit it's a shame the same wasn't done for core id vs core index
because i'm forever fixing bugs where people have swapped the two where
the mistake could have been a compiler error it is now a runtime error.

> 
> 
> I think we probably can have what you suggested above.
> Though it might be better to move rte_thread_t typedef in OS-specific
> header (rte_os.h).
> 

i don't see any benefit to this.

direct exposure of pthread_t will invariably lead to abuse by people
re-introducing dependency on pthread api leading to re-introduction of
conditionally compiled code for posix v non-posix ports.

of course the struct doesn't expressly prevent this since you can just
grovel its internals and assume the implementation but still why beg
for it by presenting no barrier entry at all.

i can address the initial concern about type sizing with a static assert
but i don't think without a much stronger argument with examples
demonstrating clear benefit i can get behind using just a typedef or
pthread_t conditionally compiled.

thanks for the feedback.

RE: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Morten Brørup]

> From: Tyler Retzlaff [mailto:roretzla@linux.microsoft.com]
> Sent: Wednesday, 11 May 2022 09.17
> 
> On Tue, May 10, 2022 at 10:52:34PM +0100, Konstantin Ananyev wrote:
> > 07/05/2022 20:47, Morten Brørup пишет:
> > >>From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> > >>Sent: Saturday, 7 May 2022 15.58
> > >>
> > >>Hi Morten,
> > >>
> > >>>>From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> > >>>>Sent: Friday, 6 May 2022 21.38
> > >>>>
> > >>>>05/05/2022 08:11, Tyler Retzlaff пишет:
> > >>>>>On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev
> wrote:
> > >>>>>>04/05/2022 16:46, Tyler Retzlaff пишет:
> > >>>>>>>Provide a portable type-safe thread identifier.
> > >>>>>>>Provide rte_thread_self for obtaining current thread
> identifier.
> > >>>>>>>
> > >>>>>>>Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> > >>>>>>>Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> > >>>>>>>Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > >>>>>>>---
> > >>>>>>>    lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> > >>>>>>>    lib/eal/unix/rte_thread.c    | 11 +++++++++++
> > >>>>>>>    lib/eal/version.map          |  3 +++
> > >>>>>>>    lib/eal/windows/rte_thread.c | 10 ++++++++++
> > >>>>>>>    4 files changed, 46 insertions(+)
> > >>>>>>>
> > >>>>>>>diff --git a/lib/eal/include/rte_thread.h
> > >>>>b/lib/eal/include/rte_thread.h
> > >>>>>>>index 8be8ed8..14478ba 100644
> > >>>>>>>--- a/lib/eal/include/rte_thread.h
> > >>>>>>>+++ b/lib/eal/include/rte_thread.h
> > >>>>>>>@@ -1,7 +1,10 @@
> > >>>>>>>    /* SPDX-License-Identifier: BSD-3-Clause
> > >>>>>>>     * Copyright(c) 2021 Mellanox Technologies, Ltd
> > >>>>>>>+ * Copyright (C) 2022 Microsoft Corporation
> > >>>>>>>     */
> > >>>>>>>+#include <stdint.h>
> > >>>>>>>+
> > >>>>>>>    #include <rte_os.h>
> > >>>>>>>    #include <rte_compat.h>
> > >>>>>>>@@ -21,10 +24,29 @@
> > >>>>>>>    #endif
> > >>>>>>>    /**
> > >>>>>>>+ * Thread id descriptor.
> > >>>>>>>+ */
> > >>>>>>>+typedef struct {
> > >>>>>>>+	uintptr_t opaque_id; /**< thread identifier */
> > >>>>>>
> > >>>>>>
> > >>>>>>I know that currently on linux typeof(pthread_id) == unsigned
> long
> > >>>>int.
> > >>>>>>Though wouldn't it be safer and cleaner to use pthread_t
> > >>explicitly
> > >>>>>>on posix-like systems?
> > >>>>>
> > >>>>>i believe the previous discussions are.
> > >>>>>
> > >>>>>* preference for reduced or no conditional compilation.
> > >>>>>* preference for sizeof(type) to be `the same' on all platforms.
> > >>>>
> > >>>>
> > >>>>It would be the same as long as sizes of pthread_t uintptr_t are
> > >>equal.
> > >>>
> > >>>They are not. pthread_t (Linux thread ID) and DWORD (Windows
> thread
> > >>ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer
> > >>size (32 or 64 bit CPU).
> > >>
> > >>What make you think pthread_t is 32-bit on linux?
> > >>  From <pthread.h> on my box:
> > >>typedef unsigned long int pthread_t;
> > >>So it is either 64-bit or 32-bit depending on arch.
> > >>Same as uintptr_t.
> > >
> > >You are right, Konstantin. I had overlooked the "long" in there.
> > >
> > >>
> > >>>
> > >>>>
> > >>>>>* preference for platform agnostic headers. i.e. don't drag
> > >>>>>     platform specific headers into the application namespace
> when
> > >>>>>     including rte_xxx.h headers.
> > >
> > >So this is an exception from the "don't hide the types" rule that
> DPDK inherited from the Kernel. In theory, this makes really good sense
> for an EAL that really is what it says (i.e. an abstraction layer). In
> reality, though, this requires that the EAL offers all the features of
> the underlying O/S that the application wants to use - otherwise, the
> application will have to access private members of the EAL structures.
> > >
> > >>>>>>Something like:
> > >>>>>>typedef struct {
> > >>>>>>#ifdef WINDOWS
> > >>>>>>	uintptr_t opaque_id;
> > >>>>>>#else
> > >>>>>>	pthread_t opaque_id;
> > >>>>>>#endif
> > >>>>>>};
> > >>>>>>AFAIK POSIX itself doesn't require pthread_t to be an
> 'arithmetic
> > >>>>type'.
> > >>>>>
> > >>>>>yes, this is correct. newer posix introduced this to allow the
> use
> > >>of
> > >>>>>structs. i assume prior reviewers are aware of the recent posix
> > >>>>>standard (or should be).
> > >>>>>
> > >>>>>this type makes no attempt to be usable on platforms that use
> > >>>>>a handle > sizeof(uintptr_t). though any platform that does is
> free
> > >>>>>to shove a pointer to struct into the handle at the cost of a
> > >>>>>dereference if that is their implementation.
> > >>>>
> > >>>>Using pthread_t directly still seems like a safest bet to me.
> > >>>>Then we can avoid doing these explicit type conversions
> before/after
> > >>>>each pthread_xxx() call and wouldn't need to worry if we'll ever
> > >>have
> > >>>>platform with bigger pthread_t (though yes, I admit it is very
> > >>>>unlikely).
> > >>>>But, if we still prefer to go ahead with 'arch-neutral' approach,
> > >>>>then I think we need to have compilation time check that
> opaque_id
> > >>>>is big enough to hold pthread_t value:
> > >>>>RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
> > >
> > >Yes, this should be in the O/S specific c files:
> > >
> > >RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(pthread_t))
> > >
> > >RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(DWORD))
> > >
> > >>>>
> > >>>
> > >>>I agree with Konstantin's concerns. All this type casting
> increases
> > >>the risk for bugs. Also, I don't understand the need to use a 64
> bit
> > >>value when thread IDs are 32 bit in both Linux and Windows.
> > >>>
> > >>>The thread handling is O/S specific code, so #ifdef is
> unavoidable.
> > >>>
> > >>>Furthermore, I don't think we should wrap O/S specific integer
> types
> > >>into structs - it adds unnecessary complexity.
> > >>>
> > >>>I would prefer:
> > >>>
> > >>>#ifdef WINDOWS
> > >>>typedef DWORD rte_thread_t;
> > >>>#else
> > >>>typedef pthread_t rte_thread_t;
> > >>>#endif
> > >
> > >OK, I was totally wrong here. But I still don't think we need to
> wrap the value into a structure, but can just use:
> 
> > >
> > >typedef uintptr_t rte_thread_t; /* And add comments about the
> underlying types, i.e. DWORD on Windows, pthread_t (not tid_t) on
> Linux/BSD. */
> 
> a struct will guarantee no implicit conversion since it is a real type.

That's a good point, which I didn't think about.

> 
> i am uncertain about why you feel it makes it more complex? it improves
> safety by enforcing the semantics intended i.e. that it is an opaque
> type.
> 
> as a side-nit it's a shame the same wasn't done for core id vs core
> index
> because i'm forever fixing bugs where people have swapped the two where
> the mistake could have been a compiler error it is now a runtime error.

I have noticed this too... I guess they started out as one, and only split into two in a later version of DPDK. This kind of legacy is difficult to improve when the API must remain stable. :-(

> 
> >
> >
> > I think we probably can have what you suggested above.
> > Though it might be better to move rte_thread_t typedef in OS-specific
> > header (rte_os.h).
> >
> 
> i don't see any benefit to this.
> 
> direct exposure of pthread_t will invariably lead to abuse by people
> re-introducing dependency on pthread api leading to re-introduction of
> conditionally compiled code for posix v non-posix ports.
> 
> of course the struct doesn't expressly prevent this since you can just
> grovel its internals and assume the implementation but still why beg
> for it by presenting no barrier entry at all.
> 
> i can address the initial concern about type sizing with a static
> assert
> but i don't think without a much stronger argument with examples
> demonstrating clear benefit i can get behind using just a typedef or
> pthread_t conditionally compiled.
> 
> thanks for the feedback.

I guess the differences of opinion mainly revolve about how opaque vs. transparent we want the rte_thread_t to be.

DPDK has a tradition, inherited from Linux, for keeping types transparent. In this case, considering your experience with working cross O/S, I will change my opinion and support your stance.

And to further avoid someone incorrectly using the private "id" member of the rte_thread_t struct, you could give it a different name for each O/S. Unfortunately, it's not C++, so you cannot make it "private" or "protected".

Re: [PATCH v5 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

11/05/2022 08:17, Tyler Retzlaff пишет:
> On Tue, May 10, 2022 at 10:52:34PM +0100, Konstantin Ananyev wrote:
>> 07/05/2022 20:47, Morten Brørup пишет:
>>>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
>>>> Sent: Saturday, 7 May 2022 15.58
>>>>
>>>> Hi Morten,
>>>>
>>>>>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
>>>>>> Sent: Friday, 6 May 2022 21.38
>>>>>>
>>>>>> 05/05/2022 08:11, Tyler Retzlaff пишет:
>>>>>>> On Wed, May 04, 2022 at 11:55:57PM +0100, Konstantin Ananyev wrote:
>>>>>>>> 04/05/2022 16:46, Tyler Retzlaff пишет:
>>>>>>>>> Provide a portable type-safe thread identifier.
>>>>>>>>> Provide rte_thread_self for obtaining current thread identifier.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
>>>>>>>>> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
>>>>>>>>> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
>>>>>>>>> ---
>>>>>>>>>     lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>>>>>>>>>     lib/eal/unix/rte_thread.c    | 11 +++++++++++
>>>>>>>>>     lib/eal/version.map          |  3 +++
>>>>>>>>>     lib/eal/windows/rte_thread.c | 10 ++++++++++
>>>>>>>>>     4 files changed, 46 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/lib/eal/include/rte_thread.h
>>>>>> b/lib/eal/include/rte_thread.h
>>>>>>>>> index 8be8ed8..14478ba 100644
>>>>>>>>> --- a/lib/eal/include/rte_thread.h
>>>>>>>>> +++ b/lib/eal/include/rte_thread.h
>>>>>>>>> @@ -1,7 +1,10 @@
>>>>>>>>>     /* SPDX-License-Identifier: BSD-3-Clause
>>>>>>>>>      * Copyright(c) 2021 Mellanox Technologies, Ltd
>>>>>>>>> + * Copyright (C) 2022 Microsoft Corporation
>>>>>>>>>      */
>>>>>>>>> +#include <stdint.h>
>>>>>>>>> +
>>>>>>>>>     #include <rte_os.h>
>>>>>>>>>     #include <rte_compat.h>
>>>>>>>>> @@ -21,10 +24,29 @@
>>>>>>>>>     #endif
>>>>>>>>>     /**
>>>>>>>>> + * Thread id descriptor.
>>>>>>>>> + */
>>>>>>>>> +typedef struct {
>>>>>>>>> +	uintptr_t opaque_id; /**< thread identifier */
>>>>>>>>
>>>>>>>>
>>>>>>>> I know that currently on linux typeof(pthread_id) == unsigned long
>>>>>> int.
>>>>>>>> Though wouldn't it be safer and cleaner to use pthread_t
>>>> explicitly
>>>>>>>> on posix-like systems?
>>>>>>>
>>>>>>> i believe the previous discussions are.
>>>>>>>
>>>>>>> * preference for reduced or no conditional compilation.
>>>>>>> * preference for sizeof(type) to be `the same' on all platforms.
>>>>>>
>>>>>>
>>>>>> It would be the same as long as sizes of pthread_t uintptr_t are
>>>> equal.
>>>>>
>>>>> They are not. pthread_t (Linux thread ID) and DWORD (Windows thread
>>>> ID) are both 32 bit, uintptr_t is 64 or 32 bit depending on pointer
>>>> size (32 or 64 bit CPU).
>>>>
>>>> What make you think pthread_t is 32-bit on linux?
>>>>   From <pthread.h> on my box:
>>>> typedef unsigned long int pthread_t;
>>>> So it is either 64-bit or 32-bit depending on arch.
>>>> Same as uintptr_t.
>>>
>>> You are right, Konstantin. I had overlooked the "long" in there.
>>>
>>>>
>>>>>
>>>>>>
>>>>>>> * preference for platform agnostic headers. i.e. don't drag
>>>>>>>      platform specific headers into the application namespace when
>>>>>>>      including rte_xxx.h headers.
>>>
>>> So this is an exception from the "don't hide the types" rule that DPDK inherited from the Kernel. In theory, this makes really good sense for an EAL that really is what it says (i.e. an abstraction layer). In reality, though, this requires that the EAL offers all the features of the underlying O/S that the application wants to use - otherwise, the application will have to access private members of the EAL structures.
>>>
>>>>>>>> Something like:
>>>>>>>> typedef struct {
>>>>>>>> #ifdef WINDOWS
>>>>>>>> 	uintptr_t opaque_id;
>>>>>>>> #else
>>>>>>>> 	pthread_t opaque_id;
>>>>>>>> #endif
>>>>>>>> };
>>>>>>>> AFAIK POSIX itself doesn't require pthread_t to be an 'arithmetic
>>>>>> type'.
>>>>>>>
>>>>>>> yes, this is correct. newer posix introduced this to allow the use
>>>> of
>>>>>>> structs. i assume prior reviewers are aware of the recent posix
>>>>>>> standard (or should be).
>>>>>>>
>>>>>>> this type makes no attempt to be usable on platforms that use
>>>>>>> a handle > sizeof(uintptr_t). though any platform that does is free
>>>>>>> to shove a pointer to struct into the handle at the cost of a
>>>>>>> dereference if that is their implementation.
>>>>>>
>>>>>> Using pthread_t directly still seems like a safest bet to me.
>>>>>> Then we can avoid doing these explicit type conversions before/after
>>>>>> each pthread_xxx() call and wouldn't need to worry if we'll ever
>>>> have
>>>>>> platform with bigger pthread_t (though yes, I admit it is very
>>>>>> unlikely).
>>>>>> But, if we still prefer to go ahead with 'arch-neutral' approach,
>>>>>> then I think we need to have compilation time check that opaque_id
>>>>>> is big enough to hold pthread_t value:
>>>>>> RTE_BUILD_BUG_ON(sizeof(pthread_t) != sizeof(opaque_id)) or so.
>>>
>>> Yes, this should be in the O/S specific c files:
>>>
>>> RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(pthread_t))
>>>
>>> RTE_BUILD_BUG_ON(sizeof(rte_thread_t) < sizeof(DWORD))
>>>
>>>>>>
>>>>>
>>>>> I agree with Konstantin's concerns. All this type casting increases
>>>> the risk for bugs. Also, I don't understand the need to use a 64 bit
>>>> value when thread IDs are 32 bit in both Linux and Windows.
>>>>>
>>>>> The thread handling is O/S specific code, so #ifdef is unavoidable.
>>>>>
>>>>> Furthermore, I don't think we should wrap O/S specific integer types
>>>> into structs - it adds unnecessary complexity.
>>>>>
>>>>> I would prefer:
>>>>>
>>>>> #ifdef WINDOWS
>>>>> typedef DWORD rte_thread_t;
>>>>> #else
>>>>> typedef pthread_t rte_thread_t;
>>>>> #endif
>>>
>>> OK, I was totally wrong here. But I still don't think we need to wrap the value into a structure, but can just use:
> 
>>>
>>> typedef uintptr_t rte_thread_t; /* And add comments about the underlying types, i.e. DWORD on Windows, pthread_t (not tid_t) on Linux/BSD. */
> 
> a struct will guarantee no implicit conversion since it is a real type.
> 
> i am uncertain about why you feel it makes it more complex? it improves
> safety by enforcing the semantics intended i.e. that it is an opaque type.
> 
> as a side-nit it's a shame the same wasn't done for core id vs core index
> because i'm forever fixing bugs where people have swapped the two where
> the mistake could have been a compiler error it is now a runtime error.
> 
>>
>>
>> I think we probably can have what you suggested above.
>> Though it might be better to move rte_thread_t typedef in OS-specific
>> header (rte_os.h).
>>
> 
> i don't see any benefit to this.
> 
> direct exposure of pthread_t will invariably lead to abuse by people
> re-introducing dependency on pthread api leading to re-introduction of
> conditionally compiled code for posix v non-posix ports.
> 
> of course the struct doesn't expressly prevent this since you can just
> grovel its internals and assume the implementation but still why beg
> for it by presenting no barrier entry at all.


Ok, if everyone else think that opaque type is a better choice here,
I wouldn't insist.


> i can address the initial concern about type sizing with a static assert


Please do.

> but i don't think without a much stronger argument with examples
> demonstrating clear benefit i can get behind using just a typedef or
> pthread_t conditionally compiled.
> 
> thanks for the feedback.

[PATCH v6 0/3] add eal functions for thread affinity and self

[Tyler Retzlaff]

this series provides basic dependencies for additional eal thread api
additions. series includes

* basic platform error number conversion.
* function to get current thread identifier.
* functions to get and set affinity with platform agnostic thread
  identifier.
* minimal unit test of get and set affinity demonstrating usage.

note: previous series introducing these functions is now superseded by
this series.
http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*

v6:
* rebase for asan flag addition to app/test/meson.build
* RTE_BUILD_BUG_ON(sizeof(pthread_t) > sizeof(uintptr_t)) to raise
  attention if pthread_t implementation exceeds storage available
  from uintptr_t.
  note:
    the macro is placed in rte_thread_self() body because it not
    valid syntax at file scope of the translation unit. ordinarily
    the macro would be used in headers but that would leak the
    pthread_t implementation detail into the public header.

v5:
* use rte_log instead of log_early. it appears that logging is
  now initialized before the call to eal_query_group_affinity.
  note: removal of log_early is off-topic for this series and
        will be done separately.
* rte_log DEBUG for failure related to limitation where all
  processors must be in the same processor group.
* remove redundant tests. get/get/compare, set/get/compare
  are both retained.

v4:
* combine patch eal/windows: translate Windows errors to errno-style
  errors into eal: implement functions for get/set thread affinity
  patch. the former introduced static functions that were not used
  without eal: implement functions for get/set thread affinity which
  would cause a build break when applied standalone.
* remove struct tag from rte_thread_t struct typedef.
* remove rte_ prefix from rte_convert_cpuset_to_affinity static
  function.

v3:
* fix memory leak on eal_create_cpu_map error paths.

v2:
* add missing boilerplate comments warning of experimental api
  for rte_thread_{set,get}_affinity_by_id().
* don't break literal format string to log_early to improve
  searchability.
* fix multi-line comment style to match file.
* return ENOTSUP instead of EINVAL from rte_convert_cpuset_to_affinity()
  if cpus in set are not part of the same processor group and note
  limitation in commit message.
* expand series to include rte_thread_self().
* modify unit test to remove use of implementation detail and
  get thread identifier use added rte_thread_self().
* move literal value to rhs when using memcmp in RTE_TEST_ASSERT

Tyler Retzlaff (3):
  eal: add basic thread ID and current thread identifier API
  eal: implement functions for get/set thread affinity
  test/threads: add unit test for thread API

 app/test/meson.build             |   2 +
 app/test/test_threads.c          |  81 +++++++++++++++++
 lib/eal/include/rte_thread.h     |  64 +++++++++++++
 lib/eal/unix/rte_thread.c        |  29 ++++++
 lib/eal/version.map              |   5 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 ++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 191 ++++++++++++++++++++++++++++++++++++++-
 9 files changed, 517 insertions(+), 48 deletions(-)
 create mode 100644 app/test/test_threads.c

-- 
1.8.3.1

[PATCH v6 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

Provide a portable type-safe thread identifier.
Provide rte_thread_self for obtaining current thread identifier.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
 lib/eal/unix/rte_thread.c    | 13 +++++++++++++
 lib/eal/version.map          |  3 +++
 lib/eal/windows/rte_thread.c | 10 ++++++++++
 4 files changed, 48 insertions(+)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 8be8ed8..14478ba 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -1,7 +1,10 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
+#include <stdint.h>
+
 #include <rte_os.h>
 #include <rte_compat.h>
 
@@ -21,10 +24,29 @@
 #endif
 
 /**
+ * Thread id descriptor.
+ */
+typedef struct {
+	uintptr_t opaque_id; /**< thread identifier */
+} rte_thread_t;
+
+/**
  * TLS key type, an opaque pointer.
  */
 typedef struct eal_tls_key *rte_thread_key;
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the id of the calling thread.
+ *
+ * @return
+ *   Return the thread id of the calling thread.
+ */
+__rte_experimental
+rte_thread_t rte_thread_self(void);
+
 #ifdef RTE_HAS_CPUSET
 
 /**
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index c34ede9..5e5beb1 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -1,5 +1,6 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <errno.h>
@@ -15,6 +16,18 @@ struct eal_tls_key {
 	pthread_key_t thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	RTE_BUILD_BUG_ON(sizeof(pthread_t) > sizeof(uintptr_t));
+
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = (uintptr_t)pthread_self();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
 {
diff --git a/lib/eal/version.map b/lib/eal/version.map
index b53eeb3..05ce8f9 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -420,6 +420,9 @@ EXPERIMENTAL {
 	rte_intr_instance_free;
 	rte_intr_type_get;
 	rte_intr_type_set;
+
+	# added in 22.07
+	rte_thread_self;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 667287c..59fed3c 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -11,6 +11,16 @@ struct eal_tls_key {
 	DWORD thread_index;
 };
 
+rte_thread_t
+rte_thread_self(void)
+{
+	rte_thread_t thread_id;
+
+	thread_id.opaque_id = GetCurrentThreadId();
+
+	return thread_id;
+}
+
 int
 rte_thread_key_create(rte_thread_key *key,
 		__rte_unused void (*destructor)(void *))
-- 
1.8.3.1

[PATCH v6 2/3] eal: implement functions for get/set thread affinity

[Tyler Retzlaff]

Implement functions for getting/setting thread affinity.
Threads can be pinned to specific cores by setting their
affinity attribute.

Windows error codes are translated to errno-style error codes.
The possible return values are chosen so that we have as
much semantic compatibility between platforms as possible.

note: convert_cpuset_to_affinity has a limitation that all cpus of
the set belong to the same processor group.

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/eal/include/rte_thread.h     |  42 +++++++++
 lib/eal/unix/rte_thread.c        |  16 ++++
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal_lcore.c      | 181 +++++++++++++++++++++++++++++----------
 lib/eal/windows/eal_windows.h    |  10 +++
 lib/eal/windows/include/rte_os.h |   2 +
 lib/eal/windows/rte_thread.c     | 181 ++++++++++++++++++++++++++++++++++++++-
 7 files changed, 386 insertions(+), 48 deletions(-)

diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
index 14478ba..7888f7a 100644
--- a/lib/eal/include/rte_thread.h
+++ b/lib/eal/include/rte_thread.h
@@ -50,6 +50,48 @@
 #ifdef RTE_HAS_CPUSET
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set the affinity of thread 'thread_id' to the cpu set
+ * specified by 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to set the affinity.
+ *
+ * @param cpuset
+ *   Pointer to CPU affinity to set.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get the affinity of thread 'thread_id' and store it
+ * in 'cpuset'.
+ *
+ * @param thread_id
+ *    Id of the thread for which to get the affinity.
+ *
+ * @param cpuset
+ *   Pointer for storing the affinity value.
+ *
+ * @return
+ *   On success, return 0.
+ *   On failure, return a positive errno-style error number.
+ */
+__rte_experimental
+int rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset);
+
+/**
  * Set core affinity of the current thread.
  * Support both EAL and non-EAL thread and update TLS.
  *
diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
index 5e5beb1..9e5fa47 100644
--- a/lib/eal/unix/rte_thread.c
+++ b/lib/eal/unix/rte_thread.c
@@ -102,3 +102,19 @@ struct eal_tls_key {
 	}
 	return pthread_getspecific(key->thread_index);
 }
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	return pthread_setaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	return pthread_getaffinity_np((pthread_t)thread_id.opaque_id,
+		sizeof(*cpuset), cpuset);
+}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 05ce8f9..d49e30b 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -422,7 +422,9 @@ EXPERIMENTAL {
 	rte_intr_type_set;
 
 	# added in 22.07
+	rte_thread_get_affinity_by_id;
 	rte_thread_self;
+	rte_thread_set_affinity_by_id;
 };
 
 INTERNAL {
diff --git a/lib/eal/windows/eal_lcore.c b/lib/eal/windows/eal_lcore.c
index 476c2d2..286fe24 100644
--- a/lib/eal/windows/eal_lcore.c
+++ b/lib/eal/windows/eal_lcore.c
@@ -1,8 +1,8 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright(c) 2019 Intel Corporation
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
-#include <pthread.h>
 #include <stdbool.h>
 #include <stdint.h>
 
@@ -27,13 +27,15 @@ struct socket_map {
 };
 
 struct cpu_map {
-	unsigned int socket_count;
 	unsigned int lcore_count;
+	unsigned int socket_count;
+	unsigned int cpu_count;
 	struct lcore_map lcores[RTE_MAX_LCORE];
 	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+	GROUP_AFFINITY cpus[CPU_SETSIZE];
 };
 
-static struct cpu_map cpu_map = { 0 };
+static struct cpu_map cpu_map;
 
 /* eal_create_cpu_map() is called before logging is initialized */
 static void
@@ -47,13 +49,115 @@ struct cpu_map {
 	va_end(va);
 }
 
+static int
+eal_query_group_affinity(void)
+{
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos = NULL;
+	unsigned int *cpu_count = &cpu_map.cpu_count;
+	DWORD infos_size = 0;
+	int ret = 0;
+	USHORT group_count;
+	KAFFINITY affinity;
+	USHORT group_no;
+	unsigned int i;
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, NULL,
+			&infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			RTE_LOG(ERR, EAL, "Cannot get group information size, error %lu\n", error);
+			rte_errno = EINVAL;
+			ret = -1;
+			goto cleanup;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;
+		ret = -1;
+		goto cleanup;
+	}
+
+	if (!GetLogicalProcessorInformationEx(RelationGroup, infos,
+			&infos_size)) {
+		RTE_LOG(ERR, EAL, "Cannot get group information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		ret = -1;
+		goto cleanup;
+	}
+
+	*cpu_count = 0;
+	group_count = infos->Group.ActiveGroupCount;
+	for (group_no = 0; group_no < group_count; group_no++) {
+		affinity = infos->Group.GroupInfo[group_no].ActiveProcessorMask;
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((affinity & ((KAFFINITY)1 << i)) == 0)
+				continue;
+			cpu_map.cpus[*cpu_count].Group = group_no;
+			cpu_map.cpus[*cpu_count].Mask = (KAFFINITY)1 << i;
+			(*cpu_count)++;
+		}
+	}
+
+cleanup:
+	free(infos);
+	return ret;
+}
+
+static bool
+eal_create_lcore_map(const SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *info)
+{
+	const unsigned int node_id = info->NumaNode.NodeNumber;
+	const GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+	struct lcore_map *lcore;
+	unsigned int socket_id;
+	unsigned int i;
+
+	/*
+	 * NUMA node may be reported multiple times if it includes
+	 * cores from different processor groups, e. g. 80 cores
+	 * of a physical processor comprise one NUMA node, but two
+	 * processor groups, because group size is limited by 32/64.
+	 */
+	for (socket_id = 0; socket_id < cpu_map.socket_count; socket_id++)
+		if (cpu_map.sockets[socket_id].node_id == node_id)
+			break;
+
+	if (socket_id == cpu_map.socket_count) {
+		if (socket_id == RTE_DIM(cpu_map.sockets))
+			return true;
+
+		cpu_map.sockets[socket_id].node_id = node_id;
+		cpu_map.socket_count++;
+	}
+
+	for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+		if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+			continue;
+
+		if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores))
+			return true;
+
+		lcore = &cpu_map.lcores[cpu_map.lcore_count];
+		lcore->socket_id = socket_id;
+		lcore->core_id = cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+		cpu_map.lcore_count++;
+	}
+	return false;
+}
+
 int
 eal_create_cpu_map(void)
 {
 	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
 	DWORD infos_size;
 	bool full = false;
+	int ret = 0;
 
+	infos = NULL;
 	infos_size = 0;
 	if (!GetLogicalProcessorInformationEx(
 			RelationNumaNode, NULL, &infos_size)) {
@@ -62,7 +166,8 @@ struct cpu_map {
 			log_early("Cannot get NUMA node info size, error %lu\n",
 				GetLastError());
 			rte_errno = ENOMEM;
-			return -1;
+			ret = -1;
+			goto exit;
 		}
 	}
 
@@ -70,7 +175,8 @@ struct cpu_map {
 	if (infos == NULL) {
 		log_early("Cannot allocate memory for NUMA node information\n");
 		rte_errno = ENOMEM;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	if (!GetLogicalProcessorInformationEx(
@@ -78,57 +184,30 @@ struct cpu_map {
 		log_early("Cannot get NUMA node information, error %lu\n",
 			GetLastError());
 		rte_errno = EINVAL;
-		return -1;
+		ret = -1;
+		goto exit;
 	}
 
 	info = infos;
 	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
-		unsigned int node_id = info->NumaNode.NodeNumber;
-		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
-		struct lcore_map *lcore;
-		unsigned int i, socket_id;
-
-		/* NUMA node may be reported multiple times if it includes
-		 * cores from different processor groups, e. g. 80 cores
-		 * of a physical processor comprise one NUMA node, but two
-		 * processor groups, because group size is limited by 32/64.
-		 */
-		for (socket_id = 0; socket_id < cpu_map.socket_count;
-		    socket_id++) {
-			if (cpu_map.sockets[socket_id].node_id == node_id)
-				break;
-		}
-
-		if (socket_id == cpu_map.socket_count) {
-			if (socket_id == RTE_DIM(cpu_map.sockets)) {
-				full = true;
-				goto exit;
-			}
-
-			cpu_map.sockets[socket_id].node_id = node_id;
-			cpu_map.socket_count++;
-		}
-
-		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
-			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
-				continue;
-
-			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
-				full = true;
-				goto exit;
-			}
-
-			lcore = &cpu_map.lcores[cpu_map.lcore_count];
-			lcore->socket_id = socket_id;
-			lcore->core_id =
-				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
-			cpu_map.lcore_count++;
+		if (eal_create_lcore_map(info)) {
+			full = true;
+			break;
 		}
 
 		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
 			(uint8_t *)info + info->Size);
 	}
 
+	if (eal_query_group_affinity()) {
+		/*
+		 * No need to set rte_errno here.
+		 * It is set by eal_query_group_affinity().
+		 */
+		ret = -1;
+		goto exit;
+	}
+
 exit:
 	if (full) {
 		/* Not a fatal error, but important for troubleshooting. */
@@ -138,7 +217,7 @@ struct cpu_map {
 
 	free(infos);
 
-	return 0;
+	return ret;
 }
 
 int
@@ -164,3 +243,11 @@ struct cpu_map {
 {
 	return cpu_map.sockets[socket_id].node_id;
 }
+
+PGROUP_AFFINITY
+eal_get_cpu_affinity(size_t cpu_index)
+{
+	RTE_VERIFY(cpu_index < CPU_SETSIZE);
+
+	return &cpu_map.cpus[cpu_index];
+}
diff --git a/lib/eal/windows/eal_windows.h b/lib/eal/windows/eal_windows.h
index e4c4670..ab25814 100644
--- a/lib/eal/windows/eal_windows.h
+++ b/lib/eal/windows/eal_windows.h
@@ -58,6 +58,16 @@
 unsigned int eal_socket_numa_node(unsigned int socket_id);
 
 /**
+ * Get pointer to the group affinity for the cpu.
+ *
+ * @param cpu_index
+ *  Index of the cpu, as it comes from rte_cpuset_t.
+ * @return
+ *  Pointer to the group affinity for the cpu.
+ */
+PGROUP_AFFINITY eal_get_cpu_affinity(size_t cpu_index);
+
+/**
  * Schedule code for execution in the interrupt thread.
  *
  * @param func
diff --git a/lib/eal/windows/include/rte_os.h b/lib/eal/windows/include/rte_os.h
index a0a3114..1c33058 100644
--- a/lib/eal/windows/include/rte_os.h
+++ b/lib/eal/windows/include/rte_os.h
@@ -14,6 +14,8 @@
 #include <stdlib.h>
 #include <string.h>
 
+#include <sched.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
index 59fed3c..a616703 100644
--- a/lib/eal/windows/rte_thread.c
+++ b/lib/eal/windows/rte_thread.c
@@ -1,16 +1,66 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2021 Mellanox Technologies, Ltd
+ * Copyright (C) 2022 Microsoft Corporation
  */
 
 #include <rte_common.h>
 #include <rte_errno.h>
 #include <rte_thread.h>
-#include <rte_windows.h>
+
+#include "eal_windows.h"
 
 struct eal_tls_key {
 	DWORD thread_index;
 };
 
+/* Translates the most common error codes related to threads */
+static int
+thread_translate_win32_error(DWORD error)
+{
+	switch (error) {
+	case ERROR_SUCCESS:
+		return 0;
+
+	case ERROR_INVALID_PARAMETER:
+		return EINVAL;
+
+	case ERROR_INVALID_HANDLE:
+		return EFAULT;
+
+	case ERROR_NOT_ENOUGH_MEMORY:
+		/* FALLTHROUGH */
+	case ERROR_NO_SYSTEM_RESOURCES:
+		return ENOMEM;
+
+	case ERROR_PRIVILEGE_NOT_HELD:
+		/* FALLTHROUGH */
+	case ERROR_ACCESS_DENIED:
+		return EACCES;
+
+	case ERROR_ALREADY_EXISTS:
+		return EEXIST;
+
+	case ERROR_POSSIBLE_DEADLOCK:
+		return EDEADLK;
+
+	case ERROR_INVALID_FUNCTION:
+		/* FALLTHROUGH */
+	case ERROR_CALL_NOT_IMPLEMENTED:
+		return ENOSYS;
+	}
+
+	return EINVAL;
+}
+
+static int
+thread_log_last_error(const char *message)
+{
+	DWORD error = GetLastError();
+	RTE_LOG(DEBUG, EAL, "GetLastError()=%lu: %s\n", error, message);
+
+	return thread_translate_win32_error(error);
+}
+
 rte_thread_t
 rte_thread_self(void)
 {
@@ -97,3 +147,132 @@ struct eal_tls_key {
 	}
 	return output;
 }
+
+static int
+convert_cpuset_to_affinity(const rte_cpuset_t *cpuset,
+		PGROUP_AFFINITY affinity)
+{
+	int ret = 0;
+	PGROUP_AFFINITY cpu_affinity = NULL;
+	unsigned int cpu_idx;
+
+	memset(affinity, 0, sizeof(GROUP_AFFINITY));
+	affinity->Group = (USHORT)-1;
+
+	/* Check that all cpus of the set belong to the same processor group and
+	 * accumulate thread affinity to be applied.
+	 */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+		if (!CPU_ISSET(cpu_idx, cpuset))
+			continue;
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if (affinity->Group == (USHORT)-1) {
+			affinity->Group = cpu_affinity->Group;
+		} else if (affinity->Group != cpu_affinity->Group) {
+			RTE_LOG(DEBUG, EAL, "All processors must belong to the same processor group\n");
+			ret = ENOTSUP;
+			goto cleanup;
+		}
+
+		affinity->Mask |= cpu_affinity->Mask;
+	}
+
+	if (affinity->Mask == 0) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+cleanup:
+	return ret;
+}
+
+int
+rte_thread_set_affinity_by_id(rte_thread_t thread_id,
+		const rte_cpuset_t *cpuset)
+{
+	int ret = 0;
+	GROUP_AFFINITY thread_affinity;
+	HANDLE thread_handle = NULL;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	ret = convert_cpuset_to_affinity(cpuset, &thread_affinity);
+	if (ret != 0) {
+		RTE_LOG(DEBUG, EAL, "Unable to convert cpuset to thread affinity\n");
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	if (!SetThreadGroupAffinity(thread_handle, &thread_affinity, NULL)) {
+		ret = thread_log_last_error("SetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+
+	return ret;
+}
+
+int
+rte_thread_get_affinity_by_id(rte_thread_t thread_id,
+		rte_cpuset_t *cpuset)
+{
+	HANDLE thread_handle = NULL;
+	PGROUP_AFFINITY cpu_affinity;
+	GROUP_AFFINITY thread_affinity;
+	unsigned int cpu_idx;
+	int ret = 0;
+
+	if (cpuset == NULL) {
+		ret = EINVAL;
+		goto cleanup;
+	}
+
+	thread_handle = OpenThread(THREAD_ALL_ACCESS, FALSE,
+		thread_id.opaque_id);
+	if (thread_handle == NULL) {
+		ret = thread_log_last_error("OpenThread()");
+		goto cleanup;
+	}
+
+	/* obtain previous thread affinity */
+	if (!GetThreadGroupAffinity(thread_handle, &thread_affinity)) {
+		ret = thread_log_last_error("GetThreadGroupAffinity()");
+		goto cleanup;
+	}
+
+	CPU_ZERO(cpuset);
+
+	/* Convert affinity to DPDK cpu set */
+	for (cpu_idx = 0; cpu_idx < CPU_SETSIZE; cpu_idx++) {
+
+		cpu_affinity = eal_get_cpu_affinity(cpu_idx);
+
+		if ((cpu_affinity->Group == thread_affinity.Group) &&
+		   ((cpu_affinity->Mask & thread_affinity.Mask) != 0)) {
+			CPU_SET(cpu_idx, cpuset);
+		}
+	}
+
+cleanup:
+	if (thread_handle != NULL) {
+		CloseHandle(thread_handle);
+		thread_handle = NULL;
+	}
+	return ret;
+}
-- 
1.8.3.1

[PATCH v6 3/3] test/threads: add unit test for thread API

[Tyler Retzlaff]

Establish unit test for testing thread api. Initial unit tests
for rte_thread_{get,set}_affinity_by_id().

Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
---
 app/test/meson.build    |  2 ++
 app/test/test_threads.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
 create mode 100644 app/test/test_threads.c

diff --git a/app/test/meson.build b/app/test/meson.build
index bb4621e..15591ce 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -133,6 +133,7 @@ test_sources = files(
         'test_tailq.c',
         'test_thash.c',
         'test_thash_perf.c',
+        'test_threads.c',
         'test_timer.c',
         'test_timer_perf.c',
         'test_timer_racecond.c',
@@ -239,6 +240,7 @@ fast_tests = [
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
+        ['threads_autotest', true, true],
         ['trace_autotest', true, true],
 ]
 
diff --git a/app/test/test_threads.c b/app/test/test_threads.c
new file mode 100644
index 0000000..e1a2ea5
--- /dev/null
+++ b/app/test/test_threads.c
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2022 Microsoft Corporation
+ */
+
+#include <string.h>
+#include <pthread.h>
+
+#include <rte_thread.h>
+#include <rte_debug.h>
+
+#include "test.h"
+
+RTE_LOG_REGISTER(threads_logtype_test, test.threads, INFO);
+
+static uint32_t thread_id_ready;
+
+static void *
+thread_main(void *arg)
+{
+	*(rte_thread_t *)arg = rte_thread_self();
+	__atomic_store_n(&thread_id_ready, 1, __ATOMIC_RELEASE);
+
+	return NULL;
+}
+
+static int
+test_thread_affinity(void)
+{
+	pthread_t id;
+	rte_thread_t thread_id;
+	rte_cpuset_t cpuset0;
+	rte_cpuset_t cpuset1;
+
+	RTE_TEST_ASSERT(pthread_create(&id, NULL, thread_main, &thread_id) == 0,
+		"Failed to create thread");
+
+	while (__atomic_load_n(&thread_id_ready, __ATOMIC_ACQUIRE) == 0)
+		;
+
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	size_t i;
+	for (i = 1; i < CPU_SETSIZE; i++)
+		if (CPU_ISSET(i, &cpuset0)) {
+			CPU_ZERO(&cpuset0);
+			CPU_SET(i, &cpuset0);
+
+			break;
+		}
+	RTE_TEST_ASSERT(rte_thread_set_affinity_by_id(thread_id, &cpuset0) == 0,
+		"Failed to set thread affinity");
+	RTE_TEST_ASSERT(rte_thread_get_affinity_by_id(thread_id, &cpuset1) == 0,
+		"Failed to get thread affinity");
+	RTE_TEST_ASSERT(memcmp(&cpuset0, &cpuset1, sizeof(rte_cpuset_t)) == 0,
+		"Affinity should be stable");
+
+	return 0;
+}
+
+static struct unit_test_suite threads_test_suite = {
+	.suite_name = "threads autotest",
+	.setup = NULL,
+	.teardown = NULL,
+	.unit_test_cases = {
+		TEST_CASE(test_thread_affinity),
+		TEST_CASES_END()
+	}
+};
+
+static int
+test_threads(void)
+{
+	return unit_test_suite_runner(&threads_test_suite);
+}
+
+REGISTER_TEST_COMMAND(threads_autotest, test_threads);
-- 
1.8.3.1

Re: [PATCH v6 1/3] eal: add basic thread ID and current thread identifier API

[Konstantin Ananyev]

12/05/2022 14:14, Tyler Retzlaff пишет:
> Provide a portable type-safe thread identifier.
> Provide rte_thread_self for obtaining current thread identifier.
> 
> Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>   lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
>   lib/eal/unix/rte_thread.c    | 13 +++++++++++++
>   lib/eal/version.map          |  3 +++
>   lib/eal/windows/rte_thread.c | 10 ++++++++++
>   4 files changed, 48 insertions(+)
> 
> diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> index 8be8ed8..14478ba 100644
> --- a/lib/eal/include/rte_thread.h
> +++ b/lib/eal/include/rte_thread.h
> @@ -1,7 +1,10 @@
>   /* SPDX-License-Identifier: BSD-3-Clause
>    * Copyright(c) 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>    */
>   
> +#include <stdint.h>
> +
>   #include <rte_os.h>
>   #include <rte_compat.h>
>   
> @@ -21,10 +24,29 @@
>   #endif
>   
>   /**
> + * Thread id descriptor.
> + */
> +typedef struct {
> +	uintptr_t opaque_id; /**< thread identifier */
> +} rte_thread_t;
> +
> +/**
>    * TLS key type, an opaque pointer.
>    */
>   typedef struct eal_tls_key *rte_thread_key;
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Get the id of the calling thread.
> + *
> + * @return
> + *   Return the thread id of the calling thread.
> + */
> +__rte_experimental
> +rte_thread_t rte_thread_self(void);
> +
>   #ifdef RTE_HAS_CPUSET
>   
>   /**
> diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
> index c34ede9..5e5beb1 100644
> --- a/lib/eal/unix/rte_thread.c
> +++ b/lib/eal/unix/rte_thread.c
> @@ -1,5 +1,6 @@
>   /* SPDX-License-Identifier: BSD-3-Clause
>    * Copyright 2021 Mellanox Technologies, Ltd
> + * Copyright (C) 2022 Microsoft Corporation
>    */
>   
>   #include <errno.h>
> @@ -15,6 +16,18 @@ struct eal_tls_key {
>   	pthread_key_t thread_index;
>   };
>   
> +rte_thread_t
> +rte_thread_self(void)
> +{
> +	RTE_BUILD_BUG_ON(sizeof(pthread_t) > sizeof(uintptr_t));

As a nit:
sizeof(pthread_t) > sizeof(thread_id.opaque_id)
seems a bit better/safer for me.

Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>


> +
> +	rte_thread_t thread_id;
> +
> +	thread_id.opaque_id = (uintptr_t)pthread_self();
> +
> +	return thread_id;
> +}
> +
>   int
>   rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
>   {
> diff --git a/lib/eal/version.map b/lib/eal/version.map
> index b53eeb3..05ce8f9 100644
> --- a/lib/eal/version.map
> +++ b/lib/eal/version.map
> @@ -420,6 +420,9 @@ EXPERIMENTAL {
>   	rte_intr_instance_free;
>   	rte_intr_type_get;
>   	rte_intr_type_set;
> +
> +	# added in 22.07
> +	rte_thread_self;
>   };
>   
>   INTERNAL {
> diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> index 667287c..59fed3c 100644
> --- a/lib/eal/windows/rte_thread.c
> +++ b/lib/eal/windows/rte_thread.c
> @@ -11,6 +11,16 @@ struct eal_tls_key {
>   	DWORD thread_index;
>   };
>   
> +rte_thread_t
> +rte_thread_self(void)
> +{
> +	rte_thread_t thread_id;
> +
> +	thread_id.opaque_id = GetCurrentThreadId();
> +
> +	return thread_id;
> +}
> +
>   int
>   rte_thread_key_create(rte_thread_key *key,
>   		__rte_unused void (*destructor)(void *))

Re: [PATCH v6 1/3] eal: add basic thread ID and current thread identifier API

[Tyler Retzlaff]

On Sun, May 15, 2022 at 11:02:30PM +0100, Konstantin Ananyev wrote:
> 12/05/2022 14:14, Tyler Retzlaff пишет:
> >Provide a portable type-safe thread identifier.
> >Provide rte_thread_self for obtaining current thread identifier.
> >
> >Signed-off-by: Narcisa Vasile <navasile@microsoft.com>
> >Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> >---
> >  lib/eal/include/rte_thread.h | 22 ++++++++++++++++++++++
> >  lib/eal/unix/rte_thread.c    | 13 +++++++++++++
> >  lib/eal/version.map          |  3 +++
> >  lib/eal/windows/rte_thread.c | 10 ++++++++++
> >  4 files changed, 48 insertions(+)
> >
> >diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h
> >index 8be8ed8..14478ba 100644
> >--- a/lib/eal/include/rte_thread.h
> >+++ b/lib/eal/include/rte_thread.h
> >@@ -1,7 +1,10 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright(c) 2021 Mellanox Technologies, Ltd
> >+ * Copyright (C) 2022 Microsoft Corporation
> >   */
> >+#include <stdint.h>
> >+
> >  #include <rte_os.h>
> >  #include <rte_compat.h>
> >@@ -21,10 +24,29 @@
> >  #endif
> >  /**
> >+ * Thread id descriptor.
> >+ */
> >+typedef struct {
> >+	uintptr_t opaque_id; /**< thread identifier */
> >+} rte_thread_t;
> >+
> >+/**
> >   * TLS key type, an opaque pointer.
> >   */
> >  typedef struct eal_tls_key *rte_thread_key;
> >+/**
> >+ * @warning
> >+ * @b EXPERIMENTAL: this API may change without prior notice.
> >+ *
> >+ * Get the id of the calling thread.
> >+ *
> >+ * @return
> >+ *   Return the thread id of the calling thread.
> >+ */
> >+__rte_experimental
> >+rte_thread_t rte_thread_self(void);
> >+
> >  #ifdef RTE_HAS_CPUSET
> >  /**
> >diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c
> >index c34ede9..5e5beb1 100644
> >--- a/lib/eal/unix/rte_thread.c
> >+++ b/lib/eal/unix/rte_thread.c
> >@@ -1,5 +1,6 @@
> >  /* SPDX-License-Identifier: BSD-3-Clause
> >   * Copyright 2021 Mellanox Technologies, Ltd
> >+ * Copyright (C) 2022 Microsoft Corporation
> >   */
> >  #include <errno.h>
> >@@ -15,6 +16,18 @@ struct eal_tls_key {
> >  	pthread_key_t thread_index;
> >  };
> >+rte_thread_t
> >+rte_thread_self(void)
> >+{
> >+	RTE_BUILD_BUG_ON(sizeof(pthread_t) > sizeof(uintptr_t));
> 
> As a nit:
> sizeof(pthread_t) > sizeof(thread_id.opaque_id)
> seems a bit better/safer for me.
> 
> Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>

i won't change it for now, but i'll keep it in mind for when i'm
touching this file in subsequent series submissions.

thanks

> 
> 
> >+
> >+	rte_thread_t thread_id;
> >+
> >+	thread_id.opaque_id = (uintptr_t)pthread_self();
> >+
> >+	return thread_id;
> >+}
> >+
> >  int
> >  rte_thread_key_create(rte_thread_key *key, void (*destructor)(void *))
> >  {
> >diff --git a/lib/eal/version.map b/lib/eal/version.map
> >index b53eeb3..05ce8f9 100644
> >--- a/lib/eal/version.map
> >+++ b/lib/eal/version.map
> >@@ -420,6 +420,9 @@ EXPERIMENTAL {
> >  	rte_intr_instance_free;
> >  	rte_intr_type_get;
> >  	rte_intr_type_set;
> >+
> >+	# added in 22.07
> >+	rte_thread_self;
> >  };
> >  INTERNAL {
> >diff --git a/lib/eal/windows/rte_thread.c b/lib/eal/windows/rte_thread.c
> >index 667287c..59fed3c 100644
> >--- a/lib/eal/windows/rte_thread.c
> >+++ b/lib/eal/windows/rte_thread.c
> >@@ -11,6 +11,16 @@ struct eal_tls_key {
> >  	DWORD thread_index;
> >  };
> >+rte_thread_t
> >+rte_thread_self(void)
> >+{
> >+	rte_thread_t thread_id;
> >+
> >+	thread_id.opaque_id = GetCurrentThreadId();
> >+
> >+	return thread_id;
> >+}
> >+
> >  int
> >  rte_thread_key_create(rte_thread_key *key,
> >  		__rte_unused void (*destructor)(void *))

Re: [PATCH v6 0/3] add eal functions for thread affinity and self

[David Marchand]

Hello Tyler,

On Thu, May 12, 2022 at 3:14 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> this series provides basic dependencies for additional eal thread api
> additions. series includes
>
> * basic platform error number conversion.
> * function to get current thread identifier.
> * functions to get and set affinity with platform agnostic thread
>   identifier.
> * minimal unit test of get and set affinity demonstrating usage.
>
> note: previous series introducing these functions is now superseded by
> this series.
> http://patches.dpdk.org/project/dpdk/list/?series=20472&state=*
>
> v6:
> * rebase for asan flag addition to app/test/meson.build
> * RTE_BUILD_BUG_ON(sizeof(pthread_t) > sizeof(uintptr_t)) to raise
>   attention if pthread_t implementation exceeds storage available
>   from uintptr_t.
>   note:
>     the macro is placed in rte_thread_self() body because it not
>     valid syntax at file scope of the translation unit. ordinarily
>     the macro would be used in headers but that would leak the
>     pthread_t implementation detail into the public header.

There was a missing update of MAINTAINERS in patch 3 that I fixed.
Series lgtm, applied, thanks.


Looking at the generated documentation, rte_thread.h is not exposed,
but this was already the case before the series.
Could you look into it?


-- 
David Marchand