From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE7BAA0C4B; Thu, 11 Nov 2021 02:34:58 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 655B6410F7; Thu, 11 Nov 2021 02:34:55 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id 4B9F340DF5 for ; Thu, 11 Nov 2021 02:34:53 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1059) id 9068820C355B; Wed, 10 Nov 2021 17:34:52 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 9068820C355B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1636594492; bh=902FgvY+BPAylWKaKY9xg/NOteJvYNaMzdVbP9RLU58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lN+iffohXmkMvxDsoG4rZrAsG6dhPnqvF1gtF6+IV0UfE6djOwCn5lpSjoUaz4Bo1 84kCT1fPhvDcEpqJv+vjDa8W/AsiQkZk4d/dGZ9nQMdXvxCOn1YHjM1hsFuoih51rJ ZpphVYVhOBiGDPf0rODtsU1Y23Jwud7zGsEmctrc= From: Narcisa Ana Maria Vasile To: dev@dpdk.org, thomas@monjalon.net, dmitry.kozliuk@gmail.com, khot@microsoft.com, navasile@microsoft.com, dmitrym@microsoft.com, roretzla@microsoft.com, talshn@nvidia.com, ocardona@microsoft.com Cc: bruce.richardson@intel.com, david.marchand@redhat.com, pallavi.kadam@intel.com Subject: [PATCH v18 0/8] eal: Add EAL API for threading Date: Wed, 10 Nov 2021 17:33:37 -0800 Message-Id: <1636594425-9692-1-git-send-email-navasile@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1636513302-7359-1-git-send-email-navasile@linux.microsoft.com> References: <1636513302-7359-1-git-send-email-navasile@linux.microsoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Narcisa Vasile EAL thread API **Problem Statement** DPDK currently uses the pthread interface to create and manage threads. Windows does not support the POSIX thread programming model, so it currently relies on a header file that hides the Windows calls under pthread matched interfaces. Given that EAL should isolate the environment specifics from the applications and libraries and mediate all the communication with the operating systems, a new EAL interface is needed for thread management. **Goals** * Introduce a generic EAL API for threading support that will remove the current Windows pthread.h shim. * Replace references to pthread_* across the DPDK codebase with the new RTE_THREAD_* API. * Allow users to choose between using the RTE_THREAD_* API or a 3rd party thread library through a configuration option. **Design plan** New API main files: * rte_thread.h (librte_eal/include) * rte_thread.c (librte_eal/windows) * rte_thread.c (librte_eal/common) **A schematic example of the design** -------------------------------------------------- lib/librte_eal/include/rte_thread.h int rte_thread_create(); lib/librte_eal/common/rte_thread.c int rte_thread_create() { return pthread_create(); } lib/librte_eal/windows/rte_thread.c int rte_thread_create() { return CreateThread(); } ----------------------------------------------------- **Thread attributes** When or after a thread is created, specific characteristics of the thread can be adjusted. Currently in DPDK most threads operate at the OS-default priority level but there are cases when increasing the priority is useful. For example, high-performance applications require elevated priority to avoid being preempted by other threads on the system. The following structure that represents thread attributes has been defined: typedef struct { enum rte_thread_priority priority; rte_cpuset_t cpuset; } rte_thread_attr_t; The *rte_thread_create()* function can optionally receive an rte_thread_attr_t object that will cause the thread to be created with the affinity and priority described by the attributes object. If no rte_thread_attr_t is passed (parameter is NULL), the default affinity and priority are used. An rte_thread_attr_t object can also be set to the default values by calling *rte_thread_attr_init()*. *Priority* is represented through an enum that currently advertises two values for priority: - RTE_THREAD_PRIORITY_NORMAL - RTE_THREAD_PRIORITY_REALTIME_CRITICAL The enum can be extended to allow for multiple priority levels. rte_thread_set_priority - sets the priority of a thread rte_thread_get_priority - retrieves the priority of a thread from the OS rte_thread_attr_set_priority - updates an rte_thread_attr_t object with a new value for priority *Affinity* is described by the already known “rte_cpuset_t” type. rte_thread_attr_set/get_affinity - sets/gets the affinity field in a rte_thread_attr_t object rte_thread_set/get_affinity – sets/gets the affinity of a thread **Errors** As different platforms have different error codes, the approach here is to translate the Windows error to POSIX-style ones to have uniformity over the values returned. **Future work** The long term plan is for EAL to provide full threading support: * Add support for conditional variables * Additional functionality offered by pthread_* (such as pthread_setname_np, etc.) v18: - Squash unit tests in corresponding patches. - Prevent priority to be set to realtime on non-Windows systems. - Use already existing affinity function in rte_thread_create() v17: - Move unrelated changes to the correct patch. - Rename RTE_STATIC_MUTEX to avoid confusion, since the mutex is still dynamically initialized behind the scenes. - Break down the unit tests into smaller patches and reorder them. - Remove duplicated code in header - Improve commit messages and cover letter. v16: - Fix warning on freebsd by adding cast - Change affinity unit test to consider ases when the requested CPU are not available on the system. - Fix priority unit test to avoid termination of thread before the priority is checked. v15: - Add try_lock mutex functionality. If the mutex is already owned by a different thread, the function returns immediately. Otherwise, the mutex will be acquired. - Add function for getting the priority of a thread. An auxiliary function that translates the OS priority to the EAL accepted ones is added. - Fix unit tests logging, add descriptive asserts that mark test failures. Verify mutex locking, verify barrier return values. Add test for statically initialized mutexes. - Fix Alpine build by removing the use of pthread_attr_set_affinity() and using pthread_set_affinity() after the thread is created. v14: - Remove patch "eal: add EAL argument for setting thread priority" This will be added later when enabling the new threading API. - Remove priority enum value "_UNDEFINED". NORMAL is used as the default. - Fix issue with thread return value. v13: - Fix syntax error in unit tests v12: - Fix freebsd warning about initializer in unit tests v11: - Add unit tests for thread API - Rebase v10: - Remove patch no. 10. It will be broken down in subpatches and sent as a different patchset that depends on this one. This is done due to the ABI breaks that would be caused by patch 10. - Replace unix/rte_thread.c with common/rte_thread.c - Remove initializations that may prevent compiler from issuing useful warnings. - Remove rte_thread_types.h and rte_windows_thread_types.h - Remove unneeded priority macros (EAL_THREAD_PRIORITY*) - Remove functions that retrieves thread handle from process handle - Remove rte_thread_cancel() until same behavior is obtained on all platforms. - Fix rte_thread_detach() function description, return value and remove empty line. - Reimplement mutex functions. Add compatible representation for mutex identifier. Add macro to replace static mutex initialization instances. - Fix commit messages (lines too long, remove unicode symbols) v9: - Sign patches v8: - Rebase - Add rte_thread_detach() API - Set default priority, when user did not specify a value v7: Based on DmitryK's review: - Change thread id representation - Change mutex id representation - Implement static mutex inititalizer for Windows - Change barrier identifier representation - Improve commit messages - Add missing doxygen comments - Split error translation function - Improve name for affinity function - Remove cpuset_size parameter - Fix eal_create_cpu_map function - Map EAL priority values to OS specific values - Add thread wrapper for start routine - Do not export rte_thread_cancel() on Windows - Cleanup, fix comments, fix typos. v6: - improve error-translation function - call the error translation function in rte_thread_value_get() v5: - update cover letter with more details on the priority argument v4: - fix function description - rebase v3: - rebase v2: - revert changes that break ABI - break up changes into smaller patches - fix coding style issues - fix issues with errors - fix parameter type in examples/kni.c Narcisa Vasile (8): eal: add basic threading functions eal: add thread attributes eal/windows: translate Windows errors to errno-style errors eal: implement functions for thread affinity management eal: implement thread priority management functions eal: add thread lifetime management eal: implement functions for thread barrier management eal: implement functions for mutex management app/test/meson.build | 2 + app/test/test_threads.c | 372 ++++++++++++++++++ lib/eal/common/meson.build | 1 + lib/eal/common/rte_thread.c | 511 +++++++++++++++++++++++++ lib/eal/include/rte_thread.h | 412 +++++++++++++++++++- lib/eal/unix/meson.build | 1 - lib/eal/unix/rte_thread.c | 92 ----- lib/eal/version.map | 22 ++ lib/eal/windows/eal_lcore.c | 176 ++++++--- lib/eal/windows/eal_windows.h | 10 + lib/eal/windows/include/sched.h | 2 +- lib/eal/windows/rte_thread.c | 656 ++++++++++++++++++++++++++++++-- 12 files changed, 2084 insertions(+), 173 deletions(-) create mode 100644 app/test/test_threads.c create mode 100644 lib/eal/common/rte_thread.c delete mode 100644 lib/eal/unix/rte_thread.c -- 2.31.0.vfs.0.1