From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B95A45C74 for ; Mon, 4 Nov 2024 09:57:30 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4F53A40B9F; Mon, 4 Nov 2024 09:57:30 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id CAACF402AF for ; Mon, 4 Nov 2024 09:57:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1730710647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5ga9Qj8HGk0Pnc4VupQkP7sMVRhTdKhJIi80l6ndoLA=; b=PY44NHQxArbsG6UDHbl67HR5zwra0Ha0hlEk1tSPHnxr0NVqbC/EGJG2MJg3yS9P0Ow2MO 4j3NWCuBCbheL9LejOdDKzGrApoqs1KzoFklEJDI5YZnlp8tIAx+/l1RcUQua0bnRwiDhH 8tyw2VxWCRAZQLuoRwa/xyxPisistRg= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-64-1rM2cb8TOMisZ0cFAp90qg-1; Mon, 04 Nov 2024 03:57:21 -0500 X-MC-Unique: 1rM2cb8TOMisZ0cFAp90qg-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4BCE61956046; Mon, 4 Nov 2024 08:57:19 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.57]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4217E300018D; Mon, 4 Nov 2024 08:57:15 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: thomas@monjalon.net, stephen@networkplumber.org, luca.boccassi@gmail.com, stable@dpdk.org, Luca Boccassi , Chengwen Feng , Tyler Retzlaff Subject: [PATCH v4] eal/unix: optimize thread creation Date: Mon, 4 Nov 2024 09:57:08 +0100 Message-ID: <20241104085708.3596816-1-david.marchand@redhat.com> In-Reply-To: <20241102100839.2325651-1-david.marchand@redhat.com> References: <20241102100839.2325651-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Setting the cpu affinity of the child thread from the parent thread is racy when using pthread_setaffinity_np, as the child thread may start running and initialize before affinity is set. On the other hand, setting the cpu affinity from the child thread itself may fail, so the parent thread waits for the child thread to report whether this call succeeded. This synchronisation point resulted in a significant slow down of rte_thread_create() (as seen in the lcores_autotest unit tests, in OBS for some ARM systems). Another option for setting cpu affinity is to use the not portable pthread_attr_setaffinity_np available in FreeBSD and glibc, but not available in musl. Fixes: b28c6196b132 ("eal/unix: fix thread creation") Cc: stable@dpdk.org Signed-off-by: David Marchand Acked-by: Luca Boccassi Acked-by: Stephen Hemminger Acked-by: Chengwen Feng --- Changes since v3: - since _np symbols are in (non standard header) pthread_np.h on FreeBSD, and since this header is unconditionnally included in rte_os.h, assumed availability of pthread_attr_setaffinity_np (added in 8.0), Changes since v2: - added pthread_attr_setaffinity_np() detection, Changes since v1: - fixed build with FreeBSD, --- lib/eal/unix/meson.build | 5 +++++ lib/eal/unix/rte_thread.c | 25 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/lib/eal/unix/meson.build b/lib/eal/unix/meson.build index cc7d67dd32..f1eb82e16a 100644 --- a/lib/eal/unix/meson.build +++ b/lib/eal/unix/meson.build @@ -11,3 +11,8 @@ sources += files( 'eal_unix_timer.c', 'rte_thread.c', ) + +if is_freebsd or cc.has_function('pthread_attr_setaffinity_np', args: '-D_GNU_SOURCE', + prefix : '#include ') + cflags += '-DRTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP' +endif diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c index 1b4c73f58e..ea629c2065 100644 --- a/lib/eal/unix/rte_thread.c +++ b/lib/eal/unix/rte_thread.c @@ -19,6 +19,7 @@ struct eal_tls_key { pthread_key_t thread_index; }; +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP struct thread_start_context { rte_thread_func thread_func; void *thread_args; @@ -28,6 +29,7 @@ struct thread_start_context { int wrapper_ret; bool wrapper_done; }; +#endif static int thread_map_priority_to_os_value(enum rte_thread_priority eal_pri, int *os_pri, @@ -88,6 +90,7 @@ thread_map_os_priority_to_eal_priority(int policy, int os_pri, return 0; } +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP static void * thread_start_wrapper(void *arg) { @@ -113,6 +116,7 @@ thread_start_wrapper(void *arg) return (void *)(uintptr_t)thread_func(thread_args); } +#endif int rte_thread_create(rte_thread_t *thread_id, @@ -126,6 +130,7 @@ rte_thread_create(rte_thread_t *thread_id, .sched_priority = 0, }; int policy = SCHED_OTHER; +#ifndef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP struct thread_start_context ctx = { .thread_func = thread_func, .thread_args = args, @@ -134,6 +139,7 @@ rte_thread_create(rte_thread_t *thread_id, .wrapper_mutex = PTHREAD_MUTEX_INITIALIZER, .wrapper_cond = PTHREAD_COND_INITIALIZER, }; +#endif if (thread_attr != NULL) { ret = pthread_attr_init(&attr); @@ -144,6 +150,16 @@ rte_thread_create(rte_thread_t *thread_id, attrp = &attr; +#ifdef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP + if (CPU_COUNT(&thread_attr->cpuset) > 0) { + ret = pthread_attr_setaffinity_np(attrp, sizeof(thread_attr->cpuset), + &thread_attr->cpuset); + if (ret != 0) { + EAL_LOG(DEBUG, "pthread_attr_setaffinity_np failed"); + goto cleanup; + } + } +#endif /* * Set the inherit scheduler parameter to explicit, * otherwise the priority attribute is ignored. @@ -178,6 +194,14 @@ rte_thread_create(rte_thread_t *thread_id, } } +#ifdef RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP + ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp, + (void *)(void *)thread_func, args); + if (ret != 0) { + EAL_LOG(DEBUG, "pthread_create failed"); + goto cleanup; + } +#else /* !RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP */ ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp, thread_start_wrapper, &ctx); if (ret != 0) { @@ -193,6 +217,7 @@ rte_thread_create(rte_thread_t *thread_id, if (ret != 0) rte_thread_join(*thread_id, NULL); +#endif /* RTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP */ cleanup: if (attrp != NULL) -- 2.46.2