From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CEA22A0093; Mon, 15 Jun 2020 08:44:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 63DE55B3C; Mon, 15 Jun 2020 08:44:02 +0200 (CEST) Received: from dal2relay195.mxroute.com (dal2relay195.mxroute.com [64.40.26.195]) by dpdk.org (Postfix) with ESMTP id C977654AE for ; Mon, 15 Jun 2020 08:44:00 +0200 (CEST) Received: from filter004.mxroute.com ([149.28.56.236] 149.28.56.236.vultr.com) (Authenticated sender: mN4UYu2MZsgR) by dal2relay195.mxroute.com (ZoneMTA) with ESMTPSA id 172b6b865170007667.001 for (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256); Mon, 15 Jun 2020 06:43:57 +0000 X-Zone-Loop: 0214815b77f46cf72560419ae790e2e81341d43c620c X-Originating-IP: [149.28.56.236] Received: from echo.mxrouting.net (echo.mxrouting.net [116.202.222.109]) by filter004.mxroute.com (Postfix) with ESMTPS id 1018B3EDA7; Mon, 15 Jun 2020 06:43:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=dWix8ksDFp4rSnyTYNrnj/VqdtoHNFzaL6lCOijxb4E=; b=JjOqOWJcvrCBRTweEfbqK3SlWG YLEgEzDNl7Tw0otiw03/kVJqs62PmtyBztE+7/1wQaY8osr2oDrVHtXu6OkZDGCLKXD2d3b5Y0rdu KtO1ukrQQRvBSkiTuvkDMhe6WQ+DRxyAofHMmiiEMsOZnJPMpZNnFgJuv+OT2arzJHC2F2OyoXZeL XhG5RNNJgcMUN8DT4PdhfQY4DeVQNk2XENBXdLdqJmRWUBh2pcI4FHD2AMlF0/wLz8nErnBuwyJLC Uc8FclPJ6o+Q5Moua/hEFKZ6nHgbmdCSFuAAFkoCvaBqO6DkH1AoXJ2RG9bBs20Eh5W9+Ky1I0tU2 j01Ps5Mw==; To: David Marchand , dev@dpdk.org Cc: John McNamara , Marko Kovacevic , Neil Horman References: <20200610144506.30505-1-david.marchand@redhat.com> <20200610144506.30505-6-david.marchand@redhat.com> From: "Kinsella, Ray" Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: <6838634a-8b99-5d33-d23e-d097a047b781@ashroe.eu> Date: Mon, 15 Jun 2020 07:43:54 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200610144506.30505-6-david.marchand@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: en-US X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH 5/7] eal: register non-EAL threads as lcore X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/06/2020 15:45, David Marchand wrote: > DPDK allows calling some part of its API from a non-EAL thread but this > has some limitations. > OVS (and other applications) has its own thread management but still > want to avoid such limitations by hacking RTE_PER_LCORE(_lcore_id) and > faking EAL threads potentially unknown of some DPDK component. > > Introduce a new API to register non-EAL thread and associate them to a > free lcore with a new EXTERNAL role. > This role denotes lcores that do not run DPDK mainloop and as such > prevents use of rte_eal_wait_lcore() and consorts. > > Signed-off-by: David Marchand > --- > doc/guides/howto/debug_troubleshoot.rst | 5 ++- > lib/librte_eal/common/eal_common_lcore.c | 44 +++++++++++++++++-- > lib/librte_eal/common/eal_common_thread.c | 45 ++++++++++++++++++++ > lib/librte_eal/common/eal_private.h | 17 ++++++++ > lib/librte_eal/include/rte_eal.h | 9 ---- > lib/librte_eal/include/rte_lcore.h | 52 +++++++++++++++++------ > lib/librte_eal/rte_eal_version.map | 4 ++ > 7 files changed, 147 insertions(+), 29 deletions(-) > > diff --git a/doc/guides/howto/debug_troubleshoot.rst b/doc/guides/howto/debug_troubleshoot.rst > index cef016b2fe..056f556a73 100644 > --- a/doc/guides/howto/debug_troubleshoot.rst > +++ b/doc/guides/howto/debug_troubleshoot.rst > @@ -307,8 +307,9 @@ Custom worker function :numref:`dtg_distributor_worker`. > > #. Configuration issue isolation > > - * Identify core role using ``rte_eal_lcore_role`` to identify RTE, OFF and > - SERVICE. Check performance functions are mapped to run on the cores. > + * Identify core role using ``rte_eal_lcore_role`` to identify RTE, OFF, > + SERVICE and EXTERNAL. Check performance functions are mapped to run on the > + cores. > > * For high-performance execution logic ensure running it on correct NUMA > and non-master core. > diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c > index 5404922a87..35d6c1295e 100644 > --- a/lib/librte_eal/common/eal_common_lcore.c > +++ b/lib/librte_eal/common/eal_common_lcore.c > @@ -6,13 +6,15 @@ > #include > #include > > -#include > -#include > -#include > -#include > #include > #include > +#include > +#include > +#include > +#include > +#include > > +#include "eal_internal_cfg.h" > #include "eal_private.h" > #include "eal_thread.h" > > @@ -209,3 +211,37 @@ rte_socket_id_by_idx(unsigned int idx) > } > return config->numa_nodes[idx]; > } > + > +rte_spinlock_t external_lcore_lock = RTE_SPINLOCK_INITIALIZER; > + > +unsigned int > +eal_lcore_external_reserve(void) > +{ > + struct rte_config *cfg = rte_eal_get_configuration(); > + unsigned int lcore_id; > + > + rte_spinlock_lock(&external_lcore_lock); > + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { > + if (rte_eal_lcore_role(lcore_id) != ROLE_OFF) > + continue; > + cfg->lcore_role[lcore_id] = ROLE_EXTERNAL; > + cfg->lcore_count++; > + break; > + } > + rte_spinlock_unlock(&external_lcore_lock); > + > + return lcore_id; > +} > + > +void > +eal_lcore_external_release(unsigned int lcore_id) > +{ > + struct rte_config *cfg = rte_eal_get_configuration(); > + > + rte_spinlock_lock(&external_lcore_lock); > + if (rte_eal_lcore_role(lcore_id) == ROLE_EXTERNAL) { > + cfg->lcore_role[lcore_id] = ROLE_OFF; > + cfg->lcore_count--; > + } > + rte_spinlock_unlock(&external_lcore_lock); > +} > diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c > index 8a973ca8ac..d57a6ec797 100644 > --- a/lib/librte_eal/common/eal_common_thread.c > +++ b/lib/librte_eal/common/eal_common_thread.c > @@ -29,6 +29,7 @@ RTE_DEFINE_PER_LCORE(int, _thread_id) = -1; > static RTE_DEFINE_PER_LCORE(unsigned int, _socket_id) = > (unsigned int)SOCKET_ID_ANY; > static RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset); > +static RTE_DEFINE_PER_LCORE(bool, thread_registered); > > unsigned rte_socket_id(void) > { > @@ -255,3 +256,47 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name, > pthread_join(*thread, NULL); > return -ret; > } > + > +void > +rte_thread_register(void) > +{ > + unsigned int lcore_id; > + rte_cpuset_t cpuset; > + > + /* EAL init flushes all lcores, we can't register before. */ > + assert(internal_config.init_complete == 1); > + > + if (RTE_PER_LCORE(thread_registered)) > + return; > + > + if (pthread_getaffinity_np(pthread_self(), sizeof(cpuset), > + &cpuset) != 0) > + CPU_ZERO(&cpuset); > + > + lcore_id = eal_lcore_external_reserve(); > + if (lcore_id >= RTE_MAX_LCORE) > + lcore_id = LCORE_ID_ANY; > + > + rte_thread_init(lcore_id, &cpuset); > + > + RTE_LOG(DEBUG, EAL, "Registered thread as lcore %u.\n", lcore_id); > + RTE_PER_LCORE(thread_registered) = true; > +} > + > +void > +rte_thread_unregister(void) > +{ > + unsigned int lcore_id; > + > + if (!RTE_PER_LCORE(thread_registered)) > + return; > + > + lcore_id = RTE_PER_LCORE(_lcore_id); > + if (lcore_id != LCORE_ID_ANY) > + eal_lcore_external_release(lcore_id); > + > + rte_thread_uninit(); > + > + RTE_LOG(DEBUG, EAL, "Unregistered thread (was lcore %u).\n", lcore_id); > + RTE_PER_LCORE(thread_registered) = false; > +} > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h > index 869ce183ad..8dd850f68a 100644 > --- a/lib/librte_eal/common/eal_private.h > +++ b/lib/librte_eal/common/eal_private.h > @@ -274,6 +274,23 @@ uint64_t get_tsc_freq(void); > */ > uint64_t get_tsc_freq_arch(void); > > +/** > + * Ask for a free lcore to associate to a non-EAL thread. > + * > + * @return > + * The id of a lcore with role ROLE_EXTERNAL or RTE_MAX_LCORE if none was > + * available. > + */ > +unsigned int eal_lcore_external_reserve(void); > + > +/** > + * Release an external lcore. > + * > + * @param lcore_id > + * The lcore with role ROLE_EXTERNAL to release. > + */ > +void eal_lcore_external_release(unsigned int lcore_id); > + > /** > * Prepare physical memory mapping > * i.e. hugepages on Linux and > diff --git a/lib/librte_eal/include/rte_eal.h b/lib/librte_eal/include/rte_eal.h > index 2edf8c6556..0913d1947c 100644 > --- a/lib/librte_eal/include/rte_eal.h > +++ b/lib/librte_eal/include/rte_eal.h > @@ -31,15 +31,6 @@ extern "C" { > /* Maximum thread_name length. */ > #define RTE_MAX_THREAD_NAME_LEN 16 > > -/** > - * The lcore role (used in RTE or not). > - */ > -enum rte_lcore_role_t { > - ROLE_RTE, > - ROLE_OFF, > - ROLE_SERVICE, > -}; > - > /** > * The type of process in a linux, multi-process setup > */ > diff --git a/lib/librte_eal/include/rte_lcore.h b/lib/librte_eal/include/rte_lcore.h > index 5c1d1926e9..d59c4edc67 100644 > --- a/lib/librte_eal/include/rte_lcore.h > +++ b/lib/librte_eal/include/rte_lcore.h > @@ -24,6 +24,16 @@ extern "C" { > > RTE_DECLARE_PER_LCORE(unsigned, _lcore_id); /**< Per thread "lcore id". */ > > +/** > + * The lcore role (used in RTE or not). > + */ > +enum rte_lcore_role_t { > + ROLE_RTE, > + ROLE_OFF, > + ROLE_SERVICE, > + ROLE_EXTERNAL, > +}; > + > /** > * Get a lcore's role. > * > @@ -34,6 +44,20 @@ RTE_DECLARE_PER_LCORE(unsigned, _lcore_id); /**< Per thread "lcore id". */ > */ > enum rte_lcore_role_t rte_eal_lcore_role(unsigned int lcore_id); > > +/** > + * Test if the core supplied has a specific role > + * > + * @param lcore_id > + * The identifier of the lcore, which MUST be between 0 and > + * RTE_MAX_LCORE-1. > + * @param role > + * The role to be checked against. > + * @return > + * Boolean value: positive if test is true; otherwise returns 0. > + */ > +int > +rte_lcore_has_role(unsigned int lcore_id, enum rte_lcore_role_t role); > + > /** > * Return the Application thread ID of the execution unit. > * > @@ -256,6 +280,20 @@ int rte_thread_setname(pthread_t id, const char *name); > __rte_experimental > int rte_thread_getname(pthread_t id, char *name, size_t len); > > +/** > + * Register current non-EAL thread as a lcore. > + */ > +__rte_experimental > +void > +rte_thread_register(void); > + > +/** > + * Unregister current thread and release lcore if one was associated. > + */ > +__rte_experimental > +void > +rte_thread_unregister(void); > + > /** > * Create a control thread. > * > @@ -283,20 +321,6 @@ rte_ctrl_thread_create(pthread_t *thread, const char *name, > const pthread_attr_t *attr, > void *(*start_routine)(void *), void *arg); > > -/** > - * Test if the core supplied has a specific role > - * > - * @param lcore_id > - * The identifier of the lcore, which MUST be between 0 and > - * RTE_MAX_LCORE-1. > - * @param role > - * The role to be checked against. > - * @return > - * Boolean value: positive if test is true; otherwise returns 0. > - */ > -int > -rte_lcore_has_role(unsigned int lcore_id, enum rte_lcore_role_t role); > - > #ifdef __cplusplus > } > #endif > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map > index fdfc3f1a88..f5b68d2815 100644 > --- a/lib/librte_eal/rte_eal_version.map > +++ b/lib/librte_eal/rte_eal_version.map > @@ -393,4 +393,8 @@ EXPERIMENTAL { > rte_trace_point_lookup; > rte_trace_regexp; > rte_trace_save; > + > + # added in 20.08 > + rte_thread_register; > + rte_thread_unregister; > }; Acked-by: Ray Kinsella