From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 25822A0093; Mon, 15 Jun 2020 08:34:22 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A1A4054AE; Mon, 15 Jun 2020 08:34:21 +0200 (CEST) Received: from qrelay73.mxroute.com (qrelay73.mxroute.com [172.82.139.73]) by dpdk.org (Postfix) with ESMTP id 45E8F4C9D for ; Mon, 15 Jun 2020 08:34:20 +0200 (CEST) Received: from filter004.mxroute.com ([149.28.56.236] 149.28.56.236.vultr.com) (Authenticated sender: mN4UYu2MZsgR) by qrelay73.mxroute.com (ZoneMTA) with ESMTPA id 172b6af86e80004840.001 for ; Mon, 15 Jun 2020 06:34:16 +0000 X-Zone-Loop: 833b90ac0d4d0185e50d819d4f6a908b62ed25b82864 X-Originating-IP: [149.28.56.236] Received: from echo.mxrouting.net (echo.mxrouting.net [116.202.222.109]) by filter004.mxroute.com (Postfix) with ESMTPS id 329AC3EDAB; Mon, 15 Jun 2020 06:34:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=4XoZDSg8TOypiVlx7FPtDr8We3WX8faXRuM8dON0ww0=; b=Cq4H4Zxloju5st/UNdkF3EpTzN a/qbAd0C+KVsywb2V/0r4LjOdl1XWNbLQCZoKRq6XQY1IlwsKYjQZ2YOjzAK3Xk7Tr3beWaObNVVF Q/4En70Xu0bxjuLGCN9D+ZZx/1+duvUErBKyptOGnjaWfYB6wtlAjZSFHE0c/hQq2rZ+JkbcK/AcT 9ZfmOMvFePK06q+JFY3L1GHZaRzU2o5UISzFtsnNK1dSlfw9rEdh/kvWLD5Isum0u+xxFqwq6vcLZ 9rdZ82MK3SJh2vs3xa402QhXl2y7vqAFXdVylhq2nHoSQpVkSm+Y9mMgDBR0stpZUN2Gj3cQitTif DkbP+Pwg==; To: David Marchand , dev@dpdk.org Cc: Neil Horman , "Richardson, Bruce" , Thomas Monjalon References: <20200610144506.30505-1-david.marchand@redhat.com> <20200610144506.30505-8-david.marchand@redhat.com> From: "Kinsella, Ray" Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: <9209a36a-c1b0-1a7f-236e-0cfeea954c54@ashroe.eu> Date: Mon, 15 Jun 2020 07:34:12 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200610144506.30505-8-david.marchand@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH 7/7] eal: add lcore hotplug notifications X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" =46rom ABI=C2=A0 PoV, you are 100%. Is the agreed term 'callback'?, not 'notifier' for example rte_dev_event_= callback_register, rte_mem_event_callback_register I did wonder however, if all these cb's would be better handled through a= EventDev event notification style approach. Ray K On 10/06/2020 15:45, David Marchand wrote: > Now that lcores can be dynamically allocated/freed, we will have to > notify DPDK components and applications of such events for cases where > per lcore context must be allocated/initialised. > > Signed-off-by: David Marchand > --- > lib/librte_eal/common/eal_common_lcore.c | 91 +++++++++++++++++++++++= > lib/librte_eal/common/eal_common_thread.c | 11 ++- > lib/librte_eal/common/eal_private.h | 26 +++++++ > lib/librte_eal/include/rte_lcore.h | 49 ++++++++++++ > lib/librte_eal/rte_eal_version.map | 2 + > 5 files changed, 178 insertions(+), 1 deletion(-) > > diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/= common/eal_common_lcore.c > index 6aca1b2fee..3a997d8115 100644 > --- a/lib/librte_eal/common/eal_common_lcore.c > +++ b/lib/librte_eal/common/eal_common_lcore.c > @@ -212,6 +212,47 @@ rte_socket_id_by_idx(unsigned int idx) > return config->numa_nodes[idx]; > } > =20 > +struct lcore_notifier { > + TAILQ_ENTRY(lcore_notifier) next; > + rte_lcore_notifier_cb cb; > + void *arg; > +}; > +static TAILQ_HEAD(lcore_notifiers_head, lcore_notifier) lcore_notifier= s =3D > + TAILQ_HEAD_INITIALIZER(lcore_notifiers); > +static rte_spinlock_t lcore_notifiers_lock =3D RTE_SPINLOCK_INITIALIZE= R; > + > +void * > +rte_lcore_notifier_register(rte_lcore_notifier_cb cb, void *arg) > +{ > + struct lcore_notifier *notifier; > + > + if (cb =3D=3D NULL) > + return NULL; > + > + notifier =3D calloc(1, sizeof(*notifier)); > + if (notifier =3D=3D NULL) > + return NULL; > + > + notifier->cb =3D cb; > + notifier->arg =3D arg; > + rte_spinlock_lock(&lcore_notifiers_lock); > + TAILQ_INSERT_TAIL(&lcore_notifiers, notifier, next); > + rte_spinlock_unlock(&lcore_notifiers_lock); > + > + return notifier; > +} > + > +void > +rte_lcore_notifier_unregister(void *handle) > +{ > + struct lcore_notifier *notifier =3D handle; > + > + rte_spinlock_lock(&lcore_notifiers_lock); > + TAILQ_REMOVE(&lcore_notifiers, notifier, next); > + rte_spinlock_unlock(&lcore_notifiers_lock); > + free(notifier); > +} > + > rte_spinlock_t external_lcore_lock =3D RTE_SPINLOCK_INITIALIZER; > =20 > unsigned int > @@ -277,3 +318,53 @@ rte_lcore_dump(FILE *f) > } > rte_spinlock_unlock(&external_lcore_lock); > } > + > +int > +eal_lcore_external_notify_allocated(unsigned int lcore_id) > +{ > + struct lcore_notifier *notifier; > + int ret =3D 0; > + > + RTE_LOG(DEBUG, EAL, "New lcore %u.\n", lcore_id); > + rte_spinlock_lock(&lcore_notifiers_lock); > + TAILQ_FOREACH(notifier, &lcore_notifiers, next) { > + if (notifier->cb(lcore_id, RTE_LCORE_EVENT_NEW_EXTERNAL, > + notifier->arg) =3D=3D 0) > + continue; > + > + /* Some notifier refused the new lcore, inform all notifiers > + * that acked it. > + */ > + RTE_LOG(DEBUG, EAL, "A lcore notifier refused new lcore %u.\n", > + lcore_id); > + > + notifier =3D TAILQ_PREV(notifier, lcore_notifiers_head, next); > + while (notifier !=3D NULL) { > + notifier->cb(lcore_id, > + RTE_LCORE_EVENT_RELEASE_EXTERNAL, > + notifier->arg); > + notifier =3D TAILQ_PREV(notifier, lcore_notifiers_head, > + next); > + } > + ret =3D -1; > + break; > + } > + rte_spinlock_unlock(&lcore_notifiers_lock); > + > + return ret; > +} > + > +void > +eal_lcore_external_notify_removed(unsigned int lcore_id) > +{ > + struct lcore_notifier *notifier; > + > + RTE_LOG(DEBUG, EAL, "Released lcore %u.\n", lcore_id); > + rte_spinlock_lock(&lcore_notifiers_lock); > + TAILQ_FOREACH_REVERSE(notifier, &lcore_notifiers, lcore_notifiers_hea= d, > + next) { > + notifier->cb(lcore_id, RTE_LCORE_EVENT_RELEASE_EXTERNAL, > + notifier->arg); > + } > + rte_spinlock_unlock(&lcore_notifiers_lock); > +} > diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal= /common/eal_common_thread.c > index a81b192ff3..f66d1ccaef 100644 > --- a/lib/librte_eal/common/eal_common_thread.c > +++ b/lib/librte_eal/common/eal_common_thread.c > @@ -285,6 +285,12 @@ rte_thread_register(void) > =20 > rte_thread_init(lcore_id, &cpuset); > =20 > + if (lcore_id !=3D LCORE_ID_ANY && > + eal_lcore_external_notify_allocated(lcore_id) < 0) { > + eal_lcore_external_release(lcore_id); > + RTE_PER_LCORE(_lcore_id) =3D lcore_id =3D LCORE_ID_ANY; > + } > + > RTE_LOG(DEBUG, EAL, "Registered thread as lcore %u.\n", lcore_id); > RTE_PER_LCORE(thread_registered) =3D true; > } > @@ -298,8 +304,11 @@ rte_thread_unregister(void) > return; > =20 > lcore_id =3D RTE_PER_LCORE(_lcore_id); > - if (lcore_id !=3D LCORE_ID_ANY) > + if (lcore_id !=3D LCORE_ID_ANY) { > + eal_lcore_external_notify_removed(lcore_id); > eal_lcore_external_release(lcore_id); > + RTE_PER_LCORE(_lcore_id) =3D LCORE_ID_ANY; > + } > =20 > rte_thread_uninit(); > =20 > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/commo= n/eal_private.h > index 8dd850f68a..649697c368 100644 > --- a/lib/librte_eal/common/eal_private.h > +++ b/lib/librte_eal/common/eal_private.h > @@ -283,6 +283,21 @@ uint64_t get_tsc_freq_arch(void); > */ > unsigned int eal_lcore_external_reserve(void); > =20 > +/** > + * Evaluate all lcore notifiers with a RTE_LCORE_EVENT_NEW_EXTERNAL ev= ent for > + * the passed lcore. > + * If an error is returned by one of them, then this change is rolled = back: > + * all previous lcore notifiers that had acked the RTE_LCORE_EVENT_NEW= _EXTERNAL > + * event receive a RTE_LCORE_EVENT_RELEASE_EXTERNAL event for the pass= ed lcore. > + * > + * @param lcore_id > + * The lcore to consider. > + * @return > + * - 0 if all notifiers agreed on the new lcore > + * - -1 if one of them refused > + */ > +int eal_lcore_external_notify_allocated(unsigned int lcore_id); > + > /** > * Release an external lcore. > * > @@ -291,6 +306,17 @@ unsigned int eal_lcore_external_reserve(void); > */ > void eal_lcore_external_release(unsigned int lcore_id); > =20 > +/** > + * Evaluate all lcore notifiers with a RTE_LCORE_EVENT_RELEASE_EXTERNA= L event > + * for the passed lcore. > + * This function must be called with a lcore that successfully passed > + * eal_lcore_external_notify_allocated(). > + * > + * @param lcore_id > + * The lcore with role ROLE_EXTERNAL to release. > + */ > +void eal_lcore_external_notify_removed(unsigned int lcore_id); > + > /** > * Prepare physical memory mapping > * i.e. hugepages on Linux and > diff --git a/lib/librte_eal/include/rte_lcore.h b/lib/librte_eal/includ= e/rte_lcore.h > index 9cf34efef4..e0fec33d5a 100644 > --- a/lib/librte_eal/include/rte_lcore.h > +++ b/lib/librte_eal/include/rte_lcore.h > @@ -238,6 +238,55 @@ __rte_experimental > void > rte_lcore_dump(FILE *f); > =20 > +enum rte_lcore_event_type { > + RTE_LCORE_EVENT_NEW_EXTERNAL, > + RTE_LCORE_EVENT_RELEASE_EXTERNAL, > +}; > + > +/** > + * Callback prototype for getting lcore events. > + * > + * @param lcore_id > + * The lcore to consider for this event. > + * @param event > + * The type of event on the lcore. > + * @param arg > + * An opaque pointer passed at notifier registration. > + * @return > + * - -1 when refusing this event, > + * - 0 otherwise. > + */ > +typedef int (*rte_lcore_notifier_cb)(unsigned int lcore_id, > + enum rte_lcore_event_type event, void *arg); > + > +/** > + * Register a lcore notifier. > + * > + * @param cb > + * The callback invoked for each lcore event with the arg argument. > + * See rte_lcore_notifier_cb description. > + * @param arg > + * An optional argument that gets passed to the callback when it get= s > + * invoked. > + * @return > + * On success, returns an opaque pointer for the created notifier. > + * NULL on failure. > + */ > +__rte_experimental > +void * > +rte_lcore_notifier_register(rte_lcore_notifier_cb cb, void *arg); > + > +/** > + * Unregister a lcore notifier. > + * > + * @param handle > + * The handle pointer returned by a former successful call to > + * rte_lcore_notifier_register. > + */ > +__rte_experimental > +void > +rte_lcore_notifier_unregister(void *handle); > + > /** > * Set core affinity of the current thread. > * Support both EAL and non-EAL thread and update TLS. > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_ea= l_version.map > index 6754d52543..1e6f2aaacc 100644 > --- a/lib/librte_eal/rte_eal_version.map > +++ b/lib/librte_eal/rte_eal_version.map > @@ -396,6 +396,8 @@ EXPERIMENTAL { > =20 > # added in 20.08 > rte_lcore_dump; > + rte_lcore_notifier_register; > + rte_lcore_notifier_unregister; > rte_thread_register; > rte_thread_unregister; > };