From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4153DA0350; Mon, 22 Jun 2020 17:49:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E7A381D8E4; Mon, 22 Jun 2020 17:49:39 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id A7BAB1D8E1 for ; Mon, 22 Jun 2020 17:49:38 +0200 (CEST) IronPort-SDR: iQnIe3ygzr6Fa4SUTHzE96zHmqwWKOttB1b7xGw08JuZDvHGBl9LeL+yx3ZsdOruZrWG1pWl9f aPSLNUlGJttQ== X-IronPort-AV: E=McAfee;i="6000,8403,9659"; a="132222331" X-IronPort-AV: E=Sophos;i="5.75,267,1589266800"; d="scan'208";a="132222331" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2020 08:49:34 -0700 IronPort-SDR: zv+Yp2G4zGNDPIHXcCXCmmApuG1d8tPxjGe26gT3eUP4NQkmx/MHUy4LqN9riVMtGVjO20AeWB e7oe6PjQxs9Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,267,1589266800"; d="scan'208";a="310966522" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by fmsmga002.fm.intel.com with ESMTP; 22 Jun 2020 08:49:33 -0700 Received: from ORSEDG001.ED.cps.intel.com (10.7.248.4) by ORSMSX109.amr.corp.intel.com (10.22.240.7) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 22 Jun 2020 08:49:33 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.175) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 22 Jun 2020 08:49:29 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ISLtHA7xU1lhpMrKIyS+dBFq2Oq533w5ny4I/5zJbJDbZTmd0iPYfqUVJTF/r9FHKaQ0Amr4nFRal0ebtxft7OflSttbaaH4lnyDhNxFYNvGiT+hgxN7Ig1PJMxp1MjHR2CU3Ridz+0vWazP+MX3V441Ss/0DHXbIhWXWMK8kKJm+3HK5srXnR0PfvukhfB1hmzd1X3iM9iYxavxFY04paI6PJ4dQ1xrAvCSj2oPxiUzkXin7Pvj+1HRQi9GEUdIHSLlKOIzqnrxvuaD3vNzzDJvryXWJEqElB773vpSaxeo2lhrq1xgZ9xwFgn9rcGtrLwyWTEasCmsjNug1Id3AQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=55pF/HDrBBHSqEqzOaOBNq+NNbl4YJXmutnW+9k7zr4=; b=KIyqfXnkjVWlmef2DBJsm9Dw0vNgjPTvY72nFza5+yiCd/qUJBT3rTIKTB9ra65bsuHXLYyDpOzB75R1nQNASWIH0j6JaAiI9uGls16Keqnjg4Fk59U2nLqIRULFBjMVgZpdqEea3MlyeLT28pOk6QFp9uKkbuyvVLDzQ7+w9C5897PzR6EzMZWucQI5ubFlqF2FGqC0xjlPbc2fawbsZMkwXxttpGaV8sfP1F5iyaCt10HoY9MhD9tj/WCuLeDpKG8So6C5NwRJrCKgPa8SnliaWXzhqNW8KMREbB2+oh9EEmONFr+3uijHULMXUb2TpBLXhFk0tx+Evo7J2sntfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=55pF/HDrBBHSqEqzOaOBNq+NNbl4YJXmutnW+9k7zr4=; b=gw+QdP4WemRuI5m9+gkx1WCQ/smoGHBvdDjQERihLuNAvTbhhXyji4Hu3KWKeSbDI4pfTwzSKAmHpWfgMCOvlBqWuqMcVTHXlZXdJtH1WbaRLfkbmWu9e2mrs7PzD2YVobPbw8xDzYMC4tFuRRPlKsKiAE8Br0IMq4dHxFjXDrE= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by BY5PR11MB4258.namprd11.prod.outlook.com (2603:10b6:a03:1c2::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3109.22; Mon, 22 Jun 2020 15:49:26 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f160:29ab:b8f9:4189]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f160:29ab:b8f9:4189%6]) with mapi id 15.20.3109.027; Mon, 22 Jun 2020 15:49:26 +0000 From: "Ananyev, Konstantin" To: David Marchand , "dev@dpdk.org" CC: "jerinjacobk@gmail.com" , "Richardson, Bruce" , "mdr@ashroe.eu" , "ktraynor@redhat.com" , "Stokes, Ian" , "i.maximets@ovn.org" , "Thomas Monjalon" , "Mcnamara, John" , "Kovacevic, Marko" , "Burakov, Anatoly" , Olivier Matz , "Andrew Rybchenko" , Neil Horman Thread-Topic: [dpdk-dev] [PATCH v3 6/9] eal: register non-EAL threads as lcores Thread-Index: AQHWSJjc02miHDhvs0aJZlTx+B7uGKjkxSPg Date: Mon, 22 Jun 2020 15:49:26 +0000 Message-ID: References: <20200610144506.30505-1-david.marchand@redhat.com> <20200622132531.21857-1-david.marchand@redhat.com> <20200622132531.21857-7-david.marchand@redhat.com> In-Reply-To: <20200622132531.21857-7-david.marchand@redhat.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 authentication-results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.198.151.184] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 96c00222-43b3-4122-93dc-08d816c3d7ab x-ms-traffictypediagnostic: BY5PR11MB4258: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5516; x-forefront-prvs: 0442E569BC x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: frso/r7Lnc4eec0xr8SBSlx0ebsFeSKodBSOpgUv6NwNkHlCagsCCI32cRXrr/85acf/jWUofosoAy2A6ksoo+EyuZYSEkU8QwPc7q6KfWAKVQl3FS9/LD+sLQIpCnmV7zNUPU1CApqPrN/Ltiaq5D2kQPW7cKzqqTjFYef+AnwmtAY8Jvdmkft++ij4Zsy9pT0L1ig6ig6/RkwEw+apcFLeTLwwnvoMxo5Xr6b215QdE1L3PyMVup6c/qjyQ3TNhrMbwto5HokS9MtTRGqg36Awl45NVSitF9OQW6tRze/RsB14ORv12KnXOZqq8TWRSWtkxwK3XhGVOt77XIEo9g== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(366004)(346002)(376002)(396003)(39860400002)(136003)(66946007)(66446008)(64756008)(66556008)(66476007)(33656002)(54906003)(5660300002)(7416002)(110136005)(316002)(7696005)(52536014)(71200400001)(86362001)(83380400001)(8676002)(55016002)(478600001)(2906002)(4326008)(26005)(6506007)(8936002)(76116006)(9686003)(186003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: cI8Dj0azqy/FNaNSyaUlKsP56DXAeTByrX4f1M1QTkPDmHV+9oezgrmtc628uxU+1i0KMJPYPJ4272L+SOQbevIJ4dhY19PqgOjFdTKq+eoo6t384o1GC4fRaoZ0MPs2Q08n3TUEznE8nUv7ZiZ+qVJkuV+XFfS5hKt6r/vBUAYl8MotywR43MjB9rZLws/4lLAO1ys4GUZGm/F/YYzeXZXDClKYjOxlXxOzg+dh+ug5aKwuoSPtFimIE1BO6oB53zgdn9HAShog3DeRxuHhIx6XjiQDcP8X+wQxabmxxm3d70affuhq+VauRoew0sHf7ld7B30B1AXJmThL9H86gYXofg4OZeHhEDEp1SF5AQ32Uh7YEuOWwsMRZwEIcFr0H5GcVaTf54uhNCJEQH+tQzKuhnTW40yWpc/aWAhFDB6pddQfe/cpvNQ37gu0tDDL2ioptB419fqrpYtIMwUG8peCidYPvB9NNOX5FosBzLduihmJOkWTAPL/csIyDYQA Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 96c00222-43b3-4122-93dc-08d816c3d7ab X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jun 2020 15:49:26.3945 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: oeGEVvtsSnh8Dvh9HU4MtRiBuRIglQAw2nWNZyEBIMydqR5jOhj5B2NBqZe53mRBqs0O2rx+D0PK00h/ljyrgD/L4vHnp758t5VU2wc2Ugc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4258 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v3 6/9] eal: register non-EAL threads as lcores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi David, > diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/co= mmon/eal_common_lcore.c > index 86d32a3dd7..7db05428e7 100644 > --- a/lib/librte_eal/common/eal_common_lcore.c > +++ b/lib/librte_eal/common/eal_common_lcore.c > @@ -6,12 +6,13 @@ > #include > #include >=20 > -#include > -#include > -#include > -#include > #include > #include > +#include > +#include > +#include > +#include > +#include >=20 > #include "eal_private.h" > #include "eal_thread.h" > @@ -220,3 +221,38 @@ rte_socket_id_by_idx(unsigned int idx) > } > return config->numa_nodes[idx]; > } > + > +static rte_spinlock_t lcore_lock =3D RTE_SPINLOCK_INITIALIZER; > + > +unsigned int > +eal_lcore_non_eal_allocate(void) > +{ > + struct rte_config *cfg =3D rte_eal_get_configuration(); > + unsigned int lcore_id; > + > + rte_spinlock_lock(&lcore_lock); I think it will break current DPDK MP modes. The problem here - rte_config (and lcore_role[]) is in shared memory, while the lock is local. Simplest way probably to move lcore_lock to rte_config. > + for (lcore_id =3D 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { > + if (cfg->lcore_role[lcore_id] !=3D ROLE_OFF) > + continue; > + cfg->lcore_role[lcore_id] =3D ROLE_NON_EAL; > + cfg->lcore_count++; > + break; > + } > + if (lcore_id =3D=3D RTE_MAX_LCORE) > + RTE_LOG(DEBUG, EAL, "No lcore available.\n"); > + rte_spinlock_unlock(&lcore_lock); > + return lcore_id; > +} > + > +void > +eal_lcore_non_eal_release(unsigned int lcore_id) > +{ > + struct rte_config *cfg =3D rte_eal_get_configuration(); > + > + rte_spinlock_lock(&lcore_lock); > + if (cfg->lcore_role[lcore_id] =3D=3D ROLE_NON_EAL) { > + cfg->lcore_role[lcore_id] =3D ROLE_OFF; > + cfg->lcore_count--; > + } > + rte_spinlock_unlock(&lcore_lock); > +} > diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/c= ommon/eal_common_thread.c > index a7ae0691bf..1cbddc4b5b 100644 > --- a/lib/librte_eal/common/eal_common_thread.c > +++ b/lib/librte_eal/common/eal_common_thread.c > @@ -236,3 +236,36 @@ rte_ctrl_thread_create(pthread_t *thread, const char= *name, > pthread_join(*thread, NULL); > return -ret; > } > + > +void > +rte_thread_register(void) > +{ > + unsigned int lcore_id; > + rte_cpuset_t cpuset; > + > + /* EAL init flushes all lcores, we can't register before. */ > + assert(internal_config.init_complete =3D=3D 1); > + if (pthread_getaffinity_np(pthread_self(), sizeof(cpuset), > + &cpuset) !=3D 0) > + CPU_ZERO(&cpuset); > + lcore_id =3D eal_lcore_non_eal_allocate(); > + if (lcore_id >=3D RTE_MAX_LCORE) > + lcore_id =3D LCORE_ID_ANY; > + rte_thread_init(lcore_id, &cpuset); So we just setting affinity to the same value, right? Not a big deal, but might be easier to allow rte_thread_init() to accept cpuset=3D=3DNULL (and just don't change thread affinity in that c= ase) > + if (lcore_id !=3D LCORE_ID_ANY) > + RTE_LOG(DEBUG, EAL, "Registered non-EAL thread as lcore %u.\n", > + lcore_id); > +} > + > +void > +rte_thread_unregister(void) > +{ > + unsigned int lcore_id =3D rte_lcore_id(); > + > + if (lcore_id !=3D LCORE_ID_ANY) > + eal_lcore_non_eal_release(lcore_id); > + rte_thread_uninit(); > + if (lcore_id !=3D LCORE_ID_ANY) > + RTE_LOG(DEBUG, EAL, "Unregistered non-EAL thread (was lcore %u).\n", > + lcore_id); > +} > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/= eal_private.h > index 0592fcd694..73238ff157 100644 > --- a/lib/librte_eal/common/eal_private.h > +++ b/lib/librte_eal/common/eal_private.h > @@ -396,6 +396,24 @@ uint64_t get_tsc_freq(void); > */ > uint64_t get_tsc_freq_arch(void); >=20 > +/** > + * Allocate a free lcore to associate to a non-EAL thread. > + * > + * @return > + * - the id of a lcore with role ROLE_NON_EAL on success. > + * - RTE_MAX_LCORE if none was available. > + */ > +unsigned int eal_lcore_non_eal_allocate(void); > + > +/** > + * Release the lcore used by a non-EAL thread. > + * Counterpart of eal_lcore_non_eal_allocate(). > + * > + * @param lcore_id > + * The lcore with role ROLE_NON_EAL to release. > + */ > +void eal_lcore_non_eal_release(unsigned int lcore_id); > + > /** > * Prepare physical memory mapping > * i.e. hugepages on Linux and > diff --git a/lib/librte_eal/include/rte_lcore.h b/lib/librte_eal/include/= rte_lcore.h > index 3968c40693..ea86220394 100644 > --- a/lib/librte_eal/include/rte_lcore.h > +++ b/lib/librte_eal/include/rte_lcore.h > @@ -31,6 +31,7 @@ enum rte_lcore_role_t { > ROLE_RTE, > ROLE_OFF, > ROLE_SERVICE, > + ROLE_NON_EAL, > }; >=20 > /** > @@ -67,7 +68,8 @@ rte_lcore_has_role(unsigned int lcore_id, enum rte_lcor= e_role_t role); > * to run threads with lcore IDs 0, 1, 2 and 3 on physical core 10.. > * > * @return > - * Logical core ID (in EAL thread) or LCORE_ID_ANY (in non-EAL thread) > + * Logical core ID (in EAL thread or registered non-EAL thread) or > + * LCORE_ID_ANY (in unregistered non-EAL thread) > */ > static inline unsigned > rte_lcore_id(void) > @@ -279,6 +281,20 @@ int rte_thread_setname(pthread_t id, const char *nam= e); > __rte_experimental > int rte_thread_getname(pthread_t id, char *name, size_t len); >=20 > +/** > + * Register current non-EAL thread as a lcore. > + */ > +__rte_experimental > +void > +rte_thread_register(void); > + > +/** > + * Unregister current thread and release lcore if one was associated. > + */ > +__rte_experimental > +void > +rte_thread_unregister(void); > + > /** > * Create a control thread. > * > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_= version.map > index 5831eea4b0..39c41d445d 100644 > --- a/lib/librte_eal/rte_eal_version.map > +++ b/lib/librte_eal/rte_eal_version.map > @@ -396,6 +396,8 @@ EXPERIMENTAL { >=20 > # added in 20.08 > __rte_trace_mem_per_thread_free; > + rte_thread_register; > + rte_thread_unregister; > }; >=20 > INTERNAL { > diff --git a/lib/librte_mempool/rte_mempool.h b/lib/librte_mempool/rte_me= mpool.h > index 652d19f9f1..9e0ee052b3 100644 > --- a/lib/librte_mempool/rte_mempool.h > +++ b/lib/librte_mempool/rte_mempool.h > @@ -28,9 +28,9 @@ > * rte_mempool_get() or rte_mempool_put() are designed to be called from= an EAL > * thread due to the internal per-lcore cache. Due to the lack of cachin= g, > * rte_mempool_get() or rte_mempool_put() performance will suffer when c= alled > - * by non-EAL threads. Instead, non-EAL threads should call > - * rte_mempool_generic_get() or rte_mempool_generic_put() with a user ca= che > - * created with rte_mempool_cache_create(). > + * by unregistered non-EAL threads. Instead, unregistered non-EAL thread= s > + * should call rte_mempool_generic_get() or rte_mempool_generic_put() wi= th a > + * user cache created with rte_mempool_cache_create(). > */ >=20 > #include > @@ -1233,7 +1233,7 @@ void rte_mempool_dump(FILE *f, struct rte_mempool *= mp); > /** > * Create a user-owned mempool cache. > * > - * This can be used by non-EAL threads to enable caching when they > + * This can be used by unregistered non-EAL threads to enable caching wh= en they > * interact with a mempool. > * > * @param size > @@ -1264,7 +1264,8 @@ rte_mempool_cache_free(struct rte_mempool_cache *ca= che); > * @param lcore_id > * The logical core id. > * @return > - * A pointer to the mempool cache or NULL if disabled or non-EAL threa= d. > + * A pointer to the mempool cache or NULL if disabled or unregistered = non-EAL > + * thread. > */ > static __rte_always_inline struct rte_mempool_cache * > rte_mempool_default_cache(struct rte_mempool *mp, unsigned lcore_id) > -- > 2.23.0