From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2082643B53; Tue, 20 Feb 2024 10:11:50 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 08F74402D1; Tue, 20 Feb 2024 10:11:50 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) by mails.dpdk.org (Postfix) with ESMTP id 2AEB6402B8 for ; Tue, 20 Feb 2024 10:11:47 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708420308; x=1739956308; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=btH8fvLwnABzMw1K7Z1vP3AtbZ2rj1rWHm3l3HPQNso=; b=MSwWXPlNDOjAokJm/L3tm8RjOkk1QGZvH7GJy3tyu4BJF4mFjlqWnuTF wbCVaTheZQtpQyfHi+MszltaisdoANYM+3zPDJHP5x/rxtkx37IYDhcIc cXJMXV6jaGA87voo2uTrk8jjp9fJDiEI4XvabpoBZF6qXzg1FXTW+HV3C W90fEVp8NSjZo9z+xCz+JqfggL2eSPNWizuVk26XwpNl9w6fOyOisg9UY WjrI6peO2VnrV8pE9s3Vg6QQHadf7flp43tW7IZ81ImaHbKzikh8FJY9e iL1bmnqeTAfPjHuexnc2OVMtH195WMRrRJuWPm1T0GB+HjEZet0vFJBVH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10989"; a="13221298" X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="13221298" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Feb 2024 01:11:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="4629791" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmviesa007.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 20 Feb 2024 01:11:47 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 20 Feb 2024 01:11:46 -0800 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Tue, 20 Feb 2024 01:11:46 -0800 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.100) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Tue, 20 Feb 2024 01:11:46 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mjq0L48lw9HxeJD04rZmV6HWVz7+XD/ACQz7IU3DN2tbTg9QAJMqfrKOBe2iae8ViD+phg4mKIzIDFG9vaS1hKH7yQwKuAZBHnI+ucvvC2ZI1L7ZYp0zyld2XIsmuURwIPkcHVLlPc4moB8xvtM9PfcqhF+prBvN5iVUyYMWDzbV7A62LNSiZQYj7sdawZW6OwJxJV8KrOA2jqxX8XCrBLXzUzTiXEmozelfQMgsBLAiDJ5Io6/gK/zSrBbs78LTfufbDNnypVlW19ROwohIQyndvWMsBH60uNMd4gBDMKg98+hJSN6wkXlhC2+sLDnhtmE1bL7hlrNv5a28m8ZP2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8irOEN+oxIlmUEofd94qYRXuW+s1Hsn08aDQuM+/10Q=; b=OaUHPmbPxEDOp62s7xT9svXmNle+afhnTlhvSx8NK1MnEUm7pJQbVUCnj81OYglYvKyU7B4EhUrnqc0Qq/5zG8E8KcJZW7UxMixMiTFLy7/L9CVX+NAXFfCaefD7Yu9c1qwHve4WPckVz0JDkenparfN12XRn0xjas8tNWq75QtzyKyWKh1FJcyXErnc8omMbPrCPbFnxhrqJMd+BS51s0wOIq3ARyEsjYlrGfmZYPeY0sfXQSEyOslr4W9eHs5tTaE3Ft5Lc91jTVpDteNnXCSG4Zp+TN7FVGRSwHOyc34eLz9VXK9B4CUjm7AbjPupM/N+pAC9mwVD2xGRREByUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by CY5PR11MB6209.namprd11.prod.outlook.com (2603:10b6:930:27::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.34; Tue, 20 Feb 2024 09:11:44 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e%7]) with mapi id 15.20.7292.036; Tue, 20 Feb 2024 09:11:44 +0000 Date: Tue, 20 Feb 2024 09:11:40 +0000 From: Bruce Richardson To: Mattias =?iso-8859-1?Q?R=F6nnblom?= CC: , , Morten =?iso-8859-1?Q?Br=F8rup?= , Stephen Hemminger Subject: Re: [RFC v3 1/6] eal: add static per-lcore memory allocation facility Message-ID: References: <20240219094036.485727-2-mattias.ronnblom@ericsson.com> <20240220084908.488252-1-mattias.ronnblom@ericsson.com> <20240220084908.488252-2-mattias.ronnblom@ericsson.com> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240220084908.488252-2-mattias.ronnblom@ericsson.com> X-ClientProxiedBy: DU2PR04CA0074.eurprd04.prod.outlook.com (2603:10a6:10:232::19) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|CY5PR11MB6209:EE_ X-MS-Office365-Filtering-Correlation-Id: d04ad676-3246-4c6b-6065-08dc31f3f53b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /ZjCLV84YFfgrbhEUmjVvGykAH3y+2fY9p8WpVC+WmkKRtAo8/IwGfkw8oau4QtgPPvTiSroVhKaZdXutyvjI4ljytv8FZQF6RMgWIG8TfdTROzl0yQrHxRCRzoXnqKsHccWuVJ7SFSARWILNikVq5bFvW6ld89N8QZ2vg3XgBFYgBJLPWi2QTMtKyXpxM4k84ur8HNalqupsCHN80ETkZDbWl40ZqUWfGncOFoxtWYrXIeDhSgq7TUldei5zbzxNPJU7cj+XuxTostWkDM54qMUy2U6uxLcYSuPVzXmXdDo+FMGG1Ki+iYtsIlJLySzfU0+UIZ7UwE+0Hh/NREg7fN1Ya0SNb/qsM0SliGnJGL41dFt8Hj5cDo/ObGSIDXGNY6PmmgP4e0aJGq/+sJ5SaE6Hgkht4jnbAVDHqLaGzv1jICFn8ZLJAtc+sqzYL96aKs4E0WUp41kfJC5BWQmMD6+YofzEdPVl3OGhFw593tOmoMiMmrA43D2PANluElHj2r8ECuF5BUIbaZDMJUhEckPIqytcWYtQHYZktva7mY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?s1jrNrcoJCmPas7o+bLT/bruxZp93arn5+bWyrvAeEtGgJrOszoHgzc9NR?= =?iso-8859-1?Q?Mk5txiP2i1nEPuEjNJ9cJ4jbfqK8nnwUpEbPCfRklDkA/c1F5Em2bJQ1uu?= =?iso-8859-1?Q?noCZUo/vgiYS/B26vPe5Plu/4o4z8p1k5mlzDxM3fSsye2vj6MrLxdnyMf?= =?iso-8859-1?Q?hCO+0H2M83b0ZG/JPcWFDqgTugV4zUoAlr80Bp8JctZERgGnNO/0ycb7Y2?= =?iso-8859-1?Q?Ig6159NGMoOJSqlv2/LGZK5M2mH/PAb+tAWyO1N0QYmvbTnKQjKqIUxvew?= =?iso-8859-1?Q?iVUleadPbYBGMsHMuGuklK6ekvD5tppoV4+LTiq5hzSdZFMYw+LfGA0KbI?= =?iso-8859-1?Q?QaRYlO53qULTvwc8GSAVC5tpEKzAsfj9ZpuuYTN1QxlyMscoe3tmLeknBw?= =?iso-8859-1?Q?01NDQN77NrO6CeSc6sTeudlxmLFnOeXdA5ZHd2oAFCt5wVOeDwcMl4H8/x?= =?iso-8859-1?Q?Vyrp17QiH4E0j138ZK/rVycaAbE+xEISIvxrmldzCD/LlHcjOpY0/DtHkX?= =?iso-8859-1?Q?VFQm1vi8RRYGKSCQWGAh+L176PLaEmsARNY53fw7obbEpt0x4Jep7zFbco?= =?iso-8859-1?Q?6L2vnOMvfjpTGiFkWRVvBS9f7Z0Wpv8+rp79qMSzTZDlhuSgKV0LMFKMmv?= =?iso-8859-1?Q?Co4QOGJIVsINkJFdTH3z5/poKsj+Ipn3YLq/qoan20sTLQNXqyRE7i9M/v?= =?iso-8859-1?Q?7VItrpd+HmpEMil+Bmg4UyyL+znBTrr2xf1WN+vuh8A/sUdbW2+z2NCBJa?= =?iso-8859-1?Q?79VzWeeVwdxtbVYXhiSgWVuOvEr2yhifWsRtiCefL9yZ3nAFumrF6Jn817?= =?iso-8859-1?Q?9APaNXw+hadshe/pTaDaXel7bT9+f6ayDP78QwSQtyd/Kw4wY+hbQZxszU?= =?iso-8859-1?Q?tEuvHOElRigdZAep5wo+VruUaT7o9YU21B/Ayz+HULBNvmapPN3JcIFzgx?= =?iso-8859-1?Q?7e2WZmLdXrnGbWIwSnRUEIbguFVwaEliuQWJet70iMgcWJ/g76ZNPhtc/k?= =?iso-8859-1?Q?BXWDBiVoXXXZZfoTeDPSQ0MkiW0+TE5AS8AEu130ZfaJgMWifF3WUz1J+Q?= =?iso-8859-1?Q?6nLrt+G8Jm3h9W+HBpvk5q/lvV3QaRSyCpQ2esZeB9Alm/wxx5bQED6tdN?= =?iso-8859-1?Q?bcDwYar3oQ/fQQc1HAMZE47CjB9otBo+EJKK8lihSQabds6F+oyXfJOU1e?= =?iso-8859-1?Q?jfiV74km9/vj8g1cZartZhwk7e0wz+kV7YzZUEgqQYzqB497q48lsncic6?= =?iso-8859-1?Q?ktQ9gaYfF08q/pwEp33Lm63Z5XIZ1DFvWLEwhonc6aW7+g1UhrTlmV8xEi?= =?iso-8859-1?Q?7e3CJEgj3SSqKsbay8egoL8kI7WiotfLkGdkykHkGJOv5GWcGWugLM90Ab?= =?iso-8859-1?Q?vRZiMOC0Qp5wT9R8D6Hw/mtTHUfR5umq9R1FdyRV6vexpne/sR4PVSMWN0?= =?iso-8859-1?Q?t6Eize1uIDtruz1zPI1A73uTrrvOjdYofincOonXdxorMvTFx2wADAabtz?= =?iso-8859-1?Q?opmFXpIVxgKO47PgIJb5S7NZ/93DY3La5ZpG3w+H0ErYqmKLoQjfxynBKI?= =?iso-8859-1?Q?QInnJPPkN/ZvgVlTR4FP6tREkRbUJLgAw4YwA8NSmN7waFVL+4AZJWIRz/?= =?iso-8859-1?Q?PrYxZQvajGE/2hjqhflabcWRM1n6HtqWC5hoMuNYdbT92oRTFk3r073Q?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: d04ad676-3246-4c6b-6065-08dc31f3f53b X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 09:11:44.3045 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hIRqESykuBtL/LelMugTY75rKp5+b/N0Up9MNIQY4uPIC47eJJgtsp6jxKJcHa3+VNLIAdJwUgcK2aUKZfTFmO/GGg7GTm37Hh/SdU0CH38= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6209 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Feb 20, 2024 at 09:49:03AM +0100, Mattias Rönnblom wrote: > Introduce DPDK per-lcore id variables, or lcore variables for short. > > An lcore variable has one value for every current and future lcore > id-equipped thread. > > The primary use case is for statically allocating > small chunks of often-used data, which is related logically, but where > there are performance benefits to reap from having updates being local > to an lcore. > > Lcore variables are similar to thread-local storage (TLS, e.g., C11 > _Thread_local), but decoupling the values' life time with that of the > threads. > > Lcore variables are also similar in terms of functionality provided by > FreeBSD kernel's DPCPU_*() family of macros and the associated > build-time machinery. DPCPU uses linker scripts, which effectively > prevents the reuse of its, otherwise seemingly viable, approach. > > The currently-prevailing way to solve the same problem as lcore > variables is to keep a module's per-lcore data as RTE_MAX_LCORE-sized > array of cache-aligned, RTE_CACHE_GUARDed structs. The benefit of > lcore variables over this approach is that data related to the same > lcore now is close (spatially, in memory), rather than data used by > the same module, which in turn avoid excessive use of padding, > polluting caches with unused data. > > RFC v3: > * Replace use of GCC-specific alignof() with alignof(). > * Update example to reflect FOREACH macro name change (in RFC v2). > > RFC v2: > * Use alignof to derive alignment requirements. (Morten Brørup) > * Change name of FOREACH to make it distinct from 's > *per-EAL-thread* RTE_LCORE_FOREACH(). (Morten Brørup) > * Allow user-specified alignment, but limit max to cache line size. > > Signed-off-by: Mattias Rönnblom > --- > config/rte_config.h | 1 + > doc/api/doxy-api-index.md | 1 + > lib/eal/common/eal_common_lcore_var.c | 82 ++++++ > lib/eal/common/meson.build | 1 + > lib/eal/include/meson.build | 1 + > lib/eal/include/rte_lcore_var.h | 375 ++++++++++++++++++++++++++ > lib/eal/version.map | 4 + > 7 files changed, 465 insertions(+) > create mode 100644 lib/eal/common/eal_common_lcore_var.c > create mode 100644 lib/eal/include/rte_lcore_var.h > > diff --git a/config/rte_config.h b/config/rte_config.h > index da265d7dd2..884482e473 100644 > --- a/config/rte_config.h > +++ b/config/rte_config.h > @@ -30,6 +30,7 @@ > /* EAL defines */ > #define RTE_CACHE_GUARD_LINES 1 > #define RTE_MAX_HEAPS 32 > +#define RTE_MAX_LCORE_VAR 1048576 > #define RTE_MAX_MEMSEG_LISTS 128 > #define RTE_MAX_MEMSEG_PER_LIST 8192 > #define RTE_MAX_MEM_MB_PER_LIST 32768 > diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md > index a6a768bd7c..bb06bb7ca1 100644 > --- a/doc/api/doxy-api-index.md > +++ b/doc/api/doxy-api-index.md > @@ -98,6 +98,7 @@ The public API headers are grouped by topics: > [interrupts](@ref rte_interrupts.h), > [launch](@ref rte_launch.h), > [lcore](@ref rte_lcore.h), > + [lcore-varible](@ref rte_lcore_var.h), > [per-lcore](@ref rte_per_lcore.h), > [service cores](@ref rte_service.h), > [keepalive](@ref rte_keepalive.h), > diff --git a/lib/eal/common/eal_common_lcore_var.c b/lib/eal/common/eal_common_lcore_var.c > new file mode 100644 > index 0000000000..dfd11cbd0b > --- /dev/null > +++ b/lib/eal/common/eal_common_lcore_var.c > @@ -0,0 +1,82 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2024 Ericsson AB > + */ > + > +#include > + > +#include > +#include > +#include > + > +#include > + > +#include "eal_private.h" > + > +#define WARN_THRESHOLD 75 > + > +/* > + * Avoid using offset zero, since it would result in a NULL-value > + * "handle" (offset) pointer, which in principle and per the API > + * definition shouldn't be an issue, but may confuse some tools and > + * users. > + */ > +#define INITIAL_OFFSET 1 > + > +char rte_lcore_var[RTE_MAX_LCORE][RTE_MAX_LCORE_VAR] __rte_cache_aligned; > + While I like the idea of improved handling for per-core variables, my main concern with this set is this definition here, which adds yet another dependency on the compile-time defined RTE_MAX_LCORE value. I believe we already have an issue with this #define where it's impossible to come up with a single value that works for all, or nearly all cases. The current default is still 128, yet DPDK needs to support systems where the number of cores is well into the hundreds, requiring workarounds of core mappings or customized builds of DPDK. Upping the value fixes those issues at the cost to memory footprint explosion for smaller systems. I'm therefore nervous about putting more dependencies on this value, when I feel we should be moving away from its use, to allow more runtime configurability of cores. For this set/feature, would it be possible to have a run-time allocated (and sized) array rather than using the RTE_MAX_LCORE value? Thanks, (and apologies for the mini-rant!) /Bruce