From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04B0F4596A; Thu, 12 Sep 2024 09:28:51 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C39FB40BA0; Thu, 12 Sep 2024 09:28:50 +0200 (CEST) Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by mails.dpdk.org (Postfix) with ESMTP id 92EFE40B94 for ; Thu, 12 Sep 2024 09:28:49 +0200 (CEST) Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-4581d15c3e3so5271991cf.0 for ; Thu, 12 Sep 2024 00:28:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726126129; x=1726730929; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vtansQU5UoYCiveDXG2z1ZXuzlT+BZTrtQ90IffNO6U=; b=jjQePqAC8fp+i2DISyW+siYmGW6Fzcsp39GPWEiGITPTAJpSUJAb3hBP0Eq4Rjp45M bssCU5wJKhbkn1QO9Fb8DELz5xpm5OaYbwonC24nRFtl79/HGlu9r3mFrjqLpnFl5rZz vlmLw64TQAjLj2dI9humqk9MwYSYd7+f0p9cjKKIno84lLV5DRhiLPJOywysmPci7TKM DcApLnq6gFdrrVjGdfsLEfWQbQABhXbHpC6bZeHDDkc6edsfmF3gzDafZof7RDmwWlfi oaAa2wFFPkQ10El+TBnJM8N+Wgp0StuRMPLf81HQiH8kEQc6HFO3x/I2LIUvyh60NNTT cZxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726126129; x=1726730929; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vtansQU5UoYCiveDXG2z1ZXuzlT+BZTrtQ90IffNO6U=; b=AzwDa8F1xaRajb43R4t4pt9YrVmcR0l+RmOTKdmauAlbDVM4yq44O259DLld6txL/+ vhz1Kb96mMln3Lg1cqH7L2SvT/LNoyIOHmKqkCeRWc4abDfIxrVqZPjRaH6D6IT9tJta EeFcwQvMyQy2PSlH3CclGrX0xxQ9peuVQ7BXaa2BoIX8EHHEBdcFKdvsGkbpQieFGV40 nzia1f2ENx+GtXi4GAxlmPmhqE8LCnD77e5bStz/UToCkl7J9ZwAp61x9l1jj/CUS+vJ vnZkIps0lU991ssvoBaBv7UtegQZp307lPP7SBjRIuzJdksMgtpqUj87oE0reb3ahzDy Gq9A== X-Forwarded-Encrypted: i=1; AJvYcCXpTF3oBq4vh+txpwMSRuE6lz3ctHPTx9e7g8QZ45QeS++SyRSaYohYP43KqjuAndLnKi0=@dpdk.org X-Gm-Message-State: AOJu0YwbYhVcAltoMlLDkFSQFzHRO5s9YHGV1YY9NM8Ar+SLeDi1t5na u1EUXWYhEUhaONWR6IO40BEb922jFqs/MhJOrver0Xgm5NQy8dze6LPmFdobfokEevEgGUZZXiK e+D4o1XA81XQeitW/YuleBnqXb3eRdZxw X-Google-Smtp-Source: AGHT+IGjd6rBZZ2jUDYYmeGzlOO5VNcioglgpu9syXUkjekdVl6LHn77Innv/ufM2DPUx0+QsA4ntt70jwyxMOX/enQ= X-Received: by 2002:a05:622a:1a12:b0:458:5161:e0ca with SMTP id d75a77b69052e-458603d12e5mr27970431cf.41.1726126128868; Thu, 12 Sep 2024 00:28:48 -0700 (PDT) MIME-Version: 1.0 References: <20240910070344.699183-2-mattias.ronnblom@ericsson.com> <20240911170430.701685-1-mattias.ronnblom@ericsson.com> <20240911170430.701685-2-mattias.ronnblom@ericsson.com> In-Reply-To: From: Jerin Jacob Date: Thu, 12 Sep 2024 12:58:22 +0530 Message-ID: Subject: Re: [PATCH v2 1/6] eal: add static per-lcore memory allocation facility To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= , Anatoly Burakov Cc: fengchengwen , =?UTF-8?Q?Mattias_R=C3=B6nnblom?= , dev@dpdk.org, =?UTF-8?Q?Morten_Br=C3=B8rup?= , Stephen Hemminger , Konstantin Ananyev , David Marchand Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Sep 12, 2024 at 11:05=E2=80=AFAM Mattias R=C3=B6nnblom wrote: > > On 2024-09-12 04:33, fengchengwen wrote: > > On 2024/9/12 1:04, Mattias R=C3=B6nnblom wrote: > >> Introduce DPDK per-lcore id variables, or lcore variables for short. > >> > >> An lcore variable has one value for every current and future lcore > >> id-equipped thread. > >> > >> The primary use case is for statically allocating > >> small, frequently-accessed data structures, for which one instance > >> should exist for each lcore. > >> > >> Lcore variables are similar to thread-local storage (TLS, e.g., C11 > >> _Thread_local), but decoupling the values' life time with that of the > >> threads. > >> > >> Lcore variables are also similar in terms of functionality provided by > >> FreeBSD kernel's DPCPU_*() family of macros and the associated > >> build-time machinery. DPCPU uses linker scripts, which effectively > >> prevents the reuse of its, otherwise seemingly viable, approach. > >> > >> The currently-prevailing way to solve the same problem as lcore > >> variables is to keep a module's per-lcore data as RTE_MAX_LCORE-sized > >> array of cache-aligned, RTE_CACHE_GUARDed structs. The benefit of > >> lcore variables over this approach is that data related to the same > >> lcore now is close (spatially, in memory), rather than data used by > >> the same module, which in turn avoid excessive use of padding, > >> polluting caches with unused data. > >> > >> Signed-off-by: Mattias R=C3=B6nnblom > >> Acked-by: Morten Br=C3=B8rup > >> > >> -- > >> > >> + > >> +#define LCORE_BUFFER_SIZE (RTE_MAX_LCORE_VAR * RTE_MAX_LCORE) > >> + > >> +static void *lcore_buffer; > >> +static size_t offset =3D RTE_MAX_LCORE_VAR; > >> + > >> +static void * > >> +lcore_var_alloc(size_t size, size_t align) > >> +{ > >> + void *handle; > >> + void *value; > >> + > >> + offset =3D RTE_ALIGN_CEIL(offset, align); > >> + > >> + if (offset + size > RTE_MAX_LCORE_VAR) { > >> +#ifdef RTE_EXEC_ENV_WINDOWS > >> + lcore_buffer =3D _aligned_malloc(LCORE_BUFFER_SIZE, > >> + RTE_CACHE_LINE_SIZE); > >> +#else > >> + lcore_buffer =3D aligned_alloc(RTE_CACHE_LINE_SIZE, > >> + LCORE_BUFFER_SIZE); > >> +#endif > >> + RTE_VERIFY(lcore_buffer !=3D NULL); > >> + > >> + offset =3D 0; > >> + } > >> + > >> + handle =3D RTE_PTR_ADD(lcore_buffer, offset); > >> + > >> + offset +=3D size; > >> + > >> + RTE_LCORE_VAR_FOREACH_VALUE(value, handle) > >> + memset(value, 0, size); > >> + > >> + EAL_LOG(DEBUG, "Allocated %"PRIuPTR" bytes of per-lcore data with= a " > >> + "%"PRIuPTR"-byte alignment", size, align); > > > > Currrent the data was malloc by libc function, I think it's mainly for = such INIT macro which will be init before main. > > But it will introduce following problem: > > 1\ it can't benefit from huge-pages. this patch may reserved many 1MBs = for each lcore, if we could place it in huge-pages it will reduce the TLB m= iss rate, especially it freq access data. > > This mechanism is for small allocations, which the sum of is also > expected to be small (although the system won't break if they aren't). > > If you have large allocations, you are better off using lazy huge page > allocations further down the initialization process. Otherwise, you will > end up using memory for RTE_MAX_LCORE instances, rather than the actual > lcore count, which could be substantially smaller. + @Anatoly Burakov If I am not wrong, DPDK huge page memory allocator (rte_malloc()), may have similar overhead glibc once. Meaning, The hugepage allocated only when needed and space is over. if so, why not use rte_malloc() if available. > > But sure, everything else being equal, you could have used huge pages > for these lcore variable values. But everything isn't equal. > > > 2\ it can't across multi-process. many of current lcore-data also don't= support multi-process, but I think it worth do that, and it will help us t= o some service recovery when sub-process failed and reboot. > > > > ... > > > > Not sure I think that's a downside. Further cementing that anti-pattern > into DPDK seems to be a bad idea to me. > > lcore variables doesn't *introduce* any of these issues, since the > mechanisms it's replacing also have these shortcomings (if you think > about them as such - I'm not sure I do).