From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Jerin Jacob" <jerinjacobk@gmail.com>,
"Mattias Rönnblom" <mattias.ronnblom@ericsson.com>
Cc: <dev@dpdk.org>, <hofors@lysator.liu.se>,
"Stephen Hemminger" <stephen@networkplumber.org>,
"Tomasz Duszynski" <tduszynski@marvell.com>
Subject: RE: [RFC v3 1/6] eal: add static per-lcore memory allocation facility
Date: Wed, 21 Feb 2024 11:31:34 +0100 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35E9F23C@smartserver.smartshare.dk> (raw)
In-Reply-To: <CALBAE1OsqXaiUB=BNOQn=t5gQ1-JZa1FLtdXVhbAwR2VOvNGqA@mail.gmail.com>
> From: Jerin Jacob [mailto:jerinjacobk@gmail.com]
> Sent: Wednesday, 21 February 2024 10.44
>
> On Tue, Feb 20, 2024 at 2:35 PM Mattias Rönnblom
> <mattias.ronnblom@ericsson.com> wrote:
> >
> > Introduce DPDK per-lcore id variables, or lcore variables for short.
> >
> > An lcore variable has one value for every current and future lcore
> > id-equipped thread.
> >
> > The primary <rte_lcore_var.h> use case is for statically allocating
> > small chunks of often-used data, which is related logically, but
> where
> > there are performance benefits to reap from having updates being
> local
> > to an lcore.
>
> I think, in order to quantify the gain, we must add a performance test
> case to measure the acces cycles with lcore variables scheme vs this
> scheme.
> Other PMU counters(Cache misses) may be interesting but we dont have
> means in DPDK to do self monitoring now like
> https://patches.dpdk.org/project/dpdk/patch/20221213104350.3218167-1-
> tduszynski@marvell.com/
>
> >
> > Lcore variables are similar to thread-local storage (TLS, e.g., C11
> > _Thread_local), but decoupling the values' life time with that of the
> > threads.
Lcore variables can be accessed by other threads, unlike TLS variables.
If a TLS variable needs to be accessed by other threads, there must also be an RTE_MAX_LCORE-sized array of pointers to the TLS variable, where each worker thread must initialize the entry pointing to its TLS variable.
> >
> > Lcore variables are also similar in terms of functionality provided
> by
> > FreeBSD kernel's DPCPU_*() family of macros and the associated
> > build-time machinery. DPCPU uses linker scripts, which effectively
> > prevents the reuse of its, otherwise seemingly viable, approach.
> >
> > The currently-prevailing way to solve the same problem as lcore
> > variables is to keep a module's per-lcore data as RTE_MAX_LCORE-sized
> > array of cache-aligned, RTE_CACHE_GUARDed structs. The benefit of
> > lcore variables over this approach is that data related to the same
> > lcore now is close (spatially, in memory), rather than data used by
> > the same module, which in turn avoid excessive use of padding,
> > polluting caches with unused data.
> >
There are 3 ways to implement per-lcore variables:
1. Thread-local storage, available via RTE_DEFINE_PER_LCORE(type, name).
2. RTE_MAX_LCORE-sized arrays.
3. Lcore variables, as provided by this patch series.
Perhaps an overview of differences and performance numbers would help understand the benefits of this patch series.
The advantages of packing more variables into the same cache line may be hard to measure without PMU counters, and could perhaps be described or estimated instead.
next prev parent reply other threads:[~2024-02-21 10:31 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-08 18:16 [RFC 0/5] Lcore variables Mattias Rönnblom
2024-02-08 18:16 ` [RFC 1/5] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-02-09 8:25 ` Morten Brørup
2024-02-09 11:46 ` Mattias Rönnblom
2024-02-09 13:04 ` Morten Brørup
2024-02-19 7:49 ` Mattias Rönnblom
2024-02-19 11:10 ` Morten Brørup
2024-02-19 14:31 ` Mattias Rönnblom
2024-02-19 15:04 ` Morten Brørup
2024-02-19 9:40 ` [RFC v2 0/5] Lcore variables Mattias Rönnblom
2024-02-19 9:40 ` [RFC v2 1/5] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 0/6] Lcore variables Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 1/6] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-02-20 9:11 ` Bruce Richardson
2024-02-20 10:47 ` Mattias Rönnblom
2024-02-20 11:39 ` Bruce Richardson
2024-02-20 13:37 ` Morten Brørup
2024-02-20 16:26 ` Mattias Rönnblom
2024-02-21 9:43 ` Jerin Jacob
2024-02-21 10:31 ` Morten Brørup [this message]
2024-02-21 14:26 ` Mattias Rönnblom
2024-02-22 9:22 ` Morten Brørup
2024-02-23 10:12 ` Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 0/6] Lcore variables Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 1/6] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-02-27 9:58 ` Morten Brørup
2024-02-27 13:44 ` Mattias Rönnblom
2024-02-27 15:05 ` Morten Brørup
2024-02-27 16:27 ` Mattias Rönnblom
2024-02-27 16:51 ` Morten Brørup
2024-02-28 10:09 ` [RFC v5 0/6] Lcore variables Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 1/6] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-03-19 12:52 ` Konstantin Ananyev
2024-03-20 10:24 ` Mattias Rönnblom
2024-03-20 14:18 ` Konstantin Ananyev
2024-05-06 8:27 ` [RFC v6 0/6] Lcore variables Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 1/6] eal: add static per-lcore memory allocation facility Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 2/6] eal: add lcore variable test suite Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 3/6] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 4/6] power: keep per-lcore " Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 5/6] service: " Mattias Rönnblom
2024-05-06 8:27 ` [RFC v6 6/6] eal: keep per-lcore power intrinsics " Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 2/6] eal: add lcore variable test suite Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 3/6] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 4/6] power: keep per-lcore " Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 5/6] service: " Mattias Rönnblom
2024-02-28 10:09 ` [RFC v5 6/6] eal: keep per-lcore power intrinsics " Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 2/6] eal: add lcore variable test suite Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 3/6] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 4/6] power: keep per-lcore " Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 5/6] service: " Mattias Rönnblom
2024-02-25 16:28 ` Mattias Rönnblom
2024-02-25 15:03 ` [RFC v4 6/6] eal: keep per-lcore power intrinsics " Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 2/6] eal: add lcore variable test suite Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 3/6] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-02-20 15:31 ` Morten Brørup
2024-02-20 8:49 ` [RFC v3 4/6] power: keep per-lcore " Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 5/6] service: " Mattias Rönnblom
2024-02-22 9:42 ` Morten Brørup
2024-02-23 10:19 ` Mattias Rönnblom
2024-02-20 8:49 ` [RFC v3 6/6] eal: keep per-lcore power intrinsics " Mattias Rönnblom
2024-02-19 9:40 ` [RFC v2 2/5] eal: add lcore variable test suite Mattias Rönnblom
2024-02-19 9:40 ` [RFC v2 3/5] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-02-19 11:22 ` Morten Brørup
2024-02-19 14:04 ` Mattias Rönnblom
2024-02-19 15:10 ` Morten Brørup
2024-02-19 9:40 ` [RFC v2 4/5] power: keep per-lcore " Mattias Rönnblom
2024-02-19 9:40 ` [RFC v2 5/5] service: " Mattias Rönnblom
2024-02-08 18:16 ` [RFC 2/5] eal: add lcore variable test suite Mattias Rönnblom
2024-02-08 18:16 ` [RFC 3/5] random: keep PRNG state in lcore variable Mattias Rönnblom
2024-02-08 18:16 ` [RFC 4/5] power: keep per-lcore " Mattias Rönnblom
2024-02-08 18:16 ` [RFC 5/5] service: " Mattias Rönnblom
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35E9F23C@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=dev@dpdk.org \
--cc=hofors@lysator.liu.se \
--cc=jerinjacobk@gmail.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=stephen@networkplumber.org \
--cc=tduszynski@marvell.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).