From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Stephen Hemminger" <stephen@networkplumber.org>
Cc: "Mattias Rönnblom" <hofors@lysator.liu.se>,
olivier.matz@6wind.com, andrew.rybchenko@oktetlabs.ru,
jerinj@marvell.com, bruce.richardson@intel.com,
thomas@monjalon.net, dev@dpdk.org
Subject: RE: [PATCH v2 2/3] mempool: include non-DPDK threads in statistics
Date: Wed, 2 Nov 2022 16:37:24 +0100 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D8747A@smartserver.smartshare.dk> (raw)
In-Reply-To: <20221102081918.102d4a2f@hermes.local>
Med venlig hilsen / Kind regards,
-Morten Brørup
> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Wednesday, 2 November 2022 16.19
> To: Morten Brørup
> Cc: Mattias Rönnblom; olivier.matz@6wind.com;
> andrew.rybchenko@oktetlabs.ru; jerinj@marvell.com;
> bruce.richardson@intel.com; thomas@monjalon.net; dev@dpdk.org
> Subject: Re: [PATCH v2 2/3] mempool: include non-DPDK threads in
> statistics
>
> On Wed, 2 Nov 2022 10:09:26 +0100
> Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > > From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> > > Sent: Wednesday, 2 November 2022 08.53
> > >
> > > On 2022-10-31 12:26, Morten Brørup wrote:
> > > > Offset the stats array index by one, and count non-DPDK threads
> at
> > > index
> > > > zero.
> > > >
> > > > This patch provides two benefits:
> > > > * Non-DPDK threads are also included in the statistics.
> > > > * A conditional in the fast path is removed. Static branch
> prediction
> > > was
> > > > correct, so the performance improvement is negligible.
> > > >
> > > > v2:
> > > > * New. No v1 of this patch in the series.
> > > >
> > > > Suggested-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> > > > ---
> > > > lib/mempool/rte_mempool.c | 2 +-
> > > > lib/mempool/rte_mempool.h | 12 ++++++------
> > > > 2 files changed, 7 insertions(+), 7 deletions(-)
> > > >
> > > > diff --git a/lib/mempool/rte_mempool.c
> b/lib/mempool/rte_mempool.c
> > > > index 62d1ce764e..e6208125e0 100644
> > > > --- a/lib/mempool/rte_mempool.c
> > > > +++ b/lib/mempool/rte_mempool.c
> > > > @@ -1272,7 +1272,7 @@ rte_mempool_dump(FILE *f, struct
> rte_mempool
> > > *mp)
> > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > > rte_mempool_ops_get_info(mp, &info);
> > > > memset(&sum, 0, sizeof(sum));
> > > > - for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
> > > > + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE + 1;
> lcore_id++) {
> > > > sum.put_bulk += mp->stats[lcore_id].put_bulk;
> > > > sum.put_objs += mp->stats[lcore_id].put_objs;
> > > > sum.put_common_pool_bulk += mp-
> > > >stats[lcore_id].put_common_pool_bulk;
> > > > diff --git a/lib/mempool/rte_mempool.h
> b/lib/mempool/rte_mempool.h
> > > > index 9c4bf5549f..16e7e62e3c 100644
> > > > --- a/lib/mempool/rte_mempool.h
> > > > +++ b/lib/mempool/rte_mempool.h
> > > > @@ -238,8 +238,11 @@ struct rte_mempool {
> > > > struct rte_mempool_memhdr_list mem_list; /**< List of
> memory
> > > chunks */
> > > >
> > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > > - /** Per-lcore statistics. */
> > > > - struct rte_mempool_debug_stats stats[RTE_MAX_LCORE];
> > > > + /** Per-lcore statistics.
> > > > + *
> > > > + * Offset by one, to include non-DPDK threads.
> > > > + */
> > > > + struct rte_mempool_debug_stats stats[RTE_MAX_LCORE + 1];
> > > > #endif
> > > > } __rte_cache_aligned;
> > > >
> > > > @@ -304,10 +307,7 @@ struct rte_mempool {
> > > > */
> > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS
> > > > #define RTE_MEMPOOL_STAT_ADD(mp, name, n) do {
> \
> > > > - unsigned __lcore_id = rte_lcore_id(); \
> > > > - if (__lcore_id < RTE_MAX_LCORE) { \
> > > > - mp->stats[__lcore_id].name += n; \
> > > > - } \
> > > > + (mp)->stats[rte_lcore_id() + 1].name += n; \
> > >
> > > This relies on LCORE_ID_ANY being UINT32_MAX, and a wrap to 0, for
> an
> > > unregistered non-EAL thread? Might be worth a comment, or better a
> > > rewrite with an explicit LCORE_ID_ANY comparison.
> >
> > The purpose of this patch is to avoid the comparison here.
> >
> > Yes, it relies on the wrap to zero, and these conditions:
> > 1. LCORE_ID_ANY being UINT32_MAX, and
> > 2. the return type of rte_lcore_id() being unsigned int, and
> > 3. unsigned int being uint32_t.
> >
> > When I wrote this, I considered it safe to assume that LCORE_ID_ANY
> will remain the unsigned equivalent of -1 using the return type of
> rte_lcore_id(). In other words: If the return type of rte_lcore_id()
> should change from 32 to 64 bit, LCORE_ID_ANY would be updated
> accordingly to UINT64_MAX.
> >
> > Because of this assumption, I didn't use [(rte_lcore_id() + 1) &
> UINT32_MAX], but just [rte_lcore_id() + 1].
> >
> > I struggled writing an appropriate comment without making it
> unacceptably long, but eventually gave up, and settled for the one-line
> comment in the structure only.
> >
> > >
> > > You anyways need a conditional. An atomic add must be used in the
> > > unregistered EAL thread case.
> >
> > Good point: The "+= n" must be atomic for non-isolated threads.
> >
> > I just took a look at how software maintained stats are handled
> elsewhere, and the first thing I found, is the IOAT DMA driver, which
> also seems to be using non-atomic increment [1] regardless if used by a
> DPDK thread or not.
> >
> > [1]: https://elixir.bootlin.com/dpdk/v22.11-
> rc2/source/drivers/dma/ioat/ioat_dmadev.c#L228
> >
> > However, doing it wrong elsewhere doesn't make it correct doing it
> wrong here too. :-)
> >
> > Atomic increments are costly, so I would rather follow your
> suggestion and reintroduce the comparison. How about:
> >
> > #define RTE_MEMPOOL_STAT_ADD(mp, name, n) do { \
> > unsigned int __lcore_id = rte_lcore_id(); \
> > if (likely(__lcore_id < RTE_MAX_LCORE)) { \
> > (mp)->stats[__lcore_id].name += n; \
> > } else {
> > rte_atomic64_add( \
> > (rte_atomic64_t*)&((mp)->stats[RTE_MAX_LCORE].name),
> n);\
> > } \
> > }
> >
> > And the structure comment could be:
> > * Plus one, for non-DPDK threads.
> >
>
> Or use rte_lcore_index() instead of rte_lcore_id()
>
rte_lcore_index() is not inline, so it would impact performance. And looking at its source code [2], it compares and branches anyway. It would effectively just be a longwinded way of converting the rte_lcore_id() return type from unsigned to signed int.
[2]: https://elixir.bootlin.com/dpdk/v22.11-rc2/source/lib/eal/common/eal_common_lcore.c#L27
next prev parent reply other threads:[~2022-11-02 15:37 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-30 11:54 [PATCH] mempool: split statistics from debug Morten Brørup
2022-10-30 14:04 ` Morten Brørup
2022-10-30 16:12 ` Stephen Hemminger
2022-10-30 20:29 ` Morten Brørup
2022-10-31 11:26 ` [PATCH v2 1/3] " Morten Brørup
2022-10-31 11:26 ` [PATCH v2 2/3] mempool: include non-DPDK threads in statistics Morten Brørup
2022-11-02 7:52 ` Mattias Rönnblom
2022-11-02 9:09 ` Morten Brørup
2022-11-02 15:19 ` Stephen Hemminger
2022-11-02 15:37 ` Morten Brørup [this message]
2022-11-02 17:53 ` Mattias Rönnblom
2022-11-03 8:59 ` Morten Brørup
2022-11-04 8:58 ` Mattias Rönnblom
2022-11-04 10:01 ` Morten Brørup
2022-11-07 7:26 ` Mattias Rönnblom
2022-11-07 8:56 ` Morten Brørup
2022-10-31 11:26 ` [PATCH v2 3/3] mempool: use cache for frequently updated statistics Morten Brørup
2022-11-02 8:01 ` Mattias Rönnblom
2022-11-02 9:29 ` Morten Brørup
2022-11-02 17:55 ` Mattias Rönnblom
2022-11-04 11:17 ` [PATCH v3 1/3] mempool: split stats from debug Morten Brørup
2022-11-04 11:17 ` [PATCH v3 2/3] mempool: add stats for unregistered non-EAL threads Morten Brørup
2022-11-04 11:17 ` [PATCH v3 3/3] mempool: use cache for frequently updated stats Morten Brørup
2022-11-04 12:03 ` [PATCH v4 1/3] mempool: split stats from debug Morten Brørup
2022-11-04 12:03 ` [PATCH v4 2/3] mempool: add stats for unregistered non-EAL threads Morten Brørup
2022-11-06 11:34 ` Andrew Rybchenko
2022-11-04 12:03 ` [PATCH v4 3/3] mempool: use cache for frequently updated stats Morten Brørup
2022-11-06 11:40 ` Andrew Rybchenko
2022-11-06 11:50 ` Morten Brørup
2022-11-06 11:59 ` Andrew Rybchenko
2022-11-06 12:16 ` Morten Brørup
2022-11-07 7:30 ` Mattias Rönnblom
2022-11-08 9:20 ` Konstantin Ananyev
2022-11-08 11:21 ` Morten Brørup
2022-11-06 11:32 ` [PATCH v4 1/3] mempool: split stats from debug Andrew Rybchenko
2022-11-09 18:18 ` [PATCH v5 " Morten Brørup
2022-11-09 18:18 ` [PATCH v5 2/3] mempool: add stats for unregistered non-EAL threads Morten Brørup
2022-11-09 18:18 ` [PATCH v5 3/3] mempool: use cache for frequently updated stats Morten Brørup
2022-11-10 16:36 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35D8747A@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=hofors@lysator.liu.se \
--cc=jerinj@marvell.com \
--cc=olivier.matz@6wind.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).