From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7DC5AA00C4; Wed, 2 Nov 2022 16:19:22 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1EACE40223; Wed, 2 Nov 2022 16:19:22 +0100 (CET) Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by mails.dpdk.org (Postfix) with ESMTP id 8F9B240041 for ; Wed, 2 Nov 2022 16:19:21 +0100 (CET) Received: by mail-pj1-f51.google.com with SMTP id l22-20020a17090a3f1600b00212fbbcfb78so2400388pjc.3 for ; Wed, 02 Nov 2022 08:19:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=gc0I/EYxS/XYAmXsBvispASZF+h7S6qrdh7OkXqpsd8=; b=VgbrNsLBbp3hMHuTC3govMrqP6rvUg5QznbuutEiMM4uL7+ofg2yYPR5NdhZXaamdN FLl7CpwhWXWWmxp2THlO5R0vZ7kxNXh7DG5GZf1c6ZmyGzsDk6TzPFzxhGxklfTxeuwK h5+WY+6Pg27Ospm67IRUBAw2sodSoqmRxQp7s4AfizOwY+Q6L7YZvV2XAKLsryHYIhpw 4TUEqO5tdeZo9GMS3PdYYnvKCi7nt2Anwes10zYKrWNZRmGjEhcOrBWnrILJ/baDEJTx 784dDKV5EgyxBi3/JqdvAQYKw5TaRScNv+uQxI0Ke9kp6zJA8B3jUlfM5ARsd6tI02q7 8buA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gc0I/EYxS/XYAmXsBvispASZF+h7S6qrdh7OkXqpsd8=; b=XtBA1AFkSoa9fOkVr1eZ4o9pOBFXSRcQynQgnas0+twMbHobXiiNQBxVrKh2jA+dm3 putZZ7hd38VFi1qhfEwN/EKO0Uo+WWo3DJF5NRAJXVIdruYK0NJKwce6Krefxunn3d// kN3HIA/YxFMTi+5xKpFFpLeHHgRJ9/h/RkXtmBOeNrRKefZgEKQ3sA0E0+DOKnlhDueB z92QlRMsWb+nR+R4vjDwxX5lUsI32pAVsIfxlfachEH4Qx86tlJhCyj7E7J9V00gCN2D vP22pxykcnUHEF31mMk1ZMvaSAlf0272NaFyDsANmvTnwVgfvlhH1YlNVkyHJhGri2xl NK9w== X-Gm-Message-State: ACrzQf1Xn3Nn7Id60VOT/SiacKNwNLYdush2qMt9j+6xWUORJMBZf5UX sDTe7wxsyU4r2vWvEP+rNiVf5g== X-Google-Smtp-Source: AMsMyM4So75N967mCgkH7L0J3Qh9kuMRFjdnjGOY8hqlkszcTOjsk9L8i2yfBlXqqJrVSTQljNEGSg== X-Received: by 2002:a17:903:1205:b0:178:ac4e:70b8 with SMTP id l5-20020a170903120500b00178ac4e70b8mr25169675plh.52.1667402360563; Wed, 02 Nov 2022 08:19:20 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id q2-20020a17090a938200b002036006d65bsm1570920pjo.39.2022.11.02.08.19.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:20 -0700 (PDT) Date: Wed, 2 Nov 2022 08:19:18 -0700 From: Stephen Hemminger To: Morten =?UTF-8?B?QnLDuHJ1cA==?= Cc: Mattias =?UTF-8?B?UsO2bm5ibG9t?= , , , , , , Subject: Re: [PATCH v2 2/3] mempool: include non-DPDK threads in statistics Message-ID: <20221102081918.102d4a2f@hermes.local> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D87473@smartserver.smartshare.dk> References: <20221030115445.2115-1-mb@smartsharesystems.com> <20221031112634.18329-1-mb@smartsharesystems.com> <20221031112634.18329-2-mb@smartsharesystems.com> <6c37816f-4136-772a-1ff2-501eb1d3a244@lysator.liu.se> <98CBD80474FA8B44BF855DF32C47DC35D87473@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 2 Nov 2022 10:09:26 +0100 Morten Br=C3=B8rup wrote: > > From: Mattias R=C3=B6nnblom [mailto:hofors@lysator.liu.se] > > Sent: Wednesday, 2 November 2022 08.53 > >=20 > > On 2022-10-31 12:26, Morten Br=C3=B8rup wrote: =20 > > > Offset the stats array index by one, and count non-DPDK threads at =20 > > index =20 > > > zero. > > > > > > This patch provides two benefits: > > > * Non-DPDK threads are also included in the statistics. > > > * A conditional in the fast path is removed. Static branch prediction= =20 > > was =20 > > > correct, so the performance improvement is negligible. > > > > > > v2: > > > * New. No v1 of this patch in the series. > > > > > > Suggested-by: Stephen Hemminger > > > Signed-off-by: Morten Br=C3=B8rup > > > --- > > > lib/mempool/rte_mempool.c | 2 +- > > > lib/mempool/rte_mempool.h | 12 ++++++------ > > > 2 files changed, 7 insertions(+), 7 deletions(-) > > > > > > diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c > > > index 62d1ce764e..e6208125e0 100644 > > > --- a/lib/mempool/rte_mempool.c > > > +++ b/lib/mempool/rte_mempool.c > > > @@ -1272,7 +1272,7 @@ rte_mempool_dump(FILE *f, struct rte_mempool =20 > > *mp) =20 > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS > > > rte_mempool_ops_get_info(mp, &info); > > > memset(&sum, 0, sizeof(sum)); > > > - for (lcore_id =3D 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { > > > + for (lcore_id =3D 0; lcore_id < RTE_MAX_LCORE + 1; lcore_id++) { > > > sum.put_bulk +=3D mp->stats[lcore_id].put_bulk; > > > sum.put_objs +=3D mp->stats[lcore_id].put_objs; > > > sum.put_common_pool_bulk +=3D mp- > > >stats[lcore_id].put_common_pool_bulk; > > > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h > > > index 9c4bf5549f..16e7e62e3c 100644 > > > --- a/lib/mempool/rte_mempool.h > > > +++ b/lib/mempool/rte_mempool.h > > > @@ -238,8 +238,11 @@ struct rte_mempool { > > > struct rte_mempool_memhdr_list mem_list; /**< List of memory =20 > > chunks */ =20 > > > > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS > > > - /** Per-lcore statistics. */ > > > - struct rte_mempool_debug_stats stats[RTE_MAX_LCORE]; > > > + /** Per-lcore statistics. > > > + * > > > + * Offset by one, to include non-DPDK threads. > > > + */ > > > + struct rte_mempool_debug_stats stats[RTE_MAX_LCORE + 1]; > > > #endif > > > } __rte_cache_aligned; > > > > > > @@ -304,10 +307,7 @@ struct rte_mempool { > > > */ > > > #ifdef RTE_LIBRTE_MEMPOOL_STATS > > > #define RTE_MEMPOOL_STAT_ADD(mp, name, n) do { \ > > > - unsigned __lcore_id =3D rte_lcore_id(); \ > > > - if (__lcore_id < RTE_MAX_LCORE) { \ > > > - mp->stats[__lcore_id].name +=3D n; \ > > > - } \ > > > + (mp)->stats[rte_lcore_id() + 1].name +=3D n; \ =20 > >=20 > > This relies on LCORE_ID_ANY being UINT32_MAX, and a wrap to 0, for an > > unregistered non-EAL thread? Might be worth a comment, or better a > > rewrite with an explicit LCORE_ID_ANY comparison. =20 >=20 > The purpose of this patch is to avoid the comparison here. >=20 > Yes, it relies on the wrap to zero, and these conditions: > 1. LCORE_ID_ANY being UINT32_MAX, and > 2. the return type of rte_lcore_id() being unsigned int, and > 3. unsigned int being uint32_t. >=20 > When I wrote this, I considered it safe to assume that LCORE_ID_ANY will = remain the unsigned equivalent of -1 using the return type of rte_lcore_id(= ). In other words: If the return type of rte_lcore_id() should change from = 32 to 64 bit, LCORE_ID_ANY would be updated accordingly to UINT64_MAX. >=20 > Because of this assumption, I didn't use [(rte_lcore_id() + 1) & UINT32_M= AX], but just [rte_lcore_id() + 1]. >=20 > I struggled writing an appropriate comment without making it unacceptably= long, but eventually gave up, and settled for the one-line comment in the = structure only. >=20 > >=20 > > You anyways need a conditional. An atomic add must be used in the > > unregistered EAL thread case. =20 >=20 > Good point: The "+=3D n" must be atomic for non-isolated threads. >=20 > I just took a look at how software maintained stats are handled elsewhere= , and the first thing I found, is the IOAT DMA driver, which also seems to = be using non-atomic increment [1] regardless if used by a DPDK thread or no= t. >=20 > [1]: https://elixir.bootlin.com/dpdk/v22.11-rc2/source/drivers/dma/ioat/i= oat_dmadev.c#L228 >=20 > However, doing it wrong elsewhere doesn't make it correct doing it wrong = here too. :-) >=20 > Atomic increments are costly, so I would rather follow your suggestion an= d reintroduce the comparison. How about: >=20 > #define RTE_MEMPOOL_STAT_ADD(mp, name, n) do { \ > unsigned int __lcore_id =3D rte_lcore_id(); \ > if (likely(__lcore_id < RTE_MAX_LCORE)) { \ > (mp)->stats[__lcore_id].name +=3D n; \ > } else { > rte_atomic64_add( \ > (rte_atomic64_t*)&((mp)->stats[RTE_MAX_LCORE].name), n);\ > } \ > } >=20 > And the structure comment could be: > * Plus one, for non-DPDK threads. >=20 Or use rte_lcore_index() instead of rte_lcore_id()