From: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
To: Bruce Richardson <bruce.richardson@intel.com>,
Ferruh Yigit <ferruh.yigit@intel.com>,
thomas@monjalon.net, jerin.jacob@caviumnetworks.com,
techboard@dpdk.org
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH 1/2] eal: add macro to mark variable mostly read only
Date: Thu, 19 Apr 2018 21:25:18 +0530 [thread overview]
Message-ID: <20180419155517.GA12194@ltp-pvn> (raw)
In-Reply-To: <20180419153723.GB41580@bricha3-MOBL.ger.corp.intel.com>
On Thu, Apr 19, 2018 at 04:37:23PM +0100, Bruce Richardson wrote:
> On Thu, Apr 19, 2018 at 08:48:33PM +0530, Pavan Nikhilesh wrote:
> > On Thu, Apr 19, 2018 at 01:09:58PM +0100, Bruce Richardson wrote:
> > > On Thu, Apr 19, 2018 at 02:50:52PM +0530, Pavan Nikhilesh wrote:
> > > > On Wed, Apr 18, 2018 at 07:03:06PM +0100, Ferruh Yigit wrote:
> > > > > On 4/18/2018 6:55 PM, Pavan Nikhilesh wrote:
> > > > > > On Wed, Apr 18, 2018 at 06:43:11PM +0100, Ferruh Yigit wrote:
> > > > > >> On 4/18/2018 4:30 PM, Pavan Nikhilesh wrote:
> > > > > >>> Add macro to mark a variable to be mostly read only and place it in a
> > > > > >>> separate section.
> > > > > >>>
> > > > > >>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@caviumnetworks.com>
> > > > > >>> ---
> > > > > >>>
> > > > > >>> Group together mostly read only data to avoid cacheline bouncing, also
> > > > > >>> useful for auditing purposes.
> > > > > >>>
> > > > > >>> lib/librte_eal/common/include/rte_common.h | 5 +++++
> > > > > >>> 1 file changed, 5 insertions(+)
> > > > > >>>
> > > > > >>> diff --git a/lib/librte_eal/common/include/rte_common.h b/lib/librte_eal/common/include/rte_common.h
> > > > > >>> index 6c5bc5a76..f2ff2e9e6 100644
> > > > > >>> --- a/lib/librte_eal/common/include/rte_common.h
> > > > > >>> +++ b/lib/librte_eal/common/include/rte_common.h
> > > > > >>> @@ -114,6 +114,11 @@ static void __attribute__((constructor(prio), used)) func(void)
> > > > > >>> */
> > > > > >>> #define __rte_noinline __attribute__((noinline))
> > > > > >>>
> > > > > >>> +/**
> > > > > >>> + * Mark a variable to be mostly read only and place it in a separate section.
> > > > > >>> + */
> > > > > >>> +#define __rte_read_mostly __attribute__((__section__(".read_mostly")))
> > > > > >>
> > > > > >
> > > > > > Hi Ferruh,
> > > > > >
> > > > > >> Hi Pavan,
> > > > > >>
> > > > > >> Is the section ".read_mostly" treated specially [1] or is this just for grouping
> > > > > >> symbols together (to reduce cacheline bouncing)?
> > > > > >
> > > > > > The section .read_mostly is not treated specially it's just for grouping
> > > > > > symbols.
> > > > >
> > > > > I have encounter with a blog post claiming this is not working:
> > > > >
> > > > > "
> > > > > The problem with the above approach is that once all the __read_mostly variables
> > > > > are grouped into one section, the remaining "non-read-mostly" variables end-up
> > > > > together too. This increases the chances that two frequently used elements (in
> > > > > the "non-read-mostly" region) will end-up competing for the same position (or
> > > > > cache-line, the basic fixed-sized block for memory<-->cache transfers) in the
> > > > > cache. Thus frequent accesses will cause excessive cache thrashing on that
> > > > > particular cache-line thereby degrading the overall system performance.
> > > > > "
> > > > >
> > > > > https://thecodeartist.blogspot.com/2011/12/why-readmostly-does-not-work-as-it.html
> > > > >
> > > >
> > > > The author is concerned about processors with less cache set-associativity,
> > > > almost all modern processors have >= 16 way set associativity. And the above
> > > > issue can happen even now when two frequently written global variables are
> > > > placed next to each other.
> > > >
> > > > Currently, we don't have much control over how the global variables are
> > > > arranged and a single addition/deletion to the global variables causes change
> > > > in alignment and in some cases minor performance regression.
> > > > Tagging them as __read_mostly we can easily identify the alignment changes
> > > > across builds by comparing map files global variable section.
> > > >
> > > > I have verified the patch-set on arm64 (16-way set-associative) and didn't
> > > > notice any performance regression.
> > > > Did you have a chance to verify if there is any performance regression?
> > > >
> > > Is there a performance improvement? It's seems a relatively strange change
> > > to me, so I'd like to know that it really improves performance in test
> > > cases.
> >
> > We had a performance regression of ~200k between 17.11 and 18.02 due enabling
> > dpaa/dpaa2 in default config this was due to new global variables being added
> > and changing the alignment.
> > Moving read mostly global variables (logtypes/device arrays) to a separate
> > section helps when tracking performance regression between builds.
> >
> So it's of use when debugging, rather than providing a performance boost in
> and of itself, right?
>
> If performance regressions are appearing, should we then see about marking
> globals with __rte_cache_align to force them all onto difference
> cachelines?
I think that would be a bit of overkill considering the number of logtype
variables.
Currently there are
29 global variables not found in any map file
List of globals
("['fw_file', 'mode_8023ad_ports', 'ecore_mz_count', 'igb_filter_rss_list', "
"'crc16_ccitt_pmull', 'igb_filter_flex_list', 'rte_crypto_devices', "
"'igb_flow_list', 'qman_clk', 'bman_ccsr_map', 'dpaa_portal_key', "
"'bman_ip_rev', 'cons_filter', 'rte_rawdevices', 'internal_config', "
"'bman_pool_max', 'qman_version', 'qman_ccsr_map', 'rte_event_devices', "
"'qman_ip_rev', 'netcfg_interface', 'igb_filter_ntuple_list', "
"'igb_filter_ethertype_list', 'skeldev_init_once', 'igb_filter_syn_list', "
"'crc32_eth_pmull', 'global_portals_used', 'virtio_hw_internal', "
"'vhost_devices']")
These are stats without including the logtype variables across all drivers.
With logtype variables included there are 70+.
prev parent reply other threads:[~2018-04-19 15:55 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-18 15:30 Pavan Nikhilesh
2018-04-18 15:30 ` [dpdk-dev] [PATCH 2/2] drivers: mark logtype variables as read mostly Pavan Nikhilesh
2018-04-18 17:43 ` [dpdk-dev] [PATCH 1/2] eal: add macro to mark variable mostly read only Ferruh Yigit
2018-04-18 17:55 ` Pavan Nikhilesh
2018-04-18 18:03 ` Ferruh Yigit
2018-04-19 9:20 ` Pavan Nikhilesh
2018-04-19 12:09 ` Bruce Richardson
2018-04-19 15:18 ` Pavan Nikhilesh
2018-04-19 15:37 ` Bruce Richardson
2018-04-19 15:55 ` Pavan Nikhilesh [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180419155517.GA12194@ltp-pvn \
--to=pbhagavatula@caviumnetworks.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=jerin.jacob@caviumnetworks.com \
--cc=techboard@dpdk.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).