From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
To: Bruce Richardson <bruce.richardson@intel.com>
Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
Stephen Hemminger <stephen@networkplumber.org>,
Yerden Zhumabekov <e_zhumabekov@sts.kz>,
"Verkamp, Daniel" <daniel.verkamp@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
Date: Mon, 12 Jun 2017 17:11:18 +0530 [thread overview]
Message-ID: <20170612114117.GA17595@jerin> (raw)
In-Reply-To: <20170612110907.GA64736@bricha3-MOBL3.ger.corp.intel.com>
-----Original Message-----
> Date: Mon, 12 Jun 2017 12:09:07 +0100
> From: Bruce Richardson <bruce.richardson@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, Stephen Hemminger
> <stephen@networkplumber.org>, Yerden Zhumabekov <e_zhumabekov@sts.kz>,
> "Verkamp, Daniel" <daniel.verkamp@intel.com>, "dev@dpdk.org"
> <dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> User-Agent: Mutt/1.8.1 (2017-04-11)
>
> On Mon, Jun 12, 2017 at 04:04:11PM +0530, Jerin Jacob wrote:
> > -----Original Message-----
> > > Date: Mon, 12 Jun 2017 10:18:39 +0000
> > > From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > CC: Stephen Hemminger <stephen@networkplumber.org>, Yerden Zhumabekov
> > > <e_zhumabekov@sts.kz>, "Richardson, Bruce" <bruce.richardson@intel.com>,
> > > "Verkamp, Daniel" <daniel.verkamp@intel.com>, "dev@dpdk.org"
> > > <dev@dpdk.org>
> > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > > > Sent: Monday, June 12, 2017 4:08 AM
> > > > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> > > > Cc: Stephen Hemminger <stephen@networkplumber.org>; Yerden Zhumabekov <e_zhumabekov@sts.kz>; Richardson, Bruce
> > > > <bruce.richardson@intel.com>; Verkamp, Daniel <daniel.verkamp@intel.com>; dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > >
> > > > -----Original Message-----
> > > > > Date: Sat, 10 Jun 2017 08:16:44 +0000
> > > > > From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> > > > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Stephen Hemminger
> > > > > <stephen@networkplumber.org>
> > > > > CC: Yerden Zhumabekov <e_zhumabekov@sts.kz>, "Richardson, Bruce"
> > > > > <bruce.richardson@intel.com>, "Verkamp, Daniel"
> > > > > <daniel.verkamp@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> > > > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > > > > > Sent: Friday, June 9, 2017 6:29 PM
> > > > > > To: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > Cc: Yerden Zhumabekov <e_zhumabekov@sts.kz>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> > > > > > <bruce.richardson@intel.com>; Verkamp, Daniel <daniel.verkamp@intel.com>; dev@dpdk.org
> > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > > >
> > > > > > -----Original Message-----
> > > > > > > Date: Fri, 9 Jun 2017 10:16:25 -0700
> > > > > > > From: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > > To: Yerden Zhumabekov <e_zhumabekov@sts.kz>
> > > > > > > Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Richardson,
> > > > > > > Bruce" <bruce.richardson@intel.com>, "Verkamp, Daniel"
> > > > > > > <daniel.verkamp@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > > > >
> > > > > > > On Fri, 9 Jun 2017 18:47:43 +0600
> > > > > > > Yerden Zhumabekov <e_zhumabekov@sts.kz> wrote:
> > > > > > >
> > > > > > > > On 06.06.2017 19:19, Ananyev, Konstantin wrote:
> > > > > > > > >
> > > > > > > > >>>> Maybe there is some deeper reason for the >= 128-byte alignment logic in rte_ring.h?
> > > > > > > > >>> Might be, would be good to hear opinion the author of that change.
> > > > > > > > >> It gives improved performance for core-2-core transfer.
> > > > > > > > > You mean empty cache-line(s) after prod/cons, correct?
> > > > > > > > > That's ok but why we can't keep them and whole rte_ring aligned on cache-line boundaries?
> > > > > > > > > Something like that:
> > > > > > > > > struct rte_ring {
> > > > > > > > > ...
> > > > > > > > > struct rte_ring_headtail prod __rte_cache_aligned;
> > > > > > > > > EMPTY_CACHE_LINE __rte_cache_aligned;
> > > > > > > > > struct rte_ring_headtail cons __rte_cache_aligned;
> > > > > > > > > EMPTY_CACHE_LINE __rte_cache_aligned;
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > Konstantin
> > > > > > > > >
> > > > > > > >
> > > > > > > > I'm curious, can anyone explain, how does it actually affect
> > > > > > > > performance? Maybe we can utilize it application code?
> > > > > > >
> > > > > > > I think it is because on Intel CPU's the CPU will speculatively fetch adjacent cache lines.
> > > > > > > If these cache lines change, then it will create false sharing.
> > > > > >
> > > > > > I see. I think, In such cases it is better to abstract as conditional
> > > > > > compilation. The above logic has worst case cache memory
> > > > > > requirement if CPU is 128B CL and no speculative prefetch.
> > >
> > > I suppose we can keep exactly the same logic as we have now:
> > > archs with 64B cache-line would have an empty cache line,
> > > for archs with 128B cacheline - no.
> > > Is that what you are looking for?
> >
> > Its valid to an arch with 128B cache-line and speculative cache prefetch.
> > (Cavium's recent SoCs comes with this property)
> > IMHO, Instead of making 128B as NOOP. We can introduce a new conditional
> > compilation flag(CONFIG_RTE_ARCH_SPECULATIVE_PREFETCH or something like
> > that) to decide the empty line and I think, In future we can use
> > the same config for similar use cases.
> >
> > Jerin
> >
> I'd rather not make it that complicated, and definitely don't like
> adding in more build time config options. Initially, I had the extra
> padding always-present, but it was felt that it made the resulting
> structure too big. For those systems with 128B cachelines, is the extra
> 256 bytes of space per ring really a problem given modern systems have
> ram in the 10's of Gigs?
I think, RAM size does not matter here. I was referring more on L1 and L2
cache size(which is very limited).i.e if you fetch the unwanted
lines then CPU have to evict fast and it will have effect on accommodating
interested lines in worker loop..I am not a fan of build time options but I
don't really see any issue with _ARCH_ specific build time options.
I think, it is good. common_base can have default value if any platform/arch wants
to override it can override it.I don't see any harm in proving that support.
>
> /Bruce
next prev parent reply other threads:[~2017-06-12 11:41 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-02 20:03 [dpdk-dev] [PATCH] " Daniel Verkamp
2017-06-02 20:12 ` [dpdk-dev] [PATCH v2] " Daniel Verkamp
2017-06-02 20:51 ` Ananyev, Konstantin
2017-06-02 22:24 ` Verkamp, Daniel
2017-06-03 10:00 ` Ananyev, Konstantin
2017-06-05 16:21 ` Verkamp, Daniel
2017-06-06 9:59 ` Ananyev, Konstantin
2017-06-06 12:42 ` Bruce Richardson
2017-06-06 13:19 ` Ananyev, Konstantin
2017-06-06 14:56 ` Bruce Richardson
2017-06-08 12:45 ` Olivier Matz
2017-06-08 13:20 ` Bruce Richardson
2017-06-08 14:05 ` Olivier Matz
2017-06-08 14:11 ` Bruce Richardson
2017-06-08 14:50 ` Ananyev, Konstantin
2017-06-08 15:24 ` Bruce Richardson
2017-06-08 15:35 ` Ananyev, Konstantin
2017-06-08 16:03 ` Bruce Richardson
2017-06-08 16:12 ` Ananyev, Konstantin
2017-06-08 16:20 ` Richardson, Bruce
2017-06-08 16:42 ` Ananyev, Konstantin
2017-06-09 9:02 ` Bruce Richardson
2017-06-12 9:02 ` Olivier Matz
2017-06-12 9:56 ` Bruce Richardson
2017-06-30 11:35 ` Olivier Matz
2017-06-09 12:47 ` Yerden Zhumabekov
2017-06-09 17:16 ` Stephen Hemminger
2017-06-09 17:28 ` Jerin Jacob
2017-06-10 8:16 ` Ananyev, Konstantin
2017-06-12 3:07 ` Jerin Jacob
2017-06-12 10:18 ` Ananyev, Konstantin
2017-06-12 10:34 ` Jerin Jacob
2017-06-12 11:09 ` Bruce Richardson
2017-06-12 11:41 ` Jerin Jacob [this message]
2017-06-12 12:17 ` Ananyev, Konstantin
2017-06-12 12:42 ` Jerin Jacob
2017-06-12 12:51 ` Ananyev, Konstantin
2017-06-12 13:06 ` Bruce Richardson
2017-06-12 13:20 ` Jerin Jacob
2017-06-30 11:36 ` Olivier Matz
2017-07-01 11:14 ` Thomas Monjalon
2017-07-01 11:25 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170612114117.GA17595@jerin \
--to=jerin.jacob@caviumnetworks.com \
--cc=bruce.richardson@intel.com \
--cc=daniel.verkamp@intel.com \
--cc=dev@dpdk.org \
--cc=e_zhumabekov@sts.kz \
--cc=konstantin.ananyev@intel.com \
--cc=stephen@networkplumber.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).