Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Mon, 12 Jun 2017 12:09:07 +0100
> From: Bruce Richardson <bruce.richardson@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, Stephen Hemminger
>  <stephen@networkplumber.org>, Yerden Zhumabekov <e_zhumabekov@sts.kz>,
>  "Verkamp, Daniel" <daniel.verkamp@intel.com>, "dev@dpdk.org"
>  <dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> User-Agent: Mutt/1.8.1 (2017-04-11)
> 
> On Mon, Jun 12, 2017 at 04:04:11PM +0530, Jerin Jacob wrote:
> > -----Original Message-----
> > > Date: Mon, 12 Jun 2017 10:18:39 +0000
> > > From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > CC: Stephen Hemminger <stephen@networkplumber.org>, Yerden Zhumabekov
> > >  <e_zhumabekov@sts.kz>, "Richardson, Bruce" <bruce.richardson@intel.com>,
> > >  "Verkamp, Daniel" <daniel.verkamp@intel.com>, "dev@dpdk.org"
> > >  <dev@dpdk.org>
> > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > 
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > > > Sent: Monday, June 12, 2017 4:08 AM
> > > > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> > > > Cc: Stephen Hemminger <stephen@networkplumber.org>; Yerden Zhumabekov <e_zhumabekov@sts.kz>; Richardson, Bruce
> > > > <bruce.richardson@intel.com>; Verkamp, Daniel <daniel.verkamp@intel.com>; dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > 
> > > > -----Original Message-----
> > > > > Date: Sat, 10 Jun 2017 08:16:44 +0000
> > > > > From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
> > > > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Stephen Hemminger
> > > > >  <stephen@networkplumber.org>
> > > > > CC: Yerden Zhumabekov <e_zhumabekov@sts.kz>, "Richardson, Bruce"
> > > > >  <bruce.richardson@intel.com>, "Verkamp, Daniel"
> > > > >  <daniel.verkamp@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> > > > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> > > > > > Sent: Friday, June 9, 2017 6:29 PM
> > > > > > To: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > Cc: Yerden Zhumabekov <e_zhumabekov@sts.kz>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> > > > > > <bruce.richardson@intel.com>; Verkamp, Daniel <daniel.verkamp@intel.com>; dev@dpdk.org
> > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > > >
> > > > > > -----Original Message-----
> > > > > > > Date: Fri, 9 Jun 2017 10:16:25 -0700
> > > > > > > From: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > > To: Yerden Zhumabekov <e_zhumabekov@sts.kz>
> > > > > > > Cc: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>, "Richardson,
> > > > > > >  Bruce" <bruce.richardson@intel.com>, "Verkamp, Daniel"
> > > > > > >  <daniel.verkamp@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
> > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation
> > > > > > >
> > > > > > > On Fri, 9 Jun 2017 18:47:43 +0600
> > > > > > > Yerden Zhumabekov <e_zhumabekov@sts.kz> wrote:
> > > > > > >
> > > > > > > > On 06.06.2017 19:19, Ananyev, Konstantin wrote:
> > > > > > > > >
> > > > > > > > >>>> Maybe there is some deeper  reason for the >= 128-byte alignment logic in rte_ring.h?
> > > > > > > > >>> Might be, would be good to hear opinion the author of that change.
> > > > > > > > >> It gives improved performance for core-2-core transfer.
> > > > > > > > > You mean empty cache-line(s) after prod/cons, correct?
> > > > > > > > > That's ok but why we can't keep them and whole rte_ring aligned on cache-line boundaries?
> > > > > > > > > Something like that:
> > > > > > > > > struct rte_ring {
> > > > > > > > >     ...
> > > > > > > > >     struct rte_ring_headtail prod __rte_cache_aligned;
> > > > > > > > >     EMPTY_CACHE_LINE   __rte_cache_aligned;
> > > > > > > > >     struct rte_ring_headtail cons __rte_cache_aligned;
> > > > > > > > >     EMPTY_CACHE_LINE   __rte_cache_aligned;
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > Konstantin
> > > > > > > > >
> > > > > > > >
> > > > > > > > I'm curious, can anyone explain, how does it actually affect
> > > > > > > > performance? Maybe we can utilize it application code?
> > > > > > >
> > > > > > > I think it is because on Intel CPU's the CPU will speculatively fetch adjacent cache lines.
> > > > > > > If these cache lines change, then it will create false sharing.
> > > > > >
> > > > > > I see. I think, In such cases it is better to abstract as conditional
> > > > > > compilation. The above logic has worst case cache memory
> > > > > > requirement if CPU is 128B CL and no speculative prefetch.
> > > 
> > > I suppose we can keep exactly the same logic as we have now:
> > > archs with 64B cache-line would have an empty cache line,
> > > for archs with 128B cacheline - no.
> > > Is that what you are looking for?
> > 
> > Its valid to an arch with 128B cache-line and speculative cache prefetch.
> > (Cavium's recent SoCs comes with this property)
> > IMHO, Instead of making 128B as NOOP. We can introduce a new conditional
> > compilation flag(CONFIG_RTE_ARCH_SPECULATIVE_PREFETCH or something like
> > that) to decide the empty line and I think, In future we can use
> > the same config for similar use cases.
> > 
> > Jerin
> > 
> I'd rather not make it that complicated, and definitely don't like
> adding in more build time config options. Initially, I had the extra
> padding always-present, but it was felt that it made the resulting
> structure too big. For those systems with 128B cachelines, is the extra
> 256 bytes of space per ring really a problem given modern systems have
> ram in the 10's of Gigs?

I think, RAM size does not matter here. I was referring more on L1 and L2
cache size(which is very limited).i.e if you fetch the unwanted
lines then CPU have to evict fast and it will have effect on accommodating
interested lines in worker loop..I am not a fan of build time options but I
don't really see any issue with _ARCH_ specific build time options.
I think, it is good. common_base can have default value if any platform/arch wants
to override it can override it.I don't see any harm in proving that support.


> 
> /Bruce