From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id ADB5BDE3 for ; Mon, 12 Jun 2017 15:06:31 +0200 (CEST) Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Jun 2017 06:06:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,333,1493708400"; d="scan'208";a="97093164" Received: from bricha3-mobl3.ger.corp.intel.com ([10.237.221.28]) by orsmga004.jf.intel.com with SMTP; 12 Jun 2017 06:06:27 -0700 Received: by (sSMTP sendmail emulation); Mon, 12 Jun 2017 14:06:26 +0100 Date: Mon, 12 Jun 2017 14:06:26 +0100 From: Bruce Richardson To: "Ananyev, Konstantin" Cc: Jerin Jacob , Stephen Hemminger , Yerden Zhumabekov , "Verkamp, Daniel" , "dev@dpdk.org" Message-ID: <20170612130626.GA68264@bricha3-MOBL3.ger.corp.intel.com> References: <20170609172854.GA2828@jerin> <2601191342CEEE43887BDE71AB9772583FB07AEC@IRSMSX109.ger.corp.intel.com> <20170612030730.GA6870@jerin> <2601191342CEEE43887BDE71AB9772583FB082EC@IRSMSX109.ger.corp.intel.com> <20170612103409.GA4354@jerin> <20170612110907.GA64736@bricha3-MOBL3.ger.corp.intel.com> <20170612114117.GA17595@jerin> <2601191342CEEE43887BDE71AB9772583FB08394@IRSMSX109.ger.corp.intel.com> <20170612124218.GA20971@jerin> <2601191342CEEE43887BDE71AB9772583FB083B9@IRSMSX109.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB9772583FB083B9@IRSMSX109.ger.corp.intel.com> Organization: Intel Research and =?iso-8859-1?Q?De=ACvel?= =?iso-8859-1?Q?opment?= Ireland Ltd. User-Agent: Mutt/1.8.1 (2017-04-11) Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jun 2017 13:06:32 -0000 On Mon, Jun 12, 2017 at 01:51:26PM +0100, Ananyev, Konstantin wrote: > > > > -----Original Message----- > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > Sent: Monday, June 12, 2017 1:42 PM > > To: Ananyev, Konstantin > > Cc: Richardson, Bruce ; Stephen Hemminger ; Yerden Zhumabekov > > ; Verkamp, Daniel ; dev@dpdk.org > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > -----Original Message----- > > > Date: Mon, 12 Jun 2017 12:17:48 +0000 > > > From: "Ananyev, Konstantin" > > > To: Jerin Jacob , "Richardson, Bruce" > > > > > > CC: Stephen Hemminger , Yerden Zhumabekov > > > , "Verkamp, Daniel" , > > > "dev@dpdk.org" > > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > -----Original Message----- > > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > > > Sent: Monday, June 12, 2017 12:41 PM > > > > To: Richardson, Bruce > > > > Cc: Ananyev, Konstantin ; Stephen Hemminger ; Yerden Zhumabekov > > > > ; Verkamp, Daniel ; dev@dpdk.org > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > -----Original Message----- > > > > > Date: Mon, 12 Jun 2017 12:09:07 +0100 > > > > > From: Bruce Richardson > > > > > To: Jerin Jacob > > > > > CC: "Ananyev, Konstantin" , Stephen Hemminger > > > > > , Yerden Zhumabekov , > > > > > "Verkamp, Daniel" , "dev@dpdk.org" > > > > > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > User-Agent: Mutt/1.8.1 (2017-04-11) > > > > > > > > > > On Mon, Jun 12, 2017 at 04:04:11PM +0530, Jerin Jacob wrote: > > > > > > -----Original Message----- > > > > > > > Date: Mon, 12 Jun 2017 10:18:39 +0000 > > > > > > > From: "Ananyev, Konstantin" > > > > > > > To: Jerin Jacob > > > > > > > CC: Stephen Hemminger , Yerden Zhumabekov > > > > > > > , "Richardson, Bruce" , > > > > > > > "Verkamp, Daniel" , "dev@dpdk.org" > > > > > > > > > > > > > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > > > > > > > Sent: Monday, June 12, 2017 4:08 AM > > > > > > > > To: Ananyev, Konstantin > > > > > > > > Cc: Stephen Hemminger ; Yerden Zhumabekov ; Richardson, Bruce > > > > > > > > ; Verkamp, Daniel ; dev@dpdk.org > > > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > Date: Sat, 10 Jun 2017 08:16:44 +0000 > > > > > > > > > From: "Ananyev, Konstantin" > > > > > > > > > To: Jerin Jacob , Stephen Hemminger > > > > > > > > > > > > > > > > > > CC: Yerden Zhumabekov , "Richardson, Bruce" > > > > > > > > > , "Verkamp, Daniel" > > > > > > > > > , "dev@dpdk.org" > > > > > > > > > Subject: RE: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > > > > > > > > > Sent: Friday, June 9, 2017 6:29 PM > > > > > > > > > > To: Stephen Hemminger > > > > > > > > > > Cc: Yerden Zhumabekov ; Ananyev, Konstantin ; Richardson, Bruce > > > > > > > > > > ; Verkamp, Daniel ; dev@dpdk.org > > > > > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > Date: Fri, 9 Jun 2017 10:16:25 -0700 > > > > > > > > > > > From: Stephen Hemminger > > > > > > > > > > > To: Yerden Zhumabekov > > > > > > > > > > > Cc: "Ananyev, Konstantin" , "Richardson, > > > > > > > > > > > Bruce" , "Verkamp, Daniel" > > > > > > > > > > > , "dev@dpdk.org" > > > > > > > > > > > Subject: Re: [dpdk-dev] [PATCH v2] ring: use aligned memzone allocation > > > > > > > > > > > > > > > > > > > > > > On Fri, 9 Jun 2017 18:47:43 +0600 > > > > > > > > > > > Yerden Zhumabekov wrote: > > > > > > > > > > > > > > > > > > > > > > > On 06.06.2017 19:19, Ananyev, Konstantin wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >>>> Maybe there is some deeper reason for the >= 128-byte alignment logic in rte_ring.h? > > > > > > > > > > > > >>> Might be, would be good to hear opinion the author of that change. > > > > > > > > > > > > >> It gives improved performance for core-2-core transfer. > > > > > > > > > > > > > You mean empty cache-line(s) after prod/cons, correct? > > > > > > > > > > > > > That's ok but why we can't keep them and whole rte_ring aligned on cache-line boundaries? > > > > > > > > > > > > > Something like that: > > > > > > > > > > > > > struct rte_ring { > > > > > > > > > > > > > ... > > > > > > > > > > > > > struct rte_ring_headtail prod __rte_cache_aligned; > > > > > > > > > > > > > EMPTY_CACHE_LINE __rte_cache_aligned; > > > > > > > > > > > > > struct rte_ring_headtail cons __rte_cache_aligned; > > > > > > > > > > > > > EMPTY_CACHE_LINE __rte_cache_aligned; > > > > > > > > > > > > > }; > > > > > > > > > > > > > > > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm curious, can anyone explain, how does it actually affect > > > > > > > > > > > > performance? Maybe we can utilize it application code? > > > > > > > > > > > > > > > > > > > > > > I think it is because on Intel CPU's the CPU will speculatively fetch adjacent cache lines. > > > > > > > > > > > If these cache lines change, then it will create false sharing. > > > > > > > > > > > > > > > > > > > > I see. I think, In such cases it is better to abstract as conditional > > > > > > > > > > compilation. The above logic has worst case cache memory > > > > > > > > > > requirement if CPU is 128B CL and no speculative prefetch. > > > > > > > > > > > > > > I suppose we can keep exactly the same logic as we have now: > > > > > > > archs with 64B cache-line would have an empty cache line, > > > > > > > for archs with 128B cacheline - no. > > > > > > > Is that what you are looking for? > > > > > > > > > > > > Its valid to an arch with 128B cache-line and speculative cache prefetch. > > > > > > (Cavium's recent SoCs comes with this property) > > > > > > IMHO, Instead of making 128B as NOOP. We can introduce a new conditional > > > > > > compilation flag(CONFIG_RTE_ARCH_SPECULATIVE_PREFETCH or something like > > > > > > that) to decide the empty line and I think, In future we can use > > > > > > the same config for similar use cases. > > > > > > > > > > > > Jerin > > > > > > > > > > > I'd rather not make it that complicated, and definitely don't like > > > > > adding in more build time config options. Initially, I had the extra > > > > > padding always-present, but it was felt that it made the resulting > > > > > structure too big. For those systems with 128B cachelines, is the extra > > > > > 256 bytes of space per ring really a problem given modern systems have > > > > > ram in the 10's of Gigs? > > > > > > > > I think, RAM size does not matter here. I was referring more on L1 and L2 > > > > cache size(which is very limited).i.e if you fetch the unwanted > > > > lines then CPU have to evict fast and it will have effect on accommodating > > > > interested lines in worker loop.. > > > > > > Not sure I understand you here - as I know, we can't control HW speculative fetch. > > > > Yes. But we can know in advance if a product family supports HW speculative fetch > > or not. Typically a product family defines this feature in arm64 case and > > we have different targets for each product family. > > > > > > It would either happen, or no depending on the actual CPU. > > > The only thing we can control here - what exactly will be fetched: > > > either empty cache line not used by anyone or cache-line with some data. > > > > Yes. If we know a given CPU does not provide HW speculative fetch in > > advance then we don't need to give empty line. > > I understand that part, what I don't understand how not providing empty cache line > (for specific HW) will improve L1/L2 cache usage? > Suppose you do have an empty line for all cases, and HW doesn't fetch next cache line. > Then it should never occur into the cache hierarchy anyway: > You never read/write to it manually, HW doesn't speculatively fetch it. > correct? > Konstantin > +1 If there are HW prefetchers that will pull in the next line, you need the extra padding. If there aren't prefetchers then having the padding has no effect on cache usage, so there is now downside to having it (from a cache point of view anyway) /Bruce