From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58]) by dpdk.org (Postfix) with ESMTP id BDC796A8B for ; Wed, 10 Dec 2014 15:30:15 +0100 (CET) Received: from nat-pool-rdu-t.redhat.com ([66.187.233.202] helo=localhost) by smtp.tuxdriver.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63) (envelope-from ) id 1XyiGv-0001pE-H8; Wed, 10 Dec 2014 09:30:01 -0500 Date: Wed, 10 Dec 2014 09:29:26 -0500 From: Neil Horman To: Bruce Richardson Message-ID: <20141210142926.GA17040@localhost.localdomain> References: <20141209141032.5fa2db0d@urahara> <20141210103225.GA10056@bricha3-MOBL3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141210103225.GA10056@bricha3-MOBL3> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Score: -2.9 (--) X-Spam-Status: No Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] A question about hugepage initialization time X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Dec 2014 14:30:17 -0000 On Wed, Dec 10, 2014 at 10:32:25AM +0000, Bruce Richardson wrote: > On Tue, Dec 09, 2014 at 02:10:32PM -0800, Stephen Hemminger wrote: > > On Tue, 9 Dec 2014 11:45:07 -0800 > > &rew wrote: > > > > > > Hey Folks, > > > > > > > > Our DPDK application deals with very large in memory data structures, and > > > > can potentially use tens or even hundreds of gigabytes of hugepage memory. > > > > During the course of development, we've noticed that as the number of huge > > > > pages increases, the memory initialization time during EAL init gets to be > > > > quite long, lasting several minutes at present. The growth in init time > > > > doesn't appear to be linear, which is concerning. > > > > > > > > This is a minor inconvenience for us and our customers, as memory > > > > initialization makes our boot times a lot longer than it would otherwise > > > > be. Also, my experience has been that really long operations often are > > > > hiding errors - what you think is merely a slow operation is actually a > > > > timeout of some sort, often due to misconfiguration. This leads to two > > > > questions: > > > > > > > > 1. Does the long initialization time suggest that there's an error > > > > happening under the covers? > > > > 2. If not, is there any simple way that we can shorten memory > > > > initialization time? > > > > > > > > Thanks in advance for your insights. > > > > > > > > -- > > > > Matt Laswell > > > > laswell@infiniteio.com > > > > infinite io, inc. > > > > > > > > > > Hello, > > > > > > please find some quick comments on the questions: > > > 1.) By our experience long initialization time is normal in case of > > > large amount of memory. However this time depends on some things: > > > - number of hugepages (pagefault handled by kernel is pretty expensive) > > > - size of hugepages (memset at initialization) > > > > > > 2.) Using 1G pages instead of 2M will reduce the initialization time > > > significantly. Using wmemset instead of memset adds an additional 20-30% > > > boost by our measurements. Or, just by touching the pages but not cleaning > > > them you can have still some more speedup. But in this case your layer or > > > the applications above need to do the cleanup at allocation time > > > (e.g. by using rte_zmalloc). > > > > > > Cheers, > > > &rew > > > > I wonder if the whole rte_malloc code is even worth it with a modern kernel > > with transparent huge pages? rte_malloc adds very little value and is less safe > > and slower than glibc or other allocators. Plus you lose the ablilty to get > > all the benefit out of valgrind or electric fence. > > While I'd dearly love to not have our own custom malloc lib to maintain, for DPDK > multiprocess, rte_malloc will be hard to replace as we would need a replacement > solution that similarly guarantees that memory mapped in process A is also > available at the same address in process B. :-( > Just out of curiosity, why even bother with multiprocess support? What you're talking about above is a multithread model, and your shoehorning multiple processes into it. Neil > /Bruce >