From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by dpdk.org (Postfix) with ESMTP id 2575C68AE for ; Tue, 9 Dec 2014 20:49:44 +0100 (CET) X-AuditID: c1b4fb30-f79d66d00000744c-a3-54875257a05b Received: from ESESSHC021.ericsson.se (Unknown_Domain [153.88.253.124]) by sesbmg22.ericsson.net (Symantec Mail Security) with SMTP id 11.BB.29772.75257845; Tue, 9 Dec 2014 20:49:43 +0100 (CET) Received: from mwlx389.dyn.sj.us.am.ericsson.se (153.88.183.153) by smtp.internal.ericsson.com (153.88.183.83) with Microsoft SMTP Server id 14.3.195.1; Tue, 9 Dec 2014 20:49:42 +0100 Date: Tue, 9 Dec 2014 11:45:07 -0800 From: &rew X-X-Sender: ethkoan@mwlx389 To: Matt Laswell In-Reply-To: Message-ID: References: User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII"; format=flowed X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprOLMWRmVeSWpSXmKPExsUyM+JvjW54UHuIwdLHehbvPm1nsli4+g6z A5PHrwVLWT02/ZvNGsAUxWWTkpqTWZZapG+XwJVxbP5CtoJHfBXv525gbmB8xt3FyMkhIWAi serKXnYIW0ziwr31bCC2kMARRokN/+Qh7N2MEs+miILYLAIqEt8eT2EBsdkE1CVuNE9ngeiV lvj+ah0ziC0ioCnx69oWsDnMAooSnxZMAqsRFnCV6F7zBqyGUyBQ4va+l2BxXgFLife9fYwQ uwIk7r3eB9YrKqArcejfHzaIGkGJkzOfsEDMtJbYNXsqaxcjB1C9usSCVtEJjIKzkFTNQlK1 gJFpFaNocWpxUm66kZFealFmcnFxfp5eXmrJJkZgSB7c8ttgB+PL546HGAU4GJV4eDektoUI sSaWFVfmHmKU5mBREuddeG5esJBAemJJanZqakFqUXxRaU5q8SFGJg5OqQZGf2PhOB+XONGw 1BKOHY5vLQ9trVj5aK8Di19vQl1gaORsnQetXasfPDcJmqn7SPrxRKZPN96r+P4Mfj4nvirK 8d7NPZemmlmsU7B4HfHSpdxYtTrjztJDs97FrM3rNp/Qeuz7Zz3T1XvecpT77pbbvdbujkCz cXvd42MCQQLxDd3nL77omderxFKckWioxVxUnAgApF5TnioCAAA= Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] A question about hugepage initialization time X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Dec 2014 19:49:44 -0000 > Hey Folks, > > Our DPDK application deals with very large in memory data structures, and > can potentially use tens or even hundreds of gigabytes of hugepage memory. > During the course of development, we've noticed that as the number of huge > pages increases, the memory initialization time during EAL init gets to be > quite long, lasting several minutes at present. The growth in init time > doesn't appear to be linear, which is concerning. > > This is a minor inconvenience for us and our customers, as memory > initialization makes our boot times a lot longer than it would otherwise > be. Also, my experience has been that really long operations often are > hiding errors - what you think is merely a slow operation is actually a > timeout of some sort, often due to misconfiguration. This leads to two > questions: > > 1. Does the long initialization time suggest that there's an error > happening under the covers? > 2. If not, is there any simple way that we can shorten memory > initialization time? > > Thanks in advance for your insights. > > -- > Matt Laswell > laswell@infiniteio.com > infinite io, inc. > Hello, please find some quick comments on the questions: 1.) By our experience long initialization time is normal in case of large amount of memory. However this time depends on some things: - number of hugepages (pagefault handled by kernel is pretty expensive) - size of hugepages (memset at initialization) 2.) Using 1G pages instead of 2M will reduce the initialization time significantly. Using wmemset instead of memset adds an additional 20-30% boost by our measurements. Or, just by touching the pages but not cleaning them you can have still some more speedup. But in this case your layer or the applications above need to do the cleanup at allocation time (e.g. by using rte_zmalloc). Cheers, &rew