From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.mhcomputing.net (master.mhcomputing.net [74.208.46.186]) by dpdk.org (Postfix) with ESMTP id AC4793F9 for ; Tue, 9 Dec 2014 20:08:34 +0100 (CET) Received: by mail.mhcomputing.net (Postfix, from userid 1000) id 2CC4480BD96; Tue, 9 Dec 2014 11:06:49 -0800 (PST) Date: Tue, 9 Dec 2014 11:06:49 -0800 From: Matthew Hall To: Matt Laswell Message-ID: <20141209190649.GA6886@mhcomputing.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] A question about hugepage initialization time X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Dec 2014 19:08:34 -0000 On Tue, Dec 09, 2014 at 10:33:59AM -0600, Matt Laswell wrote: > Our DPDK application deals with very large in memory data structures, and > can potentially use tens or even hundreds of gigabytes of hugepage memory. What you're doing is an unusual use case and this is open source code where nobody might have tested and QA'ed this yet. So my recommendation would be adding some rte_log statements to measure the various steps in the process to see what's going on. Also using the Linux Perf framework to do low-overhead sampling-based profiling, and making sure you've got everything compiled with debug symbols so you can see what's consuming the execution time. You might find that it makes sense to use some custom allocators like jemalloc alongside of the DPDK allocators, including perhaps "transparent hugepage mode" in your process, and some larger page sizes to reduce the number of pages. You can also use this handy kernel options, hugepagesz= hugepages=N . This creates guaranteed-contiguous known-good hugepages during boot which initialize much more quickly with less trouble and glitches in my experience. https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt https://www.kernel.org/doc/Documentation/vm/transhuge.txt There is no one-size-fits-all solution but these are some possibilities. Good Luck, Matthew.