From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 8FCDA2BB8 for ; Wed, 7 Jun 2017 10:22:28 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 Jun 2017 01:22:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,310,1493708400"; d="scan'208";a="1157552122" Received: from shwdeisgchi083.ccr.corp.intel.com (HELO [10.239.67.180]) ([10.239.67.180]) by fmsmga001.fm.intel.com with ESMTP; 07 Jun 2017 01:22:26 -0700 To: Imre Pinter , Marco Varlese , "users@dpdk.org" Cc: =?UTF-8?Q?Gabor_Hal=c3=a1sz?= , =?UTF-8?Q?P=c3=a9ter_Suskovics?= References: <1496311928.3871.7.camel@suse.com> <0f24fe8c-9294-9656-7338-1c09e5c83340@intel.com> From: "Tan, Jianfeng" Message-ID: <33eb312f-60b2-a0fe-3027-4cd4490f001d@intel.com> Date: Wed, 7 Jun 2017 16:22:25 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Subject: Re: [dpdk-users] Slow DPDK startup with many 1G hugepages X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 08:22:29 -0000 On 6/6/2017 11:25 PM, Imre Pinter wrote: > [...] > I've double-checked it, and if I combine Step (1) and Step (2), then OVS start end up in the following error: > EAL: Detected 32 lcore(s) > EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size > EAL: Probing VFIO support... > EAL: Not enough memory available on socket 1! Requested: 1024MB, available: 0MB > PANIC in rte_eal_init(): > Cannot init memory > > I experienced the same behavior with testpmd. > However when they use hugepages from the 'huge_qemu_1G' mountpoint, then they start properly. > > Br, > Imre Ah yes, I tried by myself and encounter similar error as you. And try to deep dive into kernel hugetlb related code to see what's going on: The hugepage allocation path is: hugetlb_fault -> hugetlb_no_page -> alloc_huge_page -> dequeue_huge_page_vma. Inside dequeue_huge_page_vma(), we can see the code logic to iterate node. But from huge_zonelist(), we can see "interleave" policy only applies to a VMA. In our case, each hugepage file is an independent VMA. As a result, we will go though all hugepages from node to another node one by one. Sorry that I take "interleave" as granted. Fortunately, there is a try to fix this: http://dpdk.org/dev/patchwork/patch/25069/. Besides, we can write a simple application which will allocate all hugepages except those used by OVS-DPDK. Thanks, Jianfeng