From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 6B6C74C9F for ; Fri, 9 Nov 2018 13:21:02 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Nov 2018 04:21:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,483,1534834800"; d="scan'208";a="278459793" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.143]) ([10.237.220.143]) by fmsmga005.fm.intel.com with ESMTP; 09 Nov 2018 04:21:00 -0800 To: jianmingfan , dev@dpdk.org Cc: Jianming Fan References: <20181109075830.27265-1-jianmingfan@126.com> <20181109092338.30097-1-jianmingfan@126.com> From: "Burakov, Anatoly" Message-ID: <6a7fcafc-5ac7-417a-fea6-2fd33b8b6c90@intel.com> Date: Fri, 9 Nov 2018 12:20:59 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20181109092338.30097-1-jianmingfan@126.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 12:21:02 -0000 On 09-Nov-18 9:23 AM, jianmingfan wrote: > --- fix coding style of the previous patch > > During procless startup, dpdk invokes clear_hugedir() to unlink all > hugepage files under /dev/hugepages. Then in map_all_hugepages(), > it invokes mmap to allocate and zero all the huge pages as configured > in /sys/kernel/mm/hugepages/xxx/nr_hugepages. > > This cause startup process extreamly slow with large size of huge page > configured. > > In our use case, we usually configure as large as 200GB hugepages in our > router. It takes more than 50s each time dpdk process startup to clear > the pages. > > To address this issue, user can turn on --reuse-map switch. With it, > dpdk will check the validity of the exiting page cache under > /dev/hugespages. If valid, the cache will be reused not deleted, > so that the os doesn't need to zero the pages again. > > However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the > os zeor page behavior. To keep things work, I add memset during > malloc_heap_alloc(). This makes sense due to the following reason. > 1) user often configure hugepage size too large to be used by the program. > In our router, 200GB is configured, but less than 2GB is actually used. > 2) dpdk users don't call heap allocation in performance-critical path. > They alloc memory during process bootup. > > Signed-off-by: Jianming Fan > --- I believe this issue is better solved by actually fixing all of the memory that DPDK leaves behind. We already have rte_eal_cleanup() call which will deallocate any EAL-allocated memory that have been reserved, and an exited application should free any memory it was using so that memory subsystem could free it back to the system, thereby not needing any cleaning of hugepages at startup. If your application does not e.g. free its mempools on exit, it should :) Chances are, the problem will go away. The only circumstance where this may not work is if you preallocated your memory using -m/--socket-mem flag. -- Thanks, Anatoly