From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by dpdk.org (Postfix) with ESMTP id CC7674C92 for ; Fri, 9 Nov 2018 17:21:04 +0100 (CET) Received: by mail-pl1-f193.google.com with SMTP id p16-v6so1147419plr.8 for ; Fri, 09 Nov 2018 08:21:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AZRGjIlwBOWpQUjXqRqMiZV6cps9PcTzT1L+2F8Ytmg=; b=Kjdzem2dvTW9KzWO+svaKjGCQBakui4JC8aZThItJYqasGFm1H1Fy+tDqhP5DkqjW3 cvFCZWI4EYvBalR9H5UYwEuucxXP24x+GBYraPXU2MmxPGdtc66q8TNGxOZ0fSfd0JnU MnAQya9Y+THajqNH2Mj80+JBtWllVpOFEWhio44es7KbQTOCdMH/tzc9trbyiX/Ks1Nf z2tDDXMOiyZn4otqjzjjdeaXzSxaCX5NR3l9z00s9PBXmUhJrjX5a5J7JHHc3gGG7pys oZ/sJDoTPegWLllbe1qqjGIOazblpFA3xHzkmNWEQpA4W1vmedwbZw0mpHEwYGJaFrdm YSKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AZRGjIlwBOWpQUjXqRqMiZV6cps9PcTzT1L+2F8Ytmg=; b=M/3/Zvon8XBBAG+V2FyImwEX5hf9Fij4PSoL/xhaMHTsITa2db8k59mK2CYwBCzlT1 tB+VoNIDD/b17CQjq5JtQxyKAOslckpSafQHVknJ2gm+Tin3+trBY0RHwSIFOPktARLT bN/5xkWhI/JFfmA1LlDG6KyXKfM83X2zjLWXT6L5/h3SZmTJXHFls+Z/MYq34u+5YCXG UxslL/oGDxowHz1NqDQkwZigL6lr7m0vmbwunWZF5eHAsliGFfF/9cOtIsYF9M6IzBh4 7DeQUom8ZaQBETqU0RX80ctUN+1L8MiJB0kJhVk18hb6wCFlybWaXwWqne3kX0FRuQqR DBAA== X-Gm-Message-State: AGRZ1gKDBpLe3jEfYgXapLonQiV1FDRYhVuF3ZxLsTMYn9VuViz5tCDN sBIfFtn6nfYwf8tvodvy5nDiNg== X-Google-Smtp-Source: AJdET5eiT70szAi7+ljTYT7ZCO6Tw44typeo2WK8KoCNEhr0FWb69CBMyU4IwZdxjt1v1B5o4rqCGA== X-Received: by 2002:a17:902:3225:: with SMTP id y34-v6mr9284487plb.227.1541780463814; Fri, 09 Nov 2018 08:21:03 -0800 (PST) Received: from xeon-e3 (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id p64-v6sm7817123pfi.22.2018.11.09.08.21.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 09 Nov 2018 08:21:03 -0800 (PST) Date: Fri, 9 Nov 2018 08:21:01 -0800 From: Stephen Hemminger To: "Burakov, Anatoly" Cc: jianmingfan , dev@dpdk.org, Jianming Fan Message-ID: <20181109082101.7ab7e746@xeon-e3> In-Reply-To: <357b1b24-68f8-2c83-ff42-6ea1dce11b9c@intel.com> References: <20181109075830.27265-1-jianmingfan@126.com> <20181109092338.30097-1-jianmingfan@126.com> <6a7fcafc-5ac7-417a-fea6-2fd33b8b6c90@intel.com> <357b1b24-68f8-2c83-ff42-6ea1dce11b9c@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 16:21:05 -0000 On Fri, 9 Nov 2018 14:03:25 +0000 "Burakov, Anatoly" wrote: > On 09-Nov-18 12:20 PM, Burakov, Anatoly wrote: > > On 09-Nov-18 9:23 AM, jianmingfan wrote: > >> --- fix coding style of the previous patch > >> > >> During procless startup, dpdk invokes clear_hugedir() to unlink all > >> hugepage files under /dev/hugepages. Then in map_all_hugepages(), > >> it invokes mmap to allocate and zero all the huge pages as configured > >> in /sys/kernel/mm/hugepages/xxx/nr_hugepages. > >> > >> This cause startup process extreamly slow with large size of huge page > >> configured. > >> > >> In our use case, we usually configure as large as 200GB hugepages in our > >> router. It takes more than 50s each time dpdk process startup to clear > >> the pages. > >> > >> To address this issue, user can turn on --reuse-map switch. With it, > >> dpdk will check the validity of the exiting page cache under > >> /dev/hugespages. If valid, the cache will be reused not deleted, > >> so that the os doesn't need to zero the pages again. > >> > >> However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the > >> os zeor page behavior. To keep things work, I add memset during > >> malloc_heap_alloc(). This makes sense due to the following reason. > >> 1) user often configure hugepage size too large to be used by the > >> program. > >> In our router, 200GB is configured, but less than 2GB is actually used. > >> 2) dpdk users don't call heap allocation in performance-critical path. > >> They alloc memory during process bootup. > >> > >> Signed-off-by: Jianming Fan > >> --- > > > > I believe this issue is better solved by actually fixing all of the > > memory that DPDK leaves behind. We already have rte_eal_cleanup() call > > which will deallocate any EAL-allocated memory that have been reserved, > > and an exited application should free any memory it was using so that > > memory subsystem could free it back to the system, thereby not needing > > any cleaning of hugepages at startup. > > > > If your application does not e.g. free its mempools on exit, it should > > :) Chances are, the problem will go away. The only circumstance where > > this may not work is if you preallocated your memory using > > -m/--socket-mem flag. > > > > To clarify - all of the above is only applicable to 18.05 and beyond. > The map_all_hugepages() function only gets called in the legacy mem > init, so this patch solves a problem that does not exist on recent DPDK > versions in the first place - faster initialization is one of the key > reasons why the new memory subsystem was developed. Applications crash and need to be restarted. It is a mistake to assume mempool is in any valid state on startup.