Re: [dpdk-dev] [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache

[Stephen Hemminger <stephen@networkplumber.org>]

On Fri, 9 Nov 2018 14:03:25 +0000
"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:

> On 09-Nov-18 12:20 PM, Burakov, Anatoly wrote:
> > On 09-Nov-18 9:23 AM, jianmingfan wrote:  
> >> --- fix coding style of the previous patch
> >>
> >> During procless startup, dpdk invokes clear_hugedir() to unlink all
> >> hugepage files under /dev/hugepages. Then in map_all_hugepages(),
> >> it invokes mmap to allocate and zero all the huge pages as configured
> >> in /sys/kernel/mm/hugepages/xxx/nr_hugepages.
> >>
> >> This cause startup process extreamly slow with large size of huge page
> >> configured.
> >>
> >> In our use case, we usually configure as large as 200GB hugepages in our
> >> router. It takes more than 50s each time dpdk process startup to clear
> >> the pages.
> >>
> >> To address this issue, user can turn on --reuse-map switch. With it,
> >> dpdk will check the validity of the exiting page cache under
> >> /dev/hugespages. If valid, the cache will be reused not deleted,
> >> so that the os doesn't need to zero the pages again.
> >>
> >> However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the
> >> os zeor page behavior. To keep things work, I add memset during
> >> malloc_heap_alloc(). This makes sense due to the following reason.
> >> 1) user often configure hugepage size too large to be used by the 
> >> program.
> >> In our router, 200GB is configured, but less than 2GB is actually used.
> >> 2) dpdk users don't call heap allocation in performance-critical path.
> >> They alloc memory during process bootup.
> >>
> >> Signed-off-by: Jianming Fan <fanjianming@jd.com>
> >> ---  
> > 
> > I believe this issue is better solved by actually fixing all of the 
> > memory that DPDK leaves behind. We already have rte_eal_cleanup() call 
> > which will deallocate any EAL-allocated memory that have been reserved, 
> > and an exited application should free any memory it was using so that 
> > memory subsystem could free it back to the system, thereby not needing 
> > any cleaning of hugepages at startup.
> > 
> > If your application does not e.g. free its mempools on exit, it should 
> > :) Chances are, the problem will go away. The only circumstance where 
> > this may not work is if you preallocated your memory using 
> > -m/--socket-mem flag.
> >   
> 
> To clarify - all of the above is only applicable to 18.05 and beyond. 
> The map_all_hugepages() function only gets called in the legacy mem 
> init, so this patch solves a problem that does not exist on recent DPDK 
> versions in the first place - faster initialization is one of the key 
> reasons why the new memory subsystem was developed.

Applications crash and need to be restarted.
It is a mistake to assume mempool is in any valid state on startup.