From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by dpdk.org (Postfix) with ESMTP id BF5AD5921 for ; Thu, 2 Apr 2015 15:41:50 +0200 (CEST) Received: by wixm2 with SMTP id m2so56239862wix.0 for ; Thu, 02 Apr 2015 06:41:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ePvfqRsKKCqholWPfXRXnG+RjF1yLtmzcFU1S8I5QIY=; b=CwPrZomtiUcqD0tlHLSvJarF95X4L6mJw94xBuqeUE6WySw0GQScBmHHG+FlMd2q6g G8/NuSrq77HgUlg+nq0WA4e5ZhJvseNcUrmasvipFoyj6vYI4H/8V7ueTgZaT0limEOu MdJp9Inus82yr658c6EaozEVVlMjV5gdLLkrxa64f3t1nU/OsMAFENlY9YkZyRiPuJgW nHESExdTbSeM0PqA9k6Y16H5Qk2jHOrvL8rBq4iOReWCqw3OMP1LaEzdkM2TBsgUJrXV Rijylfau83EukPyKtKuV+8rmgo4K5vAnDdwm7dw+9BsxrV7VVA/dnZXbWfsGfNDxv3d9 ptYQ== X-Gm-Message-State: ALoCoQl5fvNHIAOA/AJpPBmuXqtiqb51oHqz3Je5jQDSVf2IgYZG1kUbAd2bv4EpegNigVkk395B MIME-Version: 1.0 X-Received: by 10.194.2.145 with SMTP id 17mr92406768wju.79.1427982110524; Thu, 02 Apr 2015 06:41:50 -0700 (PDT) Received: by 10.194.153.130 with HTTP; Thu, 2 Apr 2015 06:41:50 -0700 (PDT) In-Reply-To: <2705739.59RNFm1sab@xps13> References: <1427974230-8572-1-git-send-email-jerry.lilijun@huawei.com> <2705739.59RNFm1sab@xps13> Date: Thu, 2 Apr 2015 08:41:50 -0500 Message-ID: From: Jay Rolette To: Thomas Monjalon Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: DPDK Subject: Re: [dpdk-dev] [PATCH] eal: decrease the memory init time with many hugepages setup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 13:41:50 -0000 On Thu, Apr 2, 2015 at 7:55 AM, Thomas Monjalon wrote: > 2015-04-02 19:30, jerry.lilijun@huawei.com: > > From: Lilijun > > > > In the function map_all_hugepages(), hugepage memory is truly allocated > by > > memset(virtaddr, 0, hugepage_sz). Then it costs about 40s to finish the > > dpdk memory initialization when 40000 2M hugepages are setup in host os. > > Yes it's something we should try to reduce. > I have a patch in my tree that does the same opto, but it is commented out right now. In our case, 2/3's of the startup time for our entire app was due to that particular call - memset(virtaddr, 0, hugepage_sz). Just zeroing 1 byte per huge page reduces that by 30% in my tests. The only reason I have it commented out is that I didn't have time to make sure there weren't side-effects for DPDK or my app. For normal shared memory on Linux, pages are initialized to zero automatically once they are touched, so the memset isn't required but I wasn't sure whether that applied to huge pages. Also wasn't sure how hugetlbfs factored into the equation. Hopefully someone can chime in on that. Would love to uncomment the opto :) > In fact we can only write one byte to finish the allocation. > > Isn't it a security hole? > Not necessarily. If the kernel pre-zeros the huge pages via CoW like normal pages, then definitely not. Even if the kernel doesn't pre-zero the pages, if DPDK takes care of properly initializing memory structures on startup as they are carved out of the huge pages, then it isn't a security hole. However, that approach is susceptible to bit rot... You can audit the code and make sure everything is kosher at first, but you have to worry about new code making assumptions about how memory is initialized. > This article speaks about "prezeroing optimizations" in Linux kernel: > http://landley.net/writing/memory-faq.txt I read through that when I was trying to figure out what whether huge pages were pre-zeroed or not. It doesn't talk about huge pages much beyond why they are useful for reducing TLB swaps. Jay