DPDK patches and discussions
 help / color / mirror / Atom feed
From: &rew <andras.kovacs@ericsson.com>
To: Matt Laswell <laswell@infiniteio.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] A question about hugepage initialization time
Date: Tue, 9 Dec 2014 11:45:07 -0800
Message-ID: <alpine.DEB.2.10.1412091130410.13009@mwlx389> (raw)
In-Reply-To: <CA+GnqArTJoVd9Hh2xZ-fFhHRnUdbgvxB5Tp+rvi2crUi0-9g9A@mail.gmail.com>

> Hey Folks,
>
> Our DPDK application deals with very large in memory data structures, and
> can potentially use tens or even hundreds of gigabytes of hugepage memory.
> During the course of development, we've noticed that as the number of huge
> pages increases, the memory initialization time during EAL init gets to be
> quite long, lasting several minutes at present.  The growth in init time
> doesn't appear to be linear, which is concerning.
>
> This is a minor inconvenience for us and our customers, as memory
> initialization makes our boot times a lot longer than it would otherwise
> be.  Also, my experience has been that really long operations often are
> hiding errors - what you think is merely a slow operation is actually a
> timeout of some sort, often due to misconfiguration. This leads to two
> questions:
>
> 1. Does the long initialization time suggest that there's an error
> happening under the covers?
> 2. If not, is there any simple way that we can shorten memory
> initialization time?
>
> Thanks in advance for your insights.
>
> --
> Matt Laswell
> laswell@infiniteio.com
> infinite io, inc.
>

Hello,

please find some quick comments on the questions:
1.) By our experience long initialization time is normal in case of 
large amount of memory. However this time depends on some things:
- number of hugepages (pagefault handled by kernel is pretty expensive)
- size of hugepages (memset at initialization)

2.) Using 1G pages instead of 2M will reduce the initialization time 
significantly. Using wmemset instead of memset adds an additional 20-30% 
boost by our measurements. Or, just by touching the pages but not cleaning 
them you can have still some more speedup. But in this case your layer or 
the applications above need to do the cleanup at allocation time 
(e.g. by using rte_zmalloc).

Cheers,
&rew

  parent reply	other threads:[~2014-12-09 19:49 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-12-09 16:33 Matt Laswell
2014-12-09 16:50 ` Burakov, Anatoly
2014-12-09 19:06 ` Matthew Hall
2014-12-09 22:05   ` Matt Laswell
2014-12-09 19:45 ` &rew [this message]
2014-12-09 22:10   ` Stephen Hemminger
2014-12-10 10:32     ` Bruce Richardson
2014-12-10 14:29       ` Neil Horman
2014-12-10 14:35         ` Bruce Richardson
2014-12-10 19:16           ` László Vadkerti
2014-12-11 10:14             ` Bruce Richardson
2014-12-12  4:07               ` László Vadkerti
2014-12-12  9:59                 ` Bruce Richardson
2014-12-12 15:50                   ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.10.1412091130410.13009@mwlx389 \
    --to=andras.kovacs@ericsson.com \
    --cc=dev@dpdk.org \
    --cc=laswell@infiniteio.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git