From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by dpdk.org (Postfix) with ESMTP id 3DBE01F3 for ; Fri, 20 Sep 2013 16:40:29 +0200 (CEST) Received: by mail-wg0-f50.google.com with SMTP id f12so599188wgh.17 for ; Fri, 20 Sep 2013 07:41:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=LGDUTmXGR1hX2P0xtPp7DXV3DVZMYLh5OOaXgCBqz18=; b=K8ldRtQk06WgXNViyptgPkSGUSbbZ/QUrCzyFqAX32KjDxnx8n6zkdOthj7Egl5gAn yMgDpXAyt93JbQd431fQUNSVEFY+2TmJUvZoQlZbLJERkv6h2DRsxmt8b5Yyfg2Jy5sA RUlnf6pV3870h0juB8vJqbyUmn/ox2ndqQJrAWBi3SVWqmxZM9YYfIQwzQmPgoUOakAJ ukWG7kfR5iYR0ahtQFXCeDcQhFLaUc3cES3x9SUXlpDbuT8MgC4CcdQmBTJnzijTR6e1 E54NZGxh9H1wPgEY27wGBRNtEta4tEUnwE+RxKej6nlEwYqTOvEVoLJTwGLwxUiJQzqo b4wA== X-Gm-Message-State: ALoCoQkn/DkP3xFvSnPGCbg8HDLnRuRe6SIhuuXNAlBejpBM5WYTqkU/j3p3k9T3aL2TXyVUG1sd MIME-Version: 1.0 X-Received: by 10.180.187.175 with SMTP id ft15mr3122314wic.20.1379688068251; Fri, 20 Sep 2013 07:41:08 -0700 (PDT) Received: by 10.216.213.72 with HTTP; Fri, 20 Sep 2013 07:41:08 -0700 (PDT) In-Reply-To: <523BEFA3.4070209@gmail.com> References: <523AACC9.8010304@gmail.com> <523BEFA3.4070209@gmail.com> Date: Fri, 20 Sep 2013 10:41:08 -0400 Message-ID: From: Robert Sanford To: Dmitry Vyal Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: dev@dpdk.org Subject: Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Sep 2013 14:40:29 -0000 It might be interesting to see that start/end address of the 256K-item (256 * 1024 * 2240 = 560 MB) mbuf memory pool. Maybe it's the first size that straddles two 1GB pages. Perhaps you should try a tool that reports cache misses, TLB misses, and related statistics. I don't know much about this area, yet, but this looks like a good starting place: http://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization -- Regards, Robert On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal wrote: > On 09/19/2013 11:39 PM, Robert Sanford wrote: > >> Hi Dmitry, >> >> The biggest drop-off seems to be from size 128K to 256K. Are you using >> 1GB huge pages already (rather than 2MB)? >> >> I would think that it would not use over 1GB until you ask for 512K mbufs >> or more. >> >> > Hi Robert, > > Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs > are 2240 bytes of size. So something strange indeed happens then we move > from ~200MB to ~400MB. Any ideas? > > Regards, > Dmitry > >