From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f174.google.com (mail-we0-f174.google.com [74.125.82.174]) by dpdk.org (Postfix) with ESMTP id 3A5001F3 for ; Fri, 20 Sep 2013 17:33:44 +0200 (CEST) Received: by mail-we0-f174.google.com with SMTP id q58so666619wes.19 for ; Fri, 20 Sep 2013 08:34:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZuIAhhZgtxsNdpsMn5ew5mb8kvUrEzhB2RRw1+uNFuM=; b=VyDjXWwXQ0Q31sR5RynmWTWuttVDq+7MZRqwnAptx0QnC/VQV0TjeiWwZJvosdgt1Y gppprAvhGFksqM/qBph8RVGTnB8bIZSWlucUCSrL13OT5sb0LsbI6BY+SATi2898I9fx C3Xfrbj0kPvMst1gAOG3O20VOcFRvVhLvQuuDklx9rpUZDLdOlveNtZg6cJULvqZsHDr KiLl6KuVAWyKYpK2yK8mTkVMVax0cEp4z5+N9wzFlRgq2doszQF6a4kNQ8euXNf4YShX WAluEYDS6ZaqGfruBsA2AA/YrwU19kUehnpXYgmWUcEZVUi+LzMMyY2JHttRl3/xF5Rx Xqjg== X-Gm-Message-State: ALoCoQkWbo1dTiWASJDMlUk4uvb97mmJiPG7quWqqRb2Vd6ZviSwX8FDlkjoryySPAYo76YZRXxU MIME-Version: 1.0 X-Received: by 10.180.187.175 with SMTP id ft15mr3318811wic.20.1379691263405; Fri, 20 Sep 2013 08:34:23 -0700 (PDT) Received: by 10.216.213.72 with HTTP; Fri, 20 Sep 2013 08:34:23 -0700 (PDT) In-Reply-To: <523BEFA3.4070209@gmail.com> References: <523AACC9.8010304@gmail.com> <523BEFA3.4070209@gmail.com> Date: Fri, 20 Sep 2013 11:34:23 -0400 Message-ID: From: Robert Sanford To: Dmitry Vyal Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: dev@dpdk.org Subject: Re: [dpdk-dev] How to fight forwarding performance regression on large mempool sizes. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Sep 2013 15:33:44 -0000 One more point, if you're not doing this already: Allocate 2^N-1 mbufs, not 2^N. According to the code and comments: "The optimum size (in terms of memory usage) for a mempool is when n is a power of two minus one: n = (2^q - 1)." The reason: rte_mempool_create(... n ...) invokes rte_ring_create(... rte_align32power(n+1) ...), i.e., it wants to allocate n+1 slots in the ring, but it has to round that up to the next power of 2. So, when you allocate 2^N bufs, its ring comes with twice as many slots (2^(N+1)). -- Robert On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal wrote: > On 09/19/2013 11:39 PM, Robert Sanford wrote: > >> Hi Dmitry, >> >> The biggest drop-off seems to be from size 128K to 256K. Are you using >> 1GB huge pages already (rather than 2MB)? >> >> I would think that it would not use over 1GB until you ask for 512K mbufs >> or more. >> >> > Hi Robert, > > Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs > are 2240 bytes of size. So something strange indeed happens then we move > from ~200MB to ~400MB. Any ideas? > > Regards, > Dmitry > >