From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <olivier.matz@6wind.com>
Received: from mail.droids-corp.org (zoll.droids-corp.org [94.23.50.67])
 by dpdk.org (Postfix) with ESMTP id 5334B8047
 for <dev@dpdk.org>; Mon,  1 Dec 2014 10:58:24 +0100 (CET)
Received: from was59-1-82-226-113-214.fbx.proxad.net ([82.226.113.214]
 helo=[192.168.0.10])
 by mail.droids-corp.org with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128)
 (Exim 4.80) (envelope-from <olivier.matz@6wind.com>)
 id 1XvNnT-0006VN-5O; Mon, 01 Dec 2014 11:01:38 +0100
Message-ID: <547C3BB7.5010309@6wind.com>
Date: Mon, 01 Dec 2014 10:58:15 +0100
From: Olivier MATZ <olivier.matz@6wind.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Icedove/24.5.0
MIME-Version: 1.0
To: Kamraan Nasim <knasim@sidebandnetworks.com>, dev@dpdk.org
References: <CAPrTskgmfxx=0hgnHBg=KMATTUhtp338vxESM=MMU6wTaW8SQQ@mail.gmail.com>
In-Reply-To: <CAPrTskgmfxx=0hgnHBg=KMATTUhtp338vxESM=MMU6wTaW8SQQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Cc: Steve Noble <snoble@sidebandnetworks.com>,
 Ashish Juneja <ajuneja@sidebandnetworks.com>
Subject: Re: [dpdk-dev] RTE mempool "used count" steadily goes down to zero
 despite steady packet throughput
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Dec 2014 09:58:24 -0000

Hi Kam,

On 11/28/2014 06:34 PM, Kamraan Nasim wrote:
> I have ~15Gbps of traffic flowing through two 10GE ports and been profiling
> the rte mempool(or rather the pktmbuf mempool) memory consumption:
> 
> I have per lcore caching disabled(cache_size is 0)
> 
> I have noticed that:
> - Mempool FREE cnt(as given byt rte_mempool_free_count()) increases
> - Mempool USED cnt(as given by rte_mempool_used_count() decreases and
> eventually drops to 0. When this happens, mempool reports itself as EMPTY
> - rx_nombuf stats for the eth ports start climbing
> - Valgrind Memcheck does not indicate any obvious leaks in RTE mempool or
> my application.
> 
> 
> I was wondering if others have come across this issue?  Or if people here
> have used ways, besides Valgrind to profile the mempool or the pkt mbuf
> pool?

First, be careful with rte_mempool_free_count(): it returns the number
of free entries in the internal ring of the mempool, which actually
corresponds to the number of allocated objects from the mempool point
of view. See:
http://dpdk.org/browse/dpdk/tree/lib/librte_mempool/rte_mempool.h

If you have the number of allocated objects increasing, and the
statistics rx_nombuf increasing when the mbuf pool is empty, it means
that you have a mbuf leak in your application. Valgrind won't see it
since it does not know about mempool alloc/free functions.

Regards,
Olivier