From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A0134A04A6; Fri, 7 Jan 2022 16:12:35 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2789740140; Fri, 7 Jan 2022 16:12:35 +0100 (CET) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 6B35740042 for <dev@dpdk.org>; Fri, 7 Jan 2022 16:12:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1641568353; x=1673104353; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=QWhEAdpiyI4/WZi9nclNKqb4w4u5MTM6EKAUhG5P3Ao=; b=fiK8uIySYf8sElXKVdGPV91D7HGLEKWalZLPpMi6jPQUb6QZ5jal3rc4 TTFoWAqZqBe1r6RppdMFaZwYIR5plKUJX4CekmfuEDeHQltwRJCiDiqOg CsUn/mixxNm8eiWXLADRFpTknncjT/eL7TNMS98y/X6dzaniHYlS8kwOR 8jQqh8r6qOxwSlmge+O4p4LQ0hm0/5d3y2B8mHN+uYmG9ZBq2Snwwlsxf i6/MG59FwaVPf//iqhXotsdBpl6uDFfFYNDAU9iO76L5j1A9isBlunLB1 hxolitj9JO7OBNO1rqYRwFMauocBtk2k6dJdtOpwSoPuGHOT6uHJQud7x A==; X-IronPort-AV: E=McAfee;i="6200,9189,10219"; a="329222128" X-IronPort-AV: E=Sophos;i="5.88,270,1635231600"; d="scan'208";a="329222128" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2022 07:12:14 -0800 X-IronPort-AV: E=Sophos;i="5.88,270,1635231600"; d="scan'208";a="527394957" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.26.15]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 07 Jan 2022 07:12:13 -0800 Date: Fri, 7 Jan 2022 15:12:09 +0000 From: Bruce Richardson <bruce.richardson@intel.com> To: Morten =?iso-8859-1?Q?Br=F8rup?= <mb@smartsharesystems.com> Cc: Olivier Matz <olivier.matz@6wind.com>, Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>, dev@dpdk.org Subject: Re: [RFC] mempool: modify flush threshold Message-ID: <YdhYSRXDTBJLpJob@bricha3-MOBL.ger.corp.intel.com> References: <98CBD80474FA8B44BF855DF32C47DC35D86DB9@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D86DB9@smartserver.smartshare.dk> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org On Tue, Dec 28, 2021 at 03:28:45PM +0100, Morten Brørup wrote: > Hi mempool maintainers and DPDK team. > > Does anyone know the reason or history why CACHE_FLUSHTHRESH_MULTIPLIER was chosen to be 1.5? I think it is counterintuitive. > > The mempool cache flush threshold was introduced in DPDK version 1.3; it was not in DPDK version 1.2. The copyright notice for rte_mempool.c says year 2012. > > > Here is my analysis: > > With the multiplier of 1.5, a mempool cache is allowed to be filled up to 50 % above than its target size before its excess entries are flushed to the mempool (thereby reducing the cache length to the target size). > > In the opposite direction, a mempool cache is allowed to be drained completely, i.e. up to 100 % below its target size. > > My instinct tells me that it would be more natural to let a mempool cache go the same amount above and below its target size, i.e. using a flush multiplier of 2 instead of 1.5. > > Also, the cache should be allowed to fill up to and including the flush threshold, so it is flushed when the threshold is exceeded, instead of when it is reached. > > Here is a simplified example: > > Imagine a cache target size of 32, corresponding to a typical packet burst. With a flush threshold of 2 (and len > threshold instead of len >= threshold), the cache could hold 1 +/-1 packet bursts. With the current multiplier it can only hold [0 .. 1.5[ packet bursts, not really providing a lot of elasticity. > Hi Morten, Interesting to see this being looked at again. The original idea of adding in some extra room above the requested value was to avoid the worst-case scenario of a pool oscillating between full and empty repeatedly due to the addition/removal of perhaps a single packet. As for why 1.5 was chosen as the value, I don't recall any particular reason for it myself. The main objective was to have a separate flush and size values so that we could go a bit above full, and when flushing, not emptying the entire cache down to zero. In terms of the behavioural points you make above, I wonder if symmetry is actually necessary or desirable in this case. After all, the ideal case is probably to keep the mempool neither full nor empty, so that both allocations or frees can be done without having to go to the underlying shared data structure. To accomodate this, the mempool will only flush when the number of elements is greater than size * 1.5, and then it only flushes elements down to size, ensuring that allocations can still take place. On allocation, new buffers are taken when we don't have enough in the cache to fullfil the request, and then the cache is filled up to size, not to the flush threshold. Now, for the scenario you describe - where the mempool cache size is set to be the same as the burst size, this scheme probably does break down, in that we don't really have any burst elasticity. However, I would query if that is a configuration that is used, since to the user it should appear correctly to provide no elasticity. Looking at testpmd, and our example apps, the standard there is a burst size of 32 and mempool cache of ~256. In OVS code, netdev-dpdk.c seems to initialize the mempool with cache size of RTE_MEMPOOL_CACHE_MAX_SIZE (through define MP_CACHE_SZ). In all these cases, I think the 1.5 threshold should work just fine for us. That said, if you want to bump it up to 2x, I can't say I'd object strongly as it should be harmless, I think. Just my initial thoughts, /Bruce