From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3ECEFA04A8; Mon, 24 Jan 2022 16:57:03 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF13F4115A; Mon, 24 Jan 2022 16:57:02 +0100 (CET) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mails.dpdk.org (Postfix) with ESMTP id 61B7440040 for ; Mon, 24 Jan 2022 16:57:01 +0100 (CET) Received: by mail-wm1-f51.google.com with SMTP id n12-20020a05600c3b8c00b0034eb13edb8eso94531wms.0 for ; Mon, 24 Jan 2022 07:57:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=Qd5UCNrN2ec47VTKgwvSmLlEAVFny1a8DMnrGeMskEM=; b=VoOvsKn+n46ks1rsdSvI3LZgo0jMZxmdE0owNjqOVYgH8hYQ1uuwzEX1N2D2trBmrm l+++qDc+ont+jUgA0REyyFddfIQw6yrrwqg39Ei72IqpheRORuiA98vbDbVW3AQ3fxeO q73VnaopV11EhZAQwzz8QXT6b2dAQT4v9wcWavfWNcu0VKUUoqO813+KIz8A3iJQ1gnx Xki68HPMrslUQ/mYIH0dzdUtcdoQzesnpnMRJIhMO5ihLCmvXPAAoGjVRilf5XfIcVGy o4CRcni31ETnVNMHxzBAKZvuutLLwCxt9qIne8wy3zb1ENkhsKpj+KWRu5YajaPy5IWr Catw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=Qd5UCNrN2ec47VTKgwvSmLlEAVFny1a8DMnrGeMskEM=; b=c8Acd7K6p6MOfXLpjf3/g34h0vh07TIA4+xqFXBtnujwk4jCNP9LZHmYlRKeMV9LCN nqDRBQ2CMOOlje8RUQPPAl4IrGFZyBo+88ff9RN/68jPwtiSHBB5gHhfKGlm9fqm7jKG kafgAAzOzK9a+JuNNzqHgG0SPyF42dtoeJlG8GxCA/pK04l/0zmhPi+qk+BjeNJrw9n7 o/FjHPOCsiLYC7Bkeb3XPI5reHFFhtVXLu6R59r1hkkke+vjlKjvVQnQRCFekjAWc6Nr FpL352fGMTM+WwvyXGvX4CA178yvOetfDnWqPHwkZI1ftSTZ8NfXCMfZjHCEjGKN/i95 dkLg== X-Gm-Message-State: AOAM530C4rzh0MJegyxJhFYb/5Xe3JQqabfg1C3CLHKmzpDseECE+W3Y uGqb1MmCmnQkXgsvckHL1CsR6AqBT8h5oA== X-Google-Smtp-Source: ABdhPJzsNI0bSm/Lpu/VGc5mlvBXynrkutNB8dADVhhf5Q+jss9hgQG8wGUDUCDdS7LsJmblNyrX/w== X-Received: by 2002:a05:600c:255:: with SMTP id 21mr2407566wmj.108.1643039821019; Mon, 24 Jan 2022 07:57:01 -0800 (PST) Received: from 6wind.com ([2a01:e0a:5ac:6460:c065:401d:87eb:9b25]) by smtp.gmail.com with ESMTPSA id p17sm13669278wrf.112.2022.01.24.07.57.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jan 2022 07:57:00 -0800 (PST) Date: Mon, 24 Jan 2022 16:56:59 +0100 From: Olivier Matz To: Bruce Richardson Cc: Morten =?iso-8859-1?Q?Br=F8rup?= , Andrew Rybchenko , dev@dpdk.org Subject: Re: [RFC] mempool: modify flush threshold Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D86DB9@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D86DF0@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Mon, Jan 10, 2022 at 09:40:48AM +0000, Bruce Richardson wrote: > On Sat, Jan 08, 2022 at 12:00:17PM +0100, Morten Brørup wrote: > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > > > Sent: Friday, 7 January 2022 16.12 > > > > > > On Tue, Dec 28, 2021 at 03:28:45PM +0100, Morten Brørup wrote: > > > > Hi mempool maintainers and DPDK team. > > > > > > > > Does anyone know the reason or history why > > > CACHE_FLUSHTHRESH_MULTIPLIER was chosen to be 1.5? I think it is > > > counterintuitive. > > > > > > > > The mempool cache flush threshold was introduced in DPDK version 1.3; > > > it was not in DPDK version 1.2. The copyright notice for rte_mempool.c > > > says year 2012. > > > > > > > > > > > > Here is my analysis: > > > > > > > > With the multiplier of 1.5, a mempool cache is allowed to be filled > > > up to 50 % above than its target size before its excess entries are > > > flushed to the mempool (thereby reducing the cache length to the target > > > size). > > > > > > > > In the opposite direction, a mempool cache is allowed to be drained > > > completely, i.e. up to 100 % below its target size. > > > > > > > > My instinct tells me that it would be more natural to let a mempool > > > cache go the same amount above and below its target size, i.e. using a > > > flush multiplier of 2 instead of 1.5. > > > > > > > > Also, the cache should be allowed to fill up to and including the > > > flush threshold, so it is flushed when the threshold is exceeded, > > > instead of when it is reached. > > > > > > > > Here is a simplified example: > > > > > > > > Imagine a cache target size of 32, corresponding to a typical packet > > > burst. With a flush threshold of 2 (and len > threshold instead of len > > > >= threshold), the cache could hold 1 +/-1 packet bursts. With the > > > current multiplier it can only hold [0 .. 1.5[ packet bursts, not > > > really providing a lot of elasticity. > > > > > > > Hi Morten, > > > > > > Interesting to see this being looked at again. The original idea of > > > adding > > > in some extra room above the requested value was to avoid the worst- > > > case > > > scenario of a pool oscillating between full and empty repeatedly due to > > > the > > > addition/removal of perhaps a single packet. As for why 1.5 was chosen > > > as > > > the value, I don't recall any particular reason for it myself. The main > > > objective was to have a separate flush and size values so that we could > > > go > > > a bit above full, and when flushing, not emptying the entire cache down > > > to > > > zero. > > > > Thanks for providing the historical background for this feature, Bruce. > > > > > > > > In terms of the behavioural points you make above, I wonder if symmetry > > > is > > > actually necessary or desirable in this case. After all, the ideal case > > > is > > > probably to keep the mempool neither full nor empty, so that both > > > allocations or frees can be done without having to go to the underlying > > > shared data structure. To accomodate this, the mempool will only flush > > > when > > > the number of elements is greater than size * 1.5, and then it only > > > flushes > > > elements down to size, ensuring that allocations can still take place. > > > On allocation, new buffers are taken when we don't have enough in the > > > cache > > > to fullfil the request, and then the cache is filled up to size, not to > > > the > > > flush threshold. > > > > I agree with the ideal case. > > > > However, it looks like the addition of the flush threshold also changed the "size" parameter to effectively become "desired length" instead. This interpretation is also supported by the flush algorithm, which doesn't flush below the "size", but to the "size". So based on interpretation, I was wondering why it is not symmetrical around the "desired length", but allowed to go 100 % below and only 50 % above. > > > > > > > > Now, for the scenario you describe - where the mempool cache size is > > > set to > > > be the same as the burst size, this scheme probably does break down, in > > > that we don't really have any burst elasticity. However, I would query > > > if > > > that is a configuration that is used, since to the user it should > > > appear > > > correctly to provide no elasticity. Looking at testpmd, and our example > > > apps, the standard there is a burst size of 32 and mempool cache of > > > ~256. > > > In OVS code, netdev-dpdk.c seems to initialize the mempool with cache > > > size > > > of RTE_MEMPOOL_CACHE_MAX_SIZE (through define MP_CACHE_SZ). In all > > > these > > > cases, I think the 1.5 threshold should work just fine for us. > > > > My example was for demonstration only, and hopefully not being used by any applications. > > > > The simplified example was intended to demonstrate the theoretical effect of the unbalance in using the 1.5 threshold. It will be similar with a cache size of 256 objects: You will be allowed to go 8 bursts (calculation: 256 objects / 32 objects per burst) below the cache "size" without retrieving objects from the backing pool, but only 4 bursts above the cache "size" before flushing objects to the backing pool. > > > > > That said, > > > if you want to bump it up to 2x, I can't say I'd object strongly as it > > > should be harmless, I think. > > > > From an academic viewpoint, 2x seems much better to me than 1.5x. And based on your background information from the introduction of the flush threshold, there was no academic reason why 1.5x was chosen - this is certainly valuable feedback. > > Just to say I agree with Morten's analysis. Having a threshold at 2x size would make more sense to me. Without the corner cases (big requests, or common pool empty), I would have suggested this, which looks more symetric than what we have today: get() fill request with as many objs as possible from cache if cache is empty, request "size" more objects from common pool fill the rest of the request from cache put() fill cache with provided objects up to 2x size if cache exceeds threshold, fill common pool with "size" objects fill the cache with the remaining objs from request But this is a quite sensitive area, and modifying these functions should be done with care, as it impacts mbuf allocation. > > I will consider providing a patch. Worst case it probably doesn't do any harm. And if anyone is really interested, they can look at the mempool debug stats to see the difference it makes for their specific application, e.g. the put_common_pool_bulk/put_bulk and get_common_pool_bulk/get_success_bulk ratios should decrease. > > Sure, thanks. > > /Bruce