From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 462ADA00C2; Fri, 14 Oct 2022 16:01:43 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 37A8C42DDF; Fri, 14 Oct 2022 16:01:43 +0200 (CEST) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mails.dpdk.org (Postfix) with ESMTP id F227D42DDF for ; Fri, 14 Oct 2022 16:01:41 +0200 (CEST) Received: by mail-wr1-f52.google.com with SMTP id j7so7725447wrr.3 for ; Fri, 14 Oct 2022 07:01:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=PBue3vAde0jKWK2E52fh1MVzhJtVIFTQZcknbzV26yM=; b=A7sU+B+Nil6Tr/l+rwIT1BNFPi6w81FrC2xR5hQn06z3AfglfrWQbyBT+5h2zSJd0R giX9R5aDuoq4YKZVmunkZhIKZjsSbkf0rnSRauEg96YVingoBlMQvsO/UwKA1B17BNMW Fe75QABjCOwZZinDdi85Lh4hcIqNhaIE5/BEvKym+68vBXdc7ECQrPZL5WXdZSs+wEOv O2oGlKyKiRY+nxsnSEjb+jIoyD2td/7n9SkyW7+YkdTqe9dlAwy9xPfeFQikfnmavFGT EhD7vjwmVguG2EFH+6jIs04DP/myjqECx4BeAbQAh6mDG4gr9r5Ha9SDigId3xC7L0j1 12YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PBue3vAde0jKWK2E52fh1MVzhJtVIFTQZcknbzV26yM=; b=SteXvabTbEYh2DTq1jvIYGhBzA9nY4oEvz++ch+diig0cj1SyrWHP8eeEKCzvjT7lg FC8GZgHNxQtfC8Isa8uc97sFOuwV4HkwEFeGXUwTXW1y8LxsWRgZPohL+TLAKEy2s2m9 sTHAitnFdt4ZApNnSaNZhqM1odDcWFza7XH38wYO4taqALJCM7b/JqGRgfujpct6CwKT JiQhCbNHRwX3St7vvZxzVZoXFaVmX2+SoBQPVrCIo3bsj0xUoXFiRz26N/UA/gj4Wadj +W3qHYAnGQFsEEOqRYowfJ+VWYoW9H6tD5lx08tGPTnZ13tKPbZttMUIvod9qyQK8WwZ 9I3A== X-Gm-Message-State: ACrzQf0wPteiTwRyer/nbq8+eYJxE/ZMAjtOS496kDfqdhxEmMtlHzCK sISDIUFXuc9uYaVUIfOoZSMu/jGBYFCoVA== X-Google-Smtp-Source: AMsMyM5fLrHJSzM3M3eTNW71NIAQBuTZhjMff3FY5qOfSP2lh5OAkBfGe3QanexVtWxnmEoIaQ2hRA== X-Received: by 2002:adf:f40e:0:b0:22e:2ce4:e6a2 with SMTP id g14-20020adff40e000000b0022e2ce4e6a2mr3746531wro.30.1665756101624; Fri, 14 Oct 2022 07:01:41 -0700 (PDT) Received: from 6wind.com ([2a01:e0a:5ac:6460:c065:401d:87eb:9b25]) by smtp.gmail.com with ESMTPSA id d20-20020a05600c34d400b003b4de550e34sm2186375wmq.40.2022.10.14.07.01.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 07:01:39 -0700 (PDT) Date: Fri, 14 Oct 2022 16:01:38 +0200 From: Olivier Matz To: Morten =?iso-8859-1?Q?Br=F8rup?= Cc: Andrew Rybchenko , dev@dpdk.org, Bruce Richardson Subject: Re: [PATCH v6 4/4] mempool: flush cache completely on overflow Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D86DB2@smartserver.smartshare.dk> <20221009133737.795377-1-andrew.rybchenko@oktetlabs.ru> <20221009133737.795377-5-andrew.rybchenko@oktetlabs.ru> <98CBD80474FA8B44BF855DF32C47DC35D873B7@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D873B7@smartserver.smartshare.dk> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Sun, Oct 09, 2022 at 04:44:08PM +0200, Morten Brørup wrote: > > From: Andrew Rybchenko [mailto:andrew.rybchenko@oktetlabs.ru] > > Sent: Sunday, 9 October 2022 15.38 > > To: Olivier Matz > > Cc: dev@dpdk.org; Morten Brørup; Bruce Richardson > > Subject: [PATCH v6 4/4] mempool: flush cache completely on overflow > > > > The cache was still full after flushing. In the opposite direction, > > i.e. when getting objects from the cache, the cache is refilled to full > > level when it crosses the low watermark (which happens to be zero). > > Similarly, the cache should be flushed to empty level when it crosses > > the high watermark (which happens to be 1.5 x the size of the cache). > > The existing flushing behaviour was suboptimal for real applications, > > because crossing the low or high watermark typically happens when the > > application is in a state where the number of put/get events are out of > > balance, e.g. when absorbing a burst of packets into a QoS queue > > (getting more mbufs from the mempool), or when a burst of packets is > > trickling out from the QoS queue (putting the mbufs back into the > > mempool). > > Now, the mempool cache is completely flushed when crossing the flush > > threshold, so only the newly put (hot) objects remain in the mempool > > cache afterwards. > > > > This bug degraded performance caused by too frequent flushing. > > > > Consider this application scenario: > > > > Either, an lcore thread in the application is in a state of balance, > > where it uses the mempool cache within its flush/refill boundaries; in > > this situation, the flush method is less important, and this fix is > > irrelevant. > > > > Or, an lcore thread in the application is out of balance (either > > permanently or temporarily), and mostly gets or puts objects from/to > > the > > mempool. If it mostly puts objects, not flushing all of the objects > > will > > cause more frequent flushing. This is the scenario addressed by this > > fix. E.g.: > > > > Cache size=256, flushthresh=384 (1.5x size), initial len=256; > > application burst len=32. > > > > If there are "size" objects in the cache after flushing, the cache is > > flushed at every 4th burst. > > > > If the cache is flushed completely, the cache is only flushed at every > > 16th burst. > > > > As you can see, this bug caused the cache to be flushed 4x too > > frequently in this example. > > > > And when/if the application thread breaks its pattern of continuously > > putting objects, and suddenly starts to get objects instead, it will > > either get objects already in the cache, or the get() function will > > refill the cache. > > > > The concept of not flushing the cache completely was probably based on > > an assumption that it is more likely for an application's lcore thread > > to get() after flushing than to put() after flushing. > > I strongly disagree with this assumption! If an application thread is > > continuously putting so much that it overflows the cache, it is much > > more likely to keep putting than it is to start getting. If in doubt, > > consider how CPU branch predictors work: When the application has done > > something many times consecutively, the branch predictor will expect > > the > > application to do the same again, rather than suddenly do something > > else. > > > > Signed-off-by: Morten Brørup > > Signed-off-by: Andrew Rybchenko > > --- > > Reviewed-by: Morten Brørup > Acked-by: Olivier Matz