From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 79755A00BE; Tue, 29 Oct 2019 18:09:15 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 071491BF4F; Tue, 29 Oct 2019 18:09:14 +0100 (CET) Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by dpdk.org (Postfix) with ESMTP id E50F71BF4C for ; Tue, 29 Oct 2019 18:09:12 +0100 (CET) Received: by mail-wr1-f66.google.com with SMTP id s1so14540973wro.0 for ; Tue, 29 Oct 2019 10:09:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=W2mW+gdDbIT9uJMAZVPThfiBmrf5xXbBIves0ba7VZ8=; b=OVhvaPmIl7p9fBQCTsI9yoZaLviWGKtQicHFtJ6peVqKYM/utWNay/5qbkuIDEygHT swGAdPb2eksV9XM234oH4LmYwF1LGtSrzO6Q+i9IRdpyUcTyT9paWfL0p92Q7dIGMXm9 7qwV2gmgnrnI3ADVdw7r1BHXj/a5FCz5jOlf584jUYtH36t1WcP3d00R6sRjS96a5Ets V/PL6wFN1w0xpHhKt27vgElKIBlqfqXxNySqw6RuRS5lbnJXRM1dnOa9vupR17lk9Srm N4NwMem92NcbTjOQM1Ss7uO3uzRlLR9qQNFP1Z50ujZSptHFqfxsyzSaKTBNkp/Ao6vx zcNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=W2mW+gdDbIT9uJMAZVPThfiBmrf5xXbBIves0ba7VZ8=; b=mHuJefj24kYk+/E67PCKarl+FxbfKZM3/X555TDhq0VyxVPoXL1rqa0dlWp0+LusHD mq4BsPUcpD+YGe77KmvoaxVcT4JWDjniLS7pRr71OMWlp+dBKgxxgT/Tc4OzIhCjAkoc v26sde5NeO9QD9AntW2zu9aO/l1Nig9jNKsOab2gDlOHbVI4lUdcZ+x2Xuq545Olj9iw i6WcMGrIdvTeb6tY9c4GlV9eK1F6ZhLPRWKYxitxkUn0QtTKZQAhx9FwZX7WnyVTMCi5 OdDjBCnXoFobyz4uRnPcGck7jrh1nttKvgajPQPxkJ1lhCN/gn3Wk5ZkAL6k25UZ9WzG cxCQ== X-Gm-Message-State: APjAAAWGg7ImBgVc0miUTw+qX3+YphcNZlC/V8r/QrYYhSQIVdRxuZgs inoI9qqDOO99jrWwvkzXXW4wjA== X-Google-Smtp-Source: APXvYqwAlC21xEbEoO37JEqFgY1v8NyHIGOUzGH2biaJ4fk6NYe2SbiKCHKmoHoccwVVNLKoqDNchw== X-Received: by 2002:adf:e9c7:: with SMTP id l7mr4427161wrn.57.1572368952459; Tue, 29 Oct 2019 10:09:12 -0700 (PDT) Received: from 6wind.com (2a01cb0c0005a6000226b0fffeed02fc.ipv6.abo.wanadoo.fr. [2a01:cb0c:5:a600:226:b0ff:feed:2fc]) by smtp.gmail.com with ESMTPSA id z9sm16425930wrv.1.2019.10.29.10.09.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2019 10:09:11 -0700 (PDT) Date: Tue, 29 Oct 2019 18:09:10 +0100 From: Olivier Matz To: Andrew Rybchenko Cc: dev@dpdk.org, Anatoly Burakov , Ferruh Yigit , "Giridharan, Ganesan" , Jerin Jacob Kollanukkaran , Kiran Kumar Kokkilagadda , Stephen Hemminger , Thomas Monjalon , Vamsi Krishna Attunuru Message-ID: <20191029170910.ukrgksaxde63ti4o@platinum> References: <20190719133845.32432-1-olivier.matz@6wind.com> <20191028140122.9592-1-olivier.matz@6wind.com> <20191028140122.9592-3-olivier.matz@6wind.com> <7b34ec2a-5ec8-043d-ec5c-811984ce6171@solarflare.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7b34ec2a-5ec8-043d-ec5c-811984ce6171@solarflare.com> User-Agent: NeoMutt/20180716 Subject: Re: [dpdk-dev] [PATCH 2/5] mempool: reduce wasted space on mempool populate X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Oct 29, 2019 at 01:09:01PM +0300, Andrew Rybchenko wrote: > On 10/28/19 5:01 PM, Olivier Matz wrote: > > The size returned by rte_mempool_op_calc_mem_size_default() is aligned > > to the specified page size. Therefore, with big pages, the returned size > > can be much more that what we really need to populate the mempool. > > > > For instance, populating a mempool that requires 1.1GB of memory with > > 1GB hugepages can result in allocating 2GB of memory. > > > > This problem is hidden most of the time due to the allocation method of > > rte_mempool_populate_default(): when try_iova_contig_mempool=true, it > > first tries to allocate an iova contiguous area, without the alignment > > constraint. If it fails, it fallbacks to an aligned allocation that does > > not require to be iova-contiguous. This can also fallback into several > > smaller aligned allocations. > > > > This commit changes rte_mempool_op_calc_mem_size_default() to relax the > > alignment constraint to a cache line and to return a smaller size. > > > > Signed-off-by: Olivier Matz > > One may be unrelated questions below > > Reviewed-by: Andrew Rybdhenko > > [snip] > > > diff --git a/lib/librte_mempool/rte_mempool_ops_default.c b/lib/librte_mempool/rte_mempool_ops_default.c > > index 4e2bfc82d..f6aea7662 100644 > > --- a/lib/librte_mempool/rte_mempool_ops_default.c > > +++ b/lib/librte_mempool/rte_mempool_ops_default.c > > @@ -12,7 +12,7 @@ rte_mempool_op_calc_mem_size_default(const struct rte_mempool *mp, > > size_t *min_chunk_size, size_t *align) > > { > > size_t total_elt_sz; > > - size_t obj_per_page, pg_num, pg_sz; > > + size_t obj_per_page, pg_sz, objs_in_last_page; > > size_t mem_size; > > total_elt_sz = mp->header_size + mp->elt_size + mp->trailer_size; > > @@ -33,14 +33,30 @@ rte_mempool_op_calc_mem_size_default(const struct rte_mempool *mp, > > mem_size = > > RTE_ALIGN_CEIL(total_elt_sz, pg_sz) * obj_num; > > } else { > > - pg_num = (obj_num + obj_per_page - 1) / obj_per_page; > > - mem_size = pg_num << pg_shift; > > + /* In the best case, the allocator will return a > > + * page-aligned address. For example, with 5 objs, > > + * the required space is as below: > > + * | page0 | page1 | page2 (last) | > > + * |obj0 |obj1 |xxx|obj2 |obj3 |xxx|obj4| > > + * <------------- mem_size -------------> > > + */ > > + objs_in_last_page = ((obj_num - 1) % obj_per_page) + 1; > > + /* room required for the last page */ > > + mem_size = objs_in_last_page * total_elt_sz; > > + /* room required for other pages */ > > + mem_size += ((obj_num - objs_in_last_page) / > > + obj_per_page) << pg_shift; > > + > > + /* In the worst case, the allocator returns a > > + * non-aligned pointer, wasting up to > > + * total_elt_sz. Add a margin for that. > > + */ > > + mem_size += total_elt_sz - 1; > > } > > } > > - *min_chunk_size = RTE_MAX((size_t)1 << pg_shift, total_elt_sz); > > - > > - *align = RTE_MAX((size_t)RTE_CACHE_LINE_SIZE, (size_t)1 << pg_shift); > > + *min_chunk_size = total_elt_sz; > > + *align = RTE_CACHE_LINE_SIZE; > > Not directly related to the patch, but may be RTE_MEMPOOL_ALIGN should be > used? Yes, and there is another one that could be changed in rte_mempool_populate_iova(). I can add a patch for that. > > > return mem_size; > > } >