From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B675A0C47; Sun, 31 Oct 2021 09:14:11 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D11004068E; Sun, 31 Oct 2021 09:14:10 +0100 (CET) Received: from smartserver.smartsharesystems.com (smartserver.smartsharesystems.com [77.243.40.215]) by mails.dpdk.org (Postfix) with ESMTP id EA8D640689 for ; Sun, 31 Oct 2021 09:14:08 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Sun, 31 Oct 2021 09:14:06 +0100 Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D86C83@smartserver.smartshare.dk> X-MimeOLE: Produced By Microsoft Exchange V6.5 X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [dpdk-dev] [RFC] mempool: implement index-based per core cache Thread-Index: AQHXtwuB3FbH85c80U2vmaazO82iNau+1ETwgAE6tgCAAv3qYIAocoNQgAFvUzA= References: <20210930172735.2675627-1-dharmik.thakkar@arm.com> From: =?iso-8859-1?Q?Morten_Br=F8rup?= To: "Honnappa Nagarahalli" , "Ananyev, Konstantin" , "Dharmik Thakkar" , "Olivier Matz" , "Andrew Rybchenko" Cc: , "nd" , "Ruifeng Wang" , "nd" , "nd" Subject: Re: [dpdk-dev] [RFC] mempool: implement index-based per core cache X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > From: Morten Br=F8rup > Sent: Saturday, 30 October 2021 12.24 >=20 > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Honnappa > > Nagarahalli > > Sent: Monday, 4 October 2021 18.36 > > > > > > > > > > > > > > > > Current mempool per core cache implementation is based on > > pointer > > > > > > For most architectures, each pointer consumes 64b Replace it > > with > > > > > > index-based implementation, where in each buffer is = addressed > > by > > > > > > (pool address + index) >=20 > I like Dharmik's suggestion very much. CPU cache is a critical and > limited resource. >=20 > DPDK has a tendency of using pointers where indexes could be used > instead. I suppose pointers provide the additional flexibility of > mixing entries from different memory pools, e.g. multiple mbuf pools. >=20 > > > > > > > > > > I don't think it is going to work: > > > > > On 64-bit systems difference between pool address and it's = elem > > > > > address could be bigger than 4GB. > > > > Are you talking about a case where the memory pool size is more > > than 4GB? > > > > > > That is one possible scenario. >=20 > That could be solved by making the index an element index instead of a > pointer offset: address =3D (pool address + index * element size). Or instead of scaling the index with the element size, which is only = known at runtime, the index could be more efficiently scaled by a = compile time constant such as RTE_MEMPOOL_ALIGN (=3D = RTE_CACHE_LINE_SIZE). With a cache line size of 64 byte, that would = allow indexing into mempools up to 256 GB in size. >=20 > > > Another possibility - user populates mempool himself with some > > external > > > memory by calling rte_mempool_populate_iova() directly. > > Is the concern that IOVA might not be contiguous for all the memory > > used by the mempool? > > > > > I suppose such situation can even occur even with normal > > > rte_mempool_create(), though it should be a really rare one. > > All in all, this feature needs to be configurable during compile > time.