From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 64180A034C; Fri, 21 Jan 2022 07:01:54 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2EA1742729; Fri, 21 Jan 2022 07:01:54 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60042.outbound.protection.outlook.com [40.107.6.42]) by mails.dpdk.org (Postfix) with ESMTP id 2224040042 for ; Fri, 21 Jan 2022 07:01:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mrJ0xz/qKxGOkVGTa/i+KfYBxpHINZOg1essPB1Jo3o=; b=mBlydhSpDKKNo8/e2f0VDMyQr1n7SlNK4PN3/Yzc0fHgST05uGl68/9BKIdp1tq4CuSr5kOZp3s6ISG8oVRZETxB2D6g09TljCS9jDs8I69v8w+TO8A+RiPdIjMmwItC5Vj8ojiI+0wr+Mqtc4K3MVBT1Wm4h3ep+bvcSAUkbmY= Received: from AS9PR06CA0087.eurprd06.prod.outlook.com (2603:10a6:20b:464::8) by AM8PR08MB5618.eurprd08.prod.outlook.com (2603:10a6:20b:1d7::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.7; Fri, 21 Jan 2022 06:01:39 +0000 Received: from AM5EUR03FT057.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:464:cafe::2c) by AS9PR06CA0087.outlook.office365.com (2603:10a6:20b:464::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.7 via Frontend Transport; Fri, 21 Jan 2022 06:01:39 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=temperror action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT057.mail.protection.outlook.com (10.152.17.44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.7 via Frontend Transport; Fri, 21 Jan 2022 06:01:37 +0000 Received: ("Tessian outbound 2877e54fe176:v113"); Fri, 21 Jan 2022 06:01:37 +0000 X-CR-MTA-TID: 64aa7808 Received: from aaca4e2ea110.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id BD300673-9388-47DF-833B-9EF110427414.1; Fri, 21 Jan 2022 06:01:27 +0000 Received: from EUR02-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id aaca4e2ea110.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 21 Jan 2022 06:01:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jPd6FZ+2UmP4MXCWjALn0BA9UQDG8v1GduFppfClWfPAOD4Rq1jrdNEBxaop2ayFZzay1IU2hxc4lWLFItVMYsPBKBjQl+u+nFE+kuVqCdfEbssRtPR+QKej8PITcIx+AeqKVOhHQl+V+2vf16GSOTYK1ZWCwsqf4LEfINEBbXxAvsK6eg8qeRfUKoXv/a2914ZCpo2WuDXQo7FN2/EBDh2Lv+oU6IIdFvoQDTsYNB0iyEpjJWLsu39Evn4sehVxV/AF7nyQH0Bkk7BSm78rD5QsHsQQSG/PNrF3QGPabJ6XaRxRM6riDwh20tFecsp+UjJEjqTbl5XFCvECMDmhLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mrJ0xz/qKxGOkVGTa/i+KfYBxpHINZOg1essPB1Jo3o=; b=TYJWjVTbT1f0/Pc5zG4tT0tk8+FoTnt1a7mpPuwiWgpgMTAhgwR0sh/qbSddHRR34AGnnSdDzUYfo1YNNO6uH5Z+MRCRniCDqp0Y70xdINFS9wUtqdsfB7ZZcANIoVPaZPbh/iTO5WcCL9PenApSNWpfgIclL7Hfjniew644vL+hOpxJujUkvetGXKVCyDiRMGzpAWV0dcapg6vg8O0V3IqWrtvomTzd+cu9VaxmxEnQe0YtKPXmhDYiis0AR+YWUQIKR+MQ/ib8z/isgOVAfotpy4lCMjTRwVzq49qh/iE7oiOAHxrYAeIGS1zv+0aqm5CJaazkIYVSLWG7FtNj9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mrJ0xz/qKxGOkVGTa/i+KfYBxpHINZOg1essPB1Jo3o=; b=mBlydhSpDKKNo8/e2f0VDMyQr1n7SlNK4PN3/Yzc0fHgST05uGl68/9BKIdp1tq4CuSr5kOZp3s6ISG8oVRZETxB2D6g09TljCS9jDs8I69v8w+TO8A+RiPdIjMmwItC5Vj8ojiI+0wr+Mqtc4K3MVBT1Wm4h3ep+bvcSAUkbmY= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by AM6PR08MB4454.eurprd08.prod.outlook.com (2603:10a6:20b:bf::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.10; Fri, 21 Jan 2022 06:01:24 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7c20:8c83:fc45:db99]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7c20:8c83:fc45:db99%5]) with mapi id 15.20.4909.012; Fri, 21 Jan 2022 06:01:24 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , Dharmik Thakkar , Olivier Matz , Andrew Rybchenko CC: "dev@dpdk.org" , nd , Ruifeng Wang , Beilei Xing , nd Subject: RE: [PATCH v2 1/1] mempool: implement index-based per core cache Thread-Topic: [PATCH v2 1/1] mempool: implement index-based per core cache Thread-Index: AQHYCD+SYdKHeDO5vkio0T5c8Zbcg6xrnQAAgACjtfA= Date: Fri, 21 Jan 2022 06:01:23 +0000 Message-ID: References: <20211224225923.806498-1-dharmik.thakkar@arm.com> <20220113053630.886638-1-dharmik.thakkar@arm.com> <20220113053630.886638-2-dharmik.thakkar@arm.com> <98CBD80474FA8B44BF855DF32C47DC35D86E14@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D86E14@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 45F197FBAE28494B84E800B3F1F11695.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-MS-Office365-Filtering-Correlation-Id: babd0005-102e-463b-635e-08d9dca37cad x-ms-traffictypediagnostic: AM6PR08MB4454:EE_|AM5EUR03FT057:EE_|AM8PR08MB5618:EE_ X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:8882;OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: I5owy4CPtKVzdnaZ0woT8RGW9te7CFmMSw83NRaLcRKv4mcREXuxOftIWeefLC4pU+Wk3GJnYcz6b8qDqisHuOtt4rk5xzGDezwz2TBO3AjvYQu43mKwYgzNkaL4wayQxhAzjz/VpbWxrgCGD0rPK+SVh8p3EXLoeuqHzQmm0q/srTunAxRQpU2N84nak3z5lY9KpnA/s52UMnHKJnprs26++PGyZyZ2eEKGd1o0Bu97UhqVdWcY1mNWXB26oNoKRgJgebr3mv57VBGmYu8iiL/2J5QlloFW/qPdEMogzDjbH8qgVMM/vUxMrNEAKpyJYqHPt8PnKsck5hm2TucA1Ka2E4Y97H4VSUh3WPEsF8TzCK3EsZAaQuFdhkoIH9vA/PkLury39MUScYJVN6hnpv6Y+G/31ldv7TvMVhfMb8ja4JtTbVGyww+f8DzoUGsfYxC3SvEJWQUIzQps2JG9diruFs5R+KKNcl9M6y/e51vHw41jCsjB7NOjeanSoQx5s2AdrDt9u2AmYyljSEHzyFbWNtJswg5iMhw9gKWA9XEwk/yB3QmcJSOXqU26iIbck55xdQU3MLJDOBjTagb5bfrW0XVauKjte+OPQA7V9XqByKqnhWlOzexfYGC3JmoUW2wgKcMikDfzHPxDrZm6GLMNcMEhBI8n7dvFXLszNU9VoK5TsqB4m+yYm03woNS6mKGpLBW6VtsTtHzkJqyZ9vJ62wPEjsVfj8PTt1FlW0MaMQ/3Bl3omBYN7TQQ1sHRt3KDjPUJ/J9nblWD9i2v8bWuFRNiRfQ8Ic7y1JMI9NE= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(316002)(9686003)(54906003)(110136005)(66476007)(66946007)(76116006)(8676002)(4326008)(66556008)(64756008)(66446008)(83380400001)(38070700005)(122000001)(55016003)(38100700002)(33656002)(45080400002)(6506007)(8936002)(71200400001)(508600001)(966005)(52536014)(186003)(26005)(86362001)(2906002)(5660300002)(7696005); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4454 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 6e98ac7c-ab2a-4d86-2009-08d9dca37462 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: p4LZAKnzh0TJ2rhUWtf3Q7uEnXuyZGf4Buf9T1EW4mSWsSUWb2pXB/C9GF7JTfDuu8/qoGuPPW0cNx71S044OQc4Ja4t+jT4xQL7nQrLnCQavrHTY2lm3T8rGh89LoR3on2dhlqshKrdX8ZQ410SFfeQW6pNqH877PrCG1cOSN5zQN+OU1x/0Y7cLb9UZAzehtruFrgb9tOljkSMkkZywxNzWmrt+MeJnihHimXHeGYyrJzX8ipCJB1ShxVRLBDMdDtq/z10EKd9veBHWdsEDi4Fu/Nd3NzZKuJiOJCsJJCun5GSpcgc8UTaCN2g6lWtMl3n1yGkH2eHJbx8t4xtU/CFnL6PIC7jDXvvrHDuCwO2neWfaamBDHXiTKT2bky9CKIQwnpgWNW4pU0TxrkB1atfYwXv3aejsnw01Gpwtwe1Z3ubr0Mz0HQEotMrY73kJbzfOJodX2SDrufFJIpBXIH5DuaEkA0VqOYBO6HfVjgRD2oSader14mfqDLl9rZ3eZwajFN4Q/UCpYCqvvEG5YkvaZikvEVl3wuU0lX6D9PVrMqHl+RMwAVm4qVnvi2lT15zw53HwnVp6es8M2iGLbCN/j9r0KFNV7KbGUQxVMSYqPZLeg7fApWEeRXGP2iFI//SdNBZVFQt84S0ycuBhdC+C1rNjkmLEP2qjkp0thH0KfvPwCZxE3bDsPyEGJugI2h4AE81C16hOfxRpyKTpW64FYrnDzLj1mRtANtUngoffECcalbYhIvOy7Djnlfhu9Bg4q9CDjwjQBmiZKcyiqFSs/Zfo8GHlO9LqZSlttI= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(508600001)(47076005)(83380400001)(55016003)(82310400004)(110136005)(63370400001)(63350400001)(54906003)(86362001)(356005)(966005)(7696005)(316002)(2906002)(336012)(45080400002)(70206006)(9686003)(6506007)(8676002)(52536014)(26005)(8936002)(33656002)(5660300002)(36860700001)(81166007)(4326008)(186003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jan 2022 06:01:37.8528 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: babd0005-102e-463b-635e-08d9dca37cad X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5618 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org >=20 > +CC Beilei as i40e maintainer >=20 > > From: Dharmik Thakkar [mailto:dharmik.thakkar@arm.com] > > Sent: Thursday, 13 January 2022 06.37 > > > > Current mempool per core cache implementation stores pointers to mbufs > > On 64b architectures, each pointer consumes 8B This patch replaces it > > with index-based implementation, where in each buffer is addressed by > > (pool base address + index) It reduces the amount of memory/cache > > required for per core cache > > > > L3Fwd performance testing reveals minor improvements in the cache > > performance (L1 and L2 misses reduced by 0.60%) with no change in > > throughput > > > > Suggested-by: Honnappa Nagarahalli > > Signed-off-by: Dharmik Thakkar > > Reviewed-by: Ruifeng Wang > > --- > > lib/mempool/rte_mempool.h | 150 +++++++++++++++++++++++++- > > lib/mempool/rte_mempool_ops_default.c | 7 ++ > > 2 files changed, 156 insertions(+), 1 deletion(-) > > > > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h > > index 1e7a3c15273c..f2403fbc97a7 100644 > > --- a/lib/mempool/rte_mempool.h > > +++ b/lib/mempool/rte_mempool.h > > @@ -50,6 +50,10 @@ > > #include > > #include > > > > +#ifdef RTE_MEMPOOL_INDEX_BASED_LCORE_CACHE > > +#include > > +#endif > > + > > #include "rte_mempool_trace_fp.h" > > > > #ifdef __cplusplus > > @@ -239,6 +243,9 @@ struct rte_mempool { > > int32_t ops_index; > > > > struct rte_mempool_cache *local_cache; /**< Per-lcore local cache */ > > +#ifdef RTE_MEMPOOL_INDEX_BASED_LCORE_CACHE > > + void *pool_base_value; /**< Base value to calculate indices */ > > +#endif > > > > uint32_t populated_size; /**< Number of populated > > objects. */ > > struct rte_mempool_objhdr_list elt_list; /**< List of objects in > > pool */ @@ -1314,7 +1321,22 @@ rte_mempool_cache_flush(struct > > rte_mempool_cache *cache, > > if (cache =3D=3D NULL || cache->len =3D=3D 0) > > return; > > rte_mempool_trace_cache_flush(cache, mp); > > + > > +#ifdef RTE_MEMPOOL_INDEX_BASED_LCORE_CACHE > > + unsigned int i; > > + unsigned int cache_len =3D cache->len; > > + void *obj_table[RTE_MEMPOOL_CACHE_MAX_SIZE * 3]; > > + void *base_value =3D mp->pool_base_value; > > + uint32_t *cache_objs =3D (uint32_t *) cache->objs; >=20 > Hi Dharmik and Honnappa, >=20 > The essence of this patch is based on recasting the type of the objs fiel= d in the > rte_mempool_cache structure from an array of pointers to an array of > uint32_t. >=20 > However, this effectively breaks the ABI, because the rte_mempool_cache > structure is public and part of the API. The patch does not change the public structure, the new member is under com= pile time flag, not sure how it breaks the ABI. >=20 > Some drivers [1] even bypass the mempool API and access the > rte_mempool_cache structure directly, assuming that the objs array in the > cache is an array of pointers. So you cannot recast the fields in the > rte_mempool_cache structure the way this patch requires. IMO, those drivers are at fault. The mempool cache structure is public only= because the APIs are inline. We should still maintain modularity and not u= se the members of structures belonging to another library directly. A simil= ar effort involving rte_ring was not accepted sometime back [1] [1] http://inbox.dpdk.org/dev/DBAPR08MB5814907968595EE56F5E20A798390@DBAPR0= 8MB5814.eurprd08.prod.outlook.com/ >=20 > Although I do consider bypassing an API's accessor functions "spaghetti > code", this driver's behavior is formally acceptable as long as the > rte_mempool_cache structure is not marked as internal. >=20 > I really liked your idea of using indexes instead of pointers, so I'm ver= y sorry to > shoot it down. :-( >=20 > [1]: E.g. the Intel i40e PMD, > http://code.dpdk.org/dpdk/latest/source/drivers/net/i40e/i40e_rxtx_vec_av= x > 512.c#L25 It is possible to throw an error when this feature is enabled in this file.= Alternatively, this PMD could implement the code for index based mempool. >=20 > -Morten