From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 72625A04A2; Sun, 23 Jan 2022 08:14:46 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ED93C40688; Sun, 23 Jan 2022 08:14:45 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 202A940040 for ; Sun, 23 Jan 2022 08:14:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1642922084; x=1674458084; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=OKHJ0rgpSqnHNsdxuQ6KEeHFzvSOXFrlAZxKASLEb4g=; b=LlEOfTiKAL52irS5UGNrt454EW4Ix9Yg82G0tdJfMFMhKDj+clYp9QDi UNeWYgYRAWLKrWNhVrsnPMMPGNvdfYJAUuQk24O7x6CBA/b8tqSNBbaM1 PJLqi5BTHA1Hg5c8VUK5wG5edRp7aU59YJRNwA2r+X8GZB1HE/GA2e315 8QwQJQDxsCNd4s9Lej8QbsVjKLlkQL2DwRFqljodxXA/2iajx1OTMDH0c TCFu28P5PUKGCFjTdZjt9pxwXpyFTmUm+q+q4qaQ8MVt2qhbZ/kJ3/ZW5 eRbZ2EbsOfuRCZvbEMYJ0f5bT9+jMdx3Qk1pYgha9390mDHnW0C5JU+M1 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10235"; a="246091760" X-IronPort-AV: E=Sophos;i="5.88,309,1635231600"; d="scan'208";a="246091760" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jan 2022 23:14:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,309,1635231600"; d="scan'208";a="695076657" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga005.jf.intel.com with ESMTP; 22 Jan 2022 23:14:34 -0800 Received: from orsmsx608.amr.corp.intel.com (10.22.229.21) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Sat, 22 Jan 2022 23:14:34 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx608.amr.corp.intel.com (10.22.229.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20 via Frontend Transport; Sat, 22 Jan 2022 23:14:34 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.177) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.20; Sat, 22 Jan 2022 23:13:54 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K5uWVmwN0RzgtHGQGcqSERC+1u6hVxzp7FO7iljnw4dCOg4c4AfOz0sexGAwKifdpxbNgMirM8mVmBhRboSDEYbTwB8YvY6ydDOi/6WXjmSsR8kfNIW52KJCzHw434fyPURlLZPzMGriGSwFRpwRtFKxUkWwwaDFF8QHtPbWHS2dZWoGFS3tNofAKgO59bx3EP2wsOyc1tYcMunMTguPytIVY1CI6iPZ5F+HxGDvY5vivmVnJNrGLH3DseFeFqpRvEfCylTKHC+5p4GqwAS5mLI/ajJXXxSZAfBJgx2JqIRKt93UGAOyMyh7ktpyaeWgCZ58nqkl7eC/9FJKEgA/wA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0k94jZUhTJdI5dj6tFgD53vqLKMq+pFlKME2i8gMNyk=; b=LlGXUJW2ifUQ0nS6iJb8j6scLQtubPvNvla2vOZYyGAS0mum59iv90ix904e5s0LzXTBShNyOtACJ82D+gkHEmFetVe0M1La2WQ7T2Ec0JsE5PxVF/Q1lxin0Zs8vqjuAi5gAQp7Jh6rHZF17xWgMrLyTGtVWe31jLhmrfAC3F+WjN50g/zO1oXKZUn3wKswOTF/WauBWdp9RPRfX6tlvRjxozJtUDyQKc6ZMzNZB74W55WymKiY3uwgOC4tz4lomyxAFSsN3CgS0tf7BYXaKel/7wIJztkOldVC5RlSHKcpelQQ41pkDigS2PfsqXvKoxhJTB1Xz4989au06BKKJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Received: from BYAPR11MB3495.namprd11.prod.outlook.com (2603:10b6:a03:8a::14) by SN6PR11MB2800.namprd11.prod.outlook.com (2603:10b6:805:5b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.7; Sun, 23 Jan 2022 07:13:51 +0000 Received: from BYAPR11MB3495.namprd11.prod.outlook.com ([fe80::8b:8b64:1a16:859c]) by BYAPR11MB3495.namprd11.prod.outlook.com ([fe80::8b:8b64:1a16:859c%3]) with mapi id 15.20.4909.017; Sun, 23 Jan 2022 07:13:51 +0000 From: "Wang, Haiyue" To: Dharmik Thakkar , Olivier Matz , Andrew Rybchenko CC: "dev@dpdk.org" , "nd@arm.com" , "honnappa.nagarahalli@arm.com" , "ruifeng.wang@arm.com" , =?iso-8859-1?Q?Morten_Br=F8rup?= , "Ananyev, Konstantin" , "Richardson, Bruce" , "Xing, Beilei" Subject: RE: [PATCH v2 1/1] mempool: implement index-based per core cache Thread-Topic: [PATCH v2 1/1] mempool: implement index-based per core cache Thread-Index: AQHYCD+ggtWfrTUygkuSfFCmwe15qaxwPWGA Date: Sun, 23 Jan 2022 07:13:50 +0000 Message-ID: References: <20211224225923.806498-1-dharmik.thakkar@arm.com> <20220113053630.886638-1-dharmik.thakkar@arm.com> <20220113053630.886638-2-dharmik.thakkar@arm.com> In-Reply-To: <20220113053630.886638-2-dharmik.thakkar@arm.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.6.200.16 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0df4f4f0-6907-491a-839c-08d9de3fe839 x-ms-traffictypediagnostic: SN6PR11MB2800:EE_ x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3276; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: onHXdjFaMjYu/byeWsHL+vvwVuUJj2sAZ090FqMOYu6DSJhgdh7lHaBWBDntw2UA5KdhsYE+n3hWuXjdYw+KrU14Fdd0O6Fpb6FelhEftJ8FjM1eNz7+lIT3pVLAk9APdGsqoW2LBhKWwC1I7scVP/3usO9+uY3MAmMiBKIM3iL7Rn60yKvbsH9bAvsj0rzeCQ/hvJOL01pIFPGoPJCZg+nXdThLLUXR1t5V6AJGRMXdyn+jpUgKY2alhrMojJNV+w5BKKeKY9Rjk3RG9uch7pqUY34ZUI8oS/N5u0YRvBnN638JbW6kGPiZF/DAnazaSRJG0nncZWxa3oUpMZxZHih9zWsF4qyi7Qkzn5J+K3ljPZNX5qy09jpVojV8MyXXAQ3rw4ozdWU5sctLu454bga60cCdon58pfYoD1lI/Zj+NSyVJqwY0q68qPn3HW5uYJXx3jfR76OH6tAxreJM8ohnRcNocPHvstHyQzgwId3HJk7LoWF4J17HnVqd3JALOdtM2X70u06AKtY4dB8SCS0SswDLRkDUqo+j3dyJ0UHTSrVO/fPWmHPOGgU6gRT8/7aPXifEow4+WBVSsGjI0XcsfbtLlqwxFt8Z0GYLfQogzys4DQlecsBLTLlK7qF8LnxkYaAgVm0RNSWrHVkd6W4k/QraTiD74LkaHCNQ5bCvY+TnIdi9cC/B/VBaZAuD+Dn847nmIvaLFKL2HZjU3A== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3495.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(26005)(9686003)(122000001)(71200400001)(86362001)(33656002)(316002)(82960400001)(52536014)(4326008)(66946007)(66446008)(107886003)(55016003)(38100700002)(64756008)(54906003)(66556008)(7696005)(508600001)(76116006)(66476007)(110136005)(8936002)(186003)(38070700005)(8676002)(53546011)(5660300002)(83380400001)(6506007)(2906002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?qnmedB25gP/1IuGzAnYHSjrIJ7oB5fFI/bqVt/c+JeEsUjgx6HeNs4TJSO?= =?iso-8859-1?Q?fTqeAWE5vBUXyWxTep9hKWjTtXo4ITXFSQEkz0IfRWuORpq9I3nD80Yr/w?= =?iso-8859-1?Q?1bAolcCAEne0NpR3jCtqwXrwwSJ0k08KMrhH/WAab3hgSQLn9gliLgZCOH?= =?iso-8859-1?Q?gzDRDzo7lYFQnuvHYRI05h4OHV0j7UR55XjV/pmnbotRHikwuidGkSg5hP?= =?iso-8859-1?Q?YvM/ETExDyBGtXRDd0tK2RR3MpZmCxitjX4LVc1cMs5zZg2ic18EkN0tYZ?= =?iso-8859-1?Q?OW5cwjYMVRZzUknqj8tyzxpj8E0Gv/AUYiRexQHD/1Ro31wTjmV4Z//NT8?= =?iso-8859-1?Q?fXay/k40c4Ip1wHsokeqzPK30Ds6aAl2PAWAUJOI6I1dL8+7rsCwK4pTYI?= =?iso-8859-1?Q?GEtdAr7tJmDDq64mAwFJKa45GCDcUVO5sajh74+WDcL2WrqyemEn3eIwzQ?= =?iso-8859-1?Q?si0YexYujbuoRunL6NIFdTB0rfnFBDVEpJzQtSvbk2CmS8s0iIdflprXSX?= =?iso-8859-1?Q?/cXK07Fsr5PH2yAnOuVoJC0rDlyUXcERLD1RCpe3IlhzzoyWuHMulBrNy5?= =?iso-8859-1?Q?rbFSnX2GaTwnpZT6BL+JNV+Ec/HdnjhZbz8aqYYwWPg/EzuYB5kGHxacsE?= =?iso-8859-1?Q?C6rsSmR92dvCbvBZH/8/XMsHziRJxZtCgpKCMvcVvUo5cLqhTzdnryyEAh?= =?iso-8859-1?Q?Bm8dlrVQ4eo2kXr0jOHTN6KiIFg+tDxKqCKNSxpeF26aCeAFfmo+xPYcZv?= =?iso-8859-1?Q?jdnqHvSL0RYy0TGqJPsjZeDjJIoayEv0ujE63vnN2wrxtkE+E9WDzXxRsA?= =?iso-8859-1?Q?pOCA8ikSq68OE/Z9PTQgk5gcL48wesoUHwRns+2DzkaYof+qB+O9Y+JfZS?= =?iso-8859-1?Q?mdpOfFnmXx3pIU8vNc+3b3AUiQI588usEIA/0V04mgFXSDjE62/mc8blIT?= =?iso-8859-1?Q?NO3bkZSaAtbBNNOj4ppeYePr/xlT0HU0iHoSEyprw4RRqPDFsMyG0ORcd2?= =?iso-8859-1?Q?g+8dRDoojEgUUBJXtAjl9+YC/d5Tza8QFYSBEH/yKIUW41EPLGs6bkHpQ7?= =?iso-8859-1?Q?cj+ZJosN83QRE14ulgT+vmhApKJP+JxL+gvT5uigGyeQydFiz/DiJuBNYM?= =?iso-8859-1?Q?1ylL6PYNT09swIdvCX04O8XC6P9izmhtSUERhJD8FjbtaoXSHB8Wr13Eki?= =?iso-8859-1?Q?4yeIb7bHNAYdon/p0w7FY0gpLebuRn7mICcqqp4LbHkVfoyWw356G87rhm?= =?iso-8859-1?Q?aL5DIUS210z/gfOJ4JIiZatXKRRH3Tt1Fk+NN9Y2CgCYu5POXZRVOWLuY4?= =?iso-8859-1?Q?zHnKOy3BjYleGjSS534nSCVw4w3fWY05CrkXggH8eiay4b7szrkoAnOQbG?= =?iso-8859-1?Q?N7Zy6rKb7e3pMLDVMUjxNTWsWHhUwjGNGm3biGetS+KN3rMonAVRskLfmh?= =?iso-8859-1?Q?nHl/4QribaxeyJYqGmCwqCH+2qWV/FFR4urMkAS90AaZRFCuyHRNAzU1pa?= =?iso-8859-1?Q?N4TDrYd9KuKYFqi/F3zS1nQcz1K/aX11DoyR6WH4NiUetPguTwGFqLYkkD?= =?iso-8859-1?Q?33xR2p8KwArrxDkIx3FnsRv1FhCWqPB52Re136ASXGj3jlOf/26h7N01pQ?= =?iso-8859-1?Q?L0pwerSQHiaOQK1UCn3a961gTCxrZ0i8yiGbEH4MCT8LxRAzuXE/fW2Q?= =?iso-8859-1?Q?=3D=3D?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3495.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0df4f4f0-6907-491a-839c-08d9de3fe839 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jan 2022 07:13:50.8316 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LcPhYHkQeM4unRNfQxODTgw9t9qYrpF2qdMRQIvJ5CCIKq1s5J3NHJOISQvD3d6q3Pb/zka0hkri1jjl4M/32Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR11MB2800 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Dharmik Thakkar > Sent: Thursday, January 13, 2022 13:37 > To: Olivier Matz ; Andrew Rybchenko > Cc: dev@dpdk.org; nd@arm.com; honnappa.nagarahalli@arm.com; ruifeng.wang@= arm.com; Dharmik Thakkar > > Subject: [PATCH v2 1/1] mempool: implement index-based per core cache >=20 > Current mempool per core cache implementation stores pointers to mbufs > On 64b architectures, each pointer consumes 8B > This patch replaces it with index-based implementation, > where in each buffer is addressed by (pool base address + index) > It reduces the amount of memory/cache required for per core cache >=20 > L3Fwd performance testing reveals minor improvements in the cache > performance (L1 and L2 misses reduced by 0.60%) > with no change in throughput >=20 > Suggested-by: Honnappa Nagarahalli > Signed-off-by: Dharmik Thakkar > Reviewed-by: Ruifeng Wang > --- > lib/mempool/rte_mempool.h | 150 +++++++++++++++++++++++++- > lib/mempool/rte_mempool_ops_default.c | 7 ++ > 2 files changed, 156 insertions(+), 1 deletion(-) >=20 > diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h > index 1e7a3c15273c..f2403fbc97a7 100644 > diff --git a/lib/mempool/rte_mempool_ops_default.c b/lib/mempool/rte_memp= ool_ops_default.c > index 22fccf9d7619..3543cad9d4ce 100644 > --- a/lib/mempool/rte_mempool_ops_default.c > +++ b/lib/mempool/rte_mempool_ops_default.c > @@ -127,6 +127,13 @@ rte_mempool_op_populate_helper(struct rte_mempool *m= p, unsigned int flags, > obj =3D va + off; > obj_cb(mp, obj_cb_arg, obj, > (iova =3D=3D RTE_BAD_IOVA) ? RTE_BAD_IOVA : (iova + off)); > +#ifdef RTE_MEMPOOL_INDEX_BASED_LCORE_CACHE > + /* Store pool base value to calculate indices for index-based > + * lcore cache implementation > + */ > + if (i =3D=3D 0) > + mp->pool_base_value =3D obj; This is wrong, the populate may run many times. ;-) I tried bellow patch to run "rte_pktmbuf_pool_create(mbuf_pool_0, 1048575, = 256, 0, 4096, 0)" This is the debug message (also, your patch will make DPDK mempool not supp= ort > 4GB): 2bfffdb40 (from last debug line 'max') - 1b3fff240 (from first line 'base a= ddr') =3D 10BFFE900 ****mempool mbuf_pool_0 (size =3D 1048575, populated_size =3D 46952, elt_si= ze =3D 4224): base addr =3D 0x1b3fff240, max =3D 0x0, diff =3D 184467440663= 94688960 (max_objs =3D 1048575) ****mempool mbuf_pool_0 (size =3D 1048575, populated_size =3D 297358, elt_s= ize =3D 4224): base addr =3D 0x1c0000040, max =3D 0x0, diff =3D 18446744066= 193358784 (max_objs =3D 1001623) ****mempool mbuf_pool_0 (size =3D 1048575, populated_size =3D 547764, elt_s= ize =3D 4224): base addr =3D 0x200000040, max =3D 0x0, diff =3D 18446744065= 119616960 (max_objs =3D 751217) ****mempool mbuf_pool_0 (size =3D 1048575, populated_size =3D 798170, elt_s= ize =3D 4224): base addr =3D 0x240000040, max =3D 0x0, diff =3D 18446744064= 045875136 (max_objs =3D 500811) ****mempool mbuf_pool_0 (size =3D 1048575, populated_size =3D 1048575, elt_= size =3D 4224): base addr =3D 0x280000040, max =3D 0x2bfffdb40, diff =3D 10= 73732352 (max_objs =3D 250405) diff --git a/lib/mempool/rte_mempool_ops_default.c b/lib/mempool/rte_mempoo= l_ops_default.c index 22fccf9d76..854067cd43 100644 --- a/lib/mempool/rte_mempool_ops_default.c +++ b/lib/mempool/rte_mempool_ops_default.c @@ -99,6 +99,7 @@ rte_mempool_op_populate_helper(struct rte_mempool *mp, un= signed int flags, unsigned int i; void *obj; int ret; + void *pool_base_value =3D NULL, *pool_max_value =3D NULL; ret =3D rte_mempool_get_page_size(mp, &pg_sz); if (ret < 0) @@ -128,9 +129,20 @@ rte_mempool_op_populate_helper(struct rte_mempool *mp,= unsigned int flags, obj_cb(mp, obj_cb_arg, obj, (iova =3D=3D RTE_BAD_IOVA) ? RTE_BAD_IOVA : (iova + = off)); rte_mempool_ops_enqueue_bulk(mp, &obj, 1); + if (i =3D=3D 0) + pool_base_value =3D obj; + else if (i =3D=3D (max_objs - 1)) + pool_max_value =3D obj; off +=3D mp->elt_size + mp->trailer_size; } + printf("****mempool %s (size =3D %u, populated_size =3D %u, elt_siz= e =3D %u): base addr =3D 0x%llx, max =3D 0x%llx, diff =3D %lu (max_objs =3D= %u)\n", + mp->name, mp->size, mp->populated_size, + mp->elt_size, + (unsigned long long) pool_base_value, + (unsigned long long) pool_max_value, + RTE_PTR_DIFF(pool_max_value, pool_base_value), max_objs); + return i; } > +#endif > rte_mempool_ops_enqueue_bulk(mp, &obj, 1); > off +=3D mp->elt_size + mp->trailer_size; > } > -- > 2.17.1