From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F802A04AC; Mon, 31 Aug 2020 22:48:19 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 663D51C0BD; Mon, 31 Aug 2020 22:48:18 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id BE7C01BE9D for ; Mon, 31 Aug 2020 22:48:16 +0200 (CEST) IronPort-SDR: VG7mqmedQMHMH7aWcp6tMl0+MqX8kstMTr06nkVA7E2s/j2WpMQ0x5DworeJq7bTddgUimBxsb fmqbK3cRS0mA== X-IronPort-AV: E=McAfee;i="6000,8403,9730"; a="144803719" X-IronPort-AV: E=Sophos;i="5.76,376,1592895600"; d="scan'208";a="144803719" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Aug 2020 13:48:14 -0700 IronPort-SDR: ViCUtiDAqit25r2nRU7dGgIvgsRwjmFBGySbShM3kTkFoOlz1D5CIXUWOh7ppiX7fziD3QbPV3 57AGJf69lzIA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,376,1592895600"; d="scan'208";a="404560653" Received: from fmsmsx605.amr.corp.intel.com ([10.18.126.85]) by fmsmga001.fm.intel.com with ESMTP; 31 Aug 2020 13:48:14 -0700 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 31 Aug 2020 13:47:21 -0700 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 31 Aug 2020 13:47:20 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 31 Aug 2020 13:47:20 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.170) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 31 Aug 2020 13:47:18 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kM3ZytfPINfpRfPVu/ueGGP40z39ydzBIIwl0NTwU6ZkX+UCDUWHYWnxWxCdzas5vViSqBFrOOmJl3Thvxz8cCtcvrH2uPLWzFIFtvF5+Bv98TWQ4nMjna5k4m199skSeVBKKlOVt220uYKu4dhuLXIEbQ9bpYKlCkKPXP6zB3hF06rNwFkEv74pBrlZlVq6ATp3o6wVjLYhtddIG5aOtlv6CysfwmaZwos35m0FmkS0vi5X2r+cseP5UGbk6NUA5bt50RV4n3C/SnVlK+B/bmpvYZbKXA3emzx4nIMyKHKCLWAHVUVkFL3T/EXDLuvyvwI7zJtS/hwJ5wf8k71pVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K5VHFhUdPOaY62yajZP0dXLPUpxlD8RKIJjKZYCn89M=; b=MN4TBCETQmeIa74joy75iG0f+P8mEy3j6pGTMPYFx6r0OPF/Ht04vIUYCpqeGcZNfSJpg35qz/SSZH8OlDvfziI36QyMP/UElMv8Iw+hFBkef3w2GRahHo5Hwu8fKED1JyJK+Vp8ErDYgC0FVTGh2sx75qhBGV1SXlW48f1SZFc3akj226L5nsOJ4lfK2SjjYGKq9yN5lc6xhCdoKA/mCe8cBfaBpOh9xDzfTAiPpZa1H+D2TJ1WaviKlp/ATFwa+pB/VrXGxQrY+FPpwPPjAevck4o+c08SxJycIaVEkaXtCRDUU8RXeHG3zbl+O4TWO9+e9f0ukNZ/kvwFIN6uTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K5VHFhUdPOaY62yajZP0dXLPUpxlD8RKIJjKZYCn89M=; b=P52efuDXUXZH+JIihtLD5n0MjdAPZt+ze62MS4HqP1gBS5JG1gWasSEP6CAQ+TQTsavP5Epwz+V/zcucl98R/bq0bEJUr/HJQXEykckY8hyW7XmCJWCyLm13kWMXZFb1IWV6/sDw/IDkI+484voj70i7pKZN5xHXgWRoM1KHpZ4= Received: from BYAPR11MB3494.namprd11.prod.outlook.com (2603:10b6:a03:86::15) by BY5PR11MB4194.namprd11.prod.outlook.com (2603:10b6:a03:1c0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.19; Mon, 31 Aug 2020 20:47:14 +0000 Received: from BYAPR11MB3494.namprd11.prod.outlook.com ([fe80::5591:b77c:6dfa:5159]) by BYAPR11MB3494.namprd11.prod.outlook.com ([fe80::5591:b77c:6dfa:5159%7]) with mapi id 15.20.3326.025; Mon, 31 Aug 2020 20:47:14 +0000 From: "Wang, Yipeng1" To: Dharmik Thakkar , "Gobriel, Sameh" , "Richardson, Bruce" , Ray Kinsella , Neil Horman CC: "dev@dpdk.org" , "nd@arm.com" Thread-Topic: [RFC v2] lib/hash: integrate RCU QSBR Thread-Index: AQHWdd4Z4JdWdKWKX0Kf7HNYFRG4calSqIZQ Date: Mon, 31 Aug 2020 20:47:14 +0000 Message-ID: References: <20190901065810.15137-1-dharmik.thakkar@arm.com> <20200819040537.1792-1-dharmik.thakkar@arm.com> In-Reply-To: <20200819040537.1792-1-dharmik.thakkar@arm.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.5.1.3 dlp-product: dlpe-windows authentication-results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [108.161.24.24] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f557dc92-c726-48c7-83d5-08d84def0af2 x-ms-traffictypediagnostic: BY5PR11MB4194: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: WKHz8zd6wCKKzD+8u7/4N88rSfnnbCM0/SRlm6ej3mc0PqMZry9y7N65v/FsMg/LAqN0QdcI4zf0Go8wuf3NYBaX+LpAxKz2CQSf3eZiEX0fhp+o3YdGUvM9x+PlfFz0YsvPJpMFOiruA8ZQNZ1XqwXxsT92gfGRIHGSICjITaLAOL2eJ3K+DPp8xDyn0UwmdLcJq9Pqp22gM3m40ZZDtNr97RwxdfBNdLKa6KJT5CjEIT6U4k5NrltXOQKqcg0ZrXarscU9Cxq2NhvXJcTezyy3R2ugoiBkTX5uLjX84ZIzm66frvgdehACAy7NPltkyexvCXTwbtyYRSx/e2RNX+35wF362o+6P2OW+dmOnVvrMjz42JKuXGYrllpNcRwvfb/W5E10xS6YmjhkwTFU88ZLdZF75rwqikVcx4apMgP5ao79TQMC+8V5hooqEHjZ0MttR0C1i2VuIaG5+IESNA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3494.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(39860400002)(136003)(396003)(376002)(66556008)(33656002)(6506007)(26005)(53546011)(966005)(71200400001)(76116006)(110136005)(54906003)(9686003)(7696005)(5660300002)(83380400001)(2906002)(52536014)(8936002)(478600001)(66946007)(66476007)(55016002)(186003)(64756008)(316002)(4326008)(66446008)(8676002)(86362001)(21314003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: 1AUrde9ZS9jprhvX42Rawze8Qq7llC9nV1PBBi3nv06EBz/Cw+wqvV9YHBLPyjWxNT+B5n7MOMWu7wEcU073iBG6uI1rJUppmnDPrGBcFIsAV1ii5WhEhS8ipEVeAssvct+y8QgAz3yoIoK/HnyTPLRdjf9/rqVDU4jJUQ6g7ZTujeV/BkTqOIGikoL0q/cxSI7+Z5dpDQEXlQTHPtC4Xvzguf4l7faJFUg9rTHRbFC0N/Vvx/pBOucdGyBpohl63Z2+B96jEiTjAoMevzoNeYDKSCbWD1NnA9vkyIiSi0CXMAWqv+BYngC9Yu5EgM2Hy7D4F23IfgN8njCtLfI1T8ODITeW5/P0rB3qHbyxcUb1nE5uZyuq+Oiym4m5k4kJct3AIKMbCB5kxtzvB2kr+kt2ddO4p4DyxFdfR3V0wOpbg6MCmnO4FO2xOx+diYuUXct2ZNRhCxqn7DbbobJG0H8YEnXtr+u/mEwG/c7HP6n645qkmpVyyiTUwjXLHzJACukF+ijhmB902INMpeWswunie3hdDUhtOy7NINJReYQEtq7LAPflyUHcj8X0ngozys8bDeEJV5TIFkY7NgR8P7ffxok+JDWHQ12hGItcNxahaRFtGryB6LzkB8JU1jGR+/4xcJf4WxakrWOk61AkLQ== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3494.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f557dc92-c726-48c7-83d5-08d84def0af2 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2020 20:47:14.7284 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 8/jTGMVwHpAOqbyNXyaHVoTKQ88PHCCDBiUlPHW9Y9IQyLQNze9jHzpIrO/MqhBf62oX1wvGisdFsNbVI0Y9Xg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4194 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Dharmik Thakkar > Sent: Tuesday, August 18, 2020 9:06 PM > To: Wang, Yipeng1 ; Gobriel, Sameh > ; Richardson, Bruce = ; > Ray Kinsella ; Neil Horman > Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar > > Subject: [RFC v2] lib/hash: integrate RCU QSBR >=20 > Integrate RCU QSBR to make it easier for the applications to use lock fre= e > algorithm. >=20 > Resource reclamation implementation was split from the original series, a= nd > has already been part of RCU library. Rework the series to base hash > integration on RCU reclamation APIs. >=20 > Refer 'Resource reclamation framework for DPDK' available at [1] to > understand various aspects of integrating RCU library into other librarie= s. >=20 > [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html >=20 > Introduce a new API rte_hash_rcu_qsbr_add for application to register a R= CU > variable that hash library will use. >=20 > Suggested-by: Honnappa Nagarahalli > Signed-off-by: Dharmik Thakkar > Reviewed-by: Ruifeng Wang > --- > v2: > - Remove defer queue related functions and use resource reclamation > APIs from the RCU QSBR library instead >=20 > - Remove patch (net/ixgbe: avoid multpile definitions of 'bool') > from the series as it is already accepted >=20 > --- > lib/Makefile | 2 +- > lib/librte_hash/Makefile | 2 +- > lib/librte_hash/meson.build | 1 + > lib/librte_hash/rte_cuckoo_hash.c | 291 +++++++++++++++++++++------ > lib/librte_hash/rte_cuckoo_hash.h | 8 + > lib/librte_hash/rte_hash.h | 75 ++++++- > lib/librte_hash/rte_hash_version.map | 2 +- > 7 files changed, 308 insertions(+), 73 deletions(-) >=20 <......> > +/** HASH RCU QSBR configuration structure. */ struct > +rte_hash_rcu_config { > + struct rte_rcu_qsbr *v; /**< RCU QSBR variable. */ > + enum rte_hash_qsbr_mode mode; > + /**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx > + * '0' for default: create defer queue for reclaim. > + */ > + uint32_t dq_size; > + /**< RCU defer queue size. > + * default: total hash table entries. > + */ > + uint32_t reclaim_thd; /**< Threshold to trigger auto reclaim. */ > + uint32_t reclaim_max; > + /**< Max entries to reclaim in one go. > + * default: RTE_HASH_RCU_DQ_RECLAIM_MAX. > + */ > + void *key_data_ptr; > + /**< Pointer passed to the free function. Typically, this is the > + * pointer to the data structure to which the resource to free > + * (key-data) belongs. This can be NULL. > + */ > + rte_hash_free_key_data free_key_data_func; > + /**< Function to call to free the resource (key-data). */ }; > + [Wang, Yipeng]=20 I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters. Personally, I incline to use variable names that match the existing qsbr pa= rameters better. For example, you could still call reclaim_thd as reclaim_limit. And _max to= be _size. Thus, people who are already familiar with qsbr can match the meanings bett= er. > /** @internal A hash table structure. */ struct rte_hash; >=20 > @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, > const void *key, hash_sig_t > * Thread safety can be enabled by setting flag during > * table creation. > * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or > - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, > + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and > + * internal RCU is NOT enabled, > * the key index returned by rte_hash_add_key_xxx APIs will not be > * freed by this API. rte_hash_free_key_with_position API must be called > * additionally to free the index associated with the key. > @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void > *key); > * Thread safety can be enabled by setting flag during > * table creation. > * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or > - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, > + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and > + * internal RCU is NOT enabled, > * the key index returned by rte_hash_add_key_xxx APIs will not be > * freed by this API. rte_hash_free_key_with_position API must be called > * additionally to free the index associated with the key. > @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash > *h, const int32_t position, > * only be called from one thread by default. Thread safety > * can be enabled by setting flag during table creation. > * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or > - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, > + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and > + * internal RCU is NOT enabled, > * the key index returned by rte_hash_del_key_xxx APIs must be freed > * using this API. This API should be called after all the readers > * have stopped referencing the entry corresponding to this key. > @@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const > void **keys, > */ > int32_t > rte_hash_iterate(const struct rte_hash *h, const void **key, void **data= , > uint32_t *next); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change without prior notice > + * > + * Associate RCU QSBR variable with an Hash object. [Wang, Yipeng] To enable RCU we need to call this func. I think you can be more explicit, e.g. "This API should be called to enable= the RCU support" > + * > + * @param h > + * the hash object to add RCU QSBR > + * @param cfg > + * RCU QSBR configuration > + * @return > + * On success - 0 > + * On error - 1 with error code set in rte_errno. > + * Possible rte_errno codes are: > + * - EINVAL - invalid pointer > + * - EEXIST - already added QSBR > + * - ENOMEM - memory allocation failure > + */ [Wang, Yipeng] Is there any further requirement for when to call this API?= =20 E.g. you could say "this API should be called immediately after rte_hash_cr= eate()" =20 > +__rte_experimental > +int rte_hash_rcu_qsbr_add(struct rte_hash *h, > + struct rte_hash_rcu_config *cfg); > #ifdef __cplusplus > } > #endif > diff --git a/lib/librte_hash/rte_hash_version.map > b/lib/librte_hash/rte_hash_version.map > index c0db81014ff9..c6d73080f478 100644 > --- a/lib/librte_hash/rte_hash_version.map > +++ b/lib/librte_hash/rte_hash_version.map > @@ -36,5 +36,5 @@ EXPERIMENTAL { > rte_hash_lookup_with_hash_bulk; > rte_hash_lookup_with_hash_bulk_data; > rte_hash_max_key_id; > - > + rte_hash_rcu_qsbr_add; > }; > -- > 2.17.1 [Wang, Yipeng]=20 Hi, Dharmik, Thanks for the patch. It generally looks good to me.=20 I guess you will revise documentation and the unit test as well after the R= FC. That is helpful for users to understand how to use hash appropriately with = the RCU lib.