From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7F2E1A0597; Wed, 8 Apr 2020 20:32:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8F88F1C1F9; Wed, 8 Apr 2020 20:32:29 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 4C2C61C1CA for ; Wed, 8 Apr 2020 20:32:27 +0200 (CEST) IronPort-SDR: pO8i/VznVhpUDIXxCLuQfwMkcJJcddXmlKl1W+bRBf1aWCyZ1SWSi4FLWNK3bSzP/b4RGBZDEY PwQejoOACnVA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2020 11:32:24 -0700 IronPort-SDR: oUGRaqg9uy1X6qalRT0p9vLYB/sfIvXZodEPHVcJ5CeD00sfNlB8Jni3UGzg+2jTbZAas7mLN/ u11IquCePUhA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,359,1580803200"; d="scan'208";a="451700648" Received: from silpixa00400072.ir.intel.com ([10.237.222.213]) by fmsmga005.fm.intel.com with ESMTP; 08 Apr 2020 11:32:23 -0700 From: Vladimir Medvedkin To: dev@dpdk.org Cc: yipeng1.wang@intel.com, sameh.gobriel@intel.com, bruce.richardson@intel.com Date: Wed, 8 Apr 2020 19:32:18 +0100 Message-Id: <1586370739-61729-1-git-send-email-vladimir.medvedkin@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1585241243-339118-1-git-send-email-vladimir.medvedkin@intel.com> References: <1585241243-339118-1-git-send-email-vladimir.medvedkin@intel.com> Subject: [dpdk-dev] [PATCH v3 1/2] hash: add hash bulk lookup with hash signatures array X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Implement rte_hash_lookup_with_hash_bulk_data() - lookup function with precomputed hash signatures. Signed-off-by: Vladimir Medvedkin --- lib/librte_hash/rte_cuckoo_hash.c | 309 ++++++++++++++++++++++++----------- lib/librte_hash/rte_hash.h | 55 +++++++ lib/librte_hash/rte_hash_version.map | 2 + 3 files changed, 269 insertions(+), 97 deletions(-) diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index 6af8ca4..a5e6677 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -1711,64 +1711,20 @@ compare_signatures(uint32_t *prim_hash_matches, uint32_t *sec_hash_matches, } } -#define PREFETCH_OFFSET 4 static inline void -__rte_hash_lookup_bulk_l(const struct rte_hash *h, const void **keys, - int32_t num_keys, int32_t *positions, - uint64_t *hit_mask, void *data[]) +__bulk_lookup_l(const struct rte_hash *h, const void **keys, + const struct rte_hash_bucket **primary_bkt, + const struct rte_hash_bucket **secondary_bkt, + uint16_t *sig, int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) { uint64_t hits = 0; int32_t i; int32_t ret; - uint32_t prim_hash[RTE_HASH_LOOKUP_BULK_MAX]; - uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX]; - uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX]; - uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; - const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; - const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; uint32_t prim_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0}; uint32_t sec_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0}; struct rte_hash_bucket *cur_bkt, *next_bkt; - /* Prefetch first keys */ - for (i = 0; i < PREFETCH_OFFSET && i < num_keys; i++) - rte_prefetch0(keys[i]); - - /* - * Prefetch rest of the keys, calculate primary and - * secondary bucket and prefetch them - */ - for (i = 0; i < (num_keys - PREFETCH_OFFSET); i++) { - rte_prefetch0(keys[i + PREFETCH_OFFSET]); - - prim_hash[i] = rte_hash_hash(h, keys[i]); - - sig[i] = get_short_sig(prim_hash[i]); - prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); - sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); - - primary_bkt[i] = &h->buckets[prim_index[i]]; - secondary_bkt[i] = &h->buckets[sec_index[i]]; - - rte_prefetch0(primary_bkt[i]); - rte_prefetch0(secondary_bkt[i]); - } - - /* Calculate and prefetch rest of the buckets */ - for (; i < num_keys; i++) { - prim_hash[i] = rte_hash_hash(h, keys[i]); - - sig[i] = get_short_sig(prim_hash[i]); - prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); - sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); - - primary_bkt[i] = &h->buckets[prim_index[i]]; - secondary_bkt[i] = &h->buckets[sec_index[i]]; - - rte_prefetch0(primary_bkt[i]); - rte_prefetch0(secondary_bkt[i]); - } - __hash_rw_reader_lock(h); /* Compare signatures and prefetch key slot of first hit */ @@ -1903,63 +1859,20 @@ __rte_hash_lookup_bulk_l(const struct rte_hash *h, const void **keys, } static inline void -__rte_hash_lookup_bulk_lf(const struct rte_hash *h, const void **keys, - int32_t num_keys, int32_t *positions, - uint64_t *hit_mask, void *data[]) +__bulk_lookup_lf(const struct rte_hash *h, const void **keys, + const struct rte_hash_bucket **primary_bkt, + const struct rte_hash_bucket **secondary_bkt, + uint16_t *sig, int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) { uint64_t hits = 0; int32_t i; int32_t ret; - uint32_t prim_hash[RTE_HASH_LOOKUP_BULK_MAX]; - uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX]; - uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX]; - uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; - const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; - const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; uint32_t prim_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0}; uint32_t sec_hitmask[RTE_HASH_LOOKUP_BULK_MAX] = {0}; struct rte_hash_bucket *cur_bkt, *next_bkt; uint32_t cnt_b, cnt_a; - /* Prefetch first keys */ - for (i = 0; i < PREFETCH_OFFSET && i < num_keys; i++) - rte_prefetch0(keys[i]); - - /* - * Prefetch rest of the keys, calculate primary and - * secondary bucket and prefetch them - */ - for (i = 0; i < (num_keys - PREFETCH_OFFSET); i++) { - rte_prefetch0(keys[i + PREFETCH_OFFSET]); - - prim_hash[i] = rte_hash_hash(h, keys[i]); - - sig[i] = get_short_sig(prim_hash[i]); - prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); - sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); - - primary_bkt[i] = &h->buckets[prim_index[i]]; - secondary_bkt[i] = &h->buckets[sec_index[i]]; - - rte_prefetch0(primary_bkt[i]); - rte_prefetch0(secondary_bkt[i]); - } - - /* Calculate and prefetch rest of the buckets */ - for (; i < num_keys; i++) { - prim_hash[i] = rte_hash_hash(h, keys[i]); - - sig[i] = get_short_sig(prim_hash[i]); - prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); - sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); - - primary_bkt[i] = &h->buckets[prim_index[i]]; - secondary_bkt[i] = &h->buckets[sec_index[i]]; - - rte_prefetch0(primary_bkt[i]); - rte_prefetch0(secondary_bkt[i]); - } - for (i = 0; i < num_keys; i++) positions[i] = -ENOENT; @@ -2124,6 +2037,91 @@ __rte_hash_lookup_bulk_lf(const struct rte_hash *h, const void **keys, *hit_mask = hits; } +#define PREFETCH_OFFSET 4 +static inline void +__bulk_lookup_prefetching_loop(const struct rte_hash *h, + const void **keys, int32_t num_keys, + uint16_t *sig, + const struct rte_hash_bucket **primary_bkt, + const struct rte_hash_bucket **secondary_bkt) +{ + int32_t i; + uint32_t prim_hash[RTE_HASH_LOOKUP_BULK_MAX]; + uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX]; + uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX]; + + /* Prefetch first keys */ + for (i = 0; i < PREFETCH_OFFSET && i < num_keys; i++) + rte_prefetch0(keys[i]); + + /* + * Prefetch rest of the keys, calculate primary and + * secondary bucket and prefetch them + */ + for (i = 0; i < (num_keys - PREFETCH_OFFSET); i++) { + rte_prefetch0(keys[i + PREFETCH_OFFSET]); + + prim_hash[i] = rte_hash_hash(h, keys[i]); + + sig[i] = get_short_sig(prim_hash[i]); + prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); + sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); + + primary_bkt[i] = &h->buckets[prim_index[i]]; + secondary_bkt[i] = &h->buckets[sec_index[i]]; + + rte_prefetch0(primary_bkt[i]); + rte_prefetch0(secondary_bkt[i]); + } + + /* Calculate and prefetch rest of the buckets */ + for (; i < num_keys; i++) { + prim_hash[i] = rte_hash_hash(h, keys[i]); + + sig[i] = get_short_sig(prim_hash[i]); + prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); + sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); + + primary_bkt[i] = &h->buckets[prim_index[i]]; + secondary_bkt[i] = &h->buckets[sec_index[i]]; + + rte_prefetch0(primary_bkt[i]); + rte_prefetch0(secondary_bkt[i]); + } +} + +static inline void +__rte_hash_lookup_bulk_l(const struct rte_hash *h, const void **keys, + int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) +{ + uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + + __bulk_lookup_prefetching_loop(h, keys, num_keys, sig, + primary_bkt, secondary_bkt); + + __bulk_lookup_l(h, keys, primary_bkt, secondary_bkt, sig, num_keys, + positions, hit_mask, data); +} + +static inline void +__rte_hash_lookup_bulk_lf(const struct rte_hash *h, const void **keys, + int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) +{ + uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + + __bulk_lookup_prefetching_loop(h, keys, num_keys, sig, + primary_bkt, secondary_bkt); + + __bulk_lookup_lf(h, keys, primary_bkt, secondary_bkt, sig, num_keys, + positions, hit_mask, data); +} + static inline void __rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, int32_t num_keys, int32_t *positions, @@ -2165,6 +2163,123 @@ rte_hash_lookup_bulk_data(const struct rte_hash *h, const void **keys, return __builtin_popcountl(*hit_mask); } + +static inline void +__rte_hash_lookup_with_hash_bulk_l(const struct rte_hash *h, + const void **keys, hash_sig_t *prim_hash, + int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) +{ + int32_t i; + uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX]; + uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX]; + uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + + /* + * Prefetch keys, calculate primary and + * secondary bucket and prefetch them + */ + for (i = 0; i < num_keys; i++) { + rte_prefetch0(keys[i]); + + sig[i] = get_short_sig(prim_hash[i]); + prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); + sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); + + primary_bkt[i] = &h->buckets[prim_index[i]]; + secondary_bkt[i] = &h->buckets[sec_index[i]]; + + rte_prefetch0(primary_bkt[i]); + rte_prefetch0(secondary_bkt[i]); + } + + __bulk_lookup_l(h, keys, primary_bkt, secondary_bkt, sig, num_keys, + positions, hit_mask, data); +} + +static inline void +__rte_hash_lookup_with_hash_bulk_lf(const struct rte_hash *h, + const void **keys, hash_sig_t *prim_hash, + int32_t num_keys, int32_t *positions, + uint64_t *hit_mask, void *data[]) +{ + int32_t i; + uint32_t prim_index[RTE_HASH_LOOKUP_BULK_MAX]; + uint32_t sec_index[RTE_HASH_LOOKUP_BULK_MAX]; + uint16_t sig[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *primary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + const struct rte_hash_bucket *secondary_bkt[RTE_HASH_LOOKUP_BULK_MAX]; + + /* + * Prefetch keys, calculate primary and + * secondary bucket and prefetch them + */ + for (i = 0; i < num_keys; i++) { + rte_prefetch0(keys[i]); + + sig[i] = get_short_sig(prim_hash[i]); + prim_index[i] = get_prim_bucket_index(h, prim_hash[i]); + sec_index[i] = get_alt_bucket_index(h, prim_index[i], sig[i]); + + primary_bkt[i] = &h->buckets[prim_index[i]]; + secondary_bkt[i] = &h->buckets[sec_index[i]]; + + rte_prefetch0(primary_bkt[i]); + rte_prefetch0(secondary_bkt[i]); + } + + __bulk_lookup_lf(h, keys, primary_bkt, secondary_bkt, sig, num_keys, + positions, hit_mask, data); +} + +static inline void +__rte_hash_lookup_with_hash_bulk(const struct rte_hash *h, const void **keys, + hash_sig_t *prim_hash, int32_t num_keys, + int32_t *positions, uint64_t *hit_mask, void *data[]) +{ + if (h->readwrite_concur_lf_support) + __rte_hash_lookup_with_hash_bulk_lf(h, keys, prim_hash, + num_keys, positions, hit_mask, data); + else + __rte_hash_lookup_with_hash_bulk_l(h, keys, prim_hash, + num_keys, positions, hit_mask, data); +} + +int +rte_hash_lookup_with_hash_bulk(const struct rte_hash *h, const void **keys, + hash_sig_t *sig, uint32_t num_keys, int32_t *positions) +{ + RETURN_IF_TRUE(((h == NULL) || (keys == NULL) || + (sig == NULL) || (num_keys == 0) || + (num_keys > RTE_HASH_LOOKUP_BULK_MAX) || + (positions == NULL)), -EINVAL); + + __rte_hash_lookup_with_hash_bulk(h, keys, sig, num_keys, + positions, NULL, NULL); + return 0; +} + +int +rte_hash_lookup_with_hash_bulk_data(const struct rte_hash *h, + const void **keys, hash_sig_t *sig, + uint32_t num_keys, uint64_t *hit_mask, void *data[]) +{ + RETURN_IF_TRUE(((h == NULL) || (keys == NULL) || + (sig == NULL) || (num_keys == 0) || + (num_keys > RTE_HASH_LOOKUP_BULK_MAX) || + (hit_mask == NULL)), -EINVAL); + + int32_t positions[num_keys]; + + __rte_hash_lookup_with_hash_bulk(h, keys, sig, num_keys, + positions, hit_mask, data); + + /* Return number of hits */ + return __builtin_popcountl(*hit_mask); +} + int32_t rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next) { diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h index ed0673b..13099d7 100644 --- a/lib/librte_hash/rte_hash.h +++ b/lib/librte_hash/rte_hash.h @@ -519,6 +519,61 @@ rte_hash_lookup_bulk_data(const struct rte_hash *h, const void **keys, uint32_t num_keys, uint64_t *hit_mask, void *data[]); /** + * Find multiple keys in the hash table with precomputed hash value array. + * This operation is multi-thread safe with regarding to other lookup threads. + * Read-write concurrency can be enabled by setting flag during + * table creation. + * + * @param h + * Hash table to look in. + * @param keys + * A pointer to a list of keys to look for. + * @param sig + * A pointer to a list of precomputed hash values for keys. + * @param num_keys + * How many keys are in the keys list (less than RTE_HASH_LOOKUP_BULK_MAX). + * @param positions + * Output containing a list of values, corresponding to the list of keys that + * can be used by the caller as an offset into an array of user data. These + * values are unique for each key, and are the same values that were returned + * when each key was added. If a key in the list was not found, then -ENOENT + * will be the value. + * @return + * -EINVAL if there's an error, otherwise 0. + */ +__rte_experimental +int +rte_hash_lookup_with_hash_bulk(const struct rte_hash *h, const void **keys, + hash_sig_t *sig, uint32_t num_keys, int32_t *positions); + +/** + * Find multiple keys in the hash table with precomputed hash value array. + * This operation is multi-thread safe with regarding to other lookup threads. + * Read-write concurrency can be enabled by setting flag during + * table creation. + * + * @param h + * Hash table to look in. + * @param keys + * A pointer to a list of keys to look for. + * @param sig + * A pointer to a list of precomputed hash values for keys. + * @param num_keys + * How many keys are in the keys list (less than RTE_HASH_LOOKUP_BULK_MAX). + * @param hit_mask + * Output containing a bitmask with all successful lookups. + * @param data + * Output containing array of data returned from all the successful lookups. + * @return + * -EINVAL if there's an error, otherwise number of successful lookups. + */ +__rte_experimental +int +rte_hash_lookup_with_hash_bulk_data(const struct rte_hash *h, + const void **keys, hash_sig_t *sig, + uint32_t num_keys, uint64_t *hit_mask, void *data[]); + +/** * Find multiple keys in the hash table. * This operation is multi-thread safe with regarding to other lookup threads. * Read-write concurrency can be enabled by setting flag during diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map index a8fbbc3..c2a9094 100644 --- a/lib/librte_hash/rte_hash_version.map +++ b/lib/librte_hash/rte_hash_version.map @@ -33,6 +33,8 @@ EXPERIMENTAL { global: rte_hash_free_key_with_position; + rte_hash_lookup_with_hash_bulk; + rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; }; -- 2.7.4