DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [RFC 0/3] RCU integration with hash library
@ 2019-09-01  6:58 Dharmik Thakkar
  2019-09-01  6:58 ` [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool' Dharmik Thakkar
                   ` (4 more replies)
  0 siblings, 5 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2019-09-01  6:58 UTC (permalink / raw)
  Cc: dev, honnappa.nagarahalli, ruifeng.wang, Dharmik Thakkar

This patchset integrates RCU QSBR support with hash library.

Note: This patchset has dependency on another patchset:
      https://patchwork.dpdk.org/cover/57813/

Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Add functional and performance test cases.

Fix 'ixgbe_ethdev.h' to avoid multiple definitions of 'bool'.
(Please note that this fix is temporary. Recommend suggesting better
solution)

Dharmik Thakkar (3):
  net/ixgbe: avoid multpile definitions of 'bool'
  lib/hash: integrate RCU QSBR
  test/hash: add tests for integrated RCU QSBR

 app/test/test_hash_readwrite_lf.c    | 700 ++++++++++++++++++++++++++-
 drivers/net/ixgbe/ixgbe_ethdev.h     |   1 +
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   2 +
 lib/librte_hash/rte_cuckoo_hash.c    | 354 +++++++++++++-
 lib/librte_hash/rte_cuckoo_hash.h    |   3 +
 lib/librte_hash/rte_hash.h           |  38 +-
 lib/librte_hash/rte_hash_version.map |   2 +-
 8 files changed, 1065 insertions(+), 37 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
  2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
@ 2019-09-01  6:58 ` Dharmik Thakkar
  2019-09-05 15:26   ` Stephen Hemminger
  2019-09-01  6:58 ` [dpdk-dev] [RFC 2/3] lib/hash: integrate RCU QSBR Dharmik Thakkar
                   ` (3 subsequent siblings)
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2019-09-01  6:58 UTC (permalink / raw)
  To: Wenzhuo Lu, Konstantin Ananyev
  Cc: dev, honnappa.nagarahalli, ruifeng.wang, Dharmik Thakkar

Compilation issue arises due to multiple definitions of 'bool'
in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included
within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This
is a temporary fix. Recommend suggesting better solution.)

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 6e9ed2e10f3c..dd4d6c022545 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -18,6 +18,7 @@
 #include <rte_flow.h>
 #include <rte_time.h>
 #include <rte_hash.h>
+#undef bool
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_tm_driver.h>
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [RFC 2/3] lib/hash: integrate RCU QSBR
  2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
  2019-09-01  6:58 ` [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool' Dharmik Thakkar
@ 2019-09-01  6:58 ` Dharmik Thakkar
  2019-09-01  6:58 ` [dpdk-dev] [RFC 3/3] test/hash: add tests for integrated " Dharmik Thakkar
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2019-09-01  6:58 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Pablo de Lara
  Cc: dev, honnappa.nagarahalli, ruifeng.wang, Dharmik Thakkar

Integrate RCU QSBR process.
(Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.)

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
---
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   2 +
 lib/librte_hash/rte_cuckoo_hash.c    | 352 +++++++++++++++++++++++++--
 lib/librte_hash/rte_cuckoo_hash.h    |   3 +
 lib/librte_hash/rte_hash.h           |  38 ++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 363 insertions(+), 36 deletions(-)

diff --git a/lib/librte_hash/Makefile b/lib/librte_hash/Makefile
index 5669d83f454f..3edc9c96beaa 100644
--- a/lib/librte_hash/Makefile
+++ b/lib/librte_hash/Makefile
@@ -8,7 +8,7 @@ LIB = librte_hash.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)
-LDLIBS += -lrte_eal -lrte_ring
+LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
 
 EXPORT_MAP := rte_hash_version.map
 
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index ebf70de89014..468668ea2488 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -2,6 +2,7 @@
 # Copyright(c) 2017 Intel Corporation
 
 version = 2
+allow_experimental_apis = true
 headers = files('rte_cmp_arm64.h',
 	'rte_cmp_x86.h',
 	'rte_crc_arm64.h',
@@ -14,6 +15,7 @@ headers = files('rte_cmp_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
 
 # rte ring reset is not yet part of stable API
 allow_experimental_apis = true
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index 87a4c01f2f9e..851c73328c8f 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -28,6 +28,7 @@
 #include <rte_compat.h>
 #include <rte_vect.h>
 #include <rte_tailq.h>
+#include <rte_rcu_qsbr.h>
 
 #include "rte_hash.h"
 #include "rte_cuckoo_hash.h"
@@ -44,6 +45,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_qs_item {
+	uint64_t token;	/**< QSBR token.*/
+	uint64_t index;	/**< key_idx and ext_bkt_idx pair*/
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -121,6 +127,68 @@ get_alt_bucket_index(const struct rte_hash *h,
 	return (cur_bkt_idx ^ sig) & h->bucket_bitmask;
 }
 
+/* Add an item into FIFO.
+ * return: 0 - success
+ */
+static int
+__rte_hash_rcu_qsbr_fifo_push(struct rte_ring *fifo,
+	struct __rte_hash_qs_item *item)
+{
+	if (rte_ring_sp_enqueue(fifo, (void *)(uintptr_t)item->token) != 0) {
+		rte_errno = ENOSPC;
+		return 1;
+	}
+	if (rte_ring_sp_enqueue(fifo, (void *)(uintptr_t)item->index) != 0) {
+		void *obj;
+		/* token needs to be dequeued when index enqueue fails */
+		rte_ring_sc_dequeue(fifo, &obj);
+		rte_errno = ENOSPC;
+		return 1;
+	}
+
+	return 0;
+}
+
+/* Remove item from FIFO.
+ * Used when data observed by rte_ring_peek.
+ */
+static void
+__rte_hash_rcu_qsbr_fifo_pop(struct rte_ring *fifo,
+	struct __rte_hash_qs_item *item)
+{
+	void *obj_token = NULL;
+	void *obj_index = NULL;
+
+	(void)rte_ring_sc_dequeue(fifo, &obj_token);
+	(void)rte_ring_sc_dequeue(fifo, &obj_index);
+
+	if (item) {
+		item->token = (uint64_t)((uintptr_t)obj_token);
+		item->index = (uint32_t)((uintptr_t)obj_index);
+	}
+}
+
+/* RCU clean up
+ * This only cleans up data associated with keys (pdata)
+ * Assumption: No reader is using the hash table
+ */
+static void
+__rte_hash_rcu_qsbr_clean_up(const struct rte_hash *h)
+{
+		struct __rte_hash_qs_item qs_item;
+		uint32_t key_to_free;
+		void *hash_data = NULL;
+		struct rte_hash_key *k = NULL;
+		while (!rte_ring_empty(h->qs_fifo)) {
+			__rte_hash_rcu_qsbr_fifo_pop(h->qs_fifo, &qs_item);
+			key_to_free = (uint32_t)(qs_item.index);
+			k = (struct rte_hash_key *) ((char *)h->key_store +
+				key_to_free * h->key_entry_size);
+			hash_data = k->pdata;
+			h->free_key_data_func(hash_data);
+		}
+}
+
 struct rte_hash *
 rte_hash_create(const struct rte_hash_parameters *params)
 {
@@ -193,11 +261,8 @@ rte_hash_create(const struct rte_hash_parameters *params)
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
 		no_free_on_del = 1;
 
-	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
+	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
-		no_free_on_del = 1;
-	}
 
 	/* Store all keys and leave the first entry as a dummy entry for lookup_bulk */
 	if (use_local_cache)
@@ -484,6 +549,13 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up
+	 * This only cleans up data associated with keys (pdata)
+	 * Assumption: No reader is using the hash table
+	 */
+	if (h->qsv && h->free_key_data_func)
+		__rte_hash_rcu_qsbr_clean_up(h);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -495,6 +567,7 @@ rte_hash_free(struct rte_hash *h)
 	rte_free(h->buckets_ext);
 	rte_free(h->tbl_chng_cnt);
 	rte_free(h->ext_bkt_to_free);
+	rte_ring_free(h->qs_fifo);
 	rte_free(h);
 	rte_free(te);
 }
@@ -567,6 +640,106 @@ __hash_rw_reader_unlock(const struct rte_hash *h)
 		rte_rwlock_read_unlock(h->readwrite_lock);
 }
 
+/* Max number of indexes to reclaim at one time. */
+#define RCU_QSBR_RECLAIM_SIZE	8
+
+/* When RCU QSBR FIFO usage is above 1/(2^RCU_QSBR_RECLAIM_LEVEL),
+ * reclaim will be triggered.
+ */
+#define RCU_QSBR_RECLAIM_LEVEL	3
+
+/* Reclaim some key/ext-bkt indexes based on quiescent state check.
+ * RCU_QSBR_RECLAIM_SIZE groups will be reclaimed at max.
+ * return: 0 - success, -EINVAL - no index reclaimed.
+ */
+static int
+__rte_hash_rcu_qsbr_reclaim_chunk(const struct rte_hash *h, uint32_t *index)
+{
+	struct __rte_hash_qs_item qs_item;
+	void *obj_token;
+	uint32_t cnt = 0;
+	uint32_t key_to_free;
+	uint32_t ext_bkt_to_free;
+	void *hash_data = NULL;
+	struct rte_hash_key *keys, *k;
+	if (h->qsv == NULL)
+		return -ENOSPC;
+
+	unsigned int lcore_id, n_slots;
+	struct lcore_cache *cached_free_slots;
+	keys = h->key_store;
+
+	/* Check reader threads quiescent state and
+	 * reclaim as much as possible.
+	 */
+	while ((cnt < RCU_QSBR_RECLAIM_SIZE) &&
+		(rte_ring_peek(h->qs_fifo, &obj_token) == 0) &&
+		(rte_rcu_qsbr_check(h->qsv, (uint64_t)((uintptr_t)obj_token),
+					false) == 1)) {
+		__rte_hash_rcu_qsbr_fifo_pop(h->qs_fifo, &qs_item);
+
+		key_to_free = (uint32_t)(qs_item.index);
+		ext_bkt_to_free = (uint32_t)(qs_item.index >> 32);
+		k = (struct rte_hash_key *) ((char *)keys +
+					key_to_free * h->key_entry_size);
+		hash_data = k->pdata;
+		h->free_key_data_func(hash_data);
+		if (h->ext_table_support) {
+			if (ext_bkt_to_free)
+				/* Recycle empty ext bkt to free list. */
+				rte_ring_sp_enqueue(h->free_ext_bkts,
+					(void *)(uintptr_t)ext_bkt_to_free);
+		}
+		/* Store key_idx for the new key in index*/
+		if (index != NULL  && *index != EMPTY_SLOT)
+			*index = key_to_free;
+
+		/* Return rest of the key indexes to free slot ring */
+		else if (h->use_local_cache) {
+			lcore_id = rte_lcore_id();
+			cached_free_slots = &h->local_free_slots[lcore_id];
+			/* Cache full, need to free it. */
+			if (cached_free_slots->len == LCORE_CACHE_SIZE) {
+				/* Need to enqueue the free slots in global ring. */
+				n_slots = rte_ring_mp_enqueue_burst(h->free_slots,
+							cached_free_slots->objs,
+							LCORE_CACHE_SIZE, NULL);
+				RETURN_IF_TRUE((n_slots == 0), -EFAULT);
+				cached_free_slots->len -= n_slots;
+			}
+			/* Put index of new free slot in cache. */
+			cached_free_slots->objs[cached_free_slots->len] =
+						(void *)((uintptr_t)key_to_free);
+			cached_free_slots->len++;
+		} else {
+			rte_ring_sp_enqueue(h->free_slots,
+					(void *)((uintptr_t)key_to_free));
+		}
+
+		cnt++;
+	}
+
+	if (cnt)
+		return 0;
+	return -ENOSPC;
+}
+
+/* Trigger reclaim when necessary.
+ * Reclaim happens when RCU QSBR queue usage is over 12.5%.
+ */
+static void
+__rte_hash_rcu_qsbr_try_reclaim(const struct rte_hash *h)
+{
+	if (h->qsv == NULL)
+		return;
+
+	if (rte_ring_count(h->qs_fifo) <
+		(rte_ring_get_capacity(h->qs_fifo) >> RCU_QSBR_RECLAIM_LEVEL))
+		return;
+
+	(void)__rte_hash_rcu_qsbr_reclaim_chunk(h, NULL);
+}
+
 void
 rte_hash_reset(struct rte_hash *h)
 {
@@ -576,6 +749,14 @@ rte_hash_reset(struct rte_hash *h)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU clean up
+	 * This only cleans up data associated with keys (pdata)
+	 * Assumption: No reader is using the hash table
+	 */
+	if (h->qsv && h->free_key_data_func)
+		__rte_hash_rcu_qsbr_clean_up(h);
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -612,6 +793,10 @@ rte_hash_reset(struct rte_hash *h)
 		for (i = 0; i < RTE_MAX_LCORE; i++)
 			h->local_free_slots[i].len = 0;
 	}
+
+	if (h->qs_fifo)
+		rte_ring_reset(h->qs_fifo);
+
 	__hash_rw_writer_unlock(h);
 }
 
@@ -915,6 +1100,19 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline int32_t
+get_free_slot(const struct rte_hash *h)
+{
+	uint32_t key_to_free;
+	__hash_rw_writer_lock(h);
+	if (__rte_hash_rcu_qsbr_reclaim_chunk(h, &key_to_free) < 0) {
+		__hash_rw_writer_unlock(h);
+		return -ENOSPC;
+	}
+	__hash_rw_writer_unlock(h);
+	return key_to_free;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -972,7 +1170,7 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 					cached_free_slots->objs,
 					LCORE_CACHE_SIZE, NULL);
 			if (n_slots == 0) {
-				return -ENOSPC;
+				goto rcu_check;
 			}
 
 			cached_free_slots->len += n_slots;
@@ -982,11 +1180,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 		cached_free_slots->len--;
 		slot_id = cached_free_slots->objs[cached_free_slots->len];
 	} else {
-		if (rte_ring_sc_dequeue(h->free_slots, &slot_id) != 0) {
-			return -ENOSPC;
-		}
+		if (rte_ring_sc_dequeue(h->free_slots, &slot_id) != 0)
+			goto rcu_check;
 	}
+	goto new_slot_found;
 
+rcu_check:
+	ret = get_free_slot(h);
+	if (ret < 0)
+		return -ENOSPC;
+	else
+		slot_id = (void *)((uintptr_t)ret);
+
+new_slot_found:
 	new_k = RTE_PTR_ADD(keys, (uintptr_t)slot_id * h->key_entry_size);
 	new_idx = (uint32_t)((uintptr_t) slot_id);
 	/* The store to application data (by the application) at *data should
@@ -1454,10 +1660,11 @@ search_and_remove(const struct rte_hash *h, const void *key,
 					key_idx * h->key_entry_size);
 			if (rte_hash_cmp_eq(key, k->key, h) == 0) {
 				bkt->sig_current[i] = NULL_SIGNATURE;
-				/* Free the key store index if
-				 * no_free_on_del is disabled.
+				/* Free the key store index if no_free_on_del
+				 * and readwrite_concur_lf_support is disabled.
 				 */
-				if (!h->no_free_on_del)
+				if (!h->readwrite_concur_lf_support &&
+					!h->no_free_on_del)
 					remove_entry(h, bkt, i);
 
 				__atomic_store_n(&bkt->key_idx[i],
@@ -1486,6 +1693,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t ext_bkt_to_free = 0;
+	uint32_t key_to_free = 0;
+	struct __rte_hash_qs_item qs_item;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1520,10 +1730,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1538,9 +1747,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 		prev_bkt->next = NULL;
 		uint32_t index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
-		 * no_free_on_del is disabled.
+		 * readwrite_concur_lf_support is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->readwrite_concur_lf_support) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1548,10 +1757,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
-			rte_ring_sp_enqueue(h->free_ext_bkts, (void *)(uintptr_t)index);
+			if (h->qsv == NULL)
+				h->ext_bkt_to_free[ret] = index;
+			/* Enqueue this index to RCU fifo if using internal RCU
+			 * and no_free_on_del is disabled.
+			 */
+			else if (!h->no_free_on_del)
+				ext_bkt_to_free = index;
+		} else
+			rte_ring_sp_enqueue(h->free_ext_bkts,
+					(void *)(uintptr_t)index);
+	}
+
+return_key:
+	if (h->readwrite_concur_lf_support && h->qsv && !h->no_free_on_del) {
+		qs_item.token = rte_rcu_qsbr_start(h->qsv);
+		/* Key index where key is stored, adding the first dummy index */
+		key_to_free = ret + 1;
+		qs_item.index = ((uint64_t)ext_bkt_to_free << 32) | key_to_free;
+		/* Push into QSBR FIFO. */
+		if (__rte_hash_rcu_qsbr_fifo_push(h->qs_fifo, &qs_item) != 0)
+			RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+
+		/* Speculatively reclaim 'deleted' indexes.
+		 * Help spread the reclaim work load across multiple calls.
+		 */
+		__rte_hash_rcu_qsbr_try_reclaim(h);
 	}
 	__hash_rw_writer_unlock(h);
 	return ret;
@@ -1610,12 +1843,42 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 	/* Out of bounds */
 	if (key_idx >= total_entries)
 		return -EINVAL;
-	if (h->ext_table_support && h->readwrite_concur_lf_support) {
-		uint32_t index = h->ext_bkt_to_free[position];
-		if (index) {
-			/* Recycle empty ext bkt to free list. */
-			rte_ring_sp_enqueue(h->free_ext_bkts, (void *)(uintptr_t)index);
-			h->ext_bkt_to_free[position] = 0;
+	if (h->readwrite_concur_lf_support) {
+		if (!h->qsv) {
+			if (h->ext_table_support) {
+				__hash_rw_writer_lock(h);
+				uint32_t index = h->ext_bkt_to_free[position];
+				if (index) {
+					/* Recycle empty ext bkt to free list. */
+					rte_ring_sp_enqueue(h->free_ext_bkts,
+						(void *)(uintptr_t)index);
+					h->ext_bkt_to_free[position] = 0;
+				}
+				__hash_rw_writer_unlock(h);
+			}
+		} else if (h->no_free_on_del) {
+			/* Push into QSBR FIFO */
+			struct __rte_hash_qs_item qs_item;
+			uint32_t ext_bkt_to_free = 0;
+			uint32_t key_to_free = 0;
+			qs_item.token = rte_rcu_qsbr_start(h->qsv);
+			key_to_free = key_idx;
+			if (h->ext_table_support)
+				ext_bkt_to_free = h->ext_bkt_to_free[position];
+			qs_item.index = ((uint64_t)ext_bkt_to_free << 32) |
+						key_to_free;
+			__hash_rw_writer_lock(h);
+			/* Push into QSBR FIFO. */
+			if (__rte_hash_rcu_qsbr_fifo_push(h->qs_fifo, &qs_item)
+				!= 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+
+			/* Speculatively reclaim 'deleted' indexes.
+			 * Help spread the reclaim work load across multiple calls.
+			 */
+			__rte_hash_rcu_qsbr_try_reclaim(h);
+			__hash_rw_writer_unlock(h);
+			return 0;
 		}
 	}
 
@@ -2225,3 +2488,40 @@ rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32
 	(*next)++;
 	return position - 1;
 }
+
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h, struct rte_rcu_qsbr *v,
+				rte_hash_free_key_data free_key_data_func)
+{
+	uint32_t qs_fifo_size;
+	char rcu_ring_name[RTE_RING_NAMESIZE];
+
+	if ((h == NULL) || (v == NULL)) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->qsv) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	/* round up qs_fifo_size to next power of two that is not less than
+	 * total hash table entries. Will store 'token' and 'index'.
+	 */
+	qs_fifo_size = 2 * rte_align32pow2(h->entries);
+
+	/* Init QSBR reclaiming FIFO. */
+	snprintf(rcu_ring_name, sizeof(rcu_ring_name), "HT_RCU_%s", h->name);
+	h->qs_fifo = rte_ring_create(rcu_ring_name, rte_align32pow2(qs_fifo_size),
+			SOCKET_ID_ANY, 0);
+	if (h->qs_fifo == NULL) {
+		RTE_LOG(ERR, HASH, "Hash QS FIFO memory allocation failed\n");
+		rte_errno = ENOMEM;
+		return 1;
+	}
+	h->qsv = v;
+	h->free_key_data_func = free_key_data_func;
+
+	return 0;
+}
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index fb19bb27dfef..c64494ee8b85 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,9 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	struct rte_rcu_qsbr *qsv;
+	struct rte_ring *qs_fifo;
+	rte_hash_free_key_data free_key_data_func;
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index 0d73370dc483..20db00784d0b 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -44,8 +45,6 @@ extern "C" {
 
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
- * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -69,6 +68,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow appication to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -267,8 +273,7 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * and should only be called from one thread by default.
  * Thread safety can be enabled by setting flag during
  * table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -296,8 +301,7 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * and should only be called from one thread by default.
  * Thread safety can be enabled by setting flag during
  * table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -350,8 +354,7 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * of the key. This operation is not multi-thread safe and should
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -545,6 +548,25 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * Configure the RCU QSBR variable to use in hash structure.
+ *
+ * @param h
+ *   Hash table
+ * @param v
+ *   RCU QSBR variable
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL if invalid pointer
+ *   - EEXIST if already added QSBR
+ *   - ENOMEM if memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h, struct rte_rcu_qsbr *v,
+			rte_hash_free_key_data free_key_data_func);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map
index 734ae28b0408..8c3b21891de0 100644
--- a/lib/librte_hash/rte_hash_version.map
+++ b/lib/librte_hash/rte_hash_version.map
@@ -58,5 +58,5 @@ EXPERIMENTAL {
 	global:
 
 	rte_hash_free_key_with_position;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [RFC 3/3] test/hash: add tests for integrated RCU QSBR
  2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
  2019-09-01  6:58 ` [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool' Dharmik Thakkar
  2019-09-01  6:58 ` [dpdk-dev] [RFC 2/3] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2019-09-01  6:58 ` Dharmik Thakkar
  2019-09-05 21:31 ` [dpdk-dev] [RFC 0/3] RCU integration with hash library Wang, Yipeng1
  2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2019-09-01  6:58 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Pablo de Lara
  Cc: dev, honnappa.nagarahalli, ruifeng.wang, Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
---
 app/test/test_hash_readwrite_lf.c | 694 +++++++++++++++++++++++++++++-
 1 file changed, 693 insertions(+), 1 deletion(-)

diff --git a/app/test/test_hash_readwrite_lf.c b/app/test/test_hash_readwrite_lf.c
index 1f2fba41f1b8..0846ed46ae52 100644
--- a/app/test/test_hash_readwrite_lf.c
+++ b/app/test/test_hash_readwrite_lf.c
@@ -13,6 +13,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_spinlock.h>
+#include <rte_rcu_qsbr.h>
 
 #include "test.h"
 
@@ -58,6 +59,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,12 +86,15 @@ struct {
 
 static rte_atomic64_t gread_cycles;
 static rte_atomic64_t greads;
+static rte_atomic64_t gwrite_cycles;
+static rte_atomic64_t gwrites;
 
 static volatile uint8_t writer_done;
 
 uint16_t enabled_core_ids[RTE_MAX_LCORE];
 
 uint8_t *scanned_bkts;
+uint8_t no_free_on_del;
 
 static inline uint16_t
 get_short_sig(const hash_sig_t hash)
@@ -173,6 +178,9 @@ init_params(int rwc_lf, int use_jhash, int htm, int ext_bkt)
 	if (ext_bkt)
 		hash_params.extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
 
+	if (no_free_on_del)
+		hash_params.extra_flag |= RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL;
+
 	hash_params.name = "tests";
 
 	handle = rte_hash_create(&hash_params);
@@ -1240,6 +1248,666 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static uint32_t *hash_data[TOTAL_ENTRY];
+static struct rte_rcu_qsbr *t;
+#define COUNTER_VALUE 4096
+#define TEST_RCU_MAX_LCORE 128
+static int
+test_rwc_rcu_reader(__attribute__((unused))void *arg)
+{
+	struct rte_rcu_qsbr *temp;
+	unsigned int i;
+	uint32_t lcore_id = rte_lcore_id();
+	uint32_t *pdata;
+	temp = t;
+
+	rte_rcu_qsbr_thread_register(temp, lcore_id);
+	do {
+		rte_rcu_qsbr_thread_online(temp, lcore_id);
+		for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks; i++) {
+			rte_rcu_qsbr_lock(temp, lcore_id);
+			if (rte_hash_lookup_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void **)&pdata) != -ENOENT) {
+				if (pdata != NULL) {
+					pdata[lcore_id] = 0;
+					while (pdata[lcore_id] < COUNTER_VALUE)
+						pdata[lcore_id]++;
+				}
+			}
+			rte_rcu_qsbr_unlock(temp, lcore_id);
+		}
+		/* Update quiescent state counter */
+		rte_rcu_qsbr_quiescent(temp, lcore_id);
+		rte_rcu_qsbr_thread_offline(temp, lcore_id);
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_unregister(temp, lcore_id);
+
+	return 0;
+}
+
+static int
+test_rwc_rcu_multi_writer(void *arg)
+{
+	int32_t pos;
+
+	uint32_t i, offset;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		pos = rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		if (no_free_on_del) {
+			if (rte_hash_free_key_with_position(tbl_rwc_test_param.h,
+								pos) < 0)
+				return -1;
+		}
+		if (pos < 0) {
+			printf("Delete key failed #%u at #%d\n",
+				tbl_rwc_test_param.keys_no_ks[i], i);
+			return -1;
+		}
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	return 0;
+}
+
+uint8_t error_state;
+static void
+free_hash_data(void *pdata)
+{
+	uint32_t i;
+	uint32_t *h_data = pdata;
+	if (h_data == NULL)
+		return;
+	for (i = 0; i < rwc_core_cnt[NUM_TEST - 1]; i++) {
+		if (h_data[i] != COUNTER_VALUE && h_data[i] != 0) {
+			printf("Reader did not complete\n");
+			__atomic_store_n(&error_state, 1, __ATOMIC_RELAXED);
+			return;
+		}
+	}
+	rte_free(pdata);
+	return;
+}
+
+/*
+ * Test integrated RCU with 'free' on hash_delete
+ * and with extended bkt feature disabled
+ */
+static int
+test_hash_rcu_free_on_del(__attribute__((unused))struct rwc_perf
+				*rwc_perf_results, int rwc_lf, int htm,
+				int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 0;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+	printf("\nTest: RCU - free on hash del - ext bkt disabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with NO 'free' on hash_delete
+ * and with extended bkt feature disabled
+ */
+static int
+test_hash_rcu_no_free_on_del(__attribute__((unused))struct rwc_perf
+				*rwc_perf_results, int rwc_lf, int htm,
+				int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 1;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - no free on hash del - ext bkt disabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with 'free' on hash_delete
+ * and with extended bkt feature enabled
+ */
+static int
+test_hash_rcu_ext_bkt_free_on_del(__attribute__((unused))struct rwc_perf
+					*rwc_perf_results, int rwc_lf, int htm,
+					int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 0;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - free on hash del - ext bkt enabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with NO 'free' on hash_delete
+ * and with extended bkt feature enabled
+ */
+static int
+test_hash_rcu_ext_bkt_no_free_on_del(__attribute__((unused))struct rwc_perf
+					*rwc_perf_results, int rwc_lf, int htm,
+					int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 1;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - no free on hash del - ext bkt enabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+static int
+test_multiwriter_perf(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	rte_atomic64_add(&gwrite_cycles, cycles);
+	rte_atomic64_add(&gwrites, tbl_rwc_test_param.single_insert);
+	return 0;
+}
+
+/*
+ * Test RCU perf:
+ * Writer(s) delete and add keys in the table.
+ */
+static int
+test_hash_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	int use_jhash = 0;
+	uint8_t write_type;
+
+	no_free_on_del = 0;
+
+	rte_atomic64_init(&gwrites);
+	rte_atomic64_init(&gwrite_cycles);
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	if (rwc_lf) {
+		uint32_t sz;
+		sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+		t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+		if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+					free_hash_data) < 0) {
+			printf("RCU init in hash failed\n");
+			return -1;
+		}
+		printf("\nTest: Writer perf with integrated RCU\n");
+	} else
+		printf("\nTest: Writer perf without integrated RCU\n");
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 1)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		rte_atomic64_clear(&gwrites);
+		rte_atomic64_clear(&gwrite_cycles);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		if (rwc_lf)
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		for (i = 1; i <= rwc_core_cnt[n]; i++)
+			rte_eal_remote_launch(test_multiwriter_perf,
+					(void *)(uintptr_t)(i - 1),
+						enabled_core_ids[i]);
+
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			rte_atomic64_read(&gwrite_cycles) /
+			rte_atomic64_read(&gwrites);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	if (rwc_lf)
+		rte_free(t);
+	return 0;
+
+err:
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	if (rwc_lf)
+		rte_free(t);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_main(void)
 {
@@ -1272,7 +1940,6 @@ test_hash_readwrite_lf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1299,6 +1966,23 @@ test_hash_readwrite_lf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		ext_bkt = 0;
+		if (test_hash_rcu_free_on_del(&rwc_lf_results, rwc_lf, htm,
+						ext_bkt) < 0)
+			return -1;
+		if (test_hash_rcu_no_free_on_del(&rwc_lf_results, rwc_lf, htm,
+						ext_bkt) < 0)
+			return -1;
+		ext_bkt = 1;
+		if (test_hash_rcu_ext_bkt_free_on_del(&rwc_lf_results, rwc_lf,
+							htm, ext_bkt) < 0)
+			return -1;
+		if (test_hash_rcu_ext_bkt_no_free_on_del(&rwc_lf_results, rwc_lf
+							, htm, ext_bkt) < 0)
+			return -1;
+		if (test_hash_writer_perf(&rwc_lf_results, rwc_lf
+							, htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
@@ -1330,9 +2014,17 @@ test_hash_readwrite_lf_main(void)
 	if (test_hash_multi_add_lookup(&rwc_non_lf_results, rwc_lf, htm,
 							ext_bkt) < 0)
 		return -1;
+	no_free_on_del = 0;
 	if (test_hash_add_ks_lookup_hit_extbkt(&rwc_non_lf_results, rwc_lf,
 						htm, ext_bkt) < 0)
 		return -1;
+	if (test_hash_writer_perf(&rwc_non_lf_results, rwc_lf,
+						htm, ext_bkt) < 0)
+		return -1;
+
+	if (error_state != 0)
+		return -1;
+
 results:
 	printf("\n\t\t\t\t\t\t********** Results summary **********\n\n");
 	int i, j, k;
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
  2019-09-01  6:58 ` [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool' Dharmik Thakkar
@ 2019-09-05 15:26   ` Stephen Hemminger
  2019-09-05 20:10     ` Dharmik Thakkar
  0 siblings, 1 reply; 49+ messages in thread
From: Stephen Hemminger @ 2019-09-05 15:26 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: Wenzhuo Lu, Konstantin Ananyev, dev, honnappa.nagarahalli, ruifeng.wang

On Sun,  1 Sep 2019 06:58:08 +0000
Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:

> Compilation issue arises due to multiple definitions of 'bool'
> in 'ixgbe_ethdev.h'.
> (Please note that this issue showed up when 'rte_rcu_qsbr.h' got included
> within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This
> is a temporary fix. Recommend suggesting better solution.)
> 
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
> index 6e9ed2e10f3c..dd4d6c022545 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> @@ -18,6 +18,7 @@
>  #include <rte_flow.h>
>  #include <rte_time.h>
>  #include <rte_hash.h>
> +#undef bool
>  #include <rte_pci.h>
>  #include <rte_bus_pci.h>
>  #include <rte_tm_driver.h>

Just fix ixgbe_ethdev.h to include stdbool itself?

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
index 6e9ed2e10f3c..cf32b1761acb 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.h
+++ b/drivers/net/ixgbe/ixgbe_ethdev.h
@@ -6,6 +6,7 @@
 #define _IXGBE_ETHDEV_H_
 
 #include <stdint.h>
+#include <stdbool.h>
 
 #include "base/ixgbe_type.h"
 #include "base/ixgbe_dcb.h"

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
  2019-09-05 15:26   ` Stephen Hemminger
@ 2019-09-05 20:10     ` Dharmik Thakkar
  2019-09-05 20:23       ` Stephen Hemminger
  0 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2019-09-05 20:10 UTC (permalink / raw)
  To: Stephen Hemminger
  Cc: Wenzhuo Lu, Konstantin Ananyev, dev, Honnappa Nagarahalli,
	Ruifeng Wang (Arm Technology China),
	nd



> On Sep 5, 2019, at 10:26 AM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Sun,  1 Sep 2019 06:58:08 +0000
> Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:
> 
>> Compilation issue arises due to multiple definitions of 'bool'
>> in 'ixgbe_ethdev.h'.
>> (Please note that this issue showed up when 'rte_rcu_qsbr.h' got included
>> within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This
>> is a temporary fix. Recommend suggesting better solution.)
>> 
>> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> ---
>> drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
>> 1 file changed, 1 insertion(+)
>> 
>> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
>> index 6e9ed2e10f3c..dd4d6c022545 100644
>> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
>> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
>> @@ -18,6 +18,7 @@
>> #include <rte_flow.h>
>> #include <rte_time.h>
>> #include <rte_hash.h>
>> +#undef bool
>> #include <rte_pci.h>
>> #include <rte_bus_pci.h>
>> #include <rte_tm_driver.h>
> 
> Just fix ixgbe_ethdev.h to include stdbool itself?
Conflicts with 'typedef int  bool;’ in ixgbe_osdep.h:86
> 
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
> index 6e9ed2e10f3c..cf32b1761acb 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> @@ -6,6 +6,7 @@
> #define _IXGBE_ETHDEV_H_
> 
> #include <stdint.h>
> +#include <stdbool.h>
> 
> #include "base/ixgbe_type.h"
> #include "base/ixgbe_dcb.h"


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
  2019-09-05 20:10     ` Dharmik Thakkar
@ 2019-09-05 20:23       ` Stephen Hemminger
  0 siblings, 0 replies; 49+ messages in thread
From: Stephen Hemminger @ 2019-09-05 20:23 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: Wenzhuo Lu, Konstantin Ananyev, dev, Honnappa Nagarahalli,
	Ruifeng Wang (Arm Technology China),
	nd

On Thu, 5 Sep 2019 20:10:05 +0000
Dharmik Thakkar <Dharmik.Thakkar@arm.com> wrote:

> > On Sep 5, 2019, at 10:26 AM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > 
> > On Sun,  1 Sep 2019 06:58:08 +0000
> > Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:
> >   
> >> Compilation issue arises due to multiple definitions of 'bool'
> >> in 'ixgbe_ethdev.h'.
> >> (Please note that this issue showed up when 'rte_rcu_qsbr.h' got included
> >> within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This
> >> is a temporary fix. Recommend suggesting better solution.)
> >> 
> >> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> >> ---
> >> drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
> >> 1 file changed, 1 insertion(+)
> >> 
> >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h
> >> index 6e9ed2e10f3c..dd4d6c022545 100644
> >> --- a/drivers/net/ixgbe/ixgbe_ethdev.h
> >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.h
> >> @@ -18,6 +18,7 @@
> >> #include <rte_flow.h>
> >> #include <rte_time.h>
> >> #include <rte_hash.h>
> >> +#undef bool
> >> #include <rte_pci.h>
> >> #include <rte_bus_pci.h>
> >> #include <rte_tm_driver.h>  
> > 
> > Just fix ixgbe_ethdev.h to include stdbool itself?  
> Conflicts with 'typedef int  bool;’ in ixgbe_osdep.h:86

Then fix the ixgbe_osdep.h. It is the problem.

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC 0/3] RCU integration with hash library
  2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
                   ` (2 preceding siblings ...)
  2019-09-01  6:58 ` [dpdk-dev] [RFC 3/3] test/hash: add tests for integrated " Dharmik Thakkar
@ 2019-09-05 21:31 ` Wang, Yipeng1
  2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
  4 siblings, 0 replies; 49+ messages in thread
From: Wang, Yipeng1 @ 2019-09-05 21:31 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: dev, honnappa.nagarahalli, ruifeng.wang, stephen, Ananyev, Konstantin

>-----Original Message-----
>From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Dharmik Thakkar
>Sent: Saturday, August 31, 2019 11:58 PM
>Cc: dev@dpdk.org; honnappa.nagarahalli@arm.com; ruifeng.wang@arm.com; Dharmik Thakkar <dharmik.thakkar@arm.com>
>Subject: [dpdk-dev] [RFC 0/3] RCU integration with hash library
>
>This patchset integrates RCU QSBR support with hash library.
>
>Note: This patchset has dependency on another patchset:
>      https://patchwork.dpdk.org/cover/57813/
>
>Refer to RCU documentation to understand various aspects of
>integrating RCU library into other libraries.
>
>Introduce a new API rte_hash_rcu_qsbr_add for application to
>register a RCU variable that hash library will use.
>
>Add functional and performance test cases.
>
>Fix 'ixgbe_ethdev.h' to avoid multiple definitions of 'bool'.
>(Please note that this fix is temporary. Recommend suggesting better
>solution)
>
>Dharmik Thakkar (3):
>  net/ixgbe: avoid multpile definitions of 'bool'
>  lib/hash: integrate RCU QSBR
>  test/hash: add tests for integrated RCU QSBR
>
> app/test/test_hash_readwrite_lf.c    | 700 ++++++++++++++++++++++++++-
> drivers/net/ixgbe/ixgbe_ethdev.h     |   1 +
> lib/librte_hash/Makefile             |   2 +-
> lib/librte_hash/meson.build          |   2 +
> lib/librte_hash/rte_cuckoo_hash.c    | 354 +++++++++++++-
> lib/librte_hash/rte_cuckoo_hash.h    |   3 +
> lib/librte_hash/rte_hash.h           |  38 +-
> lib/librte_hash/rte_hash_version.map |   2 +-
> 8 files changed, 1065 insertions(+), 37 deletions(-)
>
>--
>2.17.1

[Wang, Yipeng] 
Hi Dharmik, thanks for the code!

I agree with Stephen that many people may not use RCU,
and when they use, they may want some control in the application level.
I can see the benefit to have certain RCU helper functions inside the
libraries when people indeed uses QSBR as you designed, 
but I am not sure if the benefits justify the effort to put it in many DPDK libraries.
It adds new APIs, requires future maintenance, and so on, given the
feature can be totally realized by the user application with small cost I assume.

I believe your plan is to add similar helper functions to many other libraries. I think
we could consider them as a single proposal, and it makes sense to bring it up to
the technical committee for a broader discussion.





^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
                   ` (3 preceding siblings ...)
  2019-09-05 21:31 ` [dpdk-dev] [RFC 0/3] RCU integration with hash library Wang, Yipeng1
@ 2020-08-19  4:05 ` Dharmik Thakkar
  2020-08-25 19:59   ` Dharmik Thakkar
                     ` (2 more replies)
  4 siblings, 3 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-08-19  4:05 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd, Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock
free algorithm.

Resource reclamation implementation was split from the original
series, and has already been part of RCU library. Rework the series
to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1]
to understand various aspects of integrating RCU library
into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
v2:
 - Remove defer queue related functions and use resource reclamation
   APIs from the RCU QSBR library instead

 - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
   from the series as it is already accepted

---
 lib/Makefile                         |   2 +-
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  75 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 7 files changed, 308 insertions(+), 73 deletions(-)

diff --git a/lib/Makefile b/lib/Makefile
index 8f5b68a2d469..dffe31c829f0 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost
 DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \
 			librte_net librte_hash librte_cryptodev
 DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash
-DEPDIRS-librte_hash := librte_eal librte_ring
+DEPDIRS-librte_hash := librte_eal librte_ring librte_rcu
 DIRS-$(CONFIG_RTE_LIBRTE_EFD) += librte_efd
 DEPDIRS-librte_efd := librte_eal librte_ring librte_hash
 DIRS-$(CONFIG_RTE_LIBRTE_RIB) += librte_rib
diff --git a/lib/librte_hash/Makefile b/lib/librte_hash/Makefile
index ec9f86499262..10e697f48652 100644
--- a/lib/librte_hash/Makefile
+++ b/lib/librte_hash/Makefile
@@ -8,7 +8,7 @@ LIB = librte_hash.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)
-LDLIBS += -lrte_eal -lrte_ring
+LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
 
 EXPORT_MAP := rte_hash_version.map
 
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index 6ab46ae9d768..0977a63fd279 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index 0a6d474713a2..01c2cbe0e38b 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -612,6 +624,16 @@ rte_hash_reset(struct rte_hash *h)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU clean up. */
+	if (h->hash_rcu_cfg->v) {
+		rte_rcu_qsbr_dq_delete(h->dq);
+		h->dq = NULL;
+		if (rte_hash_rcu_qsbr_add(h, h->hash_rcu_cfg) < 0) {
+			RTE_LOG(ERR, HASH, "RCU init failed\n");
+			return;
+		}
+	}
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1053,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->hash_rcu_cfg->v) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+						    h->hash_rcu_cfg->reclaim_max,
+						    NULL, NULL, NULL)
+					== 0) {
+				slot_id = alloc_slot(h, cached_free_slots);
 			}
-
-			cached_free_slots->len += n_slots;
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1161,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->hash_rcu_cfg->v) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+						    h->hash_rcu_cfg->reclaim_max,
+						    NULL, NULL, NULL)
+					== 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
 	struct lcore_cache *cached_free_slots;
-
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1466,122 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->reclaim_thd;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = cfg->dq_size;
+	hash_rcu_cfg->reclaim_thd = cfg->reclaim_thd;
+	hash_rcu_cfg->reclaim_max = cfg->reclaim_max;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1680,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1716,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1731,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1743,32 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg->v == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg->v) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+			/* Wait for quiescent state change. */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_DQ)
+			/* Push into QSBR FIFO. */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1817,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1834,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 
 static inline void
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index 345de6bf9cfd..85be49d3bbe7 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index bff40251bc98..5431bcf4aeb1 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
+	uint32_t reclaim_max;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map
index c0db81014ff9..c6d73080f478 100644
--- a/lib/librte_hash/rte_hash_version.map
+++ b/lib/librte_hash/rte_hash_version.map
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2020-08-25 19:59   ` Dharmik Thakkar
  2020-08-31 20:47   ` Wang, Yipeng1
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
  2 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-08-25 19:59 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dpdk-dev, nd

CI has reported some compilation issues for this patch. I will resolve these issues once the RFC patch is approved.

Thank you!

> On Aug 18, 2020, at 11:05 PM, Dharmik Thakkar <Dharmik.Thakkar@arm.com> wrote:
> 
> Integrate RCU QSBR to make it easier for the applications to use lock
> free algorithm.
> 
> Resource reclamation implementation was split from the original
> series, and has already been part of RCU library. Rework the series
> to base hash integration on RCU reclamation APIs.
> 
> Refer 'Resource reclamation framework for DPDK' available at [1]
> to understand various aspects of integrating RCU library
> into other libraries.
> 
> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
> 
> Introduce a new API rte_hash_rcu_qsbr_add for application to
> register a RCU variable that hash library will use.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---
> v2:
> - Remove defer queue related functions and use resource reclamation
>   APIs from the RCU QSBR library instead
> 
> - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
>   from the series as it is already accepted
> 
> ---
> lib/Makefile                         |   2 +-
> lib/librte_hash/Makefile             |   2 +-
> lib/librte_hash/meson.build          |   1 +
> lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
> lib/librte_hash/rte_cuckoo_hash.h    |   8 +
> lib/librte_hash/rte_hash.h           |  75 ++++++-
> lib/librte_hash/rte_hash_version.map |   2 +-
> 7 files changed, 308 insertions(+), 73 deletions(-)
> 
> diff --git a/lib/Makefile b/lib/Makefile
> index 8f5b68a2d469..dffe31c829f0 100644
> --- a/lib/Makefile
> +++ b/lib/Makefile
> @@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost
> DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \
> 			librte_net librte_hash librte_cryptodev
> DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash
> -DEPDIRS-librte_hash := librte_eal librte_ring
> +DEPDIRS-librte_hash := librte_eal librte_ring librte_rcu
> DIRS-$(CONFIG_RTE_LIBRTE_EFD) += librte_efd
> DEPDIRS-librte_efd := librte_eal librte_ring librte_hash
> DIRS-$(CONFIG_RTE_LIBRTE_RIB) += librte_rib
> diff --git a/lib/librte_hash/Makefile b/lib/librte_hash/Makefile
> index ec9f86499262..10e697f48652 100644
> --- a/lib/librte_hash/Makefile
> +++ b/lib/librte_hash/Makefile
> @@ -8,7 +8,7 @@ LIB = librte_hash.a
> 
> CFLAGS += -O3
> CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)
> -LDLIBS += -lrte_eal -lrte_ring
> +LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
> 
> EXPORT_MAP := rte_hash_version.map
> 
> diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
> index 6ab46ae9d768..0977a63fd279 100644
> --- a/lib/librte_hash/meson.build
> +++ b/lib/librte_hash/meson.build
> @@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
> 
> sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
> deps += ['ring']
> +deps += ['rcu']
> diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
> index 0a6d474713a2..01c2cbe0e38b 100644
> --- a/lib/librte_hash/rte_cuckoo_hash.c
> +++ b/lib/librte_hash/rte_cuckoo_hash.c
> @@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
> };
> EAL_REGISTER_TAILQ(rte_hash_tailq)
> 
> +struct __rte_hash_rcu_dq_entry {
> +	uint32_t key_idx;
> +	uint32_t ext_bkt_idx; /**< Extended bkt index */
> +};
> +
> struct rte_hash *
> rte_hash_find_existing(const char *name)
> {
> @@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
> 
> 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
> 		readwrite_concur_lf_support = 1;
> -		/* Enable not freeing internal memory/index on delete */
> +		/* Enable not freeing internal memory/index on delete.
> +		 * If internal RCU is enabled, freeing of internal memory/index
> +		 * is done on delete
> +		 */
> 		no_free_on_del = 1;
> 	}
> 
> @@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
> 
> 	rte_mcfg_tailq_write_unlock();
> 
> +	/* RCU clean up. */
> +	if (h->dq)
> +		rte_rcu_qsbr_dq_delete(h->dq);
> +
> 	if (h->use_local_cache)
> 		rte_free(h->local_free_slots);
> 	if (h->writer_takes_lock)
> @@ -612,6 +624,16 @@ rte_hash_reset(struct rte_hash *h)
> 		return;
> 
> 	__hash_rw_writer_lock(h);
> +
> +	/* RCU clean up. */
> +	if (h->hash_rcu_cfg->v) {
> +		rte_rcu_qsbr_dq_delete(h->dq);
> +		h->dq = NULL;
> +		if (rte_hash_rcu_qsbr_add(h, h->hash_rcu_cfg) < 0) {
> +			RTE_LOG(ERR, HASH, "RCU init failed\n");
> +			return;
> +		}
> +	}
> 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
> 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
> 	*h->tbl_chng_cnt = 0;
> @@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
> 	return -ENOSPC;
> }
> 
> +static inline uint32_t
> +alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
> +{
> +	unsigned int  n_slots;
> +	uint32_t slot_id;
> +	if (h->use_local_cache) {
> +		/* Try to get a free slot from the local cache */
> +		if (cached_free_slots->len == 0) {
> +			/* Need to get another burst of free slots from global ring */
> +			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
> +					cached_free_slots->objs,
> +					sizeof(uint32_t),
> +					LCORE_CACHE_SIZE, NULL);
> +			if (n_slots == 0)
> +				return EMPTY_SLOT;
> +
> +			cached_free_slots->len += n_slots;
> +		}
> +
> +		/* Get a free slot from the local cache */
> +		cached_free_slots->len--;
> +		slot_id = cached_free_slots->objs[cached_free_slots->len];
> +	} else {
> +		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
> +						sizeof(uint32_t)) != 0)
> +			return EMPTY_SLOT;
> +	}
> +
> +	return slot_id;
> +}
> +
> static inline int32_t
> __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
> 						hash_sig_t sig, void *data)
> @@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
> 	uint32_t ext_bkt_id = 0;
> 	uint32_t slot_id;
> 	int ret;
> -	unsigned n_slots;
> 	unsigned lcore_id;
> 	unsigned int i;
> 	struct lcore_cache *cached_free_slots = NULL;
> @@ -1001,28 +1053,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
> 	if (h->use_local_cache) {
> 		lcore_id = rte_lcore_id();
> 		cached_free_slots = &h->local_free_slots[lcore_id];
> -		/* Try to get a free slot from the local cache */
> -		if (cached_free_slots->len == 0) {
> -			/* Need to get another burst of free slots from global ring */
> -			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
> -					cached_free_slots->objs,
> -					sizeof(uint32_t),
> -					LCORE_CACHE_SIZE, NULL);
> -			if (n_slots == 0) {
> -				return -ENOSPC;
> +	}
> +	slot_id = alloc_slot(h, cached_free_slots);
> +	if (slot_id == EMPTY_SLOT) {
> +		if (h->hash_rcu_cfg->v) {
> +			if (rte_rcu_qsbr_dq_reclaim(h->dq,
> +						    h->hash_rcu_cfg->reclaim_max,
> +						    NULL, NULL, NULL)
> +					== 0) {
> +				slot_id = alloc_slot(h, cached_free_slots);
> 			}
> -
> -			cached_free_slots->len += n_slots;
> 		}
> -
> -		/* Get a free slot from the local cache */
> -		cached_free_slots->len--;
> -		slot_id = cached_free_slots->objs[cached_free_slots->len];
> -	} else {
> -		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
> -						sizeof(uint32_t)) != 0) {
> +		if (slot_id == EMPTY_SLOT)
> 			return -ENOSPC;
> -		}
> 	}
> 
> 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
> @@ -1118,8 +1161,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
> 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
> 						sizeof(uint32_t)) != 0 ||
> 					ext_bkt_id == 0) {
> -		ret = -ENOSPC;
> -		goto failure;
> +		if (h->hash_rcu_cfg->v) {
> +			if (rte_rcu_qsbr_dq_reclaim(h->dq,
> +						    h->hash_rcu_cfg->reclaim_max,
> +						    NULL, NULL, NULL)
> +					== 0) {
> +				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
> +							 &ext_bkt_id,
> +							 sizeof(uint32_t));
> +			}
> +		}
> +		if (ext_bkt_id == 0) {
> +			ret = -ENOSPC;
> +			goto failure;
> +		}
> 	}
> 
> 	/* Use the first location of the new bucket */
> @@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
> 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
> }
> 
> -static inline void
> -remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
> +static int
> +free_slot(const struct rte_hash *h, uint32_t slot_id)
> {
> 	unsigned lcore_id, n_slots;
> 	struct lcore_cache *cached_free_slots;
> -
> +	/* Return key indexes to free slot ring */
> 	if (h->use_local_cache) {
> 		lcore_id = rte_lcore_id();
> 		cached_free_slots = &h->local_free_slots[lcore_id];
> @@ -1411,18 +1466,122 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
> 						cached_free_slots->objs,
> 						sizeof(uint32_t),
> 						LCORE_CACHE_SIZE, NULL);
> -			ERR_IF_TRUE((n_slots == 0),
> -				"%s: could not enqueue free slots in global ring\n",
> -				__func__);
> +			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
> 			cached_free_slots->len -= n_slots;
> 		}
> -		/* Put index of new free slot in cache. */
> -		cached_free_slots->objs[cached_free_slots->len] =
> -							bkt->key_idx[i];
> -		cached_free_slots->len++;
> +	}
> +
> +	enqueue_slot_back(h, cached_free_slots, slot_id);
> +	return 0;
> +}
> +
> +static void
> +__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
> +{
> +	void *key_data = NULL;
> +	int ret;
> +	struct rte_hash_key *keys, *k;
> +	struct rte_hash *h = (struct rte_hash *)p;
> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
> +			*((struct __rte_hash_rcu_dq_entry *)e);
> +
> +	RTE_SET_USED(n);
> +	keys = h->key_store;
> +
> +	k = (struct rte_hash_key *) ((char *)keys +
> +				rcu_dq_entry.key_idx * h->key_entry_size);
> +	key_data = k->pdata;
> +	if (h->hash_rcu_cfg->free_key_data_func)
> +		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
> +						    key_data);
> +
> +	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
> +		/* Recycle empty ext bkt to free list. */
> +		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
> +			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
> +
> +	/* Return key indexes to free slot ring */
> +	ret = free_slot(h, rcu_dq_entry.key_idx);
> +	if (ret < 0) {
> +		RTE_LOG(ERR, HASH,
> +			"%s: could not enqueue free slots in global ring\n",
> +				__func__);
> +	}
> +}
> +
> +int
> +rte_hash_rcu_qsbr_add(struct rte_hash *h,
> +				struct rte_hash_rcu_config *cfg)
> +{
> +	struct rte_rcu_qsbr_dq_parameters params = {0};
> +	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
> +	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
> +
> +	const uint32_t total_entries = h->use_local_cache ?
> +		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
> +							: h->entries + 1;
> +
> +	if ((h == NULL) || cfg == NULL) {
> +		rte_errno = EINVAL;
> +		return 1;
> +	}
> +
> +	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
> +	if (hash_rcu_cfg == NULL) {
> +		RTE_LOG(ERR, HASH, "memory allocation failed\n");
> +		return 1;
> +	}
> +
> +	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
> +		/* No other things to do. */
> +	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
> +		/* Init QSBR defer queue. */
> +		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
> +					"HASH_RCU_%s", h->name);
> +		params.name = rcu_dq_name;
> +		params.size = cfg->dq_size;
> +		if (params.size == 0)
> +			params.size = total_entries;
> +		params.trigger_reclaim_limit = cfg->reclaim_thd;
> +		if (params.max_reclaim_size == 0)
> +			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
> +		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
> +		params.free_fn = __hash_rcu_qsbr_free_resource;
> +		params.p = h;
> +		params.v = cfg->v;
> +		h->dq = rte_rcu_qsbr_dq_create(&params);
> +		if (h->dq == NULL) {
> +			rte_free(hash_rcu_cfg);
> +			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
> +			return 1;
> +		}
> 	} else {
> -		rte_ring_sp_enqueue_elem(h->free_slots,
> -				&bkt->key_idx[i], sizeof(uint32_t));
> +		rte_free(hash_rcu_cfg);
> +		rte_errno = EINVAL;
> +		return 1;
> +	}
> +
> +	hash_rcu_cfg->v = cfg->v;
> +	hash_rcu_cfg->mode = cfg->mode;
> +	hash_rcu_cfg->dq_size = cfg->dq_size;
> +	hash_rcu_cfg->reclaim_thd = cfg->reclaim_thd;
> +	hash_rcu_cfg->reclaim_max = cfg->reclaim_max;
> +	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
> +	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
> +
> +	h->hash_rcu_cfg = hash_rcu_cfg;
> +
> +	return 0;
> +}
> +
> +static inline void
> +remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
> +{
> +	int ret = free_slot(h, bkt->key_idx[i]);
> +	if (ret < 0) {
> +		RTE_LOG(ERR, HASH,
> +			"%s: could not enqueue free slots in global ring\n",
> +				__func__);
> 	}
> }
> 
> @@ -1521,6 +1680,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
> 	int pos;
> 	int32_t ret, i;
> 	uint16_t short_sig;
> +	uint32_t index = EMPTY_SLOT;
> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
> 
> 	short_sig = get_short_sig(sig);
> 	prim_bucket_idx = get_prim_bucket_index(h, sig);
> @@ -1555,10 +1716,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
> 
> /* Search last bucket to see if empty to be recycled */
> return_bkt:
> -	if (!last_bkt) {
> -		__hash_rw_writer_unlock(h);
> -		return ret;
> -	}
> +	if (!last_bkt)
> +		goto return_key;
> +
> 	while (last_bkt->next) {
> 		prev_bkt = last_bkt;
> 		last_bkt = last_bkt->next;
> @@ -1571,11 +1731,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
> 	/* found empty bucket and recycle */
> 	if (i == RTE_HASH_BUCKET_ENTRIES) {
> 		prev_bkt->next = NULL;
> -		uint32_t index = last_bkt - h->buckets_ext + 1;
> +		index = last_bkt - h->buckets_ext + 1;
> 		/* Recycle the empty bkt if
> 		 * no_free_on_del is disabled.
> 		 */
> -		if (h->no_free_on_del)
> +		if (h->no_free_on_del) {
> 			/* Store index of an empty ext bkt to be recycled
> 			 * on calling rte_hash_del_xxx APIs.
> 			 * When lock free read-write concurrency is enabled,
> @@ -1583,12 +1743,32 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
> 			 * immediately (as readers might be using it still).
> 			 * Hence freeing of the ext bkt is piggy-backed to
> 			 * freeing of the key index.
> +			 * If using external RCU, store this index in an array.
> 			 */
> -			h->ext_bkt_to_free[ret] = index;
> -		else
> +			if (h->hash_rcu_cfg->v == NULL)
> +				h->ext_bkt_to_free[ret] = index;
> +		} else
> 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
> 							sizeof(uint32_t));
> 	}
> +
> +return_key:
> +	/* Using internal RCU QSBR */
> +	if (h->hash_rcu_cfg->v) {
> +		/* Key index where key is stored, adding the first dummy index */
> +		rcu_dq_entry.key_idx = ret + 1;
> +		rcu_dq_entry.ext_bkt_idx = index;
> +		if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
> +			/* Wait for quiescent state change. */
> +			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
> +						 RTE_QSBR_THRID_INVALID);
> +			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
> +						      &rcu_dq_entry, 1);
> +		} else if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_DQ)
> +			/* Push into QSBR FIFO. */
> +			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
> +				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
> +	}
> 	__hash_rw_writer_unlock(h);
> 	return ret;
> }
> @@ -1637,8 +1817,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
> 
> 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
> 
> -	unsigned int lcore_id, n_slots;
> -	struct lcore_cache *cached_free_slots;
> 	const uint32_t total_entries = h->use_local_cache ?
> 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
> 							: h->entries + 1;
> @@ -1656,28 +1834,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
> 		}
> 	}
> 
> -	if (h->use_local_cache) {
> -		lcore_id = rte_lcore_id();
> -		cached_free_slots = &h->local_free_slots[lcore_id];
> -		/* Cache full, need to free it. */
> -		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
> -			/* Need to enqueue the free slots in global ring. */
> -			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
> -						cached_free_slots->objs,
> -						sizeof(uint32_t),
> -						LCORE_CACHE_SIZE, NULL);
> -			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
> -			cached_free_slots->len -= n_slots;
> -		}
> -		/* Put index of new free slot in cache. */
> -		cached_free_slots->objs[cached_free_slots->len] = key_idx;
> -		cached_free_slots->len++;
> -	} else {
> -		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
> -						sizeof(uint32_t));
> -	}
> +	/* Enqueue slot to cache/ring of free slots. */
> +	return free_slot(h, key_idx);
> 
> -	return 0;
> }
> 
> static inline void
> diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
> index 345de6bf9cfd..85be49d3bbe7 100644
> --- a/lib/librte_hash/rte_cuckoo_hash.h
> +++ b/lib/librte_hash/rte_cuckoo_hash.h
> @@ -168,6 +168,11 @@ struct rte_hash {
> 	struct lcore_cache *local_free_slots;
> 	/**< Local cache per lcore, storing some indexes of the free slots */
> 
> +	/* RCU config */
> +	struct rte_hash_rcu_config *hash_rcu_cfg;
> +	/**< HASH RCU QSBR configuration structure */
> +	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
> +
> 	/* Fields used in lookup */
> 
> 	uint32_t key_len __rte_cache_aligned;
> @@ -230,4 +235,7 @@ struct queue_node {
> 	int prev_slot;               /* Parent(slot) in search path */
> };
> 
> +/** @internal Default RCU defer queue entries to reclaim in one go. */
> +#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
> +
> #endif
> diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
> index bff40251bc98..5431bcf4aeb1 100644
> --- a/lib/librte_hash/rte_hash.h
> +++ b/lib/librte_hash/rte_hash.h
> @@ -15,6 +15,7 @@
> #include <stddef.h>
> 
> #include <rte_compat.h>
> +#include <rte_rcu_qsbr.h>
> 
> #ifdef __cplusplus
> extern "C" {
> @@ -45,7 +46,8 @@ extern "C" {
> /** Flag to disable freeing of key index on hash delete.
>  * Refer to rte_hash_del_xxx APIs for more details.
>  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
> - * is enabled.
> + * is enabled. However, if internal RCU is enabled, freeing of internal
> + * memory/index is done on delete
>  */
> #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
> 
> @@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
> /** Type of function used to compare the hash key. */
> typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
> 
> +/**
> + * Type of function used to free data stored in the key.
> + * Required when using internal RCU to allow application to free key-data once
> + * the key is returned to the the ring of free key-slots.
> + */
> +typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
> +
> /**
>  * Parameters used when creating the hash table.
>  */
> @@ -81,6 +90,39 @@ struct rte_hash_parameters {
> 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
> };
> 
> +/** RCU reclamation modes */
> +enum rte_hash_qsbr_mode {
> +	/** Create defer queue for reclaim. */
> +	RTE_HASH_QSBR_MODE_DQ = 0,
> +	/** Use blocking mode reclaim. No defer queue created. */
> +	RTE_HASH_QSBR_MODE_SYNC
> +};
> +
> +/** HASH RCU QSBR configuration structure. */
> +struct rte_hash_rcu_config {
> +	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
> +	enum rte_hash_qsbr_mode mode;
> +	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
> +	 * '0' for default: create defer queue for reclaim.
> +	 */
> +	uint32_t dq_size;
> +	/**< RCU defer queue size.
> +	 * default: total hash table entries.
> +	 */
> +	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
> +	uint32_t reclaim_max;
> +	/**< Max entries to reclaim in one go.
> +	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
> +	 */
> +	void *key_data_ptr;
> +	/**< Pointer passed to the free function. Typically, this is the
> +	 * pointer to the data structure to which the resource to free
> +	 * (key-data) belongs. This can be NULL.
> +	 */
> +	rte_hash_free_key_data free_key_data_func;
> +	/**< Function to call to free the resource (key-data). */
> +};
> +
> /** @internal A hash table structure. */
> struct rte_hash;
> 
> @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
>  * Thread safety can be enabled by setting flag during
>  * table creation.
>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>  * freed by this API. rte_hash_free_key_with_position API must be called
>  * additionally to free the index associated with the key.
> @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
>  * Thread safety can be enabled by setting flag during
>  * table creation.
>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>  * freed by this API. rte_hash_free_key_with_position API must be called
>  * additionally to free the index associated with the key.
> @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
>  * only be called from one thread by default. Thread safety
>  * can be enabled by setting flag during table creation.
>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>  * the key index returned by rte_hash_del_key_xxx APIs must be freed
>  * using this API. This API should be called after all the readers
>  * have stopped referencing the entry corresponding to this key.
> @@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
>  */
> int32_t
> rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Associate RCU QSBR variable with an Hash object.
> + *
> + * @param h
> + *   the hash object to add RCU QSBR
> + * @param cfg
> + *   RCU QSBR configuration
> + * @return
> + *   On success - 0
> + *   On error - 1 with error code set in rte_errno.
> + *   Possible rte_errno codes are:
> + *   - EINVAL - invalid pointer
> + *   - EEXIST - already added QSBR
> + *   - ENOMEM - memory allocation failure
> + */
> +__rte_experimental
> +int rte_hash_rcu_qsbr_add(struct rte_hash *h,
> +				struct rte_hash_rcu_config *cfg);
> #ifdef __cplusplus
> }
> #endif
> diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map
> index c0db81014ff9..c6d73080f478 100644
> --- a/lib/librte_hash/rte_hash_version.map
> +++ b/lib/librte_hash/rte_hash_version.map
> @@ -36,5 +36,5 @@ EXPERIMENTAL {
> 	rte_hash_lookup_with_hash_bulk;
> 	rte_hash_lookup_with_hash_bulk_data;
> 	rte_hash_max_key_id;
> -
> +	rte_hash_rcu_qsbr_add;
> };
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
  2020-08-25 19:59   ` Dharmik Thakkar
@ 2020-08-31 20:47   ` Wang, Yipeng1
  2020-09-01 22:01     ` Dharmik Thakkar
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
  2 siblings, 1 reply; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-08-31 20:47 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce, Ray Kinsella,
	Neil Horman
  Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Tuesday, August 18, 2020 9:06 PM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [RFC v2] lib/hash: integrate RCU QSBR
> 
> Integrate RCU QSBR to make it easier for the applications to use lock free
> algorithm.
> 
> Resource reclamation implementation was split from the original series, and
> has already been part of RCU library. Rework the series to base hash
> integration on RCU reclamation APIs.
> 
> Refer 'Resource reclamation framework for DPDK' available at [1] to
> understand various aspects of integrating RCU library into other libraries.
> 
> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
> 
> Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU
> variable that hash library will use.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---
> v2:
>  - Remove defer queue related functions and use resource reclamation
>    APIs from the RCU QSBR library instead
> 
>  - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
>    from the series as it is already accepted
> 
> ---
>  lib/Makefile                         |   2 +-
>  lib/librte_hash/Makefile             |   2 +-
>  lib/librte_hash/meson.build          |   1 +
>  lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
>  lib/librte_hash/rte_cuckoo_hash.h    |   8 +
>  lib/librte_hash/rte_hash.h           |  75 ++++++-
>  lib/librte_hash/rte_hash_version.map |   2 +-
>  7 files changed, 308 insertions(+), 73 deletions(-)
> 

<......>


> +/** HASH RCU QSBR configuration structure. */ struct
> +rte_hash_rcu_config {
> +	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
> +	enum rte_hash_qsbr_mode mode;
> +	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
> +	 * '0' for default: create defer queue for reclaim.
> +	 */
> +	uint32_t dq_size;
> +	/**< RCU defer queue size.
> +	 * default: total hash table entries.
> +	 */
> +	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
> +	uint32_t reclaim_max;
> +	/**< Max entries to reclaim in one go.
> +	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
> +	 */
> +	void *key_data_ptr;
> +	/**< Pointer passed to the free function. Typically, this is the
> +	 * pointer to the data structure to which the resource to free
> +	 * (key-data) belongs. This can be NULL.
> +	 */
> +	rte_hash_free_key_data free_key_data_func;
> +	/**< Function to call to free the resource (key-data). */ };
> +
[Wang, Yipeng] 
I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters.
Personally, I incline to use variable names that match the existing qsbr parameters better.
For example, you could still call reclaim_thd as reclaim_limit. And _max to be _size.
Thus, people who are already familiar with qsbr can match the meanings better.


>  /** @internal A hash table structure. */  struct rte_hash;
> 
> @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,
> const void *key, hash_sig_t
>   * Thread safety can be enabled by setting flag during
>   * table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_add_key_xxx APIs will not be
>   * freed by this API. rte_hash_free_key_with_position API must be called
>   * additionally to free the index associated with the key.
> @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void
> *key);
>   * Thread safety can be enabled by setting flag during
>   * table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_add_key_xxx APIs will not be
>   * freed by this API. rte_hash_free_key_with_position API must be called
>   * additionally to free the index associated with the key.
> @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash
> *h, const int32_t position,
>   * only be called from one thread by default. Thread safety
>   * can be enabled by setting flag during table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_del_key_xxx APIs must be freed
>   * using this API. This API should be called after all the readers
>   * have stopped referencing the entry corresponding to this key.
> @@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const
> void **keys,
>   */
>  int32_t
>  rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
> uint32_t *next);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Associate RCU QSBR variable with an Hash object.
[Wang, Yipeng] To enable RCU we need to call this func.
I think you can be more explicit, e.g. "This API should be called to enable the RCU support"

> + *
> + * @param h
> + *   the hash object to add RCU QSBR
> + * @param cfg
> + *   RCU QSBR configuration
> + * @return
> + *   On success - 0
> + *   On error - 1 with error code set in rte_errno.
> + *   Possible rte_errno codes are:
> + *   - EINVAL - invalid pointer
> + *   - EEXIST - already added QSBR
> + *   - ENOMEM - memory allocation failure
> + */
[Wang, Yipeng] Is there any further requirement for when to call this API? 
E.g. you could say "this API should be called immediately after rte_hash_create()"
 
> +__rte_experimental
> +int rte_hash_rcu_qsbr_add(struct rte_hash *h,
> +				struct rte_hash_rcu_config *cfg);
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/librte_hash/rte_hash_version.map
> b/lib/librte_hash/rte_hash_version.map
> index c0db81014ff9..c6d73080f478 100644
> --- a/lib/librte_hash/rte_hash_version.map
> +++ b/lib/librte_hash/rte_hash_version.map
> @@ -36,5 +36,5 @@ EXPERIMENTAL {
>  	rte_hash_lookup_with_hash_bulk;
>  	rte_hash_lookup_with_hash_bulk_data;
>  	rte_hash_max_key_id;
> -
> +	rte_hash_rcu_qsbr_add;
>  };
> --
> 2.17.1
[Wang, Yipeng] 
Hi, Dharmik,
Thanks for the patch. It generally looks good to me. 
I guess you will revise documentation and the unit test as well after the RFC.
That is helpful for users to understand how to use hash appropriately with the RCU lib.



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2020-08-31 20:47   ` Wang, Yipeng1
@ 2020-09-01 22:01     ` Dharmik Thakkar
  2020-10-16 13:53       ` David Marchand
  0 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-09-01 22:01 UTC (permalink / raw)
  To: Wang, Yipeng1
  Cc: Gobriel, Sameh, Richardson, Bruce, Ray Kinsella, Neil Horman, dev, nd

Hi Yipeng,
Thank you for the comments!

> On Aug 31, 2020, at 3:47 PM, Wang, Yipeng1 <yipeng1.wang@intel.com> wrote:
> 
>> -----Original Message-----
>> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Sent: Tuesday, August 18, 2020 9:06 PM
>> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
>> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
>> Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
>> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
>> <dharmik.thakkar@arm.com>
>> Subject: [RFC v2] lib/hash: integrate RCU QSBR
>> 
>> Integrate RCU QSBR to make it easier for the applications to use lock free
>> algorithm.
>> 
>> Resource reclamation implementation was split from the original series, and
>> has already been part of RCU library. Rework the series to base hash
>> integration on RCU reclamation APIs.
>> 
>> Refer 'Resource reclamation framework for DPDK' available at [1] to
>> understand various aspects of integrating RCU library into other libraries.
>> 
>> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>> 
>> Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU
>> variable that hash library will use.
>> 
>> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
>> ---
>> v2:
>> - Remove defer queue related functions and use resource reclamation
>>   APIs from the RCU QSBR library instead
>> 
>> - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
>>   from the series as it is already accepted
>> 
>> ---
>> lib/Makefile                         |   2 +-
>> lib/librte_hash/Makefile             |   2 +-
>> lib/librte_hash/meson.build          |   1 +
>> lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
>> lib/librte_hash/rte_cuckoo_hash.h    |   8 +
>> lib/librte_hash/rte_hash.h           |  75 ++++++-
>> lib/librte_hash/rte_hash_version.map |   2 +-
>> 7 files changed, 308 insertions(+), 73 deletions(-)
>> 
> 
> <......>
> 
> 
>> +/** HASH RCU QSBR configuration structure. */ struct
>> +rte_hash_rcu_config {
>> +	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
>> +	enum rte_hash_qsbr_mode mode;
>> +	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
>> +	 * '0' for default: create defer queue for reclaim.
>> +	 */
>> +	uint32_t dq_size;
>> +	/**< RCU defer queue size.
>> +	 * default: total hash table entries.
>> +	 */
>> +	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
>> +	uint32_t reclaim_max;
>> +	/**< Max entries to reclaim in one go.
>> +	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
>> +	 */
>> +	void *key_data_ptr;
>> +	/**< Pointer passed to the free function. Typically, this is the
>> +	 * pointer to the data structure to which the resource to free
>> +	 * (key-data) belongs. This can be NULL.
>> +	 */
>> +	rte_hash_free_key_data free_key_data_func;
>> +	/**< Function to call to free the resource (key-data). */ };
>> +
> [Wang, Yipeng] 
> I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters.
> Personally, I incline to use variable names that match the existing qsbr parameters better.
> For example, you could still call reclaim_thd as reclaim_limit. And _max to be _size.
> Thus, people who are already familiar with qsbr can match the meanings better.
> 

Makes sense. I will update it.

> 
>> /** @internal A hash table structure. */  struct rte_hash;
>> 
>> @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,
>> const void *key, hash_sig_t
>>  * Thread safety can be enabled by setting flag during
>>  * table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>>  * freed by this API. rte_hash_free_key_with_position API must be called
>>  * additionally to free the index associated with the key.
>> @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void
>> *key);
>>  * Thread safety can be enabled by setting flag during
>>  * table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>>  * freed by this API. rte_hash_free_key_with_position API must be called
>>  * additionally to free the index associated with the key.
>> @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash
>> *h, const int32_t position,
>>  * only be called from one thread by default. Thread safety
>>  * can be enabled by setting flag during table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_del_key_xxx APIs must be freed
>>  * using this API. This API should be called after all the readers
>>  * have stopped referencing the entry corresponding to this key.
>> @@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const
>> void **keys,
>>  */
>> int32_t
>> rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
>> uint32_t *next);
>> +
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice
>> + *
>> + * Associate RCU QSBR variable with an Hash object.
> [Wang, Yipeng] To enable RCU we need to call this func.
> I think you can be more explicit, e.g. "This API should be called to enable the RCU support"
> 

Yes.

>> + *
>> + * @param h
>> + *   the hash object to add RCU QSBR
>> + * @param cfg
>> + *   RCU QSBR configuration
>> + * @return
>> + *   On success - 0
>> + *   On error - 1 with error code set in rte_errno.
>> + *   Possible rte_errno codes are:
>> + *   - EINVAL - invalid pointer
>> + *   - EEXIST - already added QSBR
>> + *   - ENOMEM - memory allocation failure
>> + */
> [Wang, Yipeng] Is there any further requirement for when to call this API? 
> E.g. you could say "this API should be called immediately after rte_hash_create()"
> 

Sure, I will add further guidelines/requirements.

>> +__rte_experimental
>> +int rte_hash_rcu_qsbr_add(struct rte_hash *h,
>> +				struct rte_hash_rcu_config *cfg);
>> #ifdef __cplusplus
>> }
>> #endif
>> diff --git a/lib/librte_hash/rte_hash_version.map
>> b/lib/librte_hash/rte_hash_version.map
>> index c0db81014ff9..c6d73080f478 100644
>> --- a/lib/librte_hash/rte_hash_version.map
>> +++ b/lib/librte_hash/rte_hash_version.map
>> @@ -36,5 +36,5 @@ EXPERIMENTAL {
>> 	rte_hash_lookup_with_hash_bulk;
>> 	rte_hash_lookup_with_hash_bulk_data;
>> 	rte_hash_max_key_id;
>> -
>> +	rte_hash_rcu_qsbr_add;
>> };
>> --
>> 2.17.1
> [Wang, Yipeng] 
> Hi, Dharmik,
> Thanks for the patch. It generally looks good to me. 

That’s great. I will convert it to a patch.

> I guess you will revise documentation and the unit test as well after the RFC.
> That is helpful for users to understand how to use hash appropriately with the RCU lib.

Yes, I will add the documentation and unit test patches.

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2020-09-01 22:01     ` Dharmik Thakkar
@ 2020-10-16 13:53       ` David Marchand
  2020-10-16 15:04         ` Dharmik Thakkar
  0 siblings, 1 reply; 49+ messages in thread
From: David Marchand @ 2020-10-16 13:53 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: Wang, Yipeng1, Gobriel, Sameh, Richardson, Bruce, Ray Kinsella,
	Neil Horman, dev, nd, Honnappa Nagarahalli

Hello Dharmik,

On Wed, Sep 2, 2020 at 12:01 AM Dharmik Thakkar <Dharmik.Thakkar@arm.com> wrote:
> > [Wang, Yipeng]
> > Hi, Dharmik,
> > Thanks for the patch. It generally looks good to me.
>
> That’s great. I will convert it to a patch.
>
> > I guess you will revise documentation and the unit test as well after the RFC.
> > That is helpful for users to understand how to use hash appropriately with the RCU lib.
>
> Yes, I will add the documentation and unit test patches.

I did not see a non RFC patch.
Is this still targeting 20.11?


-- 
David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
  2020-10-16 13:53       ` David Marchand
@ 2020-10-16 15:04         ` Dharmik Thakkar
  0 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-16 15:04 UTC (permalink / raw)
  To: David Marchand
  Cc: Wang, Yipeng1, Gobriel, Sameh, Richardson, Bruce, Ray Kinsella,
	Neil Horman, dev, nd, Honnappa Nagarahalli

Hi David,

> On Oct 16, 2020, at 8:53 AM, David Marchand <david.marchand@redhat.com> wrote:
> 
> Hello Dharmik,
> 
> On Wed, Sep 2, 2020 at 12:01 AM Dharmik Thakkar <Dharmik.Thakkar@arm.com> wrote:
>>> [Wang, Yipeng]
>>> Hi, Dharmik,
>>> Thanks for the patch. It generally looks good to me.
>> 
>> That’s great. I will convert it to a patch.
>> 
>>> I guess you will revise documentation and the unit test as well after the RFC.
>>> That is helpful for users to understand how to use hash appropriately with the RCU lib.
>> 
>> Yes, I will add the documentation and unit test patches.
> 
> I did not see a non RFC patch.
> Is this still targeting 20.11?
> 

Yes, I will send out the patch series today. Sorry, it got delayed!

> 
> -- 
> David Marchand
> 


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH 0/3] hash: integrate RCU QSBR
  2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
  2020-08-25 19:59   ` Dharmik Thakkar
  2020-08-31 20:47   ` Wang, Yipeng1
@ 2020-10-16 17:38   ` Dharmik Thakkar
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 1/3] lib/hash: " Dharmik Thakkar
                       ` (4 more replies)
  2 siblings, 5 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-16 17:38 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock
free algorithm.

Resource reclamation implementation was split from the original
series, and has already been part of RCU library. Rework the series
to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1]
to understand various aspects of integrating RCU library
into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the
integration with RCU.
---
v3:
 - Add documentation
 - Add unit tests

v2:
 - Remove defer queue related functions and use resource reclamation
   APIs from the RCU QSBR library instead

 - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
   from the series as it is already accepted

Dharmik Thakkar (3):
  lib/hash: integrate RCU QSBR
  test/hash: replace rte atomic with C11 atomic APIs
  test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 300 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/rte_hash_version.map   |   2 +-
 8 files changed, 917 insertions(+), 132 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v3 1/3] lib/hash: integrate RCU QSBR
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
@ 2020-10-16 17:38     ` Dharmik Thakkar
  2020-10-19  9:43       ` Kinsella, Ray
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
                       ` (3 subsequent siblings)
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-16 17:38 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd, Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources
when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use 
lock free algorithm.
Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 doc/guides/prog_guide/hash_lib.rst   |  11 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 300 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  77 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 324 insertions(+), 75 deletions(-)

diff --git a/doc/guides/prog_guide/hash_lib.rst b/doc/guides/prog_guide/hash_lib.rst
index d06c7de2ead1..63e183ed1f08 100644
--- a/doc/guides/prog_guide/hash_lib.rst
+++ b/doc/guides/prog_guide/hash_lib.rst
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index 6ab46ae9d768..0977a63fd279 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index aad0c965be5e..30ee5c806029 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
 	struct lcore_cache *cached_free_slots;
-
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using RTE_HASH_QSBR_MODE_DQ */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 
 static inline void
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index 345de6bf9cfd..85be49d3bbe7 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index bff40251bc98..3d28f177f14a 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim. */
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.
+ * This API should be called to enable the integrated RCU QSBR support and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map
index c0db81014ff9..c6d73080f478 100644
--- a/lib/librte_hash/rte_hash_version.map
+++ b/lib/librte_hash/rte_hash_version.map
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v3 2/3] test/hash: replace rte atomic with C11 atomic APIs
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 1/3] lib/hash: " Dharmik Thakkar
@ 2020-10-16 17:38     ` Dharmik Thakkar
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
                       ` (2 subsequent siblings)
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-16 17:38 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)

diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 889799865c7b..328fa5116f65 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v3 3/3] test/hash: add tests for integrated RCU QSBR
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 1/3] lib/hash: " Dharmik Thakkar
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
@ 2020-10-16 17:38     ` Dharmik Thakkar
  2020-10-19 14:48     ` [dpdk-dev] [PATCH 0/3] hash: integrate " David Marchand
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-16 17:38 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++-
 2 files changed, 557 insertions(+), 4 deletions(-)

diff --git a/app/test/test_hash.c b/app/test/test_hash.c
index 990a1815f893..22b47b3e7728 100644
--- a/app/test/test_hash.c
+++ b/app/test/test_hash.c
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 328fa5116f65..3c278ee97c23 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -13,6 +13,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_spinlock.h>
+#include <rte_rcu_qsbr.h>
 
 #include "test.h"
 
@@ -48,6 +49,9 @@
 #define WRITE_EXT_BKT 2
 
 #define NUM_TEST 3
+
+#define QSBR_REPORTING_INTERVAL 1024
+
 static unsigned int rwc_core_cnt[NUM_TEST] = {1, 2, 4};
 
 struct rwc_perf {
@@ -58,6 +62,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,6 +89,8 @@ static struct {
 
 static uint64_t gread_cycles;
 static uint64_t greads;
+static uint64_t gwrite_cycles;
+static uint64_t gwrites;
 
 static volatile uint8_t writer_done;
 
@@ -1044,7 +1051,7 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 /*
  * Test lookup perf for multi-writer:
  * Reader(s) lookup keys present in the table and likely on the shift-path while
- * Writers add keys causing key-shiftsi.
+ * Writers add keys causing key-shifts.
  * Writers are running in parallel, on different data plane cores.
  */
 static int
@@ -1223,6 +1230,163 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static struct rte_rcu_qsbr *rv;
+
+/*
+ * Reader thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	unsigned int i, j;
+	uint32_t num_keys = tbl_rwc_test_param.count_keys_no_ks
+				- QSBR_REPORTING_INTERVAL;
+	uint32_t *keys = tbl_rwc_test_param.keys_no_ks;
+	uint32_t lcore_id = rte_lcore_id();
+	RTE_SET_USED(arg);
+
+	(void)rte_rcu_qsbr_thread_register(rv, lcore_id);
+	rte_rcu_qsbr_thread_online(rv, lcore_id);
+	do {
+		for (i = 0; i < num_keys; i += j) {
+			for (j = 0; j < QSBR_REPORTING_INTERVAL; j++)
+				rte_hash_lookup(tbl_rwc_test_param.h,
+						keys + i + j);
+			/* Update quiescent state counter */
+			rte_rcu_qsbr_quiescent(rv, lcore_id);
+		}
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_offline(rv, lcore_id);
+	(void)rte_rcu_qsbr_thread_unregister(rv, lcore_id);
+
+	return 0;
+}
+
+/*
+ * Writer thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_writer(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	__atomic_fetch_add(&gwrite_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&gwrites, tbl_rwc_test_param.single_insert,
+			   __ATOMIC_RELAXED);
+	return 0;
+}
+
+/*
+ * Writer perf test with RCU QSBR in DQ mode:
+ * Writer(s) delete and add keys in the table.
+ * Readers lookup keys in the hash table
+ */
+static int
+test_hash_rcu_qsbr_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	struct rte_hash_rcu_config rcu_config = {0};
+	uint32_t sz;
+	uint8_t pos_core;
+
+	printf("\nTest: Writer perf with integrated RCU\n");
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	rv = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	rcu_config.v = rv;
+
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, &rcu_config) < 0) {
+		printf("RCU init in hash failed\n");
+		goto err;
+	}
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 3)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		__atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED);
+		__atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		rte_rcu_qsbr_init(rv, RTE_MAX_LCORE);
+
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+
+		/* Launch 2 readers */
+		for (i = 1; i <= 2; i++)
+			rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+					      enabled_core_ids[i]);
+		pos_core = 0;
+		/* Launch writer(s) */
+		for (; i <= rwc_core_cnt[n] + 2; i++) {
+			rte_eal_remote_launch(test_hash_rcu_qsbr_writer,
+				(void *)(uintptr_t)pos_core,
+				enabled_core_ids[i]);
+			pos_core++;
+		}
+
+		/* Wait for writers to complete */
+		for (i = 3; i <= rwc_core_cnt[n] + 2; i++)
+			rte_eal_wait_lcore(enabled_core_ids[i]);
+
+		writer_done = 1;
+
+		/* Wait for readers to complete */
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			__atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) /
+			__atomic_load_n(&gwrites, __ATOMIC_RELAXED);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return 0;
+
+err:
+	writer_done = 1;
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_perf_main(void)
 {
@@ -1235,7 +1399,6 @@ test_hash_readwrite_lf_perf_main(void)
 	int rwc_lf = 0;
 	int htm;
 	int ext_bkt = 0;
-
 	if (rte_lcore_count() < 2) {
 		printf("Not enough cores for hash_readwrite_lf_perf_autotest, expecting at least 2\n");
 		return TEST_SKIPPED;
@@ -1255,7 +1418,6 @@ test_hash_readwrite_lf_perf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1282,6 +1444,9 @@ test_hash_readwrite_lf_perf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		if (test_hash_rcu_qsbr_writer_perf(&rwc_lf_results, rwc_lf,
+						   htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v3 1/3] lib/hash: integrate RCU QSBR
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 1/3] lib/hash: " Dharmik Thakkar
@ 2020-10-19  9:43       ` Kinsella, Ray
  0 siblings, 0 replies; 49+ messages in thread
From: Kinsella, Ray @ 2020-10-19  9:43 UTC (permalink / raw)
  To: Dharmik Thakkar, Yipeng Wang, Sameh Gobriel, Bruce Richardson,
	Neil Horman
  Cc: dev, nd



On 16/10/2020 18:38, Dharmik Thakkar wrote:
> Currently, users have to use external RCU mechanisms to free resources
> when using lock free hash algorithm.
> 
> Integrate RCU QSBR process to make it easier for the applications to use 
> lock free algorithm.
> Refer to RCU documentation to understand various aspects of
> integrating RCU library into other libraries.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---
>  doc/guides/prog_guide/hash_lib.rst   |  11 +-
>  lib/librte_hash/meson.build          |   1 +
>  lib/librte_hash/rte_cuckoo_hash.c    | 300 +++++++++++++++++++++------
>  lib/librte_hash/rte_cuckoo_hash.h    |   8 +
>  lib/librte_hash/rte_hash.h           |  77 ++++++-
>  lib/librte_hash/rte_hash_version.map |   2 +-
>  6 files changed, 324 insertions(+), 75 deletions(-)
> 

Acked-by: Ray Kinsella <mdr@ashroe.eu>

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH 0/3] hash: integrate RCU QSBR
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
                       ` (2 preceding siblings ...)
  2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-19 14:48     ` David Marchand
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
  4 siblings, 0 replies; 49+ messages in thread
From: David Marchand @ 2020-10-19 14:48 UTC (permalink / raw)
  To: Dharmik Thakkar; +Cc: dev, nd, Honnappa Nagarahalli

On Fri, Oct 16, 2020 at 7:39 PM Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:
>
> Integrate RCU QSBR to make it easier for the applications to use lock
> free algorithm.
>
> Resource reclamation implementation was split from the original
> series, and has already been part of RCU library. Rework the series
> to base hash integration on RCU reclamation APIs.
>
> Refer 'Resource reclamation framework for DPDK' available at [1]
> to understand various aspects of integrating RCU library
> into other libraries.
>
> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>
> Introduce a new API rte_hash_rcu_qsbr_add for application to
> register a RCU variable that hash library will use.
>
> Functional tests and performance tests are added to cover the
> integration with RCU.

clang builds are broken.

http://mails.dpdk.org/archives/test-report/2020-October/160043.html
https://travis-ci.com/github/ovsrobot/dpdk/builds/190521500


-- 
David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR
  2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
                       ` (3 preceding siblings ...)
  2020-10-19 14:48     ` [dpdk-dev] [PATCH 0/3] hash: integrate " David Marchand
@ 2020-10-19 16:35     ` Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 1/3] lib/hash: " Dharmik Thakkar
                         ` (4 more replies)
  4 siblings, 5 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-19 16:35 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock
free algorithm.

Resource reclamation implementation was split from the original
series, and has already been part of RCU library. Rework the series
to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1]
to understand various aspects of integrating RCU library
into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the
integration with RCU.
---
v4:
 - Fix clang compilation issues

v3:
 - Add documentation
 - Add unit tests

v2:
 - Remove defer queue related functions and use resource reclamation
   APIs from the RCU QSBR library instead

 - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
   from the series as it is already accepted

Dharmik Thakkar (3):
  lib/hash: integrate RCU QSBR
  test/hash: replace rte atomic with C11 atomic APIs
  test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 302 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/rte_hash_version.map   |   2 +-
 8 files changed, 918 insertions(+), 133 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v4 1/3] lib/hash: integrate RCU QSBR
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
@ 2020-10-19 16:35       ` Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
                         ` (3 subsequent siblings)
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-19 16:35 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd, Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources
when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use 
lock free algorithm.
Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
---
 doc/guides/prog_guide/hash_lib.rst   |  11 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 302 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  77 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 325 insertions(+), 76 deletions(-)

diff --git a/doc/guides/prog_guide/hash_lib.rst b/doc/guides/prog_guide/hash_lib.rst
index d06c7de2ead1..63e183ed1f08 100644
--- a/doc/guides/prog_guide/hash_lib.rst
+++ b/doc/guides/prog_guide/hash_lib.rst
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index 6ab46ae9d768..0977a63fd279 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index aad0c965be5e..b9e4d82a0c14 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
-
+	struct lcore_cache *cached_free_slots = NULL;
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using RTE_HASH_QSBR_MODE_DQ */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 
 static inline void
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index 345de6bf9cfd..85be49d3bbe7 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index bff40251bc98..3d28f177f14a 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim. */
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.
+ * This API should be called to enable the integrated RCU QSBR support and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map
index c0db81014ff9..c6d73080f478 100644
--- a/lib/librte_hash/rte_hash_version.map
+++ b/lib/librte_hash/rte_hash_version.map
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v4 2/3] test/hash: replace rte atomic with C11 atomic APIs
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 1/3] lib/hash: " Dharmik Thakkar
@ 2020-10-19 16:35       ` Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
                         ` (2 subsequent siblings)
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-19 16:35 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)

diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 889799865c7b..328fa5116f65 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v4 3/3] test/hash: add tests for integrated RCU QSBR
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 1/3] lib/hash: " Dharmik Thakkar
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
@ 2020-10-19 16:35       ` Dharmik Thakkar
  2020-10-19 21:05       ` [dpdk-dev] [PATCH v4 0/3] hash: integrate " David Marchand
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-19 16:35 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++-
 2 files changed, 557 insertions(+), 4 deletions(-)

diff --git a/app/test/test_hash.c b/app/test/test_hash.c
index 990a1815f893..22b47b3e7728 100644
--- a/app/test/test_hash.c
+++ b/app/test/test_hash.c
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 328fa5116f65..3c278ee97c23 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -13,6 +13,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_spinlock.h>
+#include <rte_rcu_qsbr.h>
 
 #include "test.h"
 
@@ -48,6 +49,9 @@
 #define WRITE_EXT_BKT 2
 
 #define NUM_TEST 3
+
+#define QSBR_REPORTING_INTERVAL 1024
+
 static unsigned int rwc_core_cnt[NUM_TEST] = {1, 2, 4};
 
 struct rwc_perf {
@@ -58,6 +62,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,6 +89,8 @@ static struct {
 
 static uint64_t gread_cycles;
 static uint64_t greads;
+static uint64_t gwrite_cycles;
+static uint64_t gwrites;
 
 static volatile uint8_t writer_done;
 
@@ -1044,7 +1051,7 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 /*
  * Test lookup perf for multi-writer:
  * Reader(s) lookup keys present in the table and likely on the shift-path while
- * Writers add keys causing key-shiftsi.
+ * Writers add keys causing key-shifts.
  * Writers are running in parallel, on different data plane cores.
  */
 static int
@@ -1223,6 +1230,163 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static struct rte_rcu_qsbr *rv;
+
+/*
+ * Reader thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	unsigned int i, j;
+	uint32_t num_keys = tbl_rwc_test_param.count_keys_no_ks
+				- QSBR_REPORTING_INTERVAL;
+	uint32_t *keys = tbl_rwc_test_param.keys_no_ks;
+	uint32_t lcore_id = rte_lcore_id();
+	RTE_SET_USED(arg);
+
+	(void)rte_rcu_qsbr_thread_register(rv, lcore_id);
+	rte_rcu_qsbr_thread_online(rv, lcore_id);
+	do {
+		for (i = 0; i < num_keys; i += j) {
+			for (j = 0; j < QSBR_REPORTING_INTERVAL; j++)
+				rte_hash_lookup(tbl_rwc_test_param.h,
+						keys + i + j);
+			/* Update quiescent state counter */
+			rte_rcu_qsbr_quiescent(rv, lcore_id);
+		}
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_offline(rv, lcore_id);
+	(void)rte_rcu_qsbr_thread_unregister(rv, lcore_id);
+
+	return 0;
+}
+
+/*
+ * Writer thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_writer(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	__atomic_fetch_add(&gwrite_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&gwrites, tbl_rwc_test_param.single_insert,
+			   __ATOMIC_RELAXED);
+	return 0;
+}
+
+/*
+ * Writer perf test with RCU QSBR in DQ mode:
+ * Writer(s) delete and add keys in the table.
+ * Readers lookup keys in the hash table
+ */
+static int
+test_hash_rcu_qsbr_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	struct rte_hash_rcu_config rcu_config = {0};
+	uint32_t sz;
+	uint8_t pos_core;
+
+	printf("\nTest: Writer perf with integrated RCU\n");
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	rv = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	rcu_config.v = rv;
+
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, &rcu_config) < 0) {
+		printf("RCU init in hash failed\n");
+		goto err;
+	}
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 3)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		__atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED);
+		__atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		rte_rcu_qsbr_init(rv, RTE_MAX_LCORE);
+
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+
+		/* Launch 2 readers */
+		for (i = 1; i <= 2; i++)
+			rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+					      enabled_core_ids[i]);
+		pos_core = 0;
+		/* Launch writer(s) */
+		for (; i <= rwc_core_cnt[n] + 2; i++) {
+			rte_eal_remote_launch(test_hash_rcu_qsbr_writer,
+				(void *)(uintptr_t)pos_core,
+				enabled_core_ids[i]);
+			pos_core++;
+		}
+
+		/* Wait for writers to complete */
+		for (i = 3; i <= rwc_core_cnt[n] + 2; i++)
+			rte_eal_wait_lcore(enabled_core_ids[i]);
+
+		writer_done = 1;
+
+		/* Wait for readers to complete */
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			__atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) /
+			__atomic_load_n(&gwrites, __ATOMIC_RELAXED);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return 0;
+
+err:
+	writer_done = 1;
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_perf_main(void)
 {
@@ -1235,7 +1399,6 @@ test_hash_readwrite_lf_perf_main(void)
 	int rwc_lf = 0;
 	int htm;
 	int ext_bkt = 0;
-
 	if (rte_lcore_count() < 2) {
 		printf("Not enough cores for hash_readwrite_lf_perf_autotest, expecting at least 2\n");
 		return TEST_SKIPPED;
@@ -1255,7 +1418,6 @@ test_hash_readwrite_lf_perf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1282,6 +1444,9 @@ test_hash_readwrite_lf_perf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		if (test_hash_rcu_qsbr_writer_perf(&rwc_lf_results, rwc_lf,
+						   htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
                         ` (2 preceding siblings ...)
  2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-19 21:05       ` David Marchand
  2020-10-20  4:08         ` Dharmik Thakkar
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
  4 siblings, 1 reply; 49+ messages in thread
From: David Marchand @ 2020-10-19 21:05 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: dev, nd, Honnappa Nagarahalli, Thomas Monjalon, Dmitry Kozlyuk

On Mon, Oct 19, 2020 at 6:37 PM Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:
>
> Integrate RCU QSBR to make it easier for the applications to use lock
> free algorithm.
>
> Resource reclamation implementation was split from the original
> series, and has already been part of RCU library. Rework the series
> to base hash integration on RCU reclamation APIs.
>
> Refer 'Resource reclamation framework for DPDK' available at [1]
> to understand various aspects of integrating RCU library
> into other libraries.
>
> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>
> Introduce a new API rte_hash_rcu_qsbr_add for application to
> register a RCU variable that hash library will use.
>
> Functional tests and performance tests are added to cover the
> integration with RCU.

This patch breaks Windows build: hash library was building fine so
far, adding a dependency on rcu broke it as rcu is not ready (yet?).


-- 
David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR
  2020-10-19 21:05       ` [dpdk-dev] [PATCH v4 0/3] hash: integrate " David Marchand
@ 2020-10-20  4:08         ` Dharmik Thakkar
  0 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20  4:08 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, nd, Honnappa Nagarahalli, thomas, Dmitry Kozlyuk



> On Oct 19, 2020, at 4:05 PM, David Marchand <david.marchand@redhat.com> wrote:
> 
> On Mon, Oct 19, 2020 at 6:37 PM Dharmik Thakkar <dharmik.thakkar@arm.com> wrote:
>> 
>> Integrate RCU QSBR to make it easier for the applications to use lock
>> free algorithm.
>> 
>> Resource reclamation implementation was split from the original
>> series, and has already been part of RCU library. Rework the series
>> to base hash integration on RCU reclamation APIs.
>> 
>> Refer 'Resource reclamation framework for DPDK' available at [1]
>> to understand various aspects of integrating RCU library
>> into other libraries.
>> 
>> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>> 
>> Introduce a new API rte_hash_rcu_qsbr_add for application to
>> register a RCU variable that hash library will use.
>> 
>> Functional tests and performance tests are added to cover the
>> integration with RCU.
> 
> This patch breaks Windows build: hash library was building fine so
> far, adding a dependency on rcu broke it as rcu is not ready (yet?).
> 

Yes, ‘rcu’ is not currently enabled for Windows. I have submitted a RFC patch [1] to enable ‘rcu’ for Windows.
I will update this patch, once ‘rcu’ support is enabled and the compilation succeeds on Windows.

[1] https://patches.dpdk.org/patch/81405/

> 
> -- 
> David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v5 0/4] hash: integrate RCU QSBR
  2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
                         ` (3 preceding siblings ...)
  2020-10-19 21:05       ` [dpdk-dev] [PATCH v4 0/3] hash: integrate " David Marchand
@ 2020-10-20 16:12       ` Dharmik Thakkar
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 1/4] rcu: build on Windows Dharmik Thakkar
                           ` (4 more replies)
  4 siblings, 5 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20 16:12 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock
free algorithm.

Resource reclamation implementation was split from the original
series, and has already been part of RCU library. Rework the series
to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1]
to understand various aspects of integrating RCU library
into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the
integration with RCU.
---
v5:
 - Enable 'rcu' for Windows

v4:
 - Fix clang compilation issues

v3:
 - Add documentation
 - Add unit tests

v2:
 - Remove defer queue related functions and use resource reclamation
   APIs from the RCU QSBR library instead

 - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
   from the series as it is already accepted

Dharmik Thakkar (4):
  rcu: build on Windows
  lib/hash: integrate RCU QSBR
  test/hash: replace rte atomic with C11 atomic APIs
  test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 302 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/version.map            |   2 +-
 lib/meson.build                        |   1 +
 9 files changed, 918 insertions(+), 133 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v5 1/4] rcu: build on Windows
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
@ 2020-10-20 16:12         ` Dharmik Thakkar
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
                           ` (3 subsequent siblings)
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20 16:12 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Build the lib for Windows.

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Tested-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/meson.build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/meson.build b/lib/meson.build
index dd55b5cb53e4..1bb019720c6a 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -41,6 +41,7 @@ if is_windows
 		'telemetry',
 		'eal',
 		'ring',
+		'rcu',
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 1/4] rcu: build on Windows Dharmik Thakkar
@ 2020-10-20 16:12         ` Dharmik Thakkar
  2020-10-21  2:42           ` Wang, Yipeng1
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
                           ` (2 subsequent siblings)
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20 16:12 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd, Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources
when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use
lock free algorithm.
Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
---
 doc/guides/prog_guide/hash_lib.rst |  11 +-
 lib/librte_hash/meson.build        |   1 +
 lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
 lib/librte_hash/rte_cuckoo_hash.h  |   8 +
 lib/librte_hash/rte_hash.h         |  77 +++++++-
 lib/librte_hash/version.map        |   2 +-
 6 files changed, 325 insertions(+), 76 deletions(-)

diff --git a/doc/guides/prog_guide/hash_lib.rst b/doc/guides/prog_guide/hash_lib.rst
index d06c7de2ead1..63e183ed1f08 100644
--- a/doc/guides/prog_guide/hash_lib.rst
+++ b/doc/guides/prog_guide/hash_lib.rst
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index 6ab46ae9d768..0977a63fd279 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index aad0c965be5e..b9e4d82a0c14 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
-
+	struct lcore_cache *cached_free_slots = NULL;
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using RTE_HASH_QSBR_MODE_DQ */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 
 static inline void
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index 345de6bf9cfd..85be49d3bbe7 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index bff40251bc98..3d28f177f14a 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim. */
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.
+ * This API should be called to enable the integrated RCU QSBR support and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/version.map b/lib/librte_hash/version.map
index c0db81014ff9..c6d73080f478 100644
--- a/lib/librte_hash/version.map
+++ b/lib/librte_hash/version.map
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 1/4] rcu: build on Windows Dharmik Thakkar
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2020-10-20 16:13         ` Dharmik Thakkar
  2020-10-21  2:52           ` Wang, Yipeng1
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20 16:13 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)

diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 889799865c7b..328fa5116f65 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
                           ` (2 preceding siblings ...)
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
@ 2020-10-20 16:13         ` Dharmik Thakkar
  2020-10-21  3:54           ` Wang, Yipeng1
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-20 16:13 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)

diff --git a/app/test/test_hash.c b/app/test/test_hash.c
index 990a1815f893..22b47b3e7728 100644
--- a/app/test/test_hash.c
+++ b/app/test/test_hash.c
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 328fa5116f65..caf18b69c1d4 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -48,6 +48,9 @@
 #define WRITE_EXT_BKT 2
 
 #define NUM_TEST 3
+
+#define QSBR_REPORTING_INTERVAL 1024
+
 static unsigned int rwc_core_cnt[NUM_TEST] = {1, 2, 4};
 
 struct rwc_perf {
@@ -58,6 +61,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,6 +88,8 @@ static struct {
 
 static uint64_t gread_cycles;
 static uint64_t greads;
+static uint64_t gwrite_cycles;
+static uint64_t gwrites;
 
 static volatile uint8_t writer_done;
 
@@ -1044,7 +1050,7 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 /*
  * Test lookup perf for multi-writer:
  * Reader(s) lookup keys present in the table and likely on the shift-path while
- * Writers add keys causing key-shiftsi.
+ * Writers add keys causing key-shifts.
  * Writers are running in parallel, on different data plane cores.
  */
 static int
@@ -1223,6 +1229,163 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static struct rte_rcu_qsbr *rv;
+
+/*
+ * Reader thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	unsigned int i, j;
+	uint32_t num_keys = tbl_rwc_test_param.count_keys_no_ks
+				- QSBR_REPORTING_INTERVAL;
+	uint32_t *keys = tbl_rwc_test_param.keys_no_ks;
+	uint32_t lcore_id = rte_lcore_id();
+	RTE_SET_USED(arg);
+
+	(void)rte_rcu_qsbr_thread_register(rv, lcore_id);
+	rte_rcu_qsbr_thread_online(rv, lcore_id);
+	do {
+		for (i = 0; i < num_keys; i += j) {
+			for (j = 0; j < QSBR_REPORTING_INTERVAL; j++)
+				rte_hash_lookup(tbl_rwc_test_param.h,
+						keys + i + j);
+			/* Update quiescent state counter */
+			rte_rcu_qsbr_quiescent(rv, lcore_id);
+		}
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_offline(rv, lcore_id);
+	(void)rte_rcu_qsbr_thread_unregister(rv, lcore_id);
+
+	return 0;
+}
+
+/*
+ * Writer thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_writer(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	__atomic_fetch_add(&gwrite_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&gwrites, tbl_rwc_test_param.single_insert,
+			   __ATOMIC_RELAXED);
+	return 0;
+}
+
+/*
+ * Writer perf test with RCU QSBR in DQ mode:
+ * Writer(s) delete and add keys in the table.
+ * Readers lookup keys in the hash table
+ */
+static int
+test_hash_rcu_qsbr_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	struct rte_hash_rcu_config rcu_config = {0};
+	uint32_t sz;
+	uint8_t pos_core;
+
+	printf("\nTest: Writer perf with integrated RCU\n");
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	rv = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	rcu_config.v = rv;
+
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, &rcu_config) < 0) {
+		printf("RCU init in hash failed\n");
+		goto err;
+	}
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 3)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		__atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED);
+		__atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		rte_rcu_qsbr_init(rv, RTE_MAX_LCORE);
+
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+
+		/* Launch 2 readers */
+		for (i = 1; i <= 2; i++)
+			rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+					      enabled_core_ids[i]);
+		pos_core = 0;
+		/* Launch writer(s) */
+		for (; i <= rwc_core_cnt[n] + 2; i++) {
+			rte_eal_remote_launch(test_hash_rcu_qsbr_writer,
+				(void *)(uintptr_t)pos_core,
+				enabled_core_ids[i]);
+			pos_core++;
+		}
+
+		/* Wait for writers to complete */
+		for (i = 3; i <= rwc_core_cnt[n] + 2; i++)
+			rte_eal_wait_lcore(enabled_core_ids[i]);
+
+		writer_done = 1;
+
+		/* Wait for readers to complete */
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			__atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) /
+			__atomic_load_n(&gwrites, __ATOMIC_RELAXED);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return 0;
+
+err:
+	writer_done = 1;
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_perf_main(void)
 {
@@ -1235,7 +1398,6 @@ test_hash_readwrite_lf_perf_main(void)
 	int rwc_lf = 0;
 	int htm;
 	int ext_bkt = 0;
-
 	if (rte_lcore_count() < 2) {
 		printf("Not enough cores for hash_readwrite_lf_perf_autotest, expecting at least 2\n");
 		return TEST_SKIPPED;
@@ -1255,7 +1417,6 @@ test_hash_readwrite_lf_perf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1282,6 +1443,9 @@ test_hash_readwrite_lf_perf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		if (test_hash_rcu_qsbr_writer_perf(&rwc_lf_results, rwc_lf,
+						   htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR
  2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2020-10-21  2:42           ` Wang, Yipeng1
  2020-10-21  4:52             ` Dharmik Thakkar
  0 siblings, 1 reply; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-10-21  2:42 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce, Ray Kinsella,
	Neil Horman
  Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Tuesday, October 20, 2020 9:13 AM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
> <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBR
> 
> Currently, users have to use external RCU mechanisms to free resources
> when using lock free hash algorithm.
> 
> Integrate RCU QSBR process to make it easier for the applications to use lock
> free algorithm.
> Refer to RCU documentation to understand various aspects of integrating RCU
> library into other libraries.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> Acked-by: Ray Kinsella <mdr@ashroe.eu>
> ---
>  doc/guides/prog_guide/hash_lib.rst |  11 +-
>  lib/librte_hash/meson.build        |   1 +
>  lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
>  lib/librte_hash/rte_cuckoo_hash.h  |   8 +
>  lib/librte_hash/rte_hash.h         |  77 +++++++-
>  lib/librte_hash/version.map        |   2 +-
>  6 files changed, 325 insertions(+), 76 deletions(-)
> 
> diff --git a/doc/guides/prog_guide/hash_lib.rst
> b/doc/guides/prog_guide/hash_lib.rst
> index d06c7de2ead1..63e183ed1f08 100644
> --- a/doc/guides/prog_guide/hash_lib.rst
> +++ b/doc/guides/prog_guide/hash_lib.rst
> @@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and
> writes the following flag values
>  *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
> flag is set, the position of the entry in the hash table is not freed upon calling
> delete(). This flag is enabled
>     by default when the lock free read/write concurrency flag is set. The
> application should free the position after all the readers have stopped
> referencing the position.
>     Where required, the application can make use of RCU mechanisms to
> determine when the readers have stopped referencing the position.
> +   RCU QSBR process is integrated within the Hash library for safe freeing of
> the position. Application has certain responsibilities while using this feature.
> +   Please refer to resource reclamation framework of :ref:`RCU library
> <RCU_Library>` for more details.

[Yipeng]: Maybe also add: rte_hash_rcu_qsbr_add() need to be called to use the embedded RCU mechanism.
Just to give user a pointer to which API to look.
> +
> 
>  Extendable Bucket Functionality support
>  ----------------------------------------
> @@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag
> is not set by default).
>  in the very unlikely case due to excessive hash collisions that a key has failed
> to be inserted, the hash table bucket is extended with a linked  list to insert
> these failed keys. This feature is important for the workloads (e.g. telco
> workloads) that need to insert up to 100% of the  hash table size and can't
> tolerate any key insertion failure (even if very few).
> -Please note that with the 'lock free read/write concurrency' flag enabled,
> users need to call 'rte_hash_free_key_with_position' API in order to free the
> empty buckets and -deleted keys, to maintain the 100% capacity guarantee.
> +Please note that with the 'lock free read/write concurrency' flag
> +enabled, users need to call 'rte_hash_free_key_with_position' API or
> configure integrated RCU QSBR (or use external RCU mechanisms) in order to
> free the empty buckets and deleted keys, to maintain the 100% capacity
> guarantee.
> 
>  Implementation Details (non Extendable Bucket Case)
>  ---------------------------------------------------
> @@ -172,7 +175,7 @@ Example of deletion:
>  Similar to lookup, the key is searched in its primary and secondary buckets. If
> the key is found, the  entry is marked as empty. If the hash table was
> configured with 'no free on delete' or 'lock free read/write concurrency',  the
> position of the key is not freed. It is the responsibility of the user to free the
> position after -readers are not referencing the position anymore.
> +readers are not referencing the position anymore. User can configure
> +integrated RCU QSBR or use external RCU mechanisms to safely free the
> +position on delete
> 
> 
>  Implementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ The
> flow table operations on the application side are described below:
>  *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free
> read/write concurrency' flags are set,
>      wait till the readers are not referencing the position returned during
> add/delete flow and then free the position.
>      RCU mechanisms can be used to find out when the readers are not
> referencing the position anymore.
> +    RCU QSBR process is integrated within the Hash library for safe freeing of
> the position. Application has certain responsibilities while using this feature.
> +    Please refer to resource reclamation framework of :ref:`RCU library
> <RCU_Library>` for more details.
> 
>  *   Lookup flow: Lookup for the flow key in the hash.
>      If the returned position is valid (flow lookup hit), use the returned position
> to access the flow entry in the flow table.
> diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build index
> 6ab46ae9d768..0977a63fd279 100644
> --- a/lib/librte_hash/meson.build
> +++ b/lib/librte_hash/meson.build
> @@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
> 
>  sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')  deps += ['ring']
> +deps += ['rcu']
> diff --git a/lib/librte_hash/rte_cuckoo_hash.c
> b/lib/librte_hash/rte_cuckoo_hash.c
> index aad0c965be5e..b9e4d82a0c14 100644
> --- a/lib/librte_hash/rte_cuckoo_hash.c
> +++ b/lib/librte_hash/rte_cuckoo_hash.c
> @@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {  };
>  EAL_REGISTER_TAILQ(rte_hash_tailq)
> 
> +struct __rte_hash_rcu_dq_entry {
> +	uint32_t key_idx;
> +	uint32_t ext_bkt_idx; /**< Extended bkt index */ };
> +
>  struct rte_hash *
>  rte_hash_find_existing(const char *name)  { @@ -210,7 +215,10 @@
> rte_hash_create(const struct rte_hash_parameters *params)
> 
>  	if (params->extra_flag &
> RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
>  		readwrite_concur_lf_support = 1;
> -		/* Enable not freeing internal memory/index on delete */
> +		/* Enable not freeing internal memory/index on delete.
> +		 * If internal RCU is enabled, freeing of internal memory/index
> +		 * is done on delete
> +		 */
>  		no_free_on_del = 1;
>  	}
> 
> @@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
> 
>  	rte_mcfg_tailq_write_unlock();
> 
> +	/* RCU clean up. */
> +	if (h->dq)
> +		rte_rcu_qsbr_dq_delete(h->dq);
> +
>  	if (h->use_local_cache)
>  		rte_free(h->local_free_slots);
>  	if (h->writer_takes_lock)
> @@ -607,11 +619,21 @@ void
>  rte_hash_reset(struct rte_hash *h)
>  {
>  	uint32_t tot_ring_cnt, i;
> +	unsigned int pending;
> 
>  	if (h == NULL)
>  		return;
> 
>  	__hash_rw_writer_lock(h);
> +
> +	/* RCU QSBR clean up. */
> +	if (h->dq) {
> +		/* Reclaim all the resources */
> +		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
> +		if (pending != 0)
> +			RTE_LOG(ERR, HASH, "RCU reclaim all resources
> failed\n");
> +	}
> +
>  	memset(h->buckets, 0, h->num_buckets * sizeof(struct
> rte_hash_bucket));
>  	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
>  	*h->tbl_chng_cnt = 0;
> @@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct
> rte_hash *h,
>  	return -ENOSPC;
>  }
> 
> +static inline uint32_t
> +alloc_slot(const struct rte_hash *h, struct lcore_cache
> +*cached_free_slots) {
> +	unsigned int  n_slots;
> +	uint32_t slot_id;

[Yipeng]: Blank line after variable declaration.

> +	if (h->use_local_cache) {
> +		/* Try to get a free slot from the local cache */
> +		if (cached_free_slots->len == 0) {
> +			/* Need to get another burst of free slots from global
> ring */
> +			n_slots = rte_ring_mc_dequeue_burst_elem(h-
> >free_slots,
> +					cached_free_slots->objs,
> +					sizeof(uint32_t),
> +					LCORE_CACHE_SIZE, NULL);
> +			if (n_slots == 0)
> +				return EMPTY_SLOT;
> +
> +			cached_free_slots->len += n_slots;
> +		}
> +
> +		/* Get a free slot from the local cache */
> +		cached_free_slots->len--;
> +		slot_id = cached_free_slots->objs[cached_free_slots->len];
> +	} else {
> +		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
> +						sizeof(uint32_t)) != 0)
> +			return EMPTY_SLOT;
> +	}
> +
> +	return slot_id;
> +}
> +
>  static inline int32_t
>  __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
>  						hash_sig_t sig, void *data)
> @@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct
> rte_hash *h, const void *key,
>  	uint32_t ext_bkt_id = 0;
>  	uint32_t slot_id;
>  	int ret;
> -	unsigned n_slots;
>  	unsigned lcore_id;
>  	unsigned int i;
>  	struct lcore_cache *cached_free_slots = NULL; @@ -1001,28
> +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h,
> const void *key,
>  	if (h->use_local_cache) {
>  		lcore_id = rte_lcore_id();
>  		cached_free_slots = &h->local_free_slots[lcore_id];
> -		/* Try to get a free slot from the local cache */
> -		if (cached_free_slots->len == 0) {
> -			/* Need to get another burst of free slots from global
> ring */
> -			n_slots = rte_ring_mc_dequeue_burst_elem(h-
> >free_slots,
> -					cached_free_slots->objs,
> -					sizeof(uint32_t),
> -					LCORE_CACHE_SIZE, NULL);
> -			if (n_slots == 0) {
> -				return -ENOSPC;
> -			}
> -
> -			cached_free_slots->len += n_slots;
> +	}
> +	slot_id = alloc_slot(h, cached_free_slots);
> +	if (slot_id == EMPTY_SLOT) {
> +		if (h->dq) {
> +			__hash_rw_writer_lock(h);
> +			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
> +					h->hash_rcu_cfg->max_reclaim_size,
> +					NULL, NULL, NULL);
> +			__hash_rw_writer_unlock(h);
> +			if (ret == 0)
> +				slot_id = alloc_slot(h, cached_free_slots);
>  		}
> -
> -		/* Get a free slot from the local cache */
> -		cached_free_slots->len--;
> -		slot_id = cached_free_slots->objs[cached_free_slots->len];
> -	} else {
> -		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
> -						sizeof(uint32_t)) != 0) {
> +		if (slot_id == EMPTY_SLOT)
>  			return -ENOSPC;
> -		}
>  	}
> 
>  	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ -
> 1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash
> *h, const void *key,
>  	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
>  						sizeof(uint32_t)) != 0 ||
>  					ext_bkt_id == 0) {
> -		ret = -ENOSPC;
> -		goto failure;
> +		if (h->dq) {
> +			if (rte_rcu_qsbr_dq_reclaim(h->dq,
> +					h->hash_rcu_cfg->max_reclaim_size,
> +					NULL, NULL, NULL) == 0) {
> +				rte_ring_sc_dequeue_elem(h-
> >free_ext_bkts,
> +							 &ext_bkt_id,
> +							 sizeof(uint32_t));
> +			}
> +		}
> +		if (ext_bkt_id == 0) {
> +			ret = -ENOSPC;
> +			goto failure;
> +		}
>  	}
> 
>  	/* Use the first location of the new bucket */ @@ -1395,12 +1450,12
> @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void
> **data)
>  	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key),
> data);  }
> 
> -static inline void
> -remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
> unsigned i)
> +static int
> +free_slot(const struct rte_hash *h, uint32_t slot_id)
>  {
>  	unsigned lcore_id, n_slots;
> -	struct lcore_cache *cached_free_slots;
> -
> +	struct lcore_cache *cached_free_slots = NULL;
> +	/* Return key indexes to free slot ring */
>  	if (h->use_local_cache) {
>  		lcore_id = rte_lcore_id();
>  		cached_free_slots = &h->local_free_slots[lcore_id]; @@ -
> 1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct
> rte_hash_bucket *bkt, unsigned i)
>  						cached_free_slots->objs,
>  						sizeof(uint32_t),
>  						LCORE_CACHE_SIZE, NULL);
> -			ERR_IF_TRUE((n_slots == 0),
> -				"%s: could not enqueue free slots in global
> ring\n",
> -				__func__);
> +			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
>  			cached_free_slots->len -= n_slots;
>  		}
> -		/* Put index of new free slot in cache. */
> -		cached_free_slots->objs[cached_free_slots->len] =
> -							bkt->key_idx[i];
> -		cached_free_slots->len++;
> +	}
> +
> +	enqueue_slot_back(h, cached_free_slots, slot_id);
> +	return 0;
> +}
> +
> +static void
> +__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n) {
> +	void *key_data = NULL;
> +	int ret;
> +	struct rte_hash_key *keys, *k;
> +	struct rte_hash *h = (struct rte_hash *)p;
> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
> +			*((struct __rte_hash_rcu_dq_entry *)e);
> +
> +	RTE_SET_USED(n);
> +	keys = h->key_store;
> +
> +	k = (struct rte_hash_key *) ((char *)keys +
> +				rcu_dq_entry.key_idx * h->key_entry_size);
> +	key_data = k->pdata;
> +	if (h->hash_rcu_cfg->free_key_data_func)
> +		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-
> >key_data_ptr,
> +						    key_data);
> +
> +	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=
> EMPTY_SLOT)
> +		/* Recycle empty ext bkt to free list. */
> +		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
> +			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
> +
> +	/* Return key indexes to free slot ring */
> +	ret = free_slot(h, rcu_dq_entry.key_idx);
> +	if (ret < 0) {
> +		RTE_LOG(ERR, HASH,
> +			"%s: could not enqueue free slots in global ring\n",
> +				__func__);
> +	}
> +}
> +
> +int
> +rte_hash_rcu_qsbr_add(struct rte_hash *h,
> +				struct rte_hash_rcu_config *cfg)
> +{
> +	struct rte_rcu_qsbr_dq_parameters params = {0};
> +	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
> +	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
> +
> +	const uint32_t total_entries = h->use_local_cache ?
> +		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
> + 1
> +							: h->entries + 1;
> +
> +	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
> +		rte_errno = EINVAL;
> +		return 1;
> +	}
> +
> +	if (h->hash_rcu_cfg) {
> +		rte_errno = EEXIST;
> +		return 1;
> +	}
> +
> +	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),
> 0);
> +	if (hash_rcu_cfg == NULL) {
> +		RTE_LOG(ERR, HASH, "memory allocation failed\n");
> +		return 1;
> +	}
> +
> +	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
> +		/* No other things to do. */
> +	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
> +		/* Init QSBR defer queue. */
> +		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
> +					"HASH_RCU_%s", h->name);
> +		params.name = rcu_dq_name;
> +		params.size = cfg->dq_size;
> +		if (params.size == 0)
> +			params.size = total_entries;
> +		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
> +		if (params.max_reclaim_size == 0)
> +			params.max_reclaim_size =
> RTE_HASH_RCU_DQ_RECLAIM_MAX;
> +		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
> +		params.free_fn = __hash_rcu_qsbr_free_resource;
> +		params.p = h;
> +		params.v = cfg->v;
> +		h->dq = rte_rcu_qsbr_dq_create(&params);
> +		if (h->dq == NULL) {
> +			rte_free(hash_rcu_cfg);
> +			RTE_LOG(ERR, HASH, "HASH defer queue creation
> failed\n");
> +			return 1;
> +		}
>  	} else {
> -		rte_ring_sp_enqueue_elem(h->free_slots,
> -				&bkt->key_idx[i], sizeof(uint32_t));
> +		rte_free(hash_rcu_cfg);
> +		rte_errno = EINVAL;
> +		return 1;
> +	}
> +
> +	hash_rcu_cfg->v = cfg->v;
> +	hash_rcu_cfg->mode = cfg->mode;
> +	hash_rcu_cfg->dq_size = params.size;
> +	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
> +	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
> +	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
> +	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
> +
> +	h->hash_rcu_cfg = hash_rcu_cfg;
> +
> +	return 0;
> +}
> +
> +static inline void
> +remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
> +unsigned i) {
> +	int ret = free_slot(h, bkt->key_idx[i]);
> +	if (ret < 0) {
> +		RTE_LOG(ERR, HASH,
> +			"%s: could not enqueue free slots in global ring\n",
> +				__func__);
>  	}
>  }
> 
> @@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct
> rte_hash *h, const void *key,
>  	int pos;
>  	int32_t ret, i;
>  	uint16_t short_sig;
> +	uint32_t index = EMPTY_SLOT;
> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
> 
>  	short_sig = get_short_sig(sig);
>  	prim_bucket_idx = get_prim_bucket_index(h, sig); @@ -1555,10
> +1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const
> void *key,
> 
>  /* Search last bucket to see if empty to be recycled */
>  return_bkt:
> -	if (!last_bkt) {
> -		__hash_rw_writer_unlock(h);
> -		return ret;
> -	}
> +	if (!last_bkt)
> +		goto return_key;
> +
>  	while (last_bkt->next) {
>  		prev_bkt = last_bkt;
>  		last_bkt = last_bkt->next;
> @@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct
> rte_hash *h, const void *key,
>  	/* found empty bucket and recycle */
>  	if (i == RTE_HASH_BUCKET_ENTRIES) {
>  		prev_bkt->next = NULL;
> -		uint32_t index = last_bkt - h->buckets_ext + 1;
> +		index = last_bkt - h->buckets_ext + 1;
>  		/* Recycle the empty bkt if
>  		 * no_free_on_del is disabled.
>  		 */
> -		if (h->no_free_on_del)
> +		if (h->no_free_on_del) {
>  			/* Store index of an empty ext bkt to be recycled
>  			 * on calling rte_hash_del_xxx APIs.
>  			 * When lock free read-write concurrency is enabled,
> @@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct
> rte_hash *h, const void *key,
>  			 * immediately (as readers might be using it still).
>  			 * Hence freeing of the ext bkt is piggy-backed to
>  			 * freeing of the key index.
> +			 * If using external RCU, store this index in an array.
>  			 */
> -			h->ext_bkt_to_free[ret] = index;
> -		else
> +			if (h->hash_rcu_cfg == NULL)
> +				h->ext_bkt_to_free[ret] = index;


[Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?

> +		} else
>  			rte_ring_sp_enqueue_elem(h->free_ext_bkts,
> &index,
>  							sizeof(uint32_t));
>  	}
> +
> +return_key:
> +	/* Using internal RCU QSBR */
> +	if (h->hash_rcu_cfg) {
> +		/* Key index where key is stored, adding the first dummy
> index */
> +		rcu_dq_entry.key_idx = ret + 1;
> +		rcu_dq_entry.ext_bkt_idx = index;
> +		if (h->dq == NULL) {
> +			/* Wait for quiescent state change if using
> +			 * RTE_HASH_QSBR_MODE_SYNC
> +			 */
> +			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
> +						 RTE_QSBR_THRID_INVALID);
> +			__hash_rcu_qsbr_free_resource((void
> *)((uintptr_t)h),
> +						      &rcu_dq_entry, 1);
> +		} else if (h->dq)
> +			/* Push into QSBR FIFO if using
> RTE_HASH_QSBR_MODE_DQ */
> +			if (rte_rcu_qsbr_dq_enqueue(h->dq,
> &rcu_dq_entry) != 0)
> +				RTE_LOG(ERR, HASH, "Failed to push QSBR
> FIFO\n");
> +	}
>  	__hash_rw_writer_unlock(h);
>  	return ret;
>  }
> @@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct
> rte_hash *h,
> 
>  	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
> 
> -	unsigned int lcore_id, n_slots;
> -	struct lcore_cache *cached_free_slots;
>  	const uint32_t total_entries = h->use_local_cache ?
>  		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
> + 1
>  							: h->entries + 1;
> @@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct
> rte_hash *h,
>  		}
>  	}
> 
> -	if (h->use_local_cache) {
> -		lcore_id = rte_lcore_id();
> -		cached_free_slots = &h->local_free_slots[lcore_id];
> -		/* Cache full, need to free it. */
> -		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
> -			/* Need to enqueue the free slots in global ring. */
> -			n_slots = rte_ring_mp_enqueue_burst_elem(h-
> >free_slots,
> -						cached_free_slots->objs,
> -						sizeof(uint32_t),
> -						LCORE_CACHE_SIZE, NULL);
> -			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
> -			cached_free_slots->len -= n_slots;
> -		}
> -		/* Put index of new free slot in cache. */
> -		cached_free_slots->objs[cached_free_slots->len] = key_idx;
> -		cached_free_slots->len++;
> -	} else {
> -		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
> -						sizeof(uint32_t));
> -	}
> +	/* Enqueue slot to cache/ring of free slots. */
> +	return free_slot(h, key_idx);
> 
> -	return 0;
>  }
> 
>  static inline void
> diff --git a/lib/librte_hash/rte_cuckoo_hash.h
> b/lib/librte_hash/rte_cuckoo_hash.h
> index 345de6bf9cfd..85be49d3bbe7 100644
> --- a/lib/librte_hash/rte_cuckoo_hash.h
> +++ b/lib/librte_hash/rte_cuckoo_hash.h
> @@ -168,6 +168,11 @@ struct rte_hash {
>  	struct lcore_cache *local_free_slots;
>  	/**< Local cache per lcore, storing some indexes of the free slots */
> 
> +	/* RCU config */
> +	struct rte_hash_rcu_config *hash_rcu_cfg;
> +	/**< HASH RCU QSBR configuration structure */
> +	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
> +
>  	/* Fields used in lookup */
> 
>  	uint32_t key_len __rte_cache_aligned;
> @@ -230,4 +235,7 @@ struct queue_node {
>  	int prev_slot;               /* Parent(slot) in search path */
>  };
> 
> +/** @internal Default RCU defer queue entries to reclaim in one go. */
> +#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
> +
>  #endif
> diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h index
> bff40251bc98..3d28f177f14a 100644
> --- a/lib/librte_hash/rte_hash.h
> +++ b/lib/librte_hash/rte_hash.h
> @@ -15,6 +15,7 @@
>  #include <stddef.h>
> 
>  #include <rte_compat.h>
> +#include <rte_rcu_qsbr.h>
> 
>  #ifdef __cplusplus
>  extern "C" {
> @@ -45,7 +46,8 @@ extern "C" {
>  /** Flag to disable freeing of key index on hash delete.
>   * Refer to rte_hash_del_xxx APIs for more details.
>   * This is enabled by default when
> RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
> - * is enabled.
> + * is enabled. However, if internal RCU is enabled, freeing of internal
> + * memory/index is done on delete
>   */
>  #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
> 
> @@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void
> *key, uint32_t key_len,
>  /** Type of function used to compare the hash key. */  typedef int
> (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
> 
> +/**
> + * Type of function used to free data stored in the key.
> + * Required when using internal RCU to allow application to free
> +key-data once
> + * the key is returned to the the ring of free key-slots.
> + */
> +typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
> +
>  /**
>   * Parameters used when creating the hash table.
>   */
> @@ -81,6 +90,39 @@ struct rte_hash_parameters {
>  	uint8_t extra_flag;		/**< Indicate if additional parameters
> are present. */
>  };
> 
> +/** RCU reclamation modes */
> +enum rte_hash_qsbr_mode {
> +	/** Create defer queue for reclaim. */
> +	RTE_HASH_QSBR_MODE_DQ = 0,
> +	/** Use blocking mode reclaim. No defer queue created. */
> +	RTE_HASH_QSBR_MODE_SYNC
> +};
> +
> +/** HASH RCU QSBR configuration structure. */ struct
> +rte_hash_rcu_config {
> +	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
> +	enum rte_hash_qsbr_mode mode;
> +	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
> +	 * '0' for default: create defer queue for reclaim.
> +	 */
> +	uint32_t dq_size;
> +	/**< RCU defer queue size.
> +	 * default: total hash table entries.
> +	 */
> +	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim.
> */
> +	uint32_t max_reclaim_size;
> +	/**< Max entries to reclaim in one go.
> +	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
> +	 */
> +	void *key_data_ptr;
> +	/**< Pointer passed to the free function. Typically, this is the
> +	 * pointer to the data structure to which the resource to free
> +	 * (key-data) belongs. This can be NULL.
> +	 */
> +	rte_hash_free_key_data free_key_data_func;
> +	/**< Function to call to free the resource (key-data). */ };
> +
>  /** @internal A hash table structure. */  struct rte_hash;
> 
> @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash
> *h, const void *key, hash_sig_t
>   * Thread safety can be enabled by setting flag during
>   * table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_add_key_xxx APIs will not be
>   * freed by this API. rte_hash_free_key_with_position API must be called
>   * additionally to free the index associated with the key.
> @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const
> void *key);
>   * Thread safety can be enabled by setting flag during
>   * table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_add_key_xxx APIs will not be
>   * freed by this API. rte_hash_free_key_with_position API must be called
>   * additionally to free the index associated with the key.
> @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct
> rte_hash *h, const int32_t position,
>   * only be called from one thread by default. Thread safety
>   * can be enabled by setting flag during table creation.
>   * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
> + * internal RCU is NOT enabled,
>   * the key index returned by rte_hash_del_key_xxx APIs must be freed
>   * using this API. This API should be called after all the readers
>   * have stopped referencing the entry corresponding to this key.
> @@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h,
> const void **keys,
>   */
>  int32_t
>  rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
> uint32_t *next);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Associate RCU QSBR variable with an Hash object.
[Yipeng]: a Hash object

> + * This API should be called to enable the integrated RCU QSBR support
> +and
> + * should be called immediately after creating the Hash object.
> + *
> + * @param h
> + *   the hash object to add RCU QSBR
> + * @param cfg
> + *   RCU QSBR configuration
> + * @return
> + *   On success - 0
> + *   On error - 1 with error code set in rte_errno.
> + *   Possible rte_errno codes are:
> + *   - EINVAL - invalid pointer
> + *   - EEXIST - already added QSBR
> + *   - ENOMEM - memory allocation failure
> + */
> +__rte_experimental
> +int rte_hash_rcu_qsbr_add(struct rte_hash *h,
> +				struct rte_hash_rcu_config *cfg);
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/librte_hash/version.map b/lib/librte_hash/version.map index
> c0db81014ff9..c6d73080f478 100644
> --- a/lib/librte_hash/version.map
> +++ b/lib/librte_hash/version.map
> @@ -36,5 +36,5 @@ EXPERIMENTAL {
>  	rte_hash_lookup_with_hash_bulk;
>  	rte_hash_lookup_with_hash_bulk_data;
>  	rte_hash_max_key_id;
> -
> +	rte_hash_rcu_qsbr_add;
>  };
> --
> 2.17.1
[Yipeng]:

Hi, Dharmik, thanks for the work! It generally looks good.
Just some minor issues to address and one question for the ext table inlined.

Thanks!





^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
@ 2020-10-21  2:52           ` Wang, Yipeng1
  0 siblings, 0 replies; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-10-21  2:52 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce; +Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Tuesday, October 20, 2020 9:13 AM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs
> 
> Replace rte_atomic APIs with C11 atomic APIs in test_hash_readwrite_lf_perf.c
> 
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---

Acked-by: Yipeng Wang <yipeng1.wang@intel.com>

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-21  3:54           ` Wang, Yipeng1
  2020-10-21  4:55             ` Dharmik Thakkar
  2020-10-21 22:34             ` Honnappa Nagarahalli
  0 siblings, 2 replies; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-10-21  3:54 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce; +Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Tuesday, October 20, 2020 9:13 AM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
> 
> Add functional and performance tests for the integrated RCU QSBR.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---
>  app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
>  app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
>  2 files changed, 556 insertions(+), 4 deletions(-)
> 
> diff --git a/app/test/test_hash.c b/app/test/test_hash.c index
> 990a1815f893..22b47b3e7728 100644
> --- a/app/test/test_hash.c
> +++ b/app/test/test_hash.c
> @@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10,
> 11, 15, 16, 21,
>  	}								\
>  } while(0)
> 
> -#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
> 	\
> +#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
>  	if (cond) {							\
>  		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
> \
>  		if (handle) rte_fbk_hash_free(handle);			\
> @@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8,
> 10, 11, 15, 16, 21,
>  	}								\
>  } while(0)
> 
> +#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
> 	\
> +	if (cond) {							\
> +		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
> \
> +		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
> 	\
> +			writer_done = 1;				\
> +			/* Wait until reader exited. */			\
> +			rte_eal_mp_wait_lcore();			\
> +		}							\
> +		if (g_handle) rte_hash_free(g_handle);			\
> +		if (g_qsv) rte_free(g_qsv);				\
> +		return -1;						\
> +	}								\
> +} while(0)
> +
>  /* 5-tuple key type */
>  struct flow_key {
>  	uint32_t ip_src;
> @@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
>  	return ret;
>  }
> 
> +static struct rte_hash *g_handle;
> +static struct rte_rcu_qsbr *g_qsv;
> +static volatile uint8_t writer_done;
> +struct flow_key g_rand_keys[9];
> +/*
> + * rte_hash_rcu_qsbr_add positive and negative tests.
> + *  - Add RCU QSBR variable to Hash
> + *  - Add another RCU QSBR variable to Hash
> + *  - Check returns
> + */
> +static int
> +test_hash_rcu_qsbr_add(void)
> +{
> +	size_t sz;
> +	struct rte_rcu_qsbr *qsv2 = NULL;
> +	int32_t status;
> +	struct rte_hash_rcu_config rcu_cfg = {0};
> +
> +	struct rte_hash_parameters params;
> +
> +	printf("\n# Running RCU QSBR add tests\n");
> +	memcpy(&params, &ut_params, sizeof(params));
> +	params.name = "test_hash_rcu_qsbr_add";
> +	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
> |
> +
> 	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
> +	g_handle = rte_hash_create(&params);
> +	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation
> failed");
> +
> +	/* Create RCU QSBR variable */
> +	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
> +	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
> +					RTE_CACHE_LINE_SIZE,
> SOCKET_ID_ANY);
> +	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
> +				 "RCU QSBR variable creation failed");
> +
> +	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
[Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.

> +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
> +				 "RCU QSBR variable initialization failed");
> +
> +	rcu_cfg.v = g_qsv;
> +	/* Invalid QSBR mode */
> +	rcu_cfg.mode = 2;
[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?

> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> +	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
> +failed");
> +
> +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose? 

> +	/* Attach RCU QSBR to hash table */
> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
> +				 "Attach RCU QSBR to hash table failed");
> +
> +	/* Create and attach another RCU QSBR to hash table */
> +	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
> +					RTE_CACHE_LINE_SIZE,
> SOCKET_ID_ANY);
> +	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
> +				 "RCU QSBR variable creation failed");
> +
> +	rcu_cfg.v = qsv2;
> +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> +	rte_free(qsv2);
> +	RETURN_IF_ERROR_RCU_QSBR(status == 0,
> +			"Attach RCU QSBR to hash table succeeded where
> failure"
> +			" is expected");
> +
> +	rte_hash_free(g_handle);
> +	rte_free(g_qsv);
> +
> +	return 0;
> +}
<...>

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR
  2020-10-21  2:42           ` Wang, Yipeng1
@ 2020-10-21  4:52             ` Dharmik Thakkar
  0 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21  4:52 UTC (permalink / raw)
  To: Wang, Yipeng1
  Cc: Gobriel, Sameh, Richardson, Bruce, Ray Kinsella, Neil Horman, dev, nd



> On Oct 20, 2020, at 9:42 PM, Wang, Yipeng1 <yipeng1.wang@intel.com> wrote:
> 
>> -----Original Message-----
>> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Sent: Tuesday, October 20, 2020 9:13 AM
>> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
>> <sameh.gobriel@intel.com>; Richardson, Bruce
>> <bruce.richardson@intel.com>; Ray Kinsella <mdr@ashroe.eu>; Neil Horman
>> <nhorman@tuxdriver.com>
>> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
>> <dharmik.thakkar@arm.com>
>> Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBR
>> 
>> Currently, users have to use external RCU mechanisms to free resources
>> when using lock free hash algorithm.
>> 
>> Integrate RCU QSBR process to make it easier for the applications to use lock
>> free algorithm.
>> Refer to RCU documentation to understand various aspects of integrating RCU
>> library into other libraries.
>> 
>> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
>> Acked-by: Ray Kinsella <mdr@ashroe.eu>
>> ---
>> doc/guides/prog_guide/hash_lib.rst |  11 +-
>> lib/librte_hash/meson.build        |   1 +
>> lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
>> lib/librte_hash/rte_cuckoo_hash.h  |   8 +
>> lib/librte_hash/rte_hash.h         |  77 +++++++-
>> lib/librte_hash/version.map        |   2 +-
>> 6 files changed, 325 insertions(+), 76 deletions(-)
>> 
>> diff --git a/doc/guides/prog_guide/hash_lib.rst
>> b/doc/guides/prog_guide/hash_lib.rst
>> index d06c7de2ead1..63e183ed1f08 100644
>> --- a/doc/guides/prog_guide/hash_lib.rst
>> +++ b/doc/guides/prog_guide/hash_lib.rst
>> @@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and
>> writes the following flag values
>> *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
>> flag is set, the position of the entry in the hash table is not freed upon calling
>> delete(). This flag is enabled
>>    by default when the lock free read/write concurrency flag is set. The
>> application should free the position after all the readers have stopped
>> referencing the position.
>>    Where required, the application can make use of RCU mechanisms to
>> determine when the readers have stopped referencing the position.
>> +   RCU QSBR process is integrated within the Hash library for safe freeing of
>> the position. Application has certain responsibilities while using this feature.
>> +   Please refer to resource reclamation framework of :ref:`RCU library
>> <RCU_Library>` for more details.
> 
> [Yipeng]: Maybe also add: rte_hash_rcu_qsbr_add() need to be called to use the embedded RCU mechanism.
> Just to give user a pointer to which API to look.

Copy.

>> +
>> 
>> Extendable Bucket Functionality support
>> ----------------------------------------
>> @@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag
>> is not set by default).
>> in the very unlikely case due to excessive hash collisions that a key has failed
>> to be inserted, the hash table bucket is extended with a linked  list to insert
>> these failed keys. This feature is important for the workloads (e.g. telco
>> workloads) that need to insert up to 100% of the  hash table size and can't
>> tolerate any key insertion failure (even if very few).
>> -Please note that with the 'lock free read/write concurrency' flag enabled,
>> users need to call 'rte_hash_free_key_with_position' API in order to free the
>> empty buckets and -deleted keys, to maintain the 100% capacity guarantee.
>> +Please note that with the 'lock free read/write concurrency' flag
>> +enabled, users need to call 'rte_hash_free_key_with_position' API or
>> configure integrated RCU QSBR (or use external RCU mechanisms) in order to
>> free the empty buckets and deleted keys, to maintain the 100% capacity
>> guarantee.
>> 
>> Implementation Details (non Extendable Bucket Case)
>> ---------------------------------------------------
>> @@ -172,7 +175,7 @@ Example of deletion:
>> Similar to lookup, the key is searched in its primary and secondary buckets. If
>> the key is found, the  entry is marked as empty. If the hash table was
>> configured with 'no free on delete' or 'lock free read/write concurrency',  the
>> position of the key is not freed. It is the responsibility of the user to free the
>> position after -readers are not referencing the position anymore.
>> +readers are not referencing the position anymore. User can configure
>> +integrated RCU QSBR or use external RCU mechanisms to safely free the
>> +position on delete
>> 
>> 
>> Implementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ The
>> flow table operations on the application side are described below:
>> *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free
>> read/write concurrency' flags are set,
>>     wait till the readers are not referencing the position returned during
>> add/delete flow and then free the position.
>>     RCU mechanisms can be used to find out when the readers are not
>> referencing the position anymore.
>> +    RCU QSBR process is integrated within the Hash library for safe freeing of
>> the position. Application has certain responsibilities while using this feature.
>> +    Please refer to resource reclamation framework of :ref:`RCU library
>> <RCU_Library>` for more details.
>> 
>> *   Lookup flow: Lookup for the flow key in the hash.
>>     If the returned position is valid (flow lookup hit), use the returned position
>> to access the flow entry in the flow table.
>> diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build index
>> 6ab46ae9d768..0977a63fd279 100644
>> --- a/lib/librte_hash/meson.build
>> +++ b/lib/librte_hash/meson.build
>> @@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
>> 
>> sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')  deps += ['ring']
>> +deps += ['rcu']
>> diff --git a/lib/librte_hash/rte_cuckoo_hash.c
>> b/lib/librte_hash/rte_cuckoo_hash.c
>> index aad0c965be5e..b9e4d82a0c14 100644
>> --- a/lib/librte_hash/rte_cuckoo_hash.c
>> +++ b/lib/librte_hash/rte_cuckoo_hash.c
>> @@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {  };
>> EAL_REGISTER_TAILQ(rte_hash_tailq)
>> 
>> +struct __rte_hash_rcu_dq_entry {
>> +	uint32_t key_idx;
>> +	uint32_t ext_bkt_idx; /**< Extended bkt index */ };
>> +
>> struct rte_hash *
>> rte_hash_find_existing(const char *name)  { @@ -210,7 +215,10 @@
>> rte_hash_create(const struct rte_hash_parameters *params)
>> 
>> 	if (params->extra_flag &
>> RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
>> 		readwrite_concur_lf_support = 1;
>> -		/* Enable not freeing internal memory/index on delete */
>> +		/* Enable not freeing internal memory/index on delete.
>> +		 * If internal RCU is enabled, freeing of internal memory/index
>> +		 * is done on delete
>> +		 */
>> 		no_free_on_del = 1;
>> 	}
>> 
>> @@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
>> 
>> 	rte_mcfg_tailq_write_unlock();
>> 
>> +	/* RCU clean up. */
>> +	if (h->dq)
>> +		rte_rcu_qsbr_dq_delete(h->dq);
>> +
>> 	if (h->use_local_cache)
>> 		rte_free(h->local_free_slots);
>> 	if (h->writer_takes_lock)
>> @@ -607,11 +619,21 @@ void
>> rte_hash_reset(struct rte_hash *h)
>> {
>> 	uint32_t tot_ring_cnt, i;
>> +	unsigned int pending;
>> 
>> 	if (h == NULL)
>> 		return;
>> 
>> 	__hash_rw_writer_lock(h);
>> +
>> +	/* RCU QSBR clean up. */
>> +	if (h->dq) {
>> +		/* Reclaim all the resources */
>> +		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
>> +		if (pending != 0)
>> +			RTE_LOG(ERR, HASH, "RCU reclaim all resources
>> failed\n");
>> +	}
>> +
>> 	memset(h->buckets, 0, h->num_buckets * sizeof(struct
>> rte_hash_bucket));
>> 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
>> 	*h->tbl_chng_cnt = 0;
>> @@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct
>> rte_hash *h,
>> 	return -ENOSPC;
>> }
>> 
>> +static inline uint32_t
>> +alloc_slot(const struct rte_hash *h, struct lcore_cache
>> +*cached_free_slots) {
>> +	unsigned int  n_slots;
>> +	uint32_t slot_id;
> 
> [Yipeng]: Blank line after variable declaration.

Copy.

> 
>> +	if (h->use_local_cache) {
>> +		/* Try to get a free slot from the local cache */
>> +		if (cached_free_slots->len == 0) {
>> +			/* Need to get another burst of free slots from global
>> ring */
>> +			n_slots = rte_ring_mc_dequeue_burst_elem(h-
>>> free_slots,
>> +					cached_free_slots->objs,
>> +					sizeof(uint32_t),
>> +					LCORE_CACHE_SIZE, NULL);
>> +			if (n_slots == 0)
>> +				return EMPTY_SLOT;
>> +
>> +			cached_free_slots->len += n_slots;
>> +		}
>> +
>> +		/* Get a free slot from the local cache */
>> +		cached_free_slots->len--;
>> +		slot_id = cached_free_slots->objs[cached_free_slots->len];
>> +	} else {
>> +		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
>> +						sizeof(uint32_t)) != 0)
>> +			return EMPTY_SLOT;
>> +	}
>> +
>> +	return slot_id;
>> +}
>> +
>> static inline int32_t
>> __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
>> 						hash_sig_t sig, void *data)
>> @@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct
>> rte_hash *h, const void *key,
>> 	uint32_t ext_bkt_id = 0;
>> 	uint32_t slot_id;
>> 	int ret;
>> -	unsigned n_slots;
>> 	unsigned lcore_id;
>> 	unsigned int i;
>> 	struct lcore_cache *cached_free_slots = NULL; @@ -1001,28
>> +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h,
>> const void *key,
>> 	if (h->use_local_cache) {
>> 		lcore_id = rte_lcore_id();
>> 		cached_free_slots = &h->local_free_slots[lcore_id];
>> -		/* Try to get a free slot from the local cache */
>> -		if (cached_free_slots->len == 0) {
>> -			/* Need to get another burst of free slots from global
>> ring */
>> -			n_slots = rte_ring_mc_dequeue_burst_elem(h-
>>> free_slots,
>> -					cached_free_slots->objs,
>> -					sizeof(uint32_t),
>> -					LCORE_CACHE_SIZE, NULL);
>> -			if (n_slots == 0) {
>> -				return -ENOSPC;
>> -			}
>> -
>> -			cached_free_slots->len += n_slots;
>> +	}
>> +	slot_id = alloc_slot(h, cached_free_slots);
>> +	if (slot_id == EMPTY_SLOT) {
>> +		if (h->dq) {
>> +			__hash_rw_writer_lock(h);
>> +			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
>> +					h->hash_rcu_cfg->max_reclaim_size,
>> +					NULL, NULL, NULL);
>> +			__hash_rw_writer_unlock(h);
>> +			if (ret == 0)
>> +				slot_id = alloc_slot(h, cached_free_slots);
>> 		}
>> -
>> -		/* Get a free slot from the local cache */
>> -		cached_free_slots->len--;
>> -		slot_id = cached_free_slots->objs[cached_free_slots->len];
>> -	} else {
>> -		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
>> -						sizeof(uint32_t)) != 0) {
>> +		if (slot_id == EMPTY_SLOT)
>> 			return -ENOSPC;
>> -		}
>> 	}
>> 
>> 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ -
>> 1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash
>> *h, const void *key,
>> 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
>> 						sizeof(uint32_t)) != 0 ||
>> 					ext_bkt_id == 0) {
>> -		ret = -ENOSPC;
>> -		goto failure;
>> +		if (h->dq) {
>> +			if (rte_rcu_qsbr_dq_reclaim(h->dq,
>> +					h->hash_rcu_cfg->max_reclaim_size,
>> +					NULL, NULL, NULL) == 0) {
>> +				rte_ring_sc_dequeue_elem(h-
>>> free_ext_bkts,
>> +							 &ext_bkt_id,
>> +							 sizeof(uint32_t));
>> +			}
>> +		}
>> +		if (ext_bkt_id == 0) {
>> +			ret = -ENOSPC;
>> +			goto failure;
>> +		}
>> 	}
>> 
>> 	/* Use the first location of the new bucket */ @@ -1395,12 +1450,12
>> @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void
>> **data)
>> 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key),
>> data);  }
>> 
>> -static inline void
>> -remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
>> unsigned i)
>> +static int
>> +free_slot(const struct rte_hash *h, uint32_t slot_id)
>> {
>> 	unsigned lcore_id, n_slots;
>> -	struct lcore_cache *cached_free_slots;
>> -
>> +	struct lcore_cache *cached_free_slots = NULL;
>> +	/* Return key indexes to free slot ring */
>> 	if (h->use_local_cache) {
>> 		lcore_id = rte_lcore_id();
>> 		cached_free_slots = &h->local_free_slots[lcore_id]; @@ -
>> 1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct
>> rte_hash_bucket *bkt, unsigned i)
>> 						cached_free_slots->objs,
>> 						sizeof(uint32_t),
>> 						LCORE_CACHE_SIZE, NULL);
>> -			ERR_IF_TRUE((n_slots == 0),
>> -				"%s: could not enqueue free slots in global
>> ring\n",
>> -				__func__);
>> +			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
>> 			cached_free_slots->len -= n_slots;
>> 		}
>> -		/* Put index of new free slot in cache. */
>> -		cached_free_slots->objs[cached_free_slots->len] =
>> -							bkt->key_idx[i];
>> -		cached_free_slots->len++;
>> +	}
>> +
>> +	enqueue_slot_back(h, cached_free_slots, slot_id);
>> +	return 0;
>> +}
>> +
>> +static void
>> +__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n) {
>> +	void *key_data = NULL;
>> +	int ret;
>> +	struct rte_hash_key *keys, *k;
>> +	struct rte_hash *h = (struct rte_hash *)p;
>> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
>> +			*((struct __rte_hash_rcu_dq_entry *)e);
>> +
>> +	RTE_SET_USED(n);
>> +	keys = h->key_store;
>> +
>> +	k = (struct rte_hash_key *) ((char *)keys +
>> +				rcu_dq_entry.key_idx * h->key_entry_size);
>> +	key_data = k->pdata;
>> +	if (h->hash_rcu_cfg->free_key_data_func)
>> +		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-
>>> key_data_ptr,
>> +						    key_data);
>> +
>> +	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=
>> EMPTY_SLOT)
>> +		/* Recycle empty ext bkt to free list. */
>> +		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
>> +			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
>> +
>> +	/* Return key indexes to free slot ring */
>> +	ret = free_slot(h, rcu_dq_entry.key_idx);
>> +	if (ret < 0) {
>> +		RTE_LOG(ERR, HASH,
>> +			"%s: could not enqueue free slots in global ring\n",
>> +				__func__);
>> +	}
>> +}
>> +
>> +int
>> +rte_hash_rcu_qsbr_add(struct rte_hash *h,
>> +				struct rte_hash_rcu_config *cfg)
>> +{
>> +	struct rte_rcu_qsbr_dq_parameters params = {0};
>> +	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
>> +	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
>> +
>> +	const uint32_t total_entries = h->use_local_cache ?
>> +		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
>> + 1
>> +							: h->entries + 1;
>> +
>> +	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
>> +		rte_errno = EINVAL;
>> +		return 1;
>> +	}
>> +
>> +	if (h->hash_rcu_cfg) {
>> +		rte_errno = EEXIST;
>> +		return 1;
>> +	}
>> +
>> +	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),
>> 0);
>> +	if (hash_rcu_cfg == NULL) {
>> +		RTE_LOG(ERR, HASH, "memory allocation failed\n");
>> +		return 1;
>> +	}
>> +
>> +	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
>> +		/* No other things to do. */
>> +	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
>> +		/* Init QSBR defer queue. */
>> +		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
>> +					"HASH_RCU_%s", h->name);
>> +		params.name = rcu_dq_name;
>> +		params.size = cfg->dq_size;
>> +		if (params.size == 0)
>> +			params.size = total_entries;
>> +		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
>> +		if (params.max_reclaim_size == 0)
>> +			params.max_reclaim_size =
>> RTE_HASH_RCU_DQ_RECLAIM_MAX;
>> +		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
>> +		params.free_fn = __hash_rcu_qsbr_free_resource;
>> +		params.p = h;
>> +		params.v = cfg->v;
>> +		h->dq = rte_rcu_qsbr_dq_create(&params);
>> +		if (h->dq == NULL) {
>> +			rte_free(hash_rcu_cfg);
>> +			RTE_LOG(ERR, HASH, "HASH defer queue creation
>> failed\n");
>> +			return 1;
>> +		}
>> 	} else {
>> -		rte_ring_sp_enqueue_elem(h->free_slots,
>> -				&bkt->key_idx[i], sizeof(uint32_t));
>> +		rte_free(hash_rcu_cfg);
>> +		rte_errno = EINVAL;
>> +		return 1;
>> +	}
>> +
>> +	hash_rcu_cfg->v = cfg->v;
>> +	hash_rcu_cfg->mode = cfg->mode;
>> +	hash_rcu_cfg->dq_size = params.size;
>> +	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
>> +	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
>> +	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
>> +	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
>> +
>> +	h->hash_rcu_cfg = hash_rcu_cfg;
>> +
>> +	return 0;
>> +}
>> +
>> +static inline void
>> +remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
>> +unsigned i) {
>> +	int ret = free_slot(h, bkt->key_idx[i]);
>> +	if (ret < 0) {
>> +		RTE_LOG(ERR, HASH,
>> +			"%s: could not enqueue free slots in global ring\n",
>> +				__func__);
>> 	}
>> }
>> 
>> @@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct
>> rte_hash *h, const void *key,
>> 	int pos;
>> 	int32_t ret, i;
>> 	uint16_t short_sig;
>> +	uint32_t index = EMPTY_SLOT;
>> +	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
>> 
>> 	short_sig = get_short_sig(sig);
>> 	prim_bucket_idx = get_prim_bucket_index(h, sig); @@ -1555,10
>> +1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const
>> void *key,
>> 
>> /* Search last bucket to see if empty to be recycled */
>> return_bkt:
>> -	if (!last_bkt) {
>> -		__hash_rw_writer_unlock(h);
>> -		return ret;
>> -	}
>> +	if (!last_bkt)
>> +		goto return_key;
>> +
>> 	while (last_bkt->next) {
>> 		prev_bkt = last_bkt;
>> 		last_bkt = last_bkt->next;
>> @@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct
>> rte_hash *h, const void *key,
>> 	/* found empty bucket and recycle */
>> 	if (i == RTE_HASH_BUCKET_ENTRIES) {
>> 		prev_bkt->next = NULL;
>> -		uint32_t index = last_bkt - h->buckets_ext + 1;
>> +		index = last_bkt - h->buckets_ext + 1;
>> 		/* Recycle the empty bkt if
>> 		 * no_free_on_del is disabled.
>> 		 */
>> -		if (h->no_free_on_del)
>> +		if (h->no_free_on_del) {
>> 			/* Store index of an empty ext bkt to be recycled
>> 			 * on calling rte_hash_del_xxx APIs.
>> 			 * When lock free read-write concurrency is enabled,
>> @@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct
>> rte_hash *h, const void *key,
>> 			 * immediately (as readers might be using it still).
>> 			 * Hence freeing of the ext bkt is piggy-backed to
>> 			 * freeing of the key index.
>> +			 * If using external RCU, store this index in an array.
>> 			 */
>> -			h->ext_bkt_to_free[ret] = index;
>> -		else
>> +			if (h->hash_rcu_cfg == NULL)
>> +				h->ext_bkt_to_free[ret] = index;
> 
> 
> [Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?

In DQ mode, ext_bkt_idx is enqueued to the defer queue along with the key_idx and recycled as
part of rte_rcu_qsbr_dq_reclaim(). In SYNC mode, it is recycled after rte_rcu_qsbr_synchronize() succeeds.

> 
>> +		} else
>> 			rte_ring_sp_enqueue_elem(h->free_ext_bkts,
>> &index,
>> 							sizeof(uint32_t));
>> 	}
>> +
>> +return_key:
>> +	/* Using internal RCU QSBR */
>> +	if (h->hash_rcu_cfg) {
>> +		/* Key index where key is stored, adding the first dummy
>> index */
>> +		rcu_dq_entry.key_idx = ret + 1;
>> +		rcu_dq_entry.ext_bkt_idx = index;
>> +		if (h->dq == NULL) {
>> +			/* Wait for quiescent state change if using
>> +			 * RTE_HASH_QSBR_MODE_SYNC
>> +			 */
>> +			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
>> +						 RTE_QSBR_THRID_INVALID);
>> +			__hash_rcu_qsbr_free_resource((void
>> *)((uintptr_t)h),
>> +						      &rcu_dq_entry, 1);
>> +		} else if (h->dq)
>> +			/* Push into QSBR FIFO if using
>> RTE_HASH_QSBR_MODE_DQ */
>> +			if (rte_rcu_qsbr_dq_enqueue(h->dq,
>> &rcu_dq_entry) != 0)
>> +				RTE_LOG(ERR, HASH, "Failed to push QSBR
>> FIFO\n");
>> +	}
>> 	__hash_rw_writer_unlock(h);
>> 	return ret;
>> }
>> @@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct
>> rte_hash *h,
>> 
>> 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
>> 
>> -	unsigned int lcore_id, n_slots;
>> -	struct lcore_cache *cached_free_slots;
>> 	const uint32_t total_entries = h->use_local_cache ?
>> 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
>> + 1
>> 							: h->entries + 1;
>> @@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct
>> rte_hash *h,
>> 		}
>> 	}
>> 
>> -	if (h->use_local_cache) {
>> -		lcore_id = rte_lcore_id();
>> -		cached_free_slots = &h->local_free_slots[lcore_id];
>> -		/* Cache full, need to free it. */
>> -		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
>> -			/* Need to enqueue the free slots in global ring. */
>> -			n_slots = rte_ring_mp_enqueue_burst_elem(h-
>>> free_slots,
>> -						cached_free_slots->objs,
>> -						sizeof(uint32_t),
>> -						LCORE_CACHE_SIZE, NULL);
>> -			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
>> -			cached_free_slots->len -= n_slots;
>> -		}
>> -		/* Put index of new free slot in cache. */
>> -		cached_free_slots->objs[cached_free_slots->len] = key_idx;
>> -		cached_free_slots->len++;
>> -	} else {
>> -		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
>> -						sizeof(uint32_t));
>> -	}
>> +	/* Enqueue slot to cache/ring of free slots. */
>> +	return free_slot(h, key_idx);
>> 
>> -	return 0;
>> }
>> 
>> static inline void
>> diff --git a/lib/librte_hash/rte_cuckoo_hash.h
>> b/lib/librte_hash/rte_cuckoo_hash.h
>> index 345de6bf9cfd..85be49d3bbe7 100644
>> --- a/lib/librte_hash/rte_cuckoo_hash.h
>> +++ b/lib/librte_hash/rte_cuckoo_hash.h
>> @@ -168,6 +168,11 @@ struct rte_hash {
>> 	struct lcore_cache *local_free_slots;
>> 	/**< Local cache per lcore, storing some indexes of the free slots */
>> 
>> +	/* RCU config */
>> +	struct rte_hash_rcu_config *hash_rcu_cfg;
>> +	/**< HASH RCU QSBR configuration structure */
>> +	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
>> +
>> 	/* Fields used in lookup */
>> 
>> 	uint32_t key_len __rte_cache_aligned;
>> @@ -230,4 +235,7 @@ struct queue_node {
>> 	int prev_slot;               /* Parent(slot) in search path */
>> };
>> 
>> +/** @internal Default RCU defer queue entries to reclaim in one go. */
>> +#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
>> +
>> #endif
>> diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h index
>> bff40251bc98..3d28f177f14a 100644
>> --- a/lib/librte_hash/rte_hash.h
>> +++ b/lib/librte_hash/rte_hash.h
>> @@ -15,6 +15,7 @@
>> #include <stddef.h>
>> 
>> #include <rte_compat.h>
>> +#include <rte_rcu_qsbr.h>
>> 
>> #ifdef __cplusplus
>> extern "C" {
>> @@ -45,7 +46,8 @@ extern "C" {
>> /** Flag to disable freeing of key index on hash delete.
>>  * Refer to rte_hash_del_xxx APIs for more details.
>>  * This is enabled by default when
>> RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
>> - * is enabled.
>> + * is enabled. However, if internal RCU is enabled, freeing of internal
>> + * memory/index is done on delete
>>  */
>> #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
>> 
>> @@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void
>> *key, uint32_t key_len,
>> /** Type of function used to compare the hash key. */  typedef int
>> (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
>> 
>> +/**
>> + * Type of function used to free data stored in the key.
>> + * Required when using internal RCU to allow application to free
>> +key-data once
>> + * the key is returned to the the ring of free key-slots.
>> + */
>> +typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
>> +
>> /**
>>  * Parameters used when creating the hash table.
>>  */
>> @@ -81,6 +90,39 @@ struct rte_hash_parameters {
>> 	uint8_t extra_flag;		/**< Indicate if additional parameters
>> are present. */
>> };
>> 
>> +/** RCU reclamation modes */
>> +enum rte_hash_qsbr_mode {
>> +	/** Create defer queue for reclaim. */
>> +	RTE_HASH_QSBR_MODE_DQ = 0,
>> +	/** Use blocking mode reclaim. No defer queue created. */
>> +	RTE_HASH_QSBR_MODE_SYNC
>> +};
>> +
>> +/** HASH RCU QSBR configuration structure. */ struct
>> +rte_hash_rcu_config {
>> +	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
>> +	enum rte_hash_qsbr_mode mode;
>> +	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
>> +	 * '0' for default: create defer queue for reclaim.
>> +	 */
>> +	uint32_t dq_size;
>> +	/**< RCU defer queue size.
>> +	 * default: total hash table entries.
>> +	 */
>> +	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim.
>> */
>> +	uint32_t max_reclaim_size;
>> +	/**< Max entries to reclaim in one go.
>> +	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
>> +	 */
>> +	void *key_data_ptr;
>> +	/**< Pointer passed to the free function. Typically, this is the
>> +	 * pointer to the data structure to which the resource to free
>> +	 * (key-data) belongs. This can be NULL.
>> +	 */
>> +	rte_hash_free_key_data free_key_data_func;
>> +	/**< Function to call to free the resource (key-data). */ };
>> +
>> /** @internal A hash table structure. */  struct rte_hash;
>> 
>> @@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash
>> *h, const void *key, hash_sig_t
>>  * Thread safety can be enabled by setting flag during
>>  * table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>>  * freed by this API. rte_hash_free_key_with_position API must be called
>>  * additionally to free the index associated with the key.
>> @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const
>> void *key);
>>  * Thread safety can be enabled by setting flag during
>>  * table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_add_key_xxx APIs will not be
>>  * freed by this API. rte_hash_free_key_with_position API must be called
>>  * additionally to free the index associated with the key.
>> @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct
>> rte_hash *h, const int32_t position,
>>  * only be called from one thread by default. Thread safety
>>  * can be enabled by setting flag during table creation.
>>  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
>> - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
>> + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
>> + * internal RCU is NOT enabled,
>>  * the key index returned by rte_hash_del_key_xxx APIs must be freed
>>  * using this API. This API should be called after all the readers
>>  * have stopped referencing the entry corresponding to this key.
>> @@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h,
>> const void **keys,
>>  */
>> int32_t
>> rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
>> uint32_t *next);
>> +
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice
>> + *
>> + * Associate RCU QSBR variable with an Hash object.
> [Yipeng]: a Hash object

Copy.

> 
>> + * This API should be called to enable the integrated RCU QSBR support
>> +and
>> + * should be called immediately after creating the Hash object.
>> + *
>> + * @param h
>> + *   the hash object to add RCU QSBR
>> + * @param cfg
>> + *   RCU QSBR configuration
>> + * @return
>> + *   On success - 0
>> + *   On error - 1 with error code set in rte_errno.
>> + *   Possible rte_errno codes are:
>> + *   - EINVAL - invalid pointer
>> + *   - EEXIST - already added QSBR
>> + *   - ENOMEM - memory allocation failure
>> + */
>> +__rte_experimental
>> +int rte_hash_rcu_qsbr_add(struct rte_hash *h,
>> +				struct rte_hash_rcu_config *cfg);
>> #ifdef __cplusplus
>> }
>> #endif
>> diff --git a/lib/librte_hash/version.map b/lib/librte_hash/version.map index
>> c0db81014ff9..c6d73080f478 100644
>> --- a/lib/librte_hash/version.map
>> +++ b/lib/librte_hash/version.map
>> @@ -36,5 +36,5 @@ EXPERIMENTAL {
>> 	rte_hash_lookup_with_hash_bulk;
>> 	rte_hash_lookup_with_hash_bulk_data;
>> 	rte_hash_max_key_id;
>> -
>> +	rte_hash_rcu_qsbr_add;
>> };
>> --
>> 2.17.1
> [Yipeng]:
> 
> Hi, Dharmik, thanks for the work! It generally looks good.
> Just some minor issues to address and one question for the ext table inlined.

Thank you for the comments, Yipeng! I will update the patches.

> 
> Thanks!


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-21  3:54           ` Wang, Yipeng1
@ 2020-10-21  4:55             ` Dharmik Thakkar
  2020-10-21 22:34             ` Honnappa Nagarahalli
  1 sibling, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21  4:55 UTC (permalink / raw)
  To: Wang, Yipeng1; +Cc: Gobriel, Sameh, Richardson, Bruce, dev, nd



> On Oct 20, 2020, at 10:54 PM, Wang, Yipeng1 <yipeng1.wang@intel.com> wrote:
> 
>> -----Original Message-----
>> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Sent: Tuesday, October 20, 2020 9:13 AM
>> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
>> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
>> <dharmik.thakkar@arm.com>
>> Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
>> 
>> Add functional and performance tests for the integrated RCU QSBR.
>> 
>> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
>> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
>> ---
>> app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
>> app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
>> 2 files changed, 556 insertions(+), 4 deletions(-)
>> 
>> diff --git a/app/test/test_hash.c b/app/test/test_hash.c index
>> 990a1815f893..22b47b3e7728 100644
>> --- a/app/test/test_hash.c
>> +++ b/app/test/test_hash.c
>> @@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10,
>> 11, 15, 16, 21,
>> 	}								\
>> } while(0)
>> 
>> -#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
>> 	\
>> +#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
>> 	if (cond) {							\
>> 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
>> \
>> 		if (handle) rte_fbk_hash_free(handle);			\
>> @@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8,
>> 10, 11, 15, 16, 21,
>> 	}								\
>> } while(0)
>> 
>> +#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
>> 	\
>> +	if (cond) {							\
>> +		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
>> \
>> +		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
>> 	\
>> +			writer_done = 1;				\
>> +			/* Wait until reader exited. */			\
>> +			rte_eal_mp_wait_lcore();			\
>> +		}							\
>> +		if (g_handle) rte_hash_free(g_handle);			\
>> +		if (g_qsv) rte_free(g_qsv);				\
>> +		return -1;						\
>> +	}								\
>> +} while(0)
>> +
>> /* 5-tuple key type */
>> struct flow_key {
>> 	uint32_t ip_src;
>> @@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
>> 	return ret;
>> }
>> 
>> +static struct rte_hash *g_handle;
>> +static struct rte_rcu_qsbr *g_qsv;
>> +static volatile uint8_t writer_done;
>> +struct flow_key g_rand_keys[9];
>> +/*
>> + * rte_hash_rcu_qsbr_add positive and negative tests.
>> + *  - Add RCU QSBR variable to Hash
>> + *  - Add another RCU QSBR variable to Hash
>> + *  - Check returns
>> + */
>> +static int
>> +test_hash_rcu_qsbr_add(void)
>> +{
>> +	size_t sz;
>> +	struct rte_rcu_qsbr *qsv2 = NULL;
>> +	int32_t status;
>> +	struct rte_hash_rcu_config rcu_cfg = {0};
>> +
>> +	struct rte_hash_parameters params;
>> +
>> +	printf("\n# Running RCU QSBR add tests\n");
>> +	memcpy(&params, &ut_params, sizeof(params));
>> +	params.name = "test_hash_rcu_qsbr_add";
>> +	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
>> |
>> +
>> 	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
>> +	g_handle = rte_hash_create(&params);
>> +	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation
>> failed");
>> +
>> +	/* Create RCU QSBR variable */
>> +	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
>> +	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
>> +					RTE_CACHE_LINE_SIZE,
>> SOCKET_ID_ANY);
>> +	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
>> +				 "RCU QSBR variable creation failed");
>> +
>> +	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
> [Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
> I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.

Yes, I think this can be hidden within rte_hash_reset() and rte_hash_rcu_qsbr_add()

> 
>> +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
>> +				 "RCU QSBR variable initialization failed");
>> +
>> +	rcu_cfg.v = g_qsv;
>> +	/* Invalid QSBR mode */
>> +	rcu_cfg.mode = 2;
> [Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?

Yes, I can use 0xxff

> 
>> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
>> +	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
>> +failed");
>> +
>> +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
> [Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?

It is available within rte_hash.h

>  
> 
>> +	/* Attach RCU QSBR to hash table */
>> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
>> +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
>> +				 "Attach RCU QSBR to hash table failed");
>> +
>> +	/* Create and attach another RCU QSBR to hash table */
>> +	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
>> +					RTE_CACHE_LINE_SIZE,
>> SOCKET_ID_ANY);
>> +	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
>> +				 "RCU QSBR variable creation failed");
>> +
>> +	rcu_cfg.v = qsv2;
>> +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
>> +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
>> +	rte_free(qsv2);
>> +	RETURN_IF_ERROR_RCU_QSBR(status == 0,
>> +			"Attach RCU QSBR to hash table succeeded where
>> failure"
>> +			" is expected");
>> +
>> +	rte_hash_free(g_handle);
>> +	rte_free(g_qsv);
>> +
>> +	return 0;
>> +}
> <...>


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-21  3:54           ` Wang, Yipeng1
  2020-10-21  4:55             ` Dharmik Thakkar
@ 2020-10-21 22:34             ` Honnappa Nagarahalli
  1 sibling, 0 replies; 49+ messages in thread
From: Honnappa Nagarahalli @ 2020-10-21 22:34 UTC (permalink / raw)
  To: Wang, Yipeng1, Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce
  Cc: dev, nd, Honnappa Nagarahalli, nd

<snip>

> >
> > Add functional and performance tests for the integrated RCU QSBR.
> >
> > Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> > Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > ---
> >  app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
> >  app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
> >  2 files changed, 556 insertions(+), 4 deletions(-)
> >
> > diff --git a/app/test/test_hash.c b/app/test/test_hash.c index
> > 990a1815f893..22b47b3e7728 100644
> > --- a/app/test/test_hash.c
> > +++ b/app/test/test_hash.c
> > @@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5,
> > 6, 7, 8, 10, 11, 15, 16, 21,
> >  	}								\
> >  } while(0)
> >
> > -#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
> > 	\
> > +#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
> >  	if (cond) {							\
> >  		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
> >  		if (handle) rte_fbk_hash_free(handle);			\
> > @@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5,
> > 6, 7, 8, 10, 11, 15, 16, 21,
> >  	}								\
> >  } while(0)
> >
> > +#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
> > 	\
> > +	if (cond) {							\
> > +		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
> > \
> > +		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
> > 	\
> > +			writer_done = 1;				\
> > +			/* Wait until reader exited. */			\
> > +			rte_eal_mp_wait_lcore();			\
> > +		}							\
> > +		if (g_handle) rte_hash_free(g_handle);			\
> > +		if (g_qsv) rte_free(g_qsv);				\
> > +		return -1;						\
> > +	}								\
> > +} while(0)
> > +
> >  /* 5-tuple key type */
> >  struct flow_key {
> >  	uint32_t ip_src;
> > @@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
> >  	return ret;
> >  }
> >
> > +static struct rte_hash *g_handle;
> > +static struct rte_rcu_qsbr *g_qsv;
> > +static volatile uint8_t writer_done;
> > +struct flow_key g_rand_keys[9];
> > +/*
> > + * rte_hash_rcu_qsbr_add positive and negative tests.
> > + *  - Add RCU QSBR variable to Hash
> > + *  - Add another RCU QSBR variable to Hash
> > + *  - Check returns
> > + */
> > +static int
> > +test_hash_rcu_qsbr_add(void)
> > +{
> > +	size_t sz;
> > +	struct rte_rcu_qsbr *qsv2 = NULL;
> > +	int32_t status;
> > +	struct rte_hash_rcu_config rcu_cfg = {0};
> > +
> > +	struct rte_hash_parameters params;
> > +
> > +	printf("\n# Running RCU QSBR add tests\n");
> > +	memcpy(&params, &ut_params, sizeof(params));
> > +	params.name = "test_hash_rcu_qsbr_add";
> > +	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
> > |
> > +
> > 	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
> > +	g_handle = rte_hash_create(&params);
> > +	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation
> > failed");
> > +
> > +	/* Create RCU QSBR variable */
> > +	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
> > +	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
> > +					RTE_CACHE_LINE_SIZE,
> > SOCKET_ID_ANY);
> > +	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
> > +				 "RCU QSBR variable creation failed");
> > +
> > +	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
> [Wang, Yipeng] It reminds me that could we hide this function in the
> rte_cuckoo_hash.c as well?
> I saw most of the rcu related functions are hidden in the hash implementation, it
> would be less confusing if we hide this one as well.
The functionality required for resource reclamation is split into various parts [1]. The functionality of creating the RCU variable, initializing it etc is left to the application. The application has better knowledge of the data plane threads, which ones are using the (hash) library etc. Hence it is better to keep this in the application.

[1] http://doc.dpdk.org/guides/prog_guide/rcu_lib.html#resource-reclamation-framework-for-dpdk

> 
> > +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
> > +				 "RCU QSBR variable initialization failed");
> > +
> > +	rcu_cfg.v = g_qsv;
> > +	/* Invalid QSBR mode */
> > +	rcu_cfg.mode = 2;
> [Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large
> number like 0xff?
> 
> > +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> > +	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
> > +failed");
> > +
> > +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
> [Wang, Yipeng] This reminds me that if there is an explanation on the difference
> of the two modes for users to easy to choose?
> 
> > +	/* Attach RCU QSBR to hash table */
> > +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> > +	RETURN_IF_ERROR_RCU_QSBR(status != 0,
> > +				 "Attach RCU QSBR to hash table failed");
> > +
> > +	/* Create and attach another RCU QSBR to hash table */
> > +	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
> > +					RTE_CACHE_LINE_SIZE,
> > SOCKET_ID_ANY);
> > +	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
> > +				 "RCU QSBR variable creation failed");
> > +
> > +	rcu_cfg.v = qsv2;
> > +	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
> > +	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
> > +	rte_free(qsv2);
> > +	RETURN_IF_ERROR_RCU_QSBR(status == 0,
> > +			"Attach RCU QSBR to hash table succeeded where
> > failure"
> > +			" is expected");
> > +
> > +	rte_hash_free(g_handle);
> > +	rte_free(g_qsv);
> > +
> > +	return 0;
> > +}
> <...>

^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR
  2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
                           ` (3 preceding siblings ...)
  2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-21 22:50         ` Dharmik Thakkar
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
                             ` (4 more replies)
  4 siblings, 5 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21 22:50 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock
free algorithm.

Resource reclamation implementation was split from the original
series, and has already been part of RCU library. Rework the series
to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1]
to understand various aspects of integrating RCU library
into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to
register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the
integration with RCU.
---
v6:
 - Update documentation
 - Remove hard-coded value

v5:
 - Enable 'rcu' for Windows

v4:
 - Fix clang compilation issues

v3:
 - Add documentation
 - Add unit tests

v2:
 - Remove defer queue related functions and use resource reclamation
   APIs from the RCU QSBR library instead

 - Remove patch (net/ixgbe: avoid multpile definitions of 'bool')
   from the series as it is already accepted

Dharmik Thakkar (4):
  rcu: build on Windows
  lib/hash: integrate RCU QSBR
  test/hash: replace rte atomic with C11 atomic APIs
  test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  12 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 303 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/version.map            |   2 +-
 lib/meson.build                        |   1 +
 9 files changed, 920 insertions(+), 133 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
@ 2020-10-21 22:50           ` Dharmik Thakkar
  2020-10-22  9:05             ` David Marchand
  2020-10-22 18:35             ` Kadam, Pallavi
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
                             ` (3 subsequent siblings)
  4 siblings, 2 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21 22:50 UTC (permalink / raw)
  Cc: dev, nd, Dharmik Thakkar

Build the lib for Windows.

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Tested-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 lib/meson.build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/meson.build b/lib/meson.build
index dd55b5cb53e4..1bb019720c6a 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -41,6 +41,7 @@ if is_windows
 		'telemetry',
 		'eal',
 		'ring',
+		'rcu',
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
@ 2020-10-21 22:50           ` Dharmik Thakkar
  2020-10-23 21:46             ` Dharmik Thakkar
  2020-10-23 22:08             ` Wang, Yipeng1
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
                             ` (2 subsequent siblings)
  4 siblings, 2 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21 22:50 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd, Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources
when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use
lock free algorithm.
Refer to RCU documentation to understand various aspects of
integrating RCU library into other libraries.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>
---
 doc/guides/prog_guide/hash_lib.rst |  12 +-
 lib/librte_hash/meson.build        |   1 +
 lib/librte_hash/rte_cuckoo_hash.c  | 303 ++++++++++++++++++++++-------
 lib/librte_hash/rte_cuckoo_hash.h  |   8 +
 lib/librte_hash/rte_hash.h         |  77 +++++++-
 lib/librte_hash/version.map        |   2 +-
 6 files changed, 327 insertions(+), 76 deletions(-)

diff --git a/doc/guides/prog_guide/hash_lib.rst b/doc/guides/prog_guide/hash_lib.rst
index d06c7de2ead1..fcdc3fa7a9ab 100644
--- a/doc/guides/prog_guide/hash_lib.rst
+++ b/doc/guides/prog_guide/hash_lib.rst
@@ -102,6 +102,10 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   For example, rte_hash_rcu_qsbr_add() need to be called to use the integrated RCU mechanism.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +113,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +176,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +290,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build
index 6ab46ae9d768..0977a63fd279 100644
--- a/lib/librte_hash/meson.build
+++ b/lib/librte_hash/meson.build
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c
index aad0c965be5e..e85e93828e3f 100644
--- a/lib/librte_hash/rte_cuckoo_hash.c
+++ b/lib/librte_hash/rte_cuckoo_hash.c
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)
 {
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,38 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1017,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1054,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1163,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1451,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
-
+	struct lcore_cache *cached_free_slots = NULL;
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1467,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1686,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1722,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */
 return_bkt:
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1737,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1749,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using RTE_HASH_QSBR_MODE_DQ */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1825,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1842,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 
 static inline void
diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h
index 345de6bf9cfd..85be49d3bbe7 100644
--- a/lib/librte_hash/rte_cuckoo_hash.h
+++ b/lib/librte_hash/rte_cuckoo_hash.h
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h
index bff40251bc98..50419c369616 100644
--- a/lib/librte_hash/rte_hash.h
+++ b/lib/librte_hash/rte_hash.h
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim. */
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with a Hash object.
+ * This API should be called to enable the integrated RCU QSBR support and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_hash/version.map b/lib/librte_hash/version.map
index c0db81014ff9..c6d73080f478 100644
--- a/lib/librte_hash/version.map
+++ b/lib/librte_hash/version.map
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v6 3/4] test/hash: replace rte atomic with C11 atomic APIs
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2020-10-21 22:50           ` Dharmik Thakkar
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
  2020-10-24  9:09           ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " David Marchand
  4 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21 22:50 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Acked-by: Yipeng Wang <yipeng1.wang@intel.com>
---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)

diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 889799865c7b..328fa5116f65 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
                             ` (2 preceding siblings ...)
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
@ 2020-10-21 22:50           ` Dharmik Thakkar
  2020-10-23 22:11             ` Wang, Yipeng1
  2020-10-24  9:09           ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " David Marchand
  4 siblings, 1 reply; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-21 22:50 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson; +Cc: dev, nd, Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)

diff --git a/app/test/test_hash.c b/app/test/test_hash.c
index 990a1815f893..cf299982aedc 100644
--- a/app/test/test_hash.c
+++ b/app/test/test_hash.c
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 0xff;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
diff --git a/app/test/test_hash_readwrite_lf_perf.c b/app/test/test_hash_readwrite_lf_perf.c
index 328fa5116f65..caf18b69c1d4 100644
--- a/app/test/test_hash_readwrite_lf_perf.c
+++ b/app/test/test_hash_readwrite_lf_perf.c
@@ -48,6 +48,9 @@
 #define WRITE_EXT_BKT 2
 
 #define NUM_TEST 3
+
+#define QSBR_REPORTING_INTERVAL 1024
+
 static unsigned int rwc_core_cnt[NUM_TEST] = {1, 2, 4};
 
 struct rwc_perf {
@@ -58,6 +61,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,6 +88,8 @@ static struct {
 
 static uint64_t gread_cycles;
 static uint64_t greads;
+static uint64_t gwrite_cycles;
+static uint64_t gwrites;
 
 static volatile uint8_t writer_done;
 
@@ -1044,7 +1050,7 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 /*
  * Test lookup perf for multi-writer:
  * Reader(s) lookup keys present in the table and likely on the shift-path while
- * Writers add keys causing key-shiftsi.
+ * Writers add keys causing key-shifts.
  * Writers are running in parallel, on different data plane cores.
  */
 static int
@@ -1223,6 +1229,163 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static struct rte_rcu_qsbr *rv;
+
+/*
+ * Reader thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	unsigned int i, j;
+	uint32_t num_keys = tbl_rwc_test_param.count_keys_no_ks
+				- QSBR_REPORTING_INTERVAL;
+	uint32_t *keys = tbl_rwc_test_param.keys_no_ks;
+	uint32_t lcore_id = rte_lcore_id();
+	RTE_SET_USED(arg);
+
+	(void)rte_rcu_qsbr_thread_register(rv, lcore_id);
+	rte_rcu_qsbr_thread_online(rv, lcore_id);
+	do {
+		for (i = 0; i < num_keys; i += j) {
+			for (j = 0; j < QSBR_REPORTING_INTERVAL; j++)
+				rte_hash_lookup(tbl_rwc_test_param.h,
+						keys + i + j);
+			/* Update quiescent state counter */
+			rte_rcu_qsbr_quiescent(rv, lcore_id);
+		}
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_offline(rv, lcore_id);
+	(void)rte_rcu_qsbr_thread_unregister(rv, lcore_id);
+
+	return 0;
+}
+
+/*
+ * Writer thread using rte_hash data structure with RCU
+ */
+static int
+test_hash_rcu_qsbr_writer(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	__atomic_fetch_add(&gwrite_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&gwrites, tbl_rwc_test_param.single_insert,
+			   __ATOMIC_RELAXED);
+	return 0;
+}
+
+/*
+ * Writer perf test with RCU QSBR in DQ mode:
+ * Writer(s) delete and add keys in the table.
+ * Readers lookup keys in the hash table
+ */
+static int
+test_hash_rcu_qsbr_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	struct rte_hash_rcu_config rcu_config = {0};
+	uint32_t sz;
+	uint8_t pos_core;
+
+	printf("\nTest: Writer perf with integrated RCU\n");
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	rv = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	rcu_config.v = rv;
+
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, &rcu_config) < 0) {
+		printf("RCU init in hash failed\n");
+		goto err;
+	}
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 3)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		__atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED);
+		__atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		rte_rcu_qsbr_init(rv, RTE_MAX_LCORE);
+
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+
+		/* Launch 2 readers */
+		for (i = 1; i <= 2; i++)
+			rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+					      enabled_core_ids[i]);
+		pos_core = 0;
+		/* Launch writer(s) */
+		for (; i <= rwc_core_cnt[n] + 2; i++) {
+			rte_eal_remote_launch(test_hash_rcu_qsbr_writer,
+				(void *)(uintptr_t)pos_core,
+				enabled_core_ids[i]);
+			pos_core++;
+		}
+
+		/* Wait for writers to complete */
+		for (i = 3; i <= rwc_core_cnt[n] + 2; i++)
+			rte_eal_wait_lcore(enabled_core_ids[i]);
+
+		writer_done = 1;
+
+		/* Wait for readers to complete */
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			__atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) /
+			__atomic_load_n(&gwrites, __ATOMIC_RELAXED);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return 0;
+
+err:
+	writer_done = 1;
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(rv);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_perf_main(void)
 {
@@ -1235,7 +1398,6 @@ test_hash_readwrite_lf_perf_main(void)
 	int rwc_lf = 0;
 	int htm;
 	int ext_bkt = 0;
-
 	if (rte_lcore_count() < 2) {
 		printf("Not enough cores for hash_readwrite_lf_perf_autotest, expecting at least 2\n");
 		return TEST_SKIPPED;
@@ -1255,7 +1417,6 @@ test_hash_readwrite_lf_perf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1282,6 +1443,9 @@ test_hash_readwrite_lf_perf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		if (test_hash_rcu_qsbr_writer_perf(&rwc_lf_results, rwc_lf,
+						   htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
-- 
2.17.1


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
@ 2020-10-22  9:05             ` David Marchand
  2020-10-22 18:35             ` Kadam, Pallavi
  1 sibling, 0 replies; 49+ messages in thread
From: David Marchand @ 2020-10-22  9:05 UTC (permalink / raw)
  To: Dharmik Thakkar, Dmitry Kozlyuk, Narcisa Ana Maria Vasile,
	Dmitry Malloy (MESHCHANINOV),
	Pallavi Kadam
  Cc: dev, nd

On Thu, Oct 22, 2020 at 12:51 AM Dharmik Thakkar
<dharmik.thakkar@arm.com> wrote:
>
> Build the lib for Windows.
>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Tested-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
>  lib/meson.build | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/lib/meson.build b/lib/meson.build
> index dd55b5cb53e4..1bb019720c6a 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -41,6 +41,7 @@ if is_windows
>                 'telemetry',
>                 'eal',
>                 'ring',
> +               'rcu',
>                 'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
>                 'cmdline',
>                 'hash',
> --
> 2.17.1
>

I only saw a test from Dmitry, pinging other Windows maintainers to
get a clear ack.
Thanks.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
  2020-10-22  9:05             ` David Marchand
@ 2020-10-22 18:35             ` Kadam, Pallavi
  1 sibling, 0 replies; 49+ messages in thread
From: Kadam, Pallavi @ 2020-10-22 18:35 UTC (permalink / raw)
  To: Dharmik Thakkar; +Cc: dev, nd


On 10/21/2020 3:50 PM, Dharmik Thakkar wrote:
> Build the lib for Windows.
>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Tested-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---

Builds successfully on Windows.

Tested-by: Pallavi Kadam <pallavi.kadam@intel.com>


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
@ 2020-10-23 21:46             ` Dharmik Thakkar
  2020-10-23 22:08             ` Wang, Yipeng1
  1 sibling, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-23 21:46 UTC (permalink / raw)
  To: Yipeng Wang, Sameh Gobriel, Bruce Richardson, Ray Kinsella, Neil Horman
  Cc: dev, nd

Hi Yipeng,

I have addressed the comments. Can you please take a look?

Thank you!

> On Oct 21, 2020, at 5:50 PM, Dharmik Thakkar <Dharmik.Thakkar@arm.com> wrote:
> 
> Currently, users have to use external RCU mechanisms to free resources
> when using lock free hash algorithm.
> 
> Integrate RCU QSBR process to make it easier for the applications to use
> lock free algorithm.
> Refer to RCU documentation to understand various aspects of
> integrating RCU library into other libraries.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> Acked-by: Ray Kinsella <mdr@ashroe.eu>
> ---


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
  2020-10-23 21:46             ` Dharmik Thakkar
@ 2020-10-23 22:08             ` Wang, Yipeng1
  1 sibling, 0 replies; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-10-23 22:08 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce, Ray Kinsella,
	Neil Horman
  Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Wednesday, October 21, 2020 3:50 PM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [PATCH v6 2/4] lib/hash: integrate RCU QSBR
> 
> Currently, users have to use external RCU mechanisms to free resources when
> using lock free hash algorithm.
> 
> Integrate RCU QSBR process to make it easier for the applications to use lock
> free algorithm.
> Refer to RCU documentation to understand various aspects of integrating RCU
> library into other libraries.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> Acked-by: Ray Kinsella <mdr@ashroe.eu>
> ---
[Wang, Yipeng] 
Thanks for revising the code.
Please check the checkpatch reported coding style issues to fix.
After that
Acked-by: Yipeng Wang <yipeng1.wang@intel.com>



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-23 22:11             ` Wang, Yipeng1
  0 siblings, 0 replies; 49+ messages in thread
From: Wang, Yipeng1 @ 2020-10-23 22:11 UTC (permalink / raw)
  To: Dharmik Thakkar, Gobriel, Sameh, Richardson, Bruce; +Cc: dev, nd

> -----Original Message-----
> From: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Sent: Wednesday, October 21, 2020 3:50 PM
> To: Wang, Yipeng1 <yipeng1.wang@intel.com>; Gobriel, Sameh
> <sameh.gobriel@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; nd@arm.com; Dharmik Thakkar
> <dharmik.thakkar@arm.com>
> Subject: [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR
> 
> Add functional and performance tests for the integrated RCU QSBR.
> 
> Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> Signed-off-by: Dharmik Thakkar <dharmik.thakkar@arm.com>
> Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
> ---
[Wang, Yipeng] 
Also please confirm the checkpatch report see if it is not a false-positive.
Then:
Acked-by: Yipeng Wang <yipeng1.wang@intel.com>


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR
  2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
                             ` (3 preceding siblings ...)
  2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
@ 2020-10-24  9:09           ` David Marchand
  2020-10-26 13:56             ` Dharmik Thakkar
  4 siblings, 1 reply; 49+ messages in thread
From: David Marchand @ 2020-10-24  9:09 UTC (permalink / raw)
  To: Dharmik Thakkar
  Cc: dev, nd, Honnappa Nagarahalli,
	Ruifeng Wang (Arm Technology China),
	Wang, Yipeng1, Dmitry Kozlyuk, Pallavi Kadam, Thomas Monjalon

On Thu, Oct 22, 2020 at 12:51 AM Dharmik Thakkar
<dharmik.thakkar@arm.com> wrote:
>
> Integrate RCU QSBR to make it easier for the applications to use lock
> free algorithm.
>
> Resource reclamation implementation was split from the original
> series, and has already been part of RCU library. Rework the series
> to base hash integration on RCU reclamation APIs.
>
> Refer 'Resource reclamation framework for DPDK' available at [1]
> to understand various aspects of integrating RCU library
> into other libraries.
>
> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>
> Introduce a new API rte_hash_rcu_qsbr_add for application to
> register a RCU variable that hash library will use.
>
> Functional tests and performance tests are added to cover the
> integration with RCU.

Fixed some style issues/checkpatch warnings.
The unit test code especially is not really checkpatch compliant,
maybe worth a separate cleanup later.

Removed unrelated changes (like removing empty lines, fixing typos in
unrelated parts).

Series applied, thanks.


Little note: checkpatch now has a check on repeated words better than
a script of mine:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
Probably worth it for people to update their checkpatch.pl.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR
  2020-10-24  9:09           ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " David Marchand
@ 2020-10-26 13:56             ` Dharmik Thakkar
  0 siblings, 0 replies; 49+ messages in thread
From: Dharmik Thakkar @ 2020-10-26 13:56 UTC (permalink / raw)
  To: David Marchand
  Cc: dev, nd, Honnappa Nagarahalli, Ruifeng Wang, Wang, Yipeng1,
	Dmitry Kozlyuk, Pallavi Kadam, thomas



> On Oct 24, 2020, at 4:09 AM, David Marchand <david.marchand@redhat.com> wrote:
> 
> On Thu, Oct 22, 2020 at 12:51 AM Dharmik Thakkar
> <dharmik.thakkar@arm.com> wrote:
>> 
>> Integrate RCU QSBR to make it easier for the applications to use lock
>> free algorithm.
>> 
>> Resource reclamation implementation was split from the original
>> series, and has already been part of RCU library. Rework the series
>> to base hash integration on RCU reclamation APIs.
>> 
>> Refer 'Resource reclamation framework for DPDK' available at [1]
>> to understand various aspects of integrating RCU library
>> into other libraries.
>> 
>> [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
>> 
>> Introduce a new API rte_hash_rcu_qsbr_add for application to
>> register a RCU variable that hash library will use.
>> 
>> Functional tests and performance tests are added to cover the
>> integration with RCU.
> 
> Fixed some style issues/checkpatch warnings.
> The unit test code especially is not really checkpatch compliant,
> maybe worth a separate cleanup later.
> 
> Removed unrelated changes (like removing empty lines, fixing typos in
> unrelated parts).
> 
> Series applied, thanks.
> 
> 
> Little note: checkpatch now has a check on repeated words better than
> a script of mine:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
> Probably worth it for people to update their checkpatch.pl.
> 

Thank you, David! Appreciate your help.

> 
> -- 
> David Marchand


^ permalink raw reply	[flat|nested] 49+ messages in thread

end of thread, other threads:[~2020-10-26 13:57 UTC | newest]

Thread overview: 49+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-01  6:58 [dpdk-dev] [RFC 0/3] RCU integration with hash library Dharmik Thakkar
2019-09-01  6:58 ` [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool' Dharmik Thakkar
2019-09-05 15:26   ` Stephen Hemminger
2019-09-05 20:10     ` Dharmik Thakkar
2019-09-05 20:23       ` Stephen Hemminger
2019-09-01  6:58 ` [dpdk-dev] [RFC 2/3] lib/hash: integrate RCU QSBR Dharmik Thakkar
2019-09-01  6:58 ` [dpdk-dev] [RFC 3/3] test/hash: add tests for integrated " Dharmik Thakkar
2019-09-05 21:31 ` [dpdk-dev] [RFC 0/3] RCU integration with hash library Wang, Yipeng1
2020-08-19  4:05 ` [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR Dharmik Thakkar
2020-08-25 19:59   ` Dharmik Thakkar
2020-08-31 20:47   ` Wang, Yipeng1
2020-09-01 22:01     ` Dharmik Thakkar
2020-10-16 13:53       ` David Marchand
2020-10-16 15:04         ` Dharmik Thakkar
2020-10-16 17:38   ` [dpdk-dev] [PATCH 0/3] hash: " Dharmik Thakkar
2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 1/3] lib/hash: " Dharmik Thakkar
2020-10-19  9:43       ` Kinsella, Ray
2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
2020-10-16 17:38     ` [dpdk-dev] [PATCH v3 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
2020-10-19 14:48     ` [dpdk-dev] [PATCH 0/3] hash: integrate " David Marchand
2020-10-19 16:35     ` [dpdk-dev] [PATCH v4 " Dharmik Thakkar
2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 1/3] lib/hash: " Dharmik Thakkar
2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 2/3] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
2020-10-19 16:35       ` [dpdk-dev] [PATCH v4 3/3] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
2020-10-19 21:05       ` [dpdk-dev] [PATCH v4 0/3] hash: integrate " David Marchand
2020-10-20  4:08         ` Dharmik Thakkar
2020-10-20 16:12       ` [dpdk-dev] [PATCH v5 0/4] " Dharmik Thakkar
2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 1/4] rcu: build on Windows Dharmik Thakkar
2020-10-20 16:12         ` [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
2020-10-21  2:42           ` Wang, Yipeng1
2020-10-21  4:52             ` Dharmik Thakkar
2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
2020-10-21  2:52           ` Wang, Yipeng1
2020-10-20 16:13         ` [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
2020-10-21  3:54           ` Wang, Yipeng1
2020-10-21  4:55             ` Dharmik Thakkar
2020-10-21 22:34             ` Honnappa Nagarahalli
2020-10-21 22:50         ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " Dharmik Thakkar
2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows Dharmik Thakkar
2020-10-22  9:05             ` David Marchand
2020-10-22 18:35             ` Kadam, Pallavi
2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR Dharmik Thakkar
2020-10-23 21:46             ` Dharmik Thakkar
2020-10-23 22:08             ` Wang, Yipeng1
2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 3/4] test/hash: replace rte atomic with C11 atomic APIs Dharmik Thakkar
2020-10-21 22:50           ` [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR Dharmik Thakkar
2020-10-23 22:11             ` Wang, Yipeng1
2020-10-24  9:09           ` [dpdk-dev] [PATCH v6 0/4] hash: integrate " David Marchand
2020-10-26 13:56             ` Dharmik Thakkar

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).