[dpdk-dev] [PATCH v4 0/7] librte_table: add key_mask parameter to

[dpdk-dev] [PATCH v4 0/7] librte_table: add key_mask parameter to

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patchset links to ABI change announced for librte_table.
The key_mask parameters has been added to the hash table
parameter structure for 8-byte key and 16-byte key extendible
bucket and LRU tables.

v2:
*updated release note

v3:
*merged release note with source code patch
*fixed build error: added missing symbol to
librte_table/rte_table_version.map

v4:
*modified rte_prefetch offsets to improve hash/lru table
lookup performance. 

Acked-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

Fan Zhang (7):
  librte_table: add key_mask parameter to 8- and 16-bytes key hash
    parameters
  librte_table: add 16 byte hash table operations with computed lookup
  app/test: modify app/test_table_combined and app/test_table_tables
  app/test-pipeline: modify pipeline test
  example/ip_pipeline: add parse_hex_string for internal use
  example/ip_pipeline/pipeline: update flow_classification pipeline
  librte_table: performance improvement on rte_prefetch offset

 app/test-pipeline/pipeline_hash.c                  |   4 +
 app/test/test_table_combined.c                     |   5 +-
 app/test/test_table_tables.c                       |   6 +-
 doc/guides/rel_notes/deprecation.rst               |   4 -
 doc/guides/rel_notes/release_2_2.rst               |   4 +
 examples/ip_pipeline/config_parse.c                |  52 +++
 .../pipeline/pipeline_flow_classification_be.c     |  56 ++-
 examples/ip_pipeline/pipeline_be.h                 |   4 +
 lib/librte_table/rte_table_hash.h                  |  20 +
 lib/librte_table/rte_table_hash_ext.c              |  10 +-
 lib/librte_table/rte_table_hash_key16.c            | 446 +++++++++++++++++++--
 lib/librte_table/rte_table_hash_key32.c            |  35 +-
 lib/librte_table/rte_table_hash_key8.c             | 105 +++--
 lib/librte_table/rte_table_hash_lru.c              |  10 +-
 lib/librte_table/rte_table_version.map             |   7 +
 15 files changed, 673 insertions(+), 95 deletions(-)

-- 
2.1.0

[dpdk-dev] [PATCH v4 1/7] librte_table: add key_mask parameter to 8- and 16-bytes key hash parameters

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patch relates to ABI change proposed for librte_table.
The key_mask parameter is added for 8-byte and 16-byte
key extendible bucket and LRU tables.The release notes
is updated and the deprecation notice is removed.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
Signed-off-by: Jasvinder Singh <jasvinder.singh@intel.com>
---
 doc/guides/rel_notes/deprecation.rst    |  4 ---
 doc/guides/rel_notes/release_2_2.rst    |  4 +++
 lib/librte_table/rte_table_hash.h       | 12 ++++++++
 lib/librte_table/rte_table_hash_key16.c | 52 ++++++++++++++++++++++++++-----
 lib/librte_table/rte_table_hash_key8.c  | 54 +++++++++++++++++++++++++++------
 lib/librte_table/rte_table_version.map  |  7 +++++
 6 files changed, 112 insertions(+), 21 deletions(-)

diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index a391ff0..16ec9f8 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -44,10 +44,6 @@ Deprecation Notices
 * librte_table: New functions for table entry bulk add/delete will be added
   to the table operations structure.
 
-* librte_table hash: Key mask parameter will be added to the hash table
-  parameter structure for 8-byte key and 16-byte key extendible bucket and
-  LRU tables.
-
 * librte_pipeline: The prototype for the pipeline input port, output port
   and table action handlers will be updated:
   the pipeline parameter will be added, the packets mask parameter will be
diff --git a/doc/guides/rel_notes/release_2_2.rst b/doc/guides/rel_notes/release_2_2.rst
index 128f956..7beba40 100644
--- a/doc/guides/rel_notes/release_2_2.rst
+++ b/doc/guides/rel_notes/release_2_2.rst
@@ -132,6 +132,10 @@ ABI Changes
 * librte_cfgfile: Allow longer names and values by increasing the constants
   CFG_NAME_LEN and CFG_VALUE_LEN to 64 and 256 respectively.
 
+* librte_table hash: The key mask parameter is added to the hash table
+  parameter structure for 8-byte key and 16-byte key extendible bucket
+  and LRU tables.
+
 
 Shared Library Versions
 -----------------------
diff --git a/lib/librte_table/rte_table_hash.h b/lib/librte_table/rte_table_hash.h
index 9181942..e2c60e1 100644
--- a/lib/librte_table/rte_table_hash.h
+++ b/lib/librte_table/rte_table_hash.h
@@ -196,6 +196,9 @@ struct rte_table_hash_key8_lru_params {
 
 	/** Byte offset within packet meta-data where the key is located */
 	uint32_t key_offset;
+
+	/** Bit-mask to be AND-ed to the key on lookup */
+	uint8_t *key_mask;
 };
 
 /** LRU hash table operations for pre-computed key signature */
@@ -226,6 +229,9 @@ struct rte_table_hash_key8_ext_params {
 
 	/** Byte offset within packet meta-data where the key is located */
 	uint32_t key_offset;
+
+	/** Bit-mask to be AND-ed to the key on lookup */
+	uint8_t *key_mask;
 };
 
 /** Extendible bucket hash table operations for pre-computed key signature */
@@ -257,6 +263,9 @@ struct rte_table_hash_key16_lru_params {
 
 	/** Byte offset within packet meta-data where the key is located */
 	uint32_t key_offset;
+
+	/** Bit-mask to be AND-ed to the key on lookup */
+	uint8_t *key_mask;
 };
 
 /** LRU hash table operations for pre-computed key signature */
@@ -284,6 +293,9 @@ struct rte_table_hash_key16_ext_params {
 
 	/** Byte offset within packet meta-data where the key is located */
 	uint32_t key_offset;
+
+	/** Bit-mask to be AND-ed to the key on lookup */
+	uint8_t *key_mask;
 };
 
 /** Extendible bucket operations for pre-computed key signature */
diff --git a/lib/librte_table/rte_table_hash_key16.c b/lib/librte_table/rte_table_hash_key16.c
index f6a3306..0d6cc55 100644
--- a/lib/librte_table/rte_table_hash_key16.c
+++ b/lib/librte_table/rte_table_hash_key16.c
@@ -85,6 +85,7 @@ struct rte_table_hash {
 	uint32_t bucket_size;
 	uint32_t signature_offset;
 	uint32_t key_offset;
+	uint64_t key_mask[2];
 	rte_table_hash_op_hash f_hash;
 	uint64_t seed;
 
@@ -164,6 +165,14 @@ rte_table_hash_create_key16_lru(void *params,
 	f->f_hash = p->f_hash;
 	f->seed = p->seed;
 
+	if (p->key_mask != NULL) {
+		f->key_mask[0] = ((uint64_t *)p->key_mask)[0];
+		f->key_mask[1] = ((uint64_t *)p->key_mask)[1];
+	} else {
+		f->key_mask[0] = 0xFFFFFFFFFFFFFFFFLLU;
+		f->key_mask[1] = 0xFFFFFFFFFFFFFFFFLLU;
+	}
+
 	for (i = 0; i < n_buckets; i++) {
 		struct rte_bucket_4_16 *bucket;
 
@@ -384,6 +393,14 @@ rte_table_hash_create_key16_ext(void *params,
 	for (i = 0; i < n_buckets_ext; i++)
 		f->stack[i] = i;
 
+	if (p->key_mask != NULL) {
+		f->key_mask[0] = (((uint64_t *)p->key_mask)[0]);
+		f->key_mask[1] = (((uint64_t *)p->key_mask)[1]);
+	} else {
+		f->key_mask[0] = 0xFFFFFFFFFFFFFFFFLLU;
+		f->key_mask[1] = 0xFFFFFFFFFFFFFFFFLLU;
+	}
+
 	return f;
 }
 
@@ -609,11 +626,14 @@ rte_table_hash_entry_delete_key16_ext(
 	void *a;						\
 	uint64_t pkt_mask;					\
 	uint64_t *key;						\
+	uint64_t hash_key_buffer[2];		\
 	uint32_t pos;						\
 								\
 	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf2, f->key_offset);\
+	hash_key_buffer[0] = key[0] & f->key_mask[0];	\
+	hash_key_buffer[1] = key[1] & f->key_mask[1];	\
 								\
-	lookup_key16_cmp(key, bucket2, pos);			\
+	lookup_key16_cmp(hash_key_buffer, bucket2, pos);	\
 								\
 	pkt_mask = (bucket2->signature[pos] & 1LLU) << pkt2_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -631,11 +651,14 @@ rte_table_hash_entry_delete_key16_ext(
 	void *a;						\
 	uint64_t pkt_mask, bucket_mask;				\
 	uint64_t *key;						\
+	uint64_t hash_key_buffer[2];		\
 	uint32_t pos;						\
 								\
 	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf2, f->key_offset);\
+	hash_key_buffer[0] = key[0] & f->key_mask[0];	\
+	hash_key_buffer[1] = key[1] & f->key_mask[1];	\
 								\
-	lookup_key16_cmp(key, bucket2, pos);			\
+	lookup_key16_cmp(hash_key_buffer, bucket2, pos);	\
 								\
 	pkt_mask = (bucket2->signature[pos] & 1LLU) << pkt2_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -658,12 +681,15 @@ rte_table_hash_entry_delete_key16_ext(
 	void *a;						\
 	uint64_t pkt_mask, bucket_mask;				\
 	uint64_t *key;						\
+	uint64_t hash_key_buffer[2];		\
 	uint32_t pos;						\
 								\
 	bucket = buckets[pkt_index];				\
 	key = keys[pkt_index];					\
+	hash_key_buffer[0] = key[0] & f->key_mask[0];	\
+	hash_key_buffer[1] = key[1] & f->key_mask[1];	\
 								\
-	lookup_key16_cmp(key, bucket, pos);			\
+	lookup_key16_cmp(hash_key_buffer, bucket, pos);	\
 								\
 	pkt_mask = (bucket->signature[pos] & 1LLU) << pkt_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -749,13 +775,19 @@ rte_table_hash_entry_delete_key16_ext(
 	void *a20, *a21;					\
 	uint64_t pkt20_mask, pkt21_mask;			\
 	uint64_t *key20, *key21;				\
+	uint64_t hash_key_buffer20[2];			\
+	uint64_t hash_key_buffer21[2];			\
 	uint32_t pos20, pos21;					\
 								\
 	key20 = RTE_MBUF_METADATA_UINT64_PTR(mbuf20, f->key_offset);\
 	key21 = RTE_MBUF_METADATA_UINT64_PTR(mbuf21, f->key_offset);\
+	hash_key_buffer20[0] = key20[0] & f->key_mask[0];	\
+	hash_key_buffer20[1] = key20[1] & f->key_mask[1];	\
+	hash_key_buffer21[0] = key21[0] & f->key_mask[0];	\
+	hash_key_buffer21[1] = key21[1] & f->key_mask[1];	\
 								\
-	lookup_key16_cmp(key20, bucket20, pos20);		\
-	lookup_key16_cmp(key21, bucket21, pos21);		\
+	lookup_key16_cmp(hash_key_buffer20, bucket20, pos20);	\
+	lookup_key16_cmp(hash_key_buffer21, bucket21, pos21);	\
 								\
 	pkt20_mask = (bucket20->signature[pos20] & 1LLU) << pkt20_index;\
 	pkt21_mask = (bucket21->signature[pos21] & 1LLU) << pkt21_index;\
@@ -778,13 +810,19 @@ rte_table_hash_entry_delete_key16_ext(
 	void *a20, *a21;					\
 	uint64_t pkt20_mask, pkt21_mask, bucket20_mask, bucket21_mask;\
 	uint64_t *key20, *key21;				\
+	uint64_t hash_key_buffer20[2];			\
+	uint64_t hash_key_buffer21[2];			\
 	uint32_t pos20, pos21;					\
 								\
 	key20 = RTE_MBUF_METADATA_UINT64_PTR(mbuf20, f->key_offset);\
 	key21 = RTE_MBUF_METADATA_UINT64_PTR(mbuf21, f->key_offset);\
+	hash_key_buffer20[0] = key20[0] & f->key_mask[0];	\
+	hash_key_buffer20[1] = key20[1] & f->key_mask[1];	\
+	hash_key_buffer21[0] = key21[0] & f->key_mask[0];	\
+	hash_key_buffer21[1] = key21[1] & f->key_mask[1];	\
 								\
-	lookup_key16_cmp(key20, bucket20, pos20);		\
-	lookup_key16_cmp(key21, bucket21, pos21);		\
+	lookup_key16_cmp(hash_key_buffer20, bucket20, pos20);	\
+	lookup_key16_cmp(hash_key_buffer21, bucket21, pos21);	\
 								\
 	pkt20_mask = (bucket20->signature[pos20] & 1LLU) << pkt20_index;\
 	pkt21_mask = (bucket21->signature[pos21] & 1LLU) << pkt21_index;\
diff --git a/lib/librte_table/rte_table_hash_key8.c b/lib/librte_table/rte_table_hash_key8.c
index b351a49..ccb20cf 100644
--- a/lib/librte_table/rte_table_hash_key8.c
+++ b/lib/librte_table/rte_table_hash_key8.c
@@ -82,6 +82,7 @@ struct rte_table_hash {
 	uint32_t bucket_size;
 	uint32_t signature_offset;
 	uint32_t key_offset;
+	uint64_t key_mask;
 	rte_table_hash_op_hash f_hash;
 	uint64_t seed;
 
@@ -160,6 +161,11 @@ rte_table_hash_create_key8_lru(void *params, int socket_id, uint32_t entry_size)
 	f->f_hash = p->f_hash;
 	f->seed = p->seed;
 
+	if (p->key_mask != NULL)
+		f->key_mask = ((uint64_t *)p->key_mask)[0];
+	else
+		f->key_mask = 0xFFFFFFFFFFFFFFFFLLU;
+
 	for (i = 0; i < n_buckets; i++) {
 		struct rte_bucket_4_8 *bucket;
 
@@ -372,6 +378,11 @@ rte_table_hash_create_key8_ext(void *params, int socket_id, uint32_t entry_size)
 	f->stack = (uint32_t *)
 		&f->memory[(n_buckets + n_buckets_ext) * f->bucket_size];
 
+	if (p->key_mask != NULL)
+		f->key_mask = ((uint64_t *)p->key_mask)[0];
+	else
+		f->key_mask = 0xFFFFFFFFFFFFFFFFLLU;
+
 	for (i = 0; i < n_buckets_ext; i++)
 		f->stack[i] = i;
 
@@ -586,9 +597,12 @@ rte_table_hash_entry_delete_key8_ext(
 	uint64_t *key;						\
 	uint64_t signature;					\
 	uint32_t bucket_index;					\
+	uint64_t hash_key_buffer;				\
 								\
 	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf1, f->key_offset);\
-	signature = f->f_hash(key, RTE_TABLE_HASH_KEY_SIZE, f->seed);\
+	hash_key_buffer = *key & f->key_mask;			\
+	signature = f->f_hash(&hash_key_buffer,			\
+		RTE_TABLE_HASH_KEY_SIZE, f->seed);		\
 	bucket_index = signature & (f->n_buckets - 1);		\
 	bucket1 = (struct rte_bucket_4_8 *)			\
 		&f->memory[bucket_index * f->bucket_size];	\
@@ -602,10 +616,12 @@ rte_table_hash_entry_delete_key8_ext(
 	uint64_t pkt_mask;					\
 	uint64_t *key;						\
 	uint32_t pos;						\
+	uint64_t hash_key_buffer;				\
 								\
 	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf2, f->key_offset);\
+	hash_key_buffer = key[0] & f->key_mask;			\
 								\
-	lookup_key8_cmp(key, bucket2, pos);			\
+	lookup_key8_cmp((&hash_key_buffer), bucket2, pos);	\
 								\
 	pkt_mask = ((bucket2->signature >> pos) & 1LLU) << pkt2_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -624,10 +640,12 @@ rte_table_hash_entry_delete_key8_ext(
 	uint64_t pkt_mask, bucket_mask;				\
 	uint64_t *key;						\
 	uint32_t pos;						\
+	uint64_t hash_key_buffer;				\
 								\
 	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf2, f->key_offset);\
+	hash_key_buffer = *key & f->key_mask;			\
 								\
-	lookup_key8_cmp(key, bucket2, pos);			\
+	lookup_key8_cmp((&hash_key_buffer), bucket2, pos);	\
 								\
 	pkt_mask = ((bucket2->signature >> pos) & 1LLU) << pkt2_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -651,11 +669,13 @@ rte_table_hash_entry_delete_key8_ext(
 	uint64_t pkt_mask, bucket_mask;				\
 	uint64_t *key;						\
 	uint32_t pos;						\
+	uint64_t hash_key_buffer;				\
 								\
 	bucket = buckets[pkt_index];				\
 	key = keys[pkt_index];					\
+	hash_key_buffer = (*key) & f->key_mask;			\
 								\
-	lookup_key8_cmp(key, bucket, pos);			\
+	lookup_key8_cmp((&hash_key_buffer), bucket, pos);	\
 								\
 	pkt_mask = ((bucket->signature >> pos) & 1LLU) << pkt_index;\
 	pkts_mask_out |= pkt_mask;				\
@@ -736,6 +756,8 @@ rte_table_hash_entry_delete_key8_ext(
 #define lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f)\
 {								\
 	uint64_t *key10, *key11;				\
+	uint64_t hash_offset_buffer10;				\
+	uint64_t hash_offset_buffer11;				\
 	uint64_t signature10, signature11;			\
 	uint32_t bucket10_index, bucket11_index;		\
 	rte_table_hash_op_hash f_hash = f->f_hash;		\
@@ -744,14 +766,18 @@ rte_table_hash_entry_delete_key8_ext(
 								\
 	key10 = RTE_MBUF_METADATA_UINT64_PTR(mbuf10, key_offset);\
 	key11 = RTE_MBUF_METADATA_UINT64_PTR(mbuf11, key_offset);\
+	hash_offset_buffer10 = *key10 & f->key_mask;		\
+	hash_offset_buffer11 = *key11 & f->key_mask;		\
 								\
-	signature10 = f_hash(key10, RTE_TABLE_HASH_KEY_SIZE, seed);\
+	signature10 = f_hash(&hash_offset_buffer10,		\
+		RTE_TABLE_HASH_KEY_SIZE, seed);			\
 	bucket10_index = signature10 & (f->n_buckets - 1);	\
 	bucket10 = (struct rte_bucket_4_8 *)			\
 		&f->memory[bucket10_index * f->bucket_size];	\
 	rte_prefetch0(bucket10);				\
 								\
-	signature11 = f_hash(key11, RTE_TABLE_HASH_KEY_SIZE, seed);\
+	signature11 = f_hash(&hash_offset_buffer11,		\
+		RTE_TABLE_HASH_KEY_SIZE, seed);			\
 	bucket11_index = signature11 & (f->n_buckets - 1);	\
 	bucket11 = (struct rte_bucket_4_8 *)			\
 		&f->memory[bucket11_index * f->bucket_size];	\
@@ -764,13 +790,17 @@ rte_table_hash_entry_delete_key8_ext(
 	void *a20, *a21;					\
 	uint64_t pkt20_mask, pkt21_mask;			\
 	uint64_t *key20, *key21;				\
+	uint64_t hash_offset_buffer20;				\
+	uint64_t hash_offset_buffer21;				\
 	uint32_t pos20, pos21;					\
 								\
 	key20 = RTE_MBUF_METADATA_UINT64_PTR(mbuf20, f->key_offset);\
 	key21 = RTE_MBUF_METADATA_UINT64_PTR(mbuf21, f->key_offset);\
+	hash_offset_buffer20 = *key20 & f->key_mask;		\
+	hash_offset_buffer21 = *key21 & f->key_mask;		\
 								\
-	lookup_key8_cmp(key20, bucket20, pos20);		\
-	lookup_key8_cmp(key21, bucket21, pos21);		\
+	lookup_key8_cmp((&hash_offset_buffer20), bucket20, pos20);\
+	lookup_key8_cmp((&hash_offset_buffer21), bucket21, pos21);\
 								\
 	pkt20_mask = ((bucket20->signature >> pos20) & 1LLU) << pkt20_index;\
 	pkt21_mask = ((bucket21->signature >> pos21) & 1LLU) << pkt21_index;\
@@ -793,13 +823,17 @@ rte_table_hash_entry_delete_key8_ext(
 	void *a20, *a21;					\
 	uint64_t pkt20_mask, pkt21_mask, bucket20_mask, bucket21_mask;\
 	uint64_t *key20, *key21;				\
+	uint64_t hash_offset_buffer20;				\
+	uint64_t hash_offset_buffer21;				\
 	uint32_t pos20, pos21;					\
 								\
 	key20 = RTE_MBUF_METADATA_UINT64_PTR(mbuf20, f->key_offset);\
 	key21 = RTE_MBUF_METADATA_UINT64_PTR(mbuf21, f->key_offset);\
+	hash_offset_buffer20 = *key20 & f->key_mask;		\
+	hash_offset_buffer21 = *key21 & f->key_mask;		\
 								\
-	lookup_key8_cmp(key20, bucket20, pos20);		\
-	lookup_key8_cmp(key21, bucket21, pos21);		\
+	lookup_key8_cmp((&hash_offset_buffer20), bucket20, pos20);\
+	lookup_key8_cmp((&hash_offset_buffer21), bucket21, pos21);\
 								\
 	pkt20_mask = ((bucket20->signature >> pos20) & 1LLU) << pkt20_index;\
 	pkt21_mask = ((bucket21->signature >> pos21) & 1LLU) << pkt21_index;\
diff --git a/lib/librte_table/rte_table_version.map b/lib/librte_table/rte_table_version.map
index d33f926..2138698 100644
--- a/lib/librte_table/rte_table_version.map
+++ b/lib/librte_table/rte_table_version.map
@@ -19,3 +19,10 @@ DPDK_2.0 {
 
 	local: *;
 };
+
+DPDK_2.2 {
+	global:
+
+	rte_table_hash_key16_ext_dosig_ops;
+
+} DPDK_2.0;
-- 
2.1.0

[dpdk-dev] [PATCH v4 2/7] librte_table: add 16 byte hash table operations with computed lookup

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patch is to adding hash table operations for key signature
computed on lookup ("do-sig") for LRU hash tables and Extendible buckets.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/librte_table/rte_table_hash.h       |   8 +
 lib/librte_table/rte_table_hash_key16.c | 359 +++++++++++++++++++++++++++++++-
 2 files changed, 364 insertions(+), 3 deletions(-)

diff --git a/lib/librte_table/rte_table_hash.h b/lib/librte_table/rte_table_hash.h
index e2c60e1..9d17516 100644
--- a/lib/librte_table/rte_table_hash.h
+++ b/lib/librte_table/rte_table_hash.h
@@ -271,6 +271,10 @@ struct rte_table_hash_key16_lru_params {
 /** LRU hash table operations for pre-computed key signature */
 extern struct rte_table_ops rte_table_hash_key16_lru_ops;
 
+/** LRU hash table operations for key signature computed on lookup
+    ("do-sig") */
+extern struct rte_table_ops rte_table_hash_key16_lru_dosig_ops;
+
 /** Extendible bucket hash table parameters */
 struct rte_table_hash_key16_ext_params {
 	/** Maximum number of entries (and keys) in the table */
@@ -301,6 +305,10 @@ struct rte_table_hash_key16_ext_params {
 /** Extendible bucket operations for pre-computed key signature */
 extern struct rte_table_ops rte_table_hash_key16_ext_ops;
 
+/** Extendible bucket hash table operations for key signature computed on
+    lookup ("do-sig") */
+extern struct rte_table_ops rte_table_hash_key16_ext_dosig_ops;
+
 /**
  * 32-byte key hash tables
  *
diff --git a/lib/librte_table/rte_table_hash_key16.c b/lib/librte_table/rte_table_hash_key16.c
index 0d6cc55..427b534 100644
--- a/lib/librte_table/rte_table_hash_key16.c
+++ b/lib/librte_table/rte_table_hash_key16.c
@@ -620,6 +620,27 @@ rte_table_hash_entry_delete_key16_ext(
 	rte_prefetch0((void *)(((uintptr_t) bucket1) + RTE_CACHE_LINE_SIZE));\
 }
 
+#define lookup1_stage1_dosig(mbuf1, bucket1, f)			\
+{								\
+	uint64_t *key;						\
+	uint64_t signature = 0;				\
+	uint32_t bucket_index;				\
+	uint64_t hash_key_buffer[2];		\
+								\
+	key = RTE_MBUF_METADATA_UINT64_PTR(mbuf1, f->key_offset);\
+								\
+	hash_key_buffer[0] = key[0] & f->key_mask[0];	\
+	hash_key_buffer[1] = key[1] & f->key_mask[1];	\
+	signature = f->f_hash(hash_key_buffer,			\
+			RTE_TABLE_HASH_KEY_SIZE, f->seed);		\
+								\
+	bucket_index = signature & (f->n_buckets - 1);		\
+	bucket1 = (struct rte_bucket_4_16 *)			\
+		&f->memory[bucket_index * f->bucket_size];	\
+	rte_prefetch0(bucket1);					\
+	rte_prefetch0((void *)(((uintptr_t) bucket1) + RTE_CACHE_LINE_SIZE));\
+}
+
 #define lookup1_stage2_lru(pkt2_index, mbuf2, bucket2,		\
 		pkts_mask_out, entries, f)			\
 {								\
@@ -769,6 +790,36 @@ rte_table_hash_entry_delete_key16_ext(
 	rte_prefetch0((void *)(((uintptr_t) bucket11) + RTE_CACHE_LINE_SIZE));\
 }
 
+#define lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f)	\
+{								\
+	uint64_t *key10, *key11;					\
+	uint64_t hash_offset_buffer[2];				\
+	uint64_t signature10, signature11;			\
+	uint32_t bucket10_index, bucket11_index;	\
+								\
+	key10 = RTE_MBUF_METADATA_UINT64_PTR(mbuf10, f->key_offset);\
+	hash_offset_buffer[0] = key10[0] & f->key_mask[0];	\
+	hash_offset_buffer[1] = key10[1] & f->key_mask[1];	\
+	signature10 = f->f_hash(hash_offset_buffer,			\
+			RTE_TABLE_HASH_KEY_SIZE, f->seed);\
+	bucket10_index = signature10 & (f->n_buckets - 1);	\
+	bucket10 = (struct rte_bucket_4_16 *)				\
+		&f->memory[bucket10_index * f->bucket_size];	\
+	rte_prefetch0(bucket10);				\
+	rte_prefetch0((void *)(((uintptr_t) bucket10) + RTE_CACHE_LINE_SIZE));\
+								\
+	key11 = RTE_MBUF_METADATA_UINT64_PTR(mbuf11, f->key_offset);\
+	hash_offset_buffer[0] = key11[0] & f->key_mask[0];	\
+	hash_offset_buffer[1] = key11[1] & f->key_mask[1];	\
+	signature11 = f->f_hash(hash_offset_buffer,			\
+			RTE_TABLE_HASH_KEY_SIZE, f->seed);\
+	bucket11_index = signature11 & (f->n_buckets - 1);	\
+	bucket11 = (struct rte_bucket_4_16 *)			\
+		&f->memory[bucket11_index * f->bucket_size];	\
+	rte_prefetch0(bucket11);				\
+	rte_prefetch0((void *)(((uintptr_t) bucket11) + RTE_CACHE_LINE_SIZE));\
+}
+
 #define lookup2_stage2_lru(pkt20_index, pkt21_index, mbuf20, mbuf21,\
 		bucket20, bucket21, pkts_mask_out, entries, f)	\
 {								\
@@ -878,7 +929,8 @@ rte_table_hash_lookup_key16_lru(
 		}
 
 		*lookup_hit_mask = pkts_mask_out;
-		RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in - __builtin_popcountll(pkts_mask_out));
+		RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f,
+			n_pkts_in - __builtin_popcountll(pkts_mask_out));
 		return 0;
 	}
 
@@ -968,11 +1020,141 @@ rte_table_hash_lookup_key16_lru(
 		bucket20, bucket21, pkts_mask_out, entries, f);
 
 	*lookup_hit_mask = pkts_mask_out;
-	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in - __builtin_popcountll(pkts_mask_out));
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in -
+		__builtin_popcountll(pkts_mask_out));
 	return 0;
 } /* rte_table_hash_lookup_key16_lru() */
 
 static int
+rte_table_hash_lookup_key16_lru_dosig(
+	void *table,
+	struct rte_mbuf **pkts,
+	uint64_t pkts_mask,
+	uint64_t *lookup_hit_mask,
+	void **entries)
+{
+	struct rte_table_hash *f = (struct rte_table_hash *) table;
+	struct rte_bucket_4_16 *bucket10, *bucket11, *bucket20, *bucket21;
+	struct rte_mbuf *mbuf00, *mbuf01, *mbuf10, *mbuf11, *mbuf20, *mbuf21;
+	uint32_t pkt00_index, pkt01_index, pkt10_index;
+	uint32_t pkt11_index, pkt20_index, pkt21_index;
+	uint64_t pkts_mask_out = 0;
+
+	__rte_unused uint32_t n_pkts_in = __builtin_popcountll(pkts_mask);
+
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_IN_ADD(f, n_pkts_in);
+
+	/* Cannot run the pipeline with less than 5 packets */
+	if (__builtin_popcountll(pkts_mask) < 5) {
+		for ( ; pkts_mask; ) {
+			struct rte_bucket_4_16 *bucket;
+			struct rte_mbuf *mbuf;
+			uint32_t pkt_index;
+
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage1_dosig(mbuf, bucket, f);
+			lookup1_stage2_lru(pkt_index, mbuf, bucket,
+				pkts_mask_out, entries, f);
+		}
+
+		*lookup_hit_mask = pkts_mask_out;
+		RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in -
+			__builtin_popcountll(pkts_mask_out));
+		return 0;
+	}
+
+	/*
+	 * Pipeline fill
+	 *
+	 */
+	/* Pipeline stage 0 */
+	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
+		pkts_mask);
+
+	/* Pipeline feed */
+	mbuf10 = mbuf00;
+	mbuf11 = mbuf01;
+	pkt10_index = pkt00_index;
+	pkt11_index = pkt01_index;
+
+	/* Pipeline stage 0 */
+	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
+		pkts_mask);
+
+	/* Pipeline stage 1 */
+	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+	/*
+	 * Pipeline run
+	 *
+	 */
+	for ( ; pkts_mask; ) {
+		/* Pipeline feed */
+		bucket20 = bucket10;
+		bucket21 = bucket11;
+		mbuf20 = mbuf10;
+		mbuf21 = mbuf11;
+		mbuf10 = mbuf00;
+		mbuf11 = mbuf01;
+		pkt20_index = pkt10_index;
+		pkt21_index = pkt11_index;
+		pkt10_index = pkt00_index;
+		pkt11_index = pkt01_index;
+
+		/* Pipeline stage 0 */
+		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
+			mbuf00, mbuf01, pkts, pkts_mask);
+
+		/* Pipeline stage 1 */
+		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+		/* Pipeline stage 2 */
+		lookup2_stage2_lru(pkt20_index, pkt21_index, mbuf20, mbuf21,
+			bucket20, bucket21, pkts_mask_out, entries, f);
+	}
+
+	/*
+	 * Pipeline flush
+	 *
+	 */
+	/* Pipeline feed */
+	bucket20 = bucket10;
+	bucket21 = bucket11;
+	mbuf20 = mbuf10;
+	mbuf21 = mbuf11;
+	mbuf10 = mbuf00;
+	mbuf11 = mbuf01;
+	pkt20_index = pkt10_index;
+	pkt21_index = pkt11_index;
+	pkt10_index = pkt00_index;
+	pkt11_index = pkt01_index;
+
+	/* Pipeline stage 1 */
+	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+	/* Pipeline stage 2 */
+	lookup2_stage2_lru(pkt20_index, pkt21_index, mbuf20, mbuf21,
+		bucket20, bucket21, pkts_mask_out, entries, f);
+
+	/* Pipeline feed */
+	bucket20 = bucket10;
+	bucket21 = bucket11;
+	mbuf20 = mbuf10;
+	mbuf21 = mbuf11;
+	pkt20_index = pkt10_index;
+	pkt21_index = pkt11_index;
+
+	/* Pipeline stage 2 */
+	lookup2_stage2_lru(pkt20_index, pkt21_index, mbuf20, mbuf21,
+		bucket20, bucket21, pkts_mask_out, entries, f);
+
+	*lookup_hit_mask = pkts_mask_out;
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in -
+		__builtin_popcountll(pkts_mask_out));
+	return 0;
+} /* rte_table_hash_lookup_key16_lru_dosig() */
+
+static int
 rte_table_hash_lookup_key16_ext(
 	void *table,
 	struct rte_mbuf **pkts,
@@ -1118,11 +1300,164 @@ grind_next_buckets:
 	}
 
 	*lookup_hit_mask = pkts_mask_out;
-	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in - __builtin_popcountll(pkts_mask_out));
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in -
+		__builtin_popcountll(pkts_mask_out));
 	return 0;
 } /* rte_table_hash_lookup_key16_ext() */
 
 static int
+rte_table_hash_lookup_key16_ext_dosig(
+	void *table,
+	struct rte_mbuf **pkts,
+	uint64_t pkts_mask,
+	uint64_t *lookup_hit_mask,
+	void **entries)
+{
+	struct rte_table_hash *f = (struct rte_table_hash *) table;
+	struct rte_bucket_4_16 *bucket10, *bucket11, *bucket20, *bucket21;
+	struct rte_mbuf *mbuf00, *mbuf01, *mbuf10, *mbuf11, *mbuf20, *mbuf21;
+	uint32_t pkt00_index, pkt01_index, pkt10_index;
+	uint32_t pkt11_index, pkt20_index, pkt21_index;
+	uint64_t pkts_mask_out = 0, buckets_mask = 0;
+	struct rte_bucket_4_16 *buckets[RTE_PORT_IN_BURST_SIZE_MAX];
+	uint64_t *keys[RTE_PORT_IN_BURST_SIZE_MAX];
+
+	__rte_unused uint32_t n_pkts_in = __builtin_popcountll(pkts_mask);
+
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_IN_ADD(f, n_pkts_in);
+
+	/* Cannot run the pipeline with less than 5 packets */
+	if (__builtin_popcountll(pkts_mask) < 5) {
+		for ( ; pkts_mask; ) {
+			struct rte_bucket_4_16 *bucket;
+			struct rte_mbuf *mbuf;
+			uint32_t pkt_index;
+
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage1_dosig(mbuf, bucket, f);
+			lookup1_stage2_ext(pkt_index, mbuf, bucket,
+				pkts_mask_out, entries, buckets_mask,
+				buckets, keys, f);
+		}
+
+		goto grind_next_buckets;
+	}
+
+	/*
+	 * Pipeline fill
+	 *
+	 */
+	/* Pipeline stage 0 */
+	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
+		pkts_mask);
+
+	/* Pipeline feed */
+	mbuf10 = mbuf00;
+	mbuf11 = mbuf01;
+	pkt10_index = pkt00_index;
+	pkt11_index = pkt01_index;
+
+	/* Pipeline stage 0 */
+	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
+		pkts_mask);
+
+	/* Pipeline stage 1 */
+	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+	/*
+	 * Pipeline run
+	 *
+	 */
+	for ( ; pkts_mask; ) {
+		/* Pipeline feed */
+		bucket20 = bucket10;
+		bucket21 = bucket11;
+		mbuf20 = mbuf10;
+		mbuf21 = mbuf11;
+		mbuf10 = mbuf00;
+		mbuf11 = mbuf01;
+		pkt20_index = pkt10_index;
+		pkt21_index = pkt11_index;
+		pkt10_index = pkt00_index;
+		pkt11_index = pkt01_index;
+
+		/* Pipeline stage 0 */
+		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
+			mbuf00, mbuf01, pkts, pkts_mask);
+
+		/* Pipeline stage 1 */
+		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+		/* Pipeline stage 2 */
+		lookup2_stage2_ext(pkt20_index, pkt21_index, mbuf20, mbuf21,
+			bucket20, bucket21, pkts_mask_out, entries,
+			buckets_mask, buckets, keys, f);
+	}
+
+	/*
+	 * Pipeline flush
+	 *
+	 */
+	/* Pipeline feed */
+	bucket20 = bucket10;
+	bucket21 = bucket11;
+	mbuf20 = mbuf10;
+	mbuf21 = mbuf11;
+	mbuf10 = mbuf00;
+	mbuf11 = mbuf01;
+	pkt20_index = pkt10_index;
+	pkt21_index = pkt11_index;
+	pkt10_index = pkt00_index;
+	pkt11_index = pkt01_index;
+
+	/* Pipeline stage 1 */
+	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
+
+	/* Pipeline stage 2 */
+	lookup2_stage2_ext(pkt20_index, pkt21_index, mbuf20, mbuf21,
+		bucket20, bucket21, pkts_mask_out, entries,
+		buckets_mask, buckets, keys, f);
+
+	/* Pipeline feed */
+	bucket20 = bucket10;
+	bucket21 = bucket11;
+	mbuf20 = mbuf10;
+	mbuf21 = mbuf11;
+	pkt20_index = pkt10_index;
+	pkt21_index = pkt11_index;
+
+	/* Pipeline stage 2 */
+	lookup2_stage2_ext(pkt20_index, pkt21_index, mbuf20, mbuf21,
+		bucket20, bucket21, pkts_mask_out, entries,
+		buckets_mask, buckets, keys, f);
+
+grind_next_buckets:
+	/* Grind next buckets */
+	for ( ; buckets_mask; ) {
+		uint64_t buckets_mask_next = 0;
+
+		for ( ; buckets_mask; ) {
+			uint64_t pkt_mask;
+			uint32_t pkt_index;
+
+			pkt_index = __builtin_ctzll(buckets_mask);
+			pkt_mask = 1LLU << pkt_index;
+			buckets_mask &= ~pkt_mask;
+
+			lookup_grinder(pkt_index, buckets, keys, pkts_mask_out,
+				entries, buckets_mask_next, f);
+		}
+
+		buckets_mask = buckets_mask_next;
+	}
+
+	*lookup_hit_mask = pkts_mask_out;
+	RTE_TABLE_HASH_KEY16_STATS_PKTS_LOOKUP_MISS(f, n_pkts_in -
+		__builtin_popcountll(pkts_mask_out));
+	return 0;
+} /* rte_table_hash_lookup_key16_ext_dosig() */
+
+static int
 rte_table_hash_key16_stats_read(void *table, struct rte_table_stats *stats, int clear)
 {
 	struct rte_table_hash *t = (struct rte_table_hash *) table;
@@ -1145,6 +1480,15 @@ struct rte_table_ops rte_table_hash_key16_lru_ops = {
 	.f_stats = rte_table_hash_key16_stats_read,
 };
 
+struct rte_table_ops rte_table_hash_key16_lru_dosig_ops = {
+	.f_create = rte_table_hash_create_key16_lru,
+	.f_free = rte_table_hash_free_key16_lru,
+	.f_add = rte_table_hash_entry_add_key16_lru,
+	.f_delete = rte_table_hash_entry_delete_key16_lru,
+	.f_lookup = rte_table_hash_lookup_key16_lru_dosig,
+	.f_stats = rte_table_hash_key16_stats_read,
+};
+
 struct rte_table_ops rte_table_hash_key16_ext_ops = {
 	.f_create = rte_table_hash_create_key16_ext,
 	.f_free = rte_table_hash_free_key16_ext,
@@ -1153,3 +1497,12 @@ struct rte_table_ops rte_table_hash_key16_ext_ops = {
 	.f_lookup = rte_table_hash_lookup_key16_ext,
 	.f_stats = rte_table_hash_key16_stats_read,
 };
+
+struct rte_table_ops rte_table_hash_key16_ext_dosig_ops = {
+	.f_create = rte_table_hash_create_key16_ext,
+	.f_free = rte_table_hash_free_key16_ext,
+	.f_add = rte_table_hash_entry_add_key16_ext,
+	.f_delete = rte_table_hash_entry_delete_key16_ext,
+	.f_lookup = rte_table_hash_lookup_key16_ext_dosig,
+	.f_stats = rte_table_hash_key16_stats_read,
+};
-- 
2.1.0

[dpdk-dev] [PATCH v4 3/7] app/test: modify app/test_table_combined and app/test_table_tables

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

Tests have been updated to work on added key_mask parameter for 8-byte
key extendible bucket and LRU tables.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 app/test/test_table_combined.c | 5 ++++-
 app/test/test_table_tables.c   | 6 ++++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/app/test/test_table_combined.c b/app/test/test_table_combined.c
index 18daeec..8bf4aeb 100644
--- a/app/test/test_table_combined.c
+++ b/app/test/test_table_combined.c
@@ -418,9 +418,9 @@ test_table_hash8lru(void)
 	struct rte_table_hash_key8_lru_params key8lru_params = {
 		.n_entries = 1<<24,
 		.f_hash = pipeline_test_hash,
-		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(0),
 		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	uint8_t key8lru[8];
@@ -479,6 +479,7 @@ test_table_hash16lru(void)
 		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(0),
 		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	uint8_t key16lru[16];
@@ -596,6 +597,7 @@ test_table_hash8ext(void)
 		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(0),
 		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	uint8_t key8ext[8];
@@ -662,6 +664,7 @@ test_table_hash16ext(void)
 		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(0),
 		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	uint8_t key16ext[16];
diff --git a/app/test/test_table_tables.c b/app/test/test_table_tables.c
index cf7c62d..b6364c4 100644
--- a/app/test/test_table_tables.c
+++ b/app/test/test_table_tables.c
@@ -669,7 +669,8 @@ test_table_hash_lru_generic(struct rte_table_ops *ops)
 		.f_hash = pipeline_test_hash,
 		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(1),
-		.key_offset = APP_METADATA_OFFSET(32)
+		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	hash_params.n_entries = 0;
@@ -784,7 +785,8 @@ test_table_hash_ext_generic(struct rte_table_ops *ops)
 		.f_hash = pipeline_test_hash,
 		.seed = 0,
 		.signature_offset = APP_METADATA_OFFSET(1),
-		.key_offset = APP_METADATA_OFFSET(32)
+		.key_offset = APP_METADATA_OFFSET(32),
+		.key_mask = NULL,
 	};
 
 	hash_params.n_entries = 0;
-- 
2.1.0

[dpdk-dev] [PATCH v4 4/7] app/test-pipeline: modify pipeline test

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

Test-pipeline has been updated to work on added
key_mask parameter for 8-byte key extendible
bucket and LRU tables.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 app/test-pipeline/pipeline_hash.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/app/test-pipeline/pipeline_hash.c b/app/test-pipeline/pipeline_hash.c
index 5e4e17f..8b888d7 100644
--- a/app/test-pipeline/pipeline_hash.c
+++ b/app/test-pipeline/pipeline_hash.c
@@ -216,6 +216,7 @@ app_main_loop_worker_pipeline_hash(void) {
 			.n_entries_ext = 1 << 23,
 			.signature_offset = APP_METADATA_OFFSET(0),
 			.key_offset = APP_METADATA_OFFSET(32),
+			.key_mask = NULL,
 			.f_hash = test_hash,
 			.seed = 0,
 		};
@@ -240,6 +241,7 @@ app_main_loop_worker_pipeline_hash(void) {
 			.n_entries = 1 << 24,
 			.signature_offset = APP_METADATA_OFFSET(0),
 			.key_offset = APP_METADATA_OFFSET(32),
+			.key_mask = NULL,
 			.f_hash = test_hash,
 			.seed = 0,
 		};
@@ -267,6 +269,7 @@ app_main_loop_worker_pipeline_hash(void) {
 			.key_offset = APP_METADATA_OFFSET(32),
 			.f_hash = test_hash,
 			.seed = 0,
+			.key_mask = NULL,
 		};
 
 		struct rte_pipeline_table_params table_params = {
@@ -291,6 +294,7 @@ app_main_loop_worker_pipeline_hash(void) {
 			.key_offset = APP_METADATA_OFFSET(32),
 			.f_hash = test_hash,
 			.seed = 0,
+			.key_mask = NULL,
 		};
 
 		struct rte_pipeline_table_params table_params = {
-- 
2.1.0

[dpdk-dev] [PATCH v4 5/7] example/ip_pipeline: add parse_hex_string for internal use

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patch adds parse_hex_string function to parse hex string to uint8_t
array.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 examples/ip_pipeline/config_parse.c | 52 +++++++++++++++++++++++++++++++++++++
 examples/ip_pipeline/pipeline_be.h  |  4 +++
 2 files changed, 56 insertions(+)

diff --git a/examples/ip_pipeline/config_parse.c b/examples/ip_pipeline/config_parse.c
index c9b78f9..ab7c518 100644
--- a/examples/ip_pipeline/config_parse.c
+++ b/examples/ip_pipeline/config_parse.c
@@ -455,6 +455,58 @@ parse_pipeline_core(uint32_t *socket,
 	return 0;
 }
 
+static uint32_t
+get_hex_val(char c)
+{
+	switch (c) {
+	case '0': case '1': case '2': case '3': case '4': case '5':
+	case '6': case '7': case '8': case '9':
+		return c - '0';
+	case 'A': case 'B': case 'C': case 'D': case 'E': case 'F':
+		return c - 'A' + 10;
+	case 'a': case 'b': case 'c': case 'd': case 'e': case 'f':
+		return c - 'a' + 10;
+	default:
+		return 0;
+	}
+}
+
+int
+parse_hex_string(char *src, uint8_t *dst, uint32_t *size)
+{
+	char *c;
+	uint32_t len, i;
+
+	/* Check input parameters */
+	if ((src == NULL) ||
+		(dst == NULL) ||
+		(size == NULL) ||
+		(*size == 0))
+		return -1;
+
+	len = strlen(src);
+	if (((len & 3) != 0) ||
+		(len > (*size) * 2))
+		return -1;
+	*size = len / 2;
+
+	for (c = src; *c != 0; c++) {
+		if ((((*c) >= '0') && ((*c) <= '9')) ||
+			(((*c) >= 'A') && ((*c) <= 'F')) ||
+			(((*c) >= 'a') && ((*c) <= 'f')))
+			continue;
+
+		return -1;
+	}
+
+	/* Convert chars to bytes */
+	for (i = 0; i < *size; i++)
+		dst[i] = get_hex_val(src[2 * i]) * 16 +
+			get_hex_val(src[2 * i + 1]);
+
+	return 0;
+}
+
 static size_t
 skip_digits(const char *src)
 {
diff --git a/examples/ip_pipeline/pipeline_be.h b/examples/ip_pipeline/pipeline_be.h
index 51f1e4f..2e46440 100644
--- a/examples/ip_pipeline/pipeline_be.h
+++ b/examples/ip_pipeline/pipeline_be.h
@@ -253,4 +253,8 @@ struct pipeline_be_ops {
 	pipeline_be_op_track f_track;
 };
 
+/* Parse hex string to uint8_t array */
+int
+parse_hex_string(char *src, uint8_t *dst, uint32_t *size);
+
 #endif
-- 
2.1.0

[dpdk-dev] [PATCH v4 6/7] example/ip_pipeline/pipeline: update flow_classification pipeline

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patch updates the flow_classification pipeline for added key_mask
parameter in 8/16-byte key hash parameters. The update provides user
optional key_mask configuration item applying to the packets.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 .../pipeline/pipeline_flow_classification_be.c     | 56 ++++++++++++++++++++--
 1 file changed, 52 insertions(+), 4 deletions(-)

diff --git a/examples/ip_pipeline/pipeline/pipeline_flow_classification_be.c b/examples/ip_pipeline/pipeline/pipeline_flow_classification_be.c
index 06a648d..e22f96f 100644
--- a/examples/ip_pipeline/pipeline/pipeline_flow_classification_be.c
+++ b/examples/ip_pipeline/pipeline/pipeline_flow_classification_be.c
@@ -37,6 +37,7 @@
 #include <rte_malloc.h>
 #include <rte_table_hash.h>
 #include <rte_byteorder.h>
+#include <pipeline.h>
 
 #include "pipeline_flow_classification_be.h"
 #include "hash_func.h"
@@ -49,6 +50,7 @@ struct pipeline_flow_classification {
 	uint32_t key_offset;
 	uint32_t key_size;
 	uint32_t hash_offset;
+	uint8_t *key_mask;
 } __rte_cache_aligned;
 
 static void *
@@ -125,8 +127,12 @@ pipeline_fc_parse_args(struct pipeline_flow_classification *p,
 	uint32_t key_offset_present = 0;
 	uint32_t key_size_present = 0;
 	uint32_t hash_offset_present = 0;
+	uint32_t key_mask_present = 0;
 
 	uint32_t i;
+	char *key_mask_str = NULL;
+
+	p->hash_offset = 0;
 
 	for (i = 0; i < params->n_args; i++) {
 		char *arg_name = params->args_name[i];
@@ -171,6 +177,20 @@ pipeline_fc_parse_args(struct pipeline_flow_classification *p,
 			continue;
 		}
 
+		/* key_mask */
+		if (strcmp(arg_name, "key_mask") == 0) {
+			if (key_mask_present)
+				return -1;
+
+			key_mask_str = strdup(arg_value);
+			if (key_mask_str == NULL)
+				return -1;
+
+			key_mask_present = 1;
+
+			continue;
+		}
+
 		/* hash_offset */
 		if (strcmp(arg_name, "hash_offset") == 0) {
 			if (hash_offset_present)
@@ -189,10 +209,23 @@ pipeline_fc_parse_args(struct pipeline_flow_classification *p,
 	/* Check that mandatory arguments are present */
 	if ((n_flows_present == 0) ||
 		(key_offset_present == 0) ||
-		(key_size_present == 0) ||
-		(hash_offset_present == 0))
+		(key_size_present == 0))
 		return -1;
 
+	if (key_mask_present) {
+		p->key_mask = rte_malloc(NULL, p->key_size, 0);
+		if (p->key_mask == NULL)
+			return -1;
+
+		if (parse_hex_string(key_mask_str, p->key_mask, &p->key_size)
+			!= 0) {
+			free(p->key_mask);
+			return -1;
+		}
+
+		free(key_mask_str);
+	}
+
 	return 0;
 }
 
@@ -297,6 +330,7 @@ static void *pipeline_fc_init(struct pipeline_params *params,
 			.signature_offset = p_fc->hash_offset,
 			.key_offset = p_fc->key_offset,
 			.f_hash = hash_func[(p_fc->key_size / 8) - 1],
+			.key_mask = p_fc->key_mask,
 			.seed = 0,
 		};
 
@@ -307,6 +341,7 @@ static void *pipeline_fc_init(struct pipeline_params *params,
 			.signature_offset = p_fc->hash_offset,
 			.key_offset = p_fc->key_offset,
 			.f_hash = hash_func[(p_fc->key_size / 8) - 1],
+			.key_mask = p_fc->key_mask,
 			.seed = 0,
 		};
 
@@ -336,12 +371,25 @@ static void *pipeline_fc_init(struct pipeline_params *params,
 
 		switch (p_fc->key_size) {
 		case 8:
-			table_params.ops = &rte_table_hash_key8_lru_ops;
+			if (p_fc->hash_offset != 0) {
+				table_params.ops =
+					&rte_table_hash_key8_ext_ops;
+			} else {
+				table_params.ops =
+					&rte_table_hash_key8_ext_dosig_ops;
+			}
 			table_params.arg_create = &table_hash_key8_params;
 			break;
+			break;
 
 		case 16:
-			table_params.ops = &rte_table_hash_key16_ext_ops;
+			if (p_fc->hash_offset != 0) {
+				table_params.ops =
+					&rte_table_hash_key16_ext_ops;
+			} else {
+				table_params.ops =
+					&rte_table_hash_key16_ext_dosig_ops;
+			}
 			table_params.arg_create = &table_hash_key16_params;
 			break;
 
-- 
2.1.0

[dpdk-dev] [PATCH v4 7/7] librte_table: performance improvement on rte_prefetch offset

[roy.fan.zhang]

From: Fan Zhang <roy.fan.zhang@intel.com>

This patch modifies rte_prefetch offsets to improve hash/lru
table lookup performance.

Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com>
---
 lib/librte_table/rte_table_hash_ext.c   | 10 ++++---
 lib/librte_table/rte_table_hash_key16.c | 51 +++++++++++++++++----------------
 lib/librte_table/rte_table_hash_key32.c | 35 +++++++++++-----------
 lib/librte_table/rte_table_hash_key8.c  | 51 +++++++++++++++++----------------
 lib/librte_table/rte_table_hash_lru.c   | 10 ++++---
 5 files changed, 85 insertions(+), 72 deletions(-)

diff --git a/lib/librte_table/rte_table_hash_ext.c b/lib/librte_table/rte_table_hash_ext.c
index 1fa15c8..854e1a5 100644
--- a/lib/librte_table/rte_table_hash_ext.c
+++ b/lib/librte_table/rte_table_hash_ext.c
@@ -648,6 +648,7 @@ static int rte_table_hash_ext_lookup_unoptimized(
 {									\
 	uint64_t pkt00_mask, pkt01_mask;				\
 	struct rte_mbuf *mbuf00, *mbuf01;				\
+	uint32_t key_offset = t->key_offset;			\
 									\
 	pkt00_index = __builtin_ctzll(pkts_mask);			\
 	pkt00_mask = 1LLU << pkt00_index;				\
@@ -659,8 +660,8 @@ static int rte_table_hash_ext_lookup_unoptimized(
 	pkts_mask &= ~pkt01_mask;					\
 	mbuf01 = pkts[pkt01_index];					\
 									\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));		\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));		\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage0_with_odd_support(t, g, pkts, pkts_mask, pkt00_index, \
@@ -668,6 +669,7 @@ static int rte_table_hash_ext_lookup_unoptimized(
 {									\
 	uint64_t pkt00_mask, pkt01_mask;				\
 	struct rte_mbuf *mbuf00, *mbuf01;				\
+	uint32_t key_offset = t->key_offset;			\
 									\
 	pkt00_index = __builtin_ctzll(pkts_mask);			\
 	pkt00_mask = 1LLU << pkt00_index;				\
@@ -681,8 +683,8 @@ static int rte_table_hash_ext_lookup_unoptimized(
 	pkts_mask &= ~pkt01_mask;					\
 	mbuf01 = pkts[pkt01_index];					\
 									\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));		\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));		\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage1(t, g, pkts, pkt10_index, pkt11_index)		\
diff --git a/lib/librte_table/rte_table_hash_key16.c b/lib/librte_table/rte_table_hash_key16.c
index 427b534..21130b9 100644
--- a/lib/librte_table/rte_table_hash_key16.c
+++ b/lib/librte_table/rte_table_hash_key16.c
@@ -595,16 +595,17 @@ rte_table_hash_entry_delete_key16_ext(
 		pos = 3;					\
 }
 
-#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask)	\
+#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask, f)	\
 {								\
 	uint64_t pkt_mask;					\
+	uint32_t key_offset = f->key_offset;\
 								\
 	pkt0_index = __builtin_ctzll(pkts_mask);		\
 	pkt_mask = 1LLU << pkt0_index;				\
 	pkts_mask &= ~pkt_mask;					\
 								\
 	mbuf0 = pkts[pkt0_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, key_offset));\
 }
 
 #define lookup1_stage1(mbuf1, bucket1, f)			\
@@ -729,36 +730,38 @@ rte_table_hash_entry_delete_key16_ext(
 }
 
 #define lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01,\
-		pkts, pkts_mask)				\
+		pkts, pkts_mask, f)				\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	pkt01_mask = 1LLU << pkt01_index;			\
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,\
-		mbuf00, mbuf01, pkts, pkts_mask)		\
+		mbuf00, mbuf01, pkts, pkts_mask, f)		\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));	\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	if (pkts_mask == 0)					\
@@ -767,7 +770,7 @@ rte_table_hash_entry_delete_key16_ext(
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));	\
 }
 
 #define lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f)	\
@@ -922,7 +925,7 @@ rte_table_hash_lookup_key16_lru(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_lru(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, f);
@@ -940,7 +943,7 @@ rte_table_hash_lookup_key16_lru(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -950,7 +953,7 @@ rte_table_hash_lookup_key16_lru(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -974,7 +977,7 @@ rte_table_hash_lookup_key16_lru(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1051,7 +1054,7 @@ rte_table_hash_lookup_key16_lru_dosig(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1_dosig(mbuf, bucket, f);
 			lookup1_stage2_lru(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, f);
@@ -1069,7 +1072,7 @@ rte_table_hash_lookup_key16_lru_dosig(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1079,7 +1082,7 @@ rte_table_hash_lookup_key16_lru_dosig(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1103,7 +1106,7 @@ rte_table_hash_lookup_key16_lru_dosig(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1181,7 +1184,7 @@ rte_table_hash_lookup_key16_ext(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_ext(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, buckets_mask,
@@ -1197,7 +1200,7 @@ rte_table_hash_lookup_key16_ext(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1207,7 +1210,7 @@ rte_table_hash_lookup_key16_ext(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1231,7 +1234,7 @@ rte_table_hash_lookup_key16_ext(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1333,7 +1336,7 @@ rte_table_hash_lookup_key16_ext_dosig(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1_dosig(mbuf, bucket, f);
 			lookup1_stage2_ext(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, buckets_mask,
@@ -1349,7 +1352,7 @@ rte_table_hash_lookup_key16_ext_dosig(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1359,7 +1362,7 @@ rte_table_hash_lookup_key16_ext_dosig(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1383,7 +1386,7 @@ rte_table_hash_lookup_key16_ext_dosig(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
diff --git a/lib/librte_table/rte_table_hash_key32.c b/lib/librte_table/rte_table_hash_key32.c
index 5fe4161..e33029a 100644
--- a/lib/librte_table/rte_table_hash_key32.c
+++ b/lib/librte_table/rte_table_hash_key32.c
@@ -591,16 +591,17 @@ rte_table_hash_entry_delete_key32_ext(
 		pos = 3;					\
 }
 
-#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask)	\
+#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask, f)	\
 {								\
 	uint64_t pkt_mask;					\
+	uint32_t key_offset = f->key_offset;	\
 								\
 	pkt0_index = __builtin_ctzll(pkts_mask);		\
 	pkt_mask = 1LLU << pkt0_index;				\
 	pkts_mask &= ~pkt_mask;					\
 								\
 	mbuf0 = pkts[pkt0_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, key_offset));\
 }
 
 #define lookup1_stage1(mbuf1, bucket1, f)			\
@@ -698,36 +699,38 @@ rte_table_hash_entry_delete_key32_ext(
 }
 
 #define lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01,\
-	pkts, pkts_mask)					\
+	pkts, pkts_mask, f)					\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	pkt01_mask = 1LLU << pkt01_index;			\
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,\
-	mbuf00, mbuf01, pkts, pkts_mask)			\
+	mbuf00, mbuf01, pkts, pkts_mask, f)			\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));	\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	if (pkts_mask == 0)					\
@@ -737,7 +740,7 @@ rte_table_hash_entry_delete_key32_ext(
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));	\
 }
 
 #define lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f)	\
@@ -852,7 +855,7 @@ rte_table_hash_lookup_key32_lru(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_lru(pkt_index, mbuf, bucket,
 					pkts_mask_out, entries, f);
@@ -869,7 +872,7 @@ rte_table_hash_lookup_key32_lru(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -879,7 +882,7 @@ rte_table_hash_lookup_key32_lru(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -903,7 +906,7 @@ rte_table_hash_lookup_key32_lru(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -981,7 +984,7 @@ rte_table_hash_lookup_key32_ext(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_ext(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, buckets_mask, buckets,
@@ -997,7 +1000,7 @@ rte_table_hash_lookup_key32_ext(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1007,7 +1010,7 @@ rte_table_hash_lookup_key32_ext(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1031,7 +1034,7 @@ rte_table_hash_lookup_key32_ext(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
diff --git a/lib/librte_table/rte_table_hash_key8.c b/lib/librte_table/rte_table_hash_key8.c
index ccb20cf..79d7184 100644
--- a/lib/librte_table/rte_table_hash_key8.c
+++ b/lib/librte_table/rte_table_hash_key8.c
@@ -568,16 +568,17 @@ rte_table_hash_entry_delete_key8_ext(
 		pos = 3;					\
 }
 
-#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask)	\
+#define lookup1_stage0(pkt0_index, mbuf0, pkts, pkts_mask, f)	\
 {								\
 	uint64_t pkt_mask;					\
+	uint32_t key_offset = f->key_offset;\
 								\
 	pkt0_index = __builtin_ctzll(pkts_mask);		\
 	pkt_mask = 1LLU << pkt0_index;				\
 	pkts_mask &= ~pkt_mask;					\
 								\
 	mbuf0 = pkts[pkt0_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf0, key_offset));	\
 }
 
 #define lookup1_stage1(mbuf1, bucket1, f)			\
@@ -693,36 +694,38 @@ rte_table_hash_entry_delete_key8_ext(
 }
 
 #define lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01,\
-	pkts, pkts_mask)					\
+	pkts, pkts_mask, f)					\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	pkt01_mask = 1LLU << pkt01_index;			\
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,\
-	mbuf00, mbuf01, pkts, pkts_mask)			\
+	mbuf00, mbuf01, pkts, pkts_mask, f)			\
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
+	uint32_t key_offset = f->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
 	pkts_mask &= ~pkt00_mask;				\
 								\
 	mbuf00 = pkts[pkt00_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
 								\
 	pkt01_index = __builtin_ctzll(pkts_mask);		\
 	if (pkts_mask == 0)					\
@@ -732,7 +735,7 @@ rte_table_hash_entry_delete_key8_ext(
 	pkts_mask &= ~pkt01_mask;				\
 								\
 	mbuf01 = pkts[pkt01_index];				\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f)	\
@@ -882,7 +885,7 @@ rte_table_hash_lookup_key8_lru(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_lru(pkt_index, mbuf, bucket,
 					pkts_mask_out, entries, f);
@@ -899,7 +902,7 @@ rte_table_hash_lookup_key8_lru(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -909,7 +912,7 @@ rte_table_hash_lookup_key8_lru(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -933,7 +936,7 @@ rte_table_hash_lookup_key8_lru(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1008,7 +1011,7 @@ rte_table_hash_lookup_key8_lru_dosig(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1_dosig(mbuf, bucket, f);
 			lookup1_stage2_lru(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, f);
@@ -1025,7 +1028,7 @@ rte_table_hash_lookup_key8_lru_dosig(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1035,7 +1038,7 @@ rte_table_hash_lookup_key8_lru_dosig(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1059,7 +1062,7 @@ rte_table_hash_lookup_key8_lru_dosig(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1136,7 +1139,7 @@ rte_table_hash_lookup_key8_ext(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1(mbuf, bucket, f);
 			lookup1_stage2_ext(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, buckets_mask, buckets,
@@ -1152,7 +1155,7 @@ rte_table_hash_lookup_key8_ext(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1162,7 +1165,7 @@ rte_table_hash_lookup_key8_ext(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1186,7 +1189,7 @@ rte_table_hash_lookup_key8_ext(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1286,7 +1289,7 @@ rte_table_hash_lookup_key8_ext_dosig(
 			struct rte_mbuf *mbuf;
 			uint32_t pkt_index;
 
-			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask);
+			lookup1_stage0(pkt_index, mbuf, pkts, pkts_mask, f);
 			lookup1_stage1_dosig(mbuf, bucket, f);
 			lookup1_stage2_ext(pkt_index, mbuf, bucket,
 				pkts_mask_out, entries, buckets_mask,
@@ -1302,7 +1305,7 @@ rte_table_hash_lookup_key8_ext_dosig(
 	 */
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline feed */
 	mbuf10 = mbuf00;
@@ -1312,7 +1315,7 @@ rte_table_hash_lookup_key8_ext_dosig(
 
 	/* Pipeline stage 0 */
 	lookup2_stage0(pkt00_index, pkt01_index, mbuf00, mbuf01, pkts,
-		pkts_mask);
+		pkts_mask, f);
 
 	/* Pipeline stage 1 */
 	lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
@@ -1336,7 +1339,7 @@ rte_table_hash_lookup_key8_ext_dosig(
 
 		/* Pipeline stage 0 */
 		lookup2_stage0_with_odd_support(pkt00_index, pkt01_index,
-			mbuf00, mbuf01, pkts, pkts_mask);
+			mbuf00, mbuf01, pkts, pkts_mask, f);
 
 		/* Pipeline stage 1 */
 		lookup2_stage1_dosig(mbuf10, mbuf11, bucket10, bucket11, f);
diff --git a/lib/librte_table/rte_table_hash_lru.c b/lib/librte_table/rte_table_hash_lru.c
index 1640dc9..b9b88e9 100644
--- a/lib/librte_table/rte_table_hash_lru.c
+++ b/lib/librte_table/rte_table_hash_lru.c
@@ -576,6 +576,7 @@ static int rte_table_hash_lru_lookup_unoptimized(
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
 	struct rte_mbuf *mbuf00, *mbuf01;			\
+	uint32_t key_offset = t->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
@@ -587,8 +588,8 @@ static int rte_table_hash_lru_lookup_unoptimized(
 	pkts_mask &= ~pkt01_mask;				\
 	mbuf01 = pkts[pkt01_index];				\
 								\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage0_with_odd_support(t, g, pkts, pkts_mask, pkt00_index, \
@@ -596,6 +597,7 @@ static int rte_table_hash_lru_lookup_unoptimized(
 {								\
 	uint64_t pkt00_mask, pkt01_mask;			\
 	struct rte_mbuf *mbuf00, *mbuf01;			\
+	uint32_t key_offset = t->key_offset;		\
 								\
 	pkt00_index = __builtin_ctzll(pkts_mask);		\
 	pkt00_mask = 1LLU << pkt00_index;			\
@@ -610,8 +612,8 @@ static int rte_table_hash_lru_lookup_unoptimized(
 	pkts_mask &= ~pkt01_mask;				\
 	mbuf01 = pkts[pkt01_index];				\
 								\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, 0));	\
-	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, 0));	\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf00, key_offset));\
+	rte_prefetch0(RTE_MBUF_METADATA_UINT8_PTR(mbuf01, key_offset));\
 }
 
 #define lookup2_stage1(t, g, pkts, pkt10_index, pkt11_index)	\
-- 
2.1.0

Re: [dpdk-dev] [PATCH v4 0/7] librte_table: add key_mask parameter to

[Thomas Monjalon]

2015-10-28 17:11, roy.fan.zhang@intel.com:
> From: Fan Zhang <roy.fan.zhang@intel.com>
> 
> This patchset links to ABI change announced for librte_table.
> The key_mask parameters has been added to the hash table
> parameter structure for 8-byte key and 16-byte key extendible
> bucket and LRU tables.
> 
> v2:
> *updated release note
> 
> v3:
> *merged release note with source code patch
> *fixed build error: added missing symbol to
> librte_table/rte_table_version.map
> 
> v4:
> *modified rte_prefetch offsets to improve hash/lru table
> lookup performance. 
> 
> Acked-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

Please put this line in each commit.

> Fan Zhang (7):
>   librte_table: add key_mask parameter to 8- and 16-bytes key hash
>     parameters
>   librte_table: add 16 byte hash table operations with computed lookup
>   app/test: modify app/test_table_combined and app/test_table_tables
>   app/test-pipeline: modify pipeline test
>   example/ip_pipeline: add parse_hex_string for internal use
>   example/ip_pipeline/pipeline: update flow_classification pipeline
>   librte_table: performance improvement on rte_prefetch offset

Applied, thanks