DPDK patches and discussions
 help / color / mirror / Atom feed
From: Yipeng Wang <yipeng1.wang@intel.com>
To: pablo.de.lara.guarch@intel.com
Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com,
	honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com,
	brijesh.s.singh@gmail.com
Subject: [dpdk-dev] [PATCH v4 0/8] Add read-write concurrency to rte_hash library
Date: Mon,  9 Jul 2018 03:44:55 -0700	[thread overview]
Message-ID: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> (raw)
In-Reply-To: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com>

This patch set adds the read-write concurrency support in rte_hash.
A new flag value is added to indicate if read-write concurrency is needed
during creation time. Test cases are implemented to do functional and
performance tests.

The new concurrency model is based on rte_rwlock. When Intel TSX is
available and the users indicate to use it, the TM version of the
rte_rwlock will be called. Both multi-writer and read-write concurrency
are protected by the rte_rwlock instead of the x86 specific RTM
instructions, so the x86 specific header rte_cuckoo_hash_x86.h is removed
and the code is infused into the main .c file.

A new rte_hash_count API is proposed to count how many keys are inserted
into the hash table.

v3->v4:
1. Change commit message titles as Pablo suggested. (Pablo)
2. hash: remove unnecessary changes in commit 4. (Pablo)
3. test: remove unnecessary double blank lines. (Pablo)
4. Add Pablo's ack in commit message.

v2->v3:
1. hash: Concurrency bug fix: after beginning cuckoo path moving,
the last empty slot needs to be verified again in case other writers
raced into this slot and occupy it. A new commit is added to do this
bug fix since it applies to master head as well.
2. hash: Concurrency bug fix: if cuckoo path is detected to be invalid,
the current slot needs to be emptied since it is duplicated to its
target bucket.
3. hash: "const" is used for types in multiple locations. (Pablo)
4. hash: rte_malloc used for readwriter lock used wrong align
argument. Similar fix applies to master head so a new commit is
created. (Pablo)
5. hash: ring size calculation fix is moved to front. (Pablo)
6. hash: search-and-remove function is refactored to be more aligned
with other search function. (Pablo)
7. test: using jhash in functional test for read-write concurrency.
It is because jhash with sequential keys incur more cuckoo path.
8. Multiple coding style, typo, commit message fixes. (Pablo)

v1->v2:
1. Split each commit into two commits for easier review (Pablo).
2. Add more comments in various places (Pablo).
3. hash: In key insertion function, move duplicated key checking to
earlier location and protect it using locks. Checking duplicated key
should happen first and data updates should be protected.
4. hash: In lookup bulk function, put signature comparison in lock,
since writers could happen between signature match on two buckets.
5. hash: Add write locks to reset function as well to protect resets.
5. test: Fix 32-bit compilation error in read-write test (Pablo).
6. test: Check total physical core count in read-write test. Don't
test with thread count that larger than physical core count.
7. Other minor fixes such as typos (Pablo).


Yipeng Wang (8):
  hash: fix multiwriter lock memory allocation
  hash: fix a multi-writer race condition
  hash: fix key slot size accuracy
  hash: make duplicated code into functions
  hash: add read and write concurrency support
  test: add tests in hash table perf test
  test: add test case for read write concurrency
  hash: add new API function to query the key count

 lib/librte_hash/meson.build           |   1 -
 lib/librte_hash/rte_cuckoo_hash.c     | 701 +++++++++++++++++++++-------------
 lib/librte_hash/rte_cuckoo_hash.h     |  18 +-
 lib/librte_hash/rte_cuckoo_hash_x86.h | 164 --------
 lib/librte_hash/rte_hash.h            |  14 +
 lib/librte_hash/rte_hash_version.map  |   8 +
 test/test/Makefile                    |   1 +
 test/test/test_hash.c                 |  12 +
 test/test/test_hash_multiwriter.c     |   9 +
 test/test/test_hash_perf.c            |  36 +-
 test/test/test_hash_readwrite.c       | 637 ++++++++++++++++++++++++++++++
 11 files changed, 1156 insertions(+), 445 deletions(-)
 delete mode 100644 lib/librte_hash/rte_cuckoo_hash_x86.h
 create mode 100644 test/test/test_hash_readwrite.c

-- 
2.7.4

  parent reply	other threads:[~2018-07-09 17:52 UTC|newest]

Thread overview: 65+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-06-08 10:51 [dpdk-dev] [PATCH v1 0/3] " Yipeng Wang
2018-06-08 10:51 ` [dpdk-dev] [PATCH v1 1/3] hash: add read and write concurrency support Yipeng Wang
2018-06-26 14:59   ` De Lara Guarch, Pablo
2018-06-08 10:51 ` [dpdk-dev] [PATCH v1 2/3] test: add test case for read write concurrency Yipeng Wang
2018-06-26 15:48   ` De Lara Guarch, Pablo
2018-06-08 10:51 ` [dpdk-dev] [PATCH v1 3/3] hash: add new API function to query the key count Yipeng Wang
2018-06-26 16:11   ` De Lara Guarch, Pablo
2018-06-29 12:24 ` [dpdk-dev] [PATCH v2 0/6] Add read-write concurrency to rte_hash library Yipeng Wang
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 1/6] hash: make duplicated code into functions Yipeng Wang
2018-07-06 10:04     ` De Lara Guarch, Pablo
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 2/6] hash: add read and write concurrency support Yipeng Wang
2018-07-06 17:11     ` De Lara Guarch, Pablo
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 3/6] test: add tests in hash table perf test Yipeng Wang
2018-07-06 17:17     ` De Lara Guarch, Pablo
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 4/6] test: add test case for read write concurrency Yipeng Wang
2018-07-06 17:31     ` De Lara Guarch, Pablo
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 5/6] hash: fix to have more accurate key slot size Yipeng Wang
2018-07-06 17:32     ` De Lara Guarch, Pablo
2018-06-29 12:24   ` [dpdk-dev] [PATCH v2 6/6] hash: add new API function to query the key count Yipeng Wang
2018-07-06 17:36     ` De Lara Guarch, Pablo
2018-07-06 19:46 ` [dpdk-dev] [PATCH v3 0/8] Add read-write concurrency to rte_hash library Yipeng Wang
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 1/8] hash: fix multiwriter lock memory allocation Yipeng Wang
2018-07-09 11:26     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 2/8] hash: fix a multi-writer bug Yipeng Wang
2018-07-09 14:16     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 3/8] hash: fix to have more accurate key slot size Yipeng Wang
2018-07-09 14:20     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 4/8] hash: make duplicated code into functions Yipeng Wang
2018-07-09 14:25     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 5/8] hash: add read and write concurrency support Yipeng Wang
2018-07-09 14:28     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 6/8] test: add tests in hash table perf test Yipeng Wang
2018-07-09 15:33     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 7/8] test: add test case for read write concurrency Yipeng Wang
2018-07-09 16:24     ` De Lara Guarch, Pablo
2018-07-06 19:46   ` [dpdk-dev] [PATCH v3 8/8] hash: add new API function to query the key count Yipeng Wang
2018-07-09 16:22     ` De Lara Guarch, Pablo
2018-07-09 10:44 ` Yipeng Wang [this message]
2018-07-09 10:44   ` [dpdk-dev] [PATCH v4 1/8] hash: fix multiwriter lock memory allocation Yipeng Wang
2018-07-09 10:44   ` [dpdk-dev] [PATCH v4 2/8] hash: fix a multi-writer race condition Yipeng Wang
2018-07-09 10:44   ` [dpdk-dev] [PATCH v4 3/8] hash: fix key slot size accuracy Yipeng Wang
2018-07-09 10:44   ` [dpdk-dev] [PATCH v4 4/8] hash: make duplicated code into functions Yipeng Wang
2018-07-09 10:45   ` [dpdk-dev] [PATCH v4 5/8] hash: add read and write concurrency support Yipeng Wang
2018-07-09 10:45   ` [dpdk-dev] [PATCH v4 6/8] test: add tests in hash table perf test Yipeng Wang
2018-07-09 10:45   ` [dpdk-dev] [PATCH v4 7/8] test: add test case for read write concurrency Yipeng Wang
2018-07-09 10:45   ` [dpdk-dev] [PATCH v4 8/8] hash: add new API function to query the key count Yipeng Wang
2018-07-10 18:00   ` [dpdk-dev] [PATCH v4 0/8] Add read-write concurrency to rte_hash library Honnappa Nagarahalli
2018-07-12  1:31     ` Wang, Yipeng1
2018-07-12  2:36       ` Honnappa Nagarahalli
2018-07-13  1:47         ` Wang, Yipeng1
2018-07-10 16:59 ` [dpdk-dev] [PATCH v5 " Yipeng Wang
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 1/8] hash: fix multiwriter lock memory allocation Yipeng Wang
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 2/8] hash: fix a multi-writer race condition Yipeng Wang
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 3/8] hash: fix key slot size accuracy Yipeng Wang
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 4/8] hash: make duplicated code into functions Yipeng Wang
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 5/8] hash: add read and write concurrency support Yipeng Wang
2018-07-11 20:49     ` Stephen Hemminger
2018-07-12  1:22       ` Wang, Yipeng1
2018-07-12 20:30         ` Thomas Monjalon
2018-07-13  1:55           ` Wang, Yipeng1
2018-08-17 12:51             ` ASM
2018-07-10 16:59   ` [dpdk-dev] [PATCH v5 6/8] test: add tests in hash table perf test Yipeng Wang
2018-07-10 17:00   ` [dpdk-dev] [PATCH v5 7/8] test: add test case for read write concurrency Yipeng Wang
2018-07-10 17:00   ` [dpdk-dev] [PATCH v5 8/8] hash: add new API function to query the key count Yipeng Wang
2018-07-12 21:03   ` [dpdk-dev] [PATCH v5 0/8] Add read-write concurrency to rte_hash library Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1531133103-437316-1-git-send-email-yipeng1.wang@intel.com \
    --to=yipeng1.wang@intel.com \
    --cc=brijesh.s.singh@gmail.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=pablo.de.lara.guarch@intel.com \
    --cc=vguvva@caviumnetworks.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).