DPDK patches and discussions
 help / color / Atom feed
* [dpdk-dev] [RFC 0/6] New sync modes for ring
@ 2020-02-24 11:35 Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
                   ` (7 more replies)
  0 siblings, 8 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Upfront note - that RFC is not a complete patch.
It introduces an ABI breakage, plus it doesn't update ring_elem
code properly, etc.
I plan to deal with all these things in later versions.
Right now I seek an initial feedback about proposed ideas.
Would also ask people to repeat performance tests (see below)
on their platforms to confirm the impact.

More and more customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
LHP is quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot.
This is a well-known problem for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve this problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly.
So this RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
    only one thread at a time is allowed to perform given op.
    As another enhancement provide ability to split enqueue/dequeue
    operation into two phases:
      - enqueue/dequeue start
      - enqueue/dequeue finish
    That allows user to inspect objects in the ring without removing
    them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.

Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by ring_stress_*autotest
(first patch in these series).

X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj

                                                MP/MC      HTS     RTS
1thread@1core(--lcores=6-7)                     8.00       8.15    8.99
2thread@2core(--lcores=6-8)                     19.14      19.61   20.35
4thread@4core(--lcores=6-10)                    29.43      29.79   31.82
8thread@8core(--lcores=6-14)                    110.59     192.81  119.50
16thread@16core(--lcores=6-22)                  461.03     813.12  495.59
32thread/@32core(--lcores='6-22,55-70')         982.90     1972.38 1160.51

2thread@1core(--lcores='6,(10-11)@7'            20140.50   23.58   25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8'  153680.60  76.88   80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  280314.32  294.72  318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59  1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68

8thread@2core(--lcores='6,(10-17)@(7,8))'       321085.98  298.59  307.47
16thread@4core(--lcores='6,(20-35)@(7-10))'     1900705.61 575.35  678.29
32thread@4core(--lcores='6,(20-51)@(7-10))'     5510445.85 2164.36 2714.12

i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj

                                                MP/MC      HTS     RTS
1thread@1core(--lcores=6-7)                     7.85       12.13   11.31
2thread@2core(--lcores=6-8)                     17.89      24.52   21.86
8thread@8core(--lcores=6-14)                    32.58      354.20  54.58
32thread/@32core(--lcores='6-22,55-70')         813.77     6072.41 2169.91

2thread@1core(--lcores='6,(10-11)@7'            16095.00   36.06   34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  1140354.54 346.61  361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65

8thread@2core(--lcores='6,(10-17)@(7,8))'       594358.61  332.70  357.74
32thread@4core(--lcores='6,(20-51)@(7-10))'     5319896.86 2836.44 3028.87

Konstantin Ananyev (6):
  test/ring: add contention stress test
  ring: rework ring layout to allow new sync schemes
  ring: introduce RTS ring mode
  test/ring: add contention stress test for RTS ring
  ring: introduce HTS ring mode
  test/ring: add contention stress test for HTS ring

 app/test/Makefile                      |   3 +
 app/test/meson.build                   |   3 +
 app/test/test_pdump.c                  |   6 +-
 app/test/test_ring_hts_stress.c        |  28 ++
 app/test/test_ring_rts_stress.c        |  28 ++
 app/test/test_ring_stress.c            |  27 ++
 app/test/test_ring_stress.h            | 477 +++++++++++++++++++
 lib/librte_pdump/rte_pdump.c           |   2 +-
 lib/librte_port/rte_port_ring.c        |  12 +-
 lib/librte_ring/Makefile               |   4 +-
 lib/librte_ring/meson.build            |   4 +-
 lib/librte_ring/rte_ring.c             |  84 +++-
 lib/librte_ring/rte_ring.h             | 619 +++++++++++++++++++++++--
 lib/librte_ring/rte_ring_elem.h        |   8 +-
 lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
 lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
 16 files changed, 1721 insertions(+), 52 deletions(-)
 create mode 100644 app/test/test_ring_hts_stress.c
 create mode 100644 app/test/test_ring_rts_stress.c
 create mode 100644 app/test/test_ring_stress.c
 create mode 100644 app/test/test_ring_stress.h
 create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
 create mode 100644 lib/librte_ring/rte_ring_rts_generic.h

-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 1/6] test/ring: add contention stress test
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes Konstantin Ananyev
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Introduce new test-case to measure ring perfomance under contention
(miltiple producers/consumers).
Starts dequeue/enqueue loop on all available slave lcores.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/Makefile           |   1 +
 app/test/meson.build        |   1 +
 app/test/test_ring_stress.c |  27 ++
 app/test/test_ring_stress.h | 477 ++++++++++++++++++++++++++++++++++++
 4 files changed, 506 insertions(+)
 create mode 100644 app/test/test_ring_stress.c
 create mode 100644 app/test/test_ring_stress.h

diff --git a/app/test/Makefile b/app/test/Makefile
index 1f080d162..4f586d95f 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -78,6 +78,7 @@ SRCS-y += test_rand_perf.c
 
 SRCS-y += test_ring.c
 SRCS-y += test_ring_perf.c
+SRCS-y += test_ring_stress.c
 SRCS-y += test_pmd_perf.c
 
 ifeq ($(CONFIG_RTE_LIBRTE_TABLE),y)
diff --git a/app/test/meson.build b/app/test/meson.build
index 0a2ce710f..84dde28ad 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files('commands.c',
 	'test_rib6.c',
 	'test_ring.c',
 	'test_ring_perf.c',
+	'test_ring_stress.c',
 	'test_rwlock.c',
 	'test_sched.c',
 	'test_service_cores.c',
diff --git a/app/test/test_ring_stress.c b/app/test/test_ring_stress.c
new file mode 100644
index 000000000..5689e06c8
--- /dev/null
+++ b/app/test/test_ring_stress.c
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "test_ring_stress.h"
+
+static inline uint32_t
+_st_ring_dequeue_bulk(struct rte_ring *r, void **obj, uint32_t n,
+	uint32_t *avail)
+{
+	return rte_ring_mc_dequeue_bulk(r, obj, n, avail);
+}
+
+static inline uint32_t
+_st_ring_enqueue_bulk(struct rte_ring *r, void * const *obj, uint32_t n,
+	uint32_t *free)
+{
+	return rte_ring_mp_enqueue_bulk(r, obj, n, free);
+}
+
+static int
+_st_ring_init(struct rte_ring *r, const char *name, uint32_t num)
+{
+	return rte_ring_init(r, name, num, 0);
+}
+
+REGISTER_TEST_COMMAND(ring_stress_autotest, test_ring_stress);
diff --git a/app/test/test_ring_stress.h b/app/test/test_ring_stress.h
new file mode 100644
index 000000000..c6f0bc9f1
--- /dev/null
+++ b/app/test/test_ring_stress.h
@@ -0,0 +1,477 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdalign.h>
+#include <string.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <rte_ring.h>
+#include <rte_cycles.h>
+#include <rte_launch.h>
+#include <rte_pause.h>
+#include <rte_random.h>
+#include <rte_malloc.h>
+
+#include "test.h"
+
+/*
+ * Measures performance of ring enqueue/dequeue under high contention
+ */
+
+#define RING_NAME	"RING_STRESS"
+#define BULK_NUM	32
+#define RING_SIZE	(2 * BULK_NUM * RTE_MAX_LCORE)
+
+enum {
+	WRK_CMD_STOP,
+	WRK_CMD_RUN,
+};
+
+static volatile uint32_t wrk_cmd __rte_cache_aligned;
+
+/* test run-time in seconds */
+static const uint32_t run_time = 60;
+
+struct lcore_stat {
+	uint64_t nb_cycle;
+	struct {
+		uint64_t nb_call;
+		uint64_t nb_obj;
+		uint64_t nb_cycle;
+		uint64_t max_cycle;
+		uint64_t min_cycle;
+	} op;
+};
+
+struct lcore_arg {
+	struct rte_ring *rng;
+	struct lcore_stat stats;
+} __rte_cache_aligned;
+
+struct ring_elem {
+	uint32_t cnt[RTE_CACHE_LINE_SIZE / sizeof(uint32_t)];
+} __rte_cache_aligned;
+
+/*
+ * redefinable functions
+ */
+static uint32_t
+_st_ring_dequeue_bulk(struct rte_ring *r, void **obj, uint32_t n,
+	uint32_t *avail);
+
+static uint32_t
+_st_ring_enqueue_bulk(struct rte_ring *r, void * const *obj, uint32_t n,
+	uint32_t *free);
+
+static int
+_st_ring_init(struct rte_ring *r, const char *name, uint32_t num);
+
+
+static void
+lcore_stat_update(struct lcore_stat *ls, uint64_t call, uint64_t obj,
+	uint64_t tm, int32_t prcs)
+{
+	ls->op.nb_call += call;
+	ls->op.nb_obj += obj;
+	ls->op.nb_cycle += tm;
+	if (prcs) {
+		ls->op.max_cycle = RTE_MAX(ls->op.max_cycle, tm);
+		ls->op.min_cycle = RTE_MIN(ls->op.min_cycle, tm);
+	}
+}
+
+static void
+lcore_op_stat_aggr(struct lcore_stat *ms, const struct lcore_stat *ls)
+{
+
+	ms->op.nb_call += ls->op.nb_call;
+	ms->op.nb_obj += ls->op.nb_obj;
+	ms->op.nb_cycle += ls->op.nb_cycle;
+	ms->op.max_cycle = RTE_MAX(ms->op.max_cycle, ls->op.max_cycle);
+	ms->op.min_cycle = RTE_MIN(ms->op.min_cycle, ls->op.min_cycle);
+}
+
+static void
+lcore_stat_aggr(struct lcore_stat *ms, const struct lcore_stat *ls)
+{
+	ms->nb_cycle = RTE_MAX(ms->nb_cycle, ls->nb_cycle);
+	lcore_op_stat_aggr(ms, ls);
+}
+
+static void
+lcore_stat_dump(FILE *f, uint32_t lc, const struct lcore_stat *ls)
+{
+	long double st;
+
+	st = (long double)rte_get_timer_hz() / US_PER_S;
+
+	if (lc == UINT32_MAX)
+		fprintf(f, "%s(AGGREGATE)={\n", __func__);
+	else
+		fprintf(f, "%s(lc=%u)={\n", __func__, lc);
+
+	fprintf(f, "\tnb_cycle=%" PRIu64 "(%.2Lf usec),\n",
+		ls->nb_cycle, (long double)ls->nb_cycle / st);
+
+	fprintf(f, "\tDEQ+ENQ={\n");
+
+	fprintf(f, "\t\tnb_call=%" PRIu64 ",\n", ls->op.nb_call);
+	fprintf(f, "\t\tnb_obj=%" PRIu64 ",\n", ls->op.nb_obj);
+	fprintf(f, "\t\tnb_cycle=%" PRIu64 ",\n", ls->op.nb_cycle);
+	fprintf(f, "\t\tobj/call(avg): %.2Lf\n",
+		(long double)ls->op.nb_obj / ls->op.nb_call);
+	fprintf(f, "\t\tcycles/obj(avg): %.2Lf\n",
+		(long double)ls->op.nb_cycle / ls->op.nb_obj);
+	fprintf(f, "\t\tcycles/call(avg): %.2Lf\n",
+		(long double)ls->op.nb_cycle / ls->op.nb_call);
+
+	/* if min/max cycles per call stats was collected */
+	if (ls->op.min_cycle != UINT64_MAX) {
+		fprintf(f, "\t\tmax cycles/call=%" PRIu64 "(%.2Lf usec),\n",
+			ls->op.max_cycle,
+			(long double)ls->op.max_cycle / st);
+		fprintf(f, "\t\tmin cycles/call=%" PRIu64 "(%.2Lf usec),\n",
+			ls->op.min_cycle,
+			(long double)ls->op.min_cycle / st);
+	}
+
+	fprintf(f, "\t},\n");
+	fprintf(f, "};\n");
+}
+
+static void
+fill_ring_elm(struct ring_elem *elm, uint32_t fill)
+{
+	uint32_t i;
+
+	for (i = 0; i != RTE_DIM(elm->cnt); i++)
+		elm->cnt[i] = fill;
+}
+
+static int32_t
+check_updt_elem(struct ring_elem *elm[], uint32_t num,
+	const struct ring_elem *check, const struct ring_elem *fill)
+{
+	uint32_t i;
+
+	static rte_spinlock_t dump_lock;
+
+	for (i = 0; i != num; i++) {
+		if (memcmp(check, elm[i], sizeof(*check)) != 0) {
+			rte_spinlock_lock(&dump_lock);
+			printf("%s(lc=%u, num=%u) failed at %u-th iter, "
+				"offending object: %p\n",
+				__func__, rte_lcore_id(), num, i, elm[i]);
+			rte_memdump(stdout, "expected", check, sizeof(*check));
+			rte_memdump(stdout, "result", elm[i], sizeof(elm[i]));
+			rte_spinlock_unlock(&dump_lock);
+			return -EINVAL;
+		}
+		memcpy(elm[i], fill, sizeof(*elm[i]));
+	}
+
+	return 0;
+}
+
+static int
+check_ring_op(uint32_t exp, uint32_t res, uint32_t lc,
+	const char *fname, const char *opname)
+{
+	if (exp != res) {
+		printf("%s(lc=%u) failure: %s expected: %u, returned %u\n",
+			fname, lc, opname, exp, res);
+		return -ENOSPC;
+	}
+	return 0;
+}
+
+static int
+test_worker_prcs(void *arg)
+{
+	int32_t rc;
+	uint32_t lc, n, num;
+	uint64_t cl, tm0, tm1;
+	struct lcore_arg *la;
+	struct ring_elem def_elm, loc_elm;
+	struct ring_elem *obj[2 * BULK_NUM];
+
+	la = arg;
+	lc = rte_lcore_id();
+
+	fill_ring_elm(&def_elm, UINT32_MAX);
+	fill_ring_elm(&loc_elm, lc);
+
+	while (wrk_cmd != WRK_CMD_RUN) {
+		rte_smp_rmb();
+		rte_pause();
+	}
+
+	cl = rte_rdtsc_precise();
+
+	do {
+		/* num in interval [7/8, 11/8] of BULK_NUM */
+		num = 7 * BULK_NUM / 8 + rte_rand() % (BULK_NUM / 2);
+
+		/* reset all pointer values */
+		memset(obj, 0, sizeof(obj));
+
+		/* dequeue num elems */
+		tm0 = rte_rdtsc_precise();
+		n = _st_ring_dequeue_bulk(la->rng, (void **)obj, num, NULL);
+		tm0 = rte_rdtsc_precise() - tm0;
+
+		/* check return value and objects */
+		rc = check_ring_op(num, n, lc, __func__,
+			RTE_STR(_st_ring_dequeue_bulk));
+		if (rc == 0)
+			rc = check_updt_elem(obj, num, &def_elm, &loc_elm);
+		if (rc != 0)
+			break;
+
+		/* enqueue num elems */
+		rte_compiler_barrier();
+		rc = check_updt_elem(obj, num, &loc_elm, &def_elm);
+		if (rc != 0)
+			break;
+
+		tm1 = rte_rdtsc_precise();
+		n = _st_ring_enqueue_bulk(la->rng, (void **)obj, num, NULL);
+		tm1 = rte_rdtsc_precise() - tm1;
+
+		/* check return value */
+		rc = check_ring_op(num, n, lc, __func__,
+			RTE_STR(_st_ring_enqueue_bulk));
+		if (rc != 0)
+			break;
+
+		lcore_stat_update(&la->stats, 1, num, tm0 + tm1, 1);
+
+	} while (wrk_cmd == WRK_CMD_RUN);
+
+	la->stats.nb_cycle = rte_rdtsc_precise() - cl;
+	return rc;
+}
+
+static int
+test_worker_avg(void *arg)
+{
+	int32_t rc;
+	uint32_t lc, n, num;
+	uint64_t cl;
+	struct lcore_arg *la;
+	struct ring_elem def_elm, loc_elm;
+	struct ring_elem *obj[2 * BULK_NUM];
+
+	la = arg;
+	lc = rte_lcore_id();
+
+	fill_ring_elm(&def_elm, UINT32_MAX);
+	fill_ring_elm(&loc_elm, lc);
+
+	while (wrk_cmd != WRK_CMD_RUN) {
+		rte_smp_rmb();
+		rte_pause();
+	}
+
+	cl = rte_rdtsc_precise();
+
+	do {
+		/* num in interval [7/8, 11/8] of BULK_NUM */
+		num = 7 * BULK_NUM / 8 + rte_rand() % (BULK_NUM / 2);
+
+		/* reset all pointer values */
+		memset(obj, 0, sizeof(obj));
+
+		/* dequeue num elems */
+		n = _st_ring_dequeue_bulk(la->rng, (void **)obj, num, NULL);
+
+		/* check return value and objects */
+		rc = check_ring_op(num, n, lc, __func__,
+			RTE_STR(_st_ring_dequeue_bulk));
+		if (rc == 0)
+			rc = check_updt_elem(obj, num, &def_elm, &loc_elm);
+		if (rc != 0)
+			break;
+
+		/* enqueue num elems */
+		rte_compiler_barrier();
+		rc = check_updt_elem(obj, num, &loc_elm, &def_elm);
+		if (rc != 0)
+			break;
+
+		n = _st_ring_enqueue_bulk(la->rng, (void **)obj, num, NULL);
+
+		/* check return value */
+		rc = check_ring_op(num, n, lc, __func__,
+			RTE_STR(_st_ring_enqueue_bulk));
+		if (rc != 0)
+			break;
+
+		lcore_stat_update(&la->stats, 1, num, 0, 0);
+
+	} while (wrk_cmd == WRK_CMD_RUN);
+
+	/* final stats update */
+	cl = rte_rdtsc_precise() - cl;
+	lcore_stat_update(&la->stats, 0, 0, cl, 0);
+	la->stats.nb_cycle = cl;
+
+	return rc;
+}
+
+static void
+mt1_fini(struct rte_ring *rng, void *data)
+{
+	rte_free(rng);
+	rte_free(data);
+}
+
+static int
+mt1_init(struct rte_ring **rng, void **data, uint32_t num)
+{
+	int32_t rc;
+	size_t sz;
+	uint32_t i, nr;
+	struct rte_ring *r;
+	struct ring_elem *elm;
+	void *p;
+
+	*rng = NULL;
+	*data = NULL;
+
+	sz = num * sizeof(*elm);
+	elm = rte_zmalloc(NULL, sz, alignof(*elm));
+
+	/* alloc ring */
+	nr = 2 * num;
+	sz = rte_ring_get_memsize(nr);
+	r = rte_zmalloc(NULL, sz, alignof(*r));
+	if (r == NULL) {
+		printf("%s: alloca(%zu) for FIFO with %u elems failed",
+			__func__, sz, nr);
+		return -ENOMEM;
+	}
+	rc = _st_ring_init(r, RING_NAME, nr);
+	if (rc != 0) {
+		printf("%s: _st_ring_init(%p, %u) failed, error: %d(%s)\n",
+			__func__, r, nr, rc, strerror(-rc));
+		return rc;
+	}
+
+	for (i = 0; i != num; i++) {
+		fill_ring_elm(elm + i, UINT32_MAX);
+		p = elm + i;
+		if (_st_ring_enqueue_bulk(r, &p, 1, NULL) != 1)
+			break;
+	}
+
+	if (i != num) {
+		printf("%s: _st_ring_enqueue(%p, %u) returned %u\n",
+			__func__, r, num, i);
+		return -ENOSPC;
+	}
+
+	*rng = r;
+	*data = elm;
+	return 0;
+}
+
+static int
+test_mt1(int (*test)(void *))
+{
+	int32_t rc;
+	uint32_t lc, mc;
+	struct rte_ring *r;
+	void *data;
+	struct lcore_arg arg[RTE_MAX_LCORE];
+
+	static const struct lcore_stat init_stat = {
+		.op.min_cycle = UINT64_MAX,
+	};
+
+	rc = mt1_init(&r, &data, RING_SIZE);
+	if (rc != 0) {
+		mt1_fini(r, data);
+		return rc;
+	}
+
+	memset(arg, 0, sizeof(arg));
+
+	/* launch on all slaves */
+	RTE_LCORE_FOREACH_SLAVE(lc) {
+		arg[lc].rng = r;
+		arg[lc].stats = init_stat;
+		rte_eal_remote_launch(test, &arg[lc], lc);
+	}
+
+	/* signal worker to start test */
+	wrk_cmd = WRK_CMD_RUN;
+	rte_smp_wmb();
+
+	usleep(run_time * US_PER_S);
+
+	/* signal worker to start test */
+	wrk_cmd = WRK_CMD_STOP;
+	rte_smp_wmb();
+
+	/* wait for slaves and collect stats. */
+	mc = rte_lcore_id();
+	arg[mc].stats = init_stat;
+
+	rc = 0;
+	RTE_LCORE_FOREACH_SLAVE(lc) {
+		rc |= rte_eal_wait_lcore(lc);
+		lcore_stat_dump(stdout, lc, &arg[lc].stats);
+		lcore_stat_aggr(&arg[mc].stats, &arg[lc].stats);
+	}
+
+	lcore_stat_dump(stdout, UINT32_MAX, &arg[mc].stats);
+	mt1_fini(r, data);
+	return rc;
+}
+
+static int
+test_ring_stress(void)
+{
+	int32_t rc;
+	uint32_t i, k;
+
+	const struct {
+		const char *name;
+		int (*func)(int (*)(void *));
+		int (*wfunc)(void *arg);
+	} tests[] = {
+		{
+			.name = "MT-WRK_ENQ_DEQ-MST_NONE-PRCS",
+			.func = test_mt1,
+			.wfunc = test_worker_prcs,
+		},
+		{
+			.name = "MT-WRK_ENQ_DEQ-MST_NONE-AVG",
+			.func = test_mt1,
+			.wfunc = test_worker_avg,
+		},
+	};
+
+	for (i = 0, k = 0; i != RTE_DIM(tests); i++) {
+
+		printf("TEST %s START\n", tests[i].name);
+
+		rc = tests[i].func(tests[i].wfunc);
+		k += (rc == 0);
+
+		if (rc != 0)
+			printf("TEST-CASE %s FAILED\n", tests[i].name);
+		else
+			printf("TEST-CASE %s OK\n", tests[i].name);
+	}
+
+	printf("Number of tests:\t%u\nSuccess:\t%u\nFailed:\t%u\n",
+		i, k, i - k);
+	return (k != i);
+}
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode Konstantin Ananyev
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Change from *single* to *sync_type* to allow different
synchronisation schemes to be applied.
Change layout to make sure that *sync_type* and *tail*
will always reside on same offsets.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/test_pdump.c           |  6 +--
 lib/librte_pdump/rte_pdump.c    |  2 +-
 lib/librte_port/rte_port_ring.c | 12 +++---
 lib/librte_ring/rte_ring.c      |  6 ++-
 lib/librte_ring/rte_ring.h      | 76 ++++++++++++++++++++++++---------
 lib/librte_ring/rte_ring_elem.h |  8 ++--
 6 files changed, 71 insertions(+), 39 deletions(-)

diff --git a/app/test/test_pdump.c b/app/test/test_pdump.c
index ad183184c..6a1180bcb 100644
--- a/app/test/test_pdump.c
+++ b/app/test/test_pdump.c
@@ -57,8 +57,7 @@ run_pdump_client_tests(void)
 	if (ret < 0)
 		return -1;
 	mp->flags = 0x0000;
-	ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
-				      RING_F_SP_ENQ | RING_F_SC_DEQ);
+	ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
 	if (ring_client == NULL) {
 		printf("rte_ring_create SR0 failed");
 		return -1;
@@ -71,9 +70,6 @@ run_pdump_client_tests(void)
 	}
 	rte_eth_dev_probing_finish(eth_dev);
 
-	ring_client->prod.single = 0;
-	ring_client->cons.single = 0;
-
 	printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
 
 	for (itr = 0; itr < NUM_ITR; itr++) {
diff --git a/lib/librte_pdump/rte_pdump.c b/lib/librte_pdump/rte_pdump.c
index 8a01ac510..65364f2c5 100644
--- a/lib/librte_pdump/rte_pdump.c
+++ b/lib/librte_pdump/rte_pdump.c
@@ -380,7 +380,7 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct rte_mempool *mp)
 		rte_errno = EINVAL;
 		return -1;
 	}
-	if (ring->prod.single || ring->cons.single) {
+	if (rte_ring_prod_single(ring) || rte_ring_cons_single(ring)) {
 		PDUMP_LOG(ERR, "ring with either SP or SC settings"
 		" is not valid for pdump, should have MP and MC settings\n");
 		rte_errno = EINVAL;
diff --git a/lib/librte_port/rte_port_ring.c b/lib/librte_port/rte_port_ring.c
index 47fcdd06a..2f6c050fa 100644
--- a/lib/librte_port/rte_port_ring.c
+++ b/lib/librte_port/rte_port_ring.c
@@ -44,8 +44,8 @@ rte_port_ring_reader_create_internal(void *params, int socket_id,
 	/* Check input parameters */
 	if ((conf == NULL) ||
 		(conf->ring == NULL) ||
-		(conf->ring->cons.single && is_multi) ||
-		(!(conf->ring->cons.single) && !is_multi)) {
+		(rte_ring_cons_single(conf->ring) && is_multi) ||
+		(!rte_ring_cons_single(conf->ring) && !is_multi)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
 	}
@@ -171,8 +171,8 @@ rte_port_ring_writer_create_internal(void *params, int socket_id,
 	/* Check input parameters */
 	if ((conf == NULL) ||
 		(conf->ring == NULL) ||
-		(conf->ring->prod.single && is_multi) ||
-		(!(conf->ring->prod.single) && !is_multi) ||
+		(rte_ring_prod_single(conf->ring) && is_multi) ||
+		(!rte_ring_prod_single(conf->ring) && !is_multi) ||
 		(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
@@ -440,8 +440,8 @@ rte_port_ring_writer_nodrop_create_internal(void *params, int socket_id,
 	/* Check input parameters */
 	if ((conf == NULL) ||
 		(conf->ring == NULL) ||
-		(conf->ring->prod.single && is_multi) ||
-		(!(conf->ring->prod.single) && !is_multi) ||
+		(rte_ring_prod_single(conf->ring) && is_multi) ||
+		(!rte_ring_prod_single(conf->ring) && !is_multi) ||
 		(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 77e5de099..fa5733907 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -106,8 +106,10 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.single = (flags & RING_F_SP_ENQ) ? __IS_SP : __IS_MP;
-	r->cons.single = (flags & RING_F_SC_DEQ) ? __IS_SC : __IS_MC;
+	r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
+		RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+	r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
+		RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index 18fc5d845..aa8c628eb 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -61,11 +61,22 @@ enum rte_ring_queue_behavior {
 #define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - \
 			   sizeof(RTE_RING_MZ_PREFIX) + 1)
 
-/* structure to hold a pair of head/tail values and other metadata */
+/** prod/cons sync types */
+enum {
+	RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
+	RTE_RING_SYNC_ST,     /**< single thread only */
+};
+
+/**
+ * structure to hold a pair of head/tail values and other metadata.
+ * used by RTE_RING_SYNC_MT, RTE_RING_SYNC_ST sync types.
+ * Depending on sync_type format of that structure might be different,
+ * but offset for *sync_type* and *tail* values should remain the same.
+ */
 struct rte_ring_headtail {
-	volatile uint32_t head;  /**< Prod/consumer head. */
-	volatile uint32_t tail;  /**< Prod/consumer tail. */
-	uint32_t single;         /**< True if single prod/cons */
+	uint32_t sync_type;                      /**< sync type of prod/cons */
+	volatile uint32_t tail __rte_aligned(8); /**< prod/consumer tail. */
+	volatile uint32_t head;                  /**< prod/consumer head. */
 };
 
 /**
@@ -116,11 +127,10 @@ struct rte_ring {
 #define RING_F_EXACT_SZ 0x0004
 #define RTE_RING_SZ_MASK  (0x7fffffffU) /**< Ring size mask */
 
-/* @internal defines for passing to the enqueue dequeue worker functions */
-#define __IS_SP 1
-#define __IS_MP 0
-#define __IS_SC 1
-#define __IS_MC 0
+#define __IS_SP RTE_RING_SYNC_ST
+#define __IS_MP RTE_RING_SYNC_MT
+#define __IS_SC RTE_RING_SYNC_ST
+#define __IS_MC RTE_RING_SYNC_MT
 
 /**
  * Calculate the memory size needed for a ring
@@ -420,7 +430,7 @@ rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 			 unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_MP, free_space);
+			RTE_RING_SYNC_MT, free_space);
 }
 
 /**
@@ -443,7 +453,7 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 			 unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_SP, free_space);
+			RTE_RING_SYNC_ST, free_space);
 }
 
 /**
@@ -470,7 +480,7 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 		      unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			r->prod.single, free_space);
+			r->prod.sync_type, free_space);
 }
 
 /**
@@ -554,7 +564,7 @@ rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table,
 		unsigned int n, unsigned int *available)
 {
 	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_MC, available);
+			RTE_RING_SYNC_MT, available);
 }
 
 /**
@@ -578,7 +588,7 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
 		unsigned int n, unsigned int *available)
 {
 	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			__IS_SC, available);
+			RTE_RING_SYNC_ST, available);
 }
 
 /**
@@ -605,7 +615,7 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
 		unsigned int *available)
 {
 	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-				r->cons.single, available);
+				r->cons.sync_type, available);
 }
 
 /**
@@ -777,6 +787,30 @@ rte_ring_get_capacity(const struct rte_ring *r)
 	return r->capacity;
 }
 
+static inline uint32_t
+rte_ring_get_prod_sync_type(const struct rte_ring *r)
+{
+	return r->prod.sync_type;
+}
+
+static inline int
+rte_ring_prod_single(const struct rte_ring *r)
+{
+	return (rte_ring_get_prod_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
+static inline uint32_t
+rte_ring_get_cons_sync_type(const struct rte_ring *r)
+{
+	return r->cons.sync_type;
+}
+
+static inline int
+rte_ring_cons_single(const struct rte_ring *r)
+{
+	return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST);
+}
+
 /**
  * Dump the status of all rings on the console
  *
@@ -820,7 +854,7 @@ rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 			 unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_MP, free_space);
+			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, free_space);
 }
 
 /**
@@ -843,7 +877,7 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 			 unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_SP, free_space);
+			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
 }
 
 /**
@@ -870,7 +904,7 @@ rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 		      unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
-			r->prod.single, free_space);
+			r->prod.sync_type, free_space);
 }
 
 /**
@@ -898,7 +932,7 @@ rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table,
 		unsigned int n, unsigned int *available)
 {
 	return __rte_ring_do_dequeue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_MC, available);
+			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_MT, available);
 }
 
 /**
@@ -923,7 +957,7 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
 		unsigned int n, unsigned int *available)
 {
 	return __rte_ring_do_dequeue(r, obj_table, n,
-			RTE_RING_QUEUE_VARIABLE, __IS_SC, available);
+			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
 }
 
 /**
@@ -951,7 +985,7 @@ rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
 {
 	return __rte_ring_do_dequeue(r, obj_table, n,
 				RTE_RING_QUEUE_VARIABLE,
-				r->cons.single, available);
+				r->cons.sync_type, available);
 }
 
 #ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_elem.h b/lib/librte_ring/rte_ring_elem.h
index 3976757ed..ff7a28ea5 100644
--- a/lib/librte_ring/rte_ring_elem.h
+++ b/lib/librte_ring/rte_ring_elem.h
@@ -570,7 +570,7 @@ rte_ring_enqueue_bulk_elem(struct rte_ring *r, const void *obj_table,
 		unsigned int esize, unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
-			RTE_RING_QUEUE_FIXED, r->prod.single, free_space);
+			RTE_RING_QUEUE_FIXED, r->prod.sync_type, free_space);
 }
 
 /**
@@ -734,7 +734,7 @@ rte_ring_dequeue_bulk_elem(struct rte_ring *r, void *obj_table,
 		unsigned int esize, unsigned int n, unsigned int *available)
 {
 	return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
-			RTE_RING_QUEUE_FIXED, r->cons.single, available);
+			RTE_RING_QUEUE_FIXED, r->cons.sync_type, available);
 }
 
 /**
@@ -902,7 +902,7 @@ rte_ring_enqueue_burst_elem(struct rte_ring *r, const void *obj_table,
 		unsigned int esize, unsigned int n, unsigned int *free_space)
 {
 	return __rte_ring_do_enqueue_elem(r, obj_table, esize, n,
-			RTE_RING_QUEUE_VARIABLE, r->prod.single, free_space);
+			RTE_RING_QUEUE_VARIABLE, r->prod.sync_type, free_space);
 }
 
 /**
@@ -995,7 +995,7 @@ rte_ring_dequeue_burst_elem(struct rte_ring *r, void *obj_table,
 {
 	return __rte_ring_do_dequeue_elem(r, obj_table, esize, n,
 				RTE_RING_QUEUE_VARIABLE,
-				r->cons.single, available);
+				r->cons.sync_type, available);
 }
 
 #ifdef __cplusplus
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring Konstantin Ananyev
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Introduce relaxed tail sync (RTS) mode for MT ring synchronization.
Aim to reduce stall times in case when ring is used on
overcommited cpus (multiple active threads on the same cpu).
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_ring/Makefile               |   3 +-
 lib/librte_ring/meson.build            |   3 +-
 lib/librte_ring/rte_ring.c             |  75 ++++++-
 lib/librte_ring/rte_ring.h             | 300 +++++++++++++++++++++++--
 lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++++++++++++
 5 files changed, 598 insertions(+), 23 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_rts_generic.h

diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 917c560ad..4f90344f4 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -18,6 +18,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
 SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
 					rte_ring_elem.h \
 					rte_ring_generic.h \
-					rte_ring_c11_mem.h
+					rte_ring_c11_mem.h \
+					rte_ring_rts_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index f2f3ccc88..dc8d7dbea 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -5,7 +5,8 @@ sources = files('rte_ring.c')
 headers = files('rte_ring.h',
 		'rte_ring_elem.h',
 		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h')
+		'rte_ring_generic.h',
+		'rte_ring_rts_generic.h')
 
 # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
 allow_experimental_apis = true
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index fa5733907..1ce0af3e5 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -45,6 +45,9 @@ EAL_REGISTER_TAILQ(rte_ring_tailq)
 /* true if x is a power of 2 */
 #define POWEROF2(x) ((((x)-1) & (x)) == 0)
 
+/* by default set head/tail distance as 1/8 of ring capacity */
+#define HTD_MAX_DEF	8
+
 /* return the size of memory occupied by a ring */
 ssize_t
 rte_ring_get_memsize_elem(unsigned int esize, unsigned int count)
@@ -82,8 +85,56 @@ rte_ring_get_memsize(unsigned int count)
 void
 rte_ring_reset(struct rte_ring *r)
 {
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+	memset((void *)(uintptr_t)&r->prod.tail, 0,
+		offsetof(struct rte_ring, pad1) -
+		offsetof(struct rte_ring, prod.tail));
+	memset((void *)(uintptr_t)&r->cons.tail, 0,
+		offsetof(struct rte_ring, pad2) -
+		offsetof(struct rte_ring, cons.tail));
+}
+
+/*
+ * helper function, calculates sync_type values for prod and cons
+ * based on input flags. Returns zero at success or negative
+ * errno value otherwise.
+ */
+static int
+get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)
+{
+	static const uint32_t prod_st_flags =
+		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+	static const uint32_t cons_st_flags =
+		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+
+	switch (flags & prod_st_flags) {
+	case 0:
+		*prod_st = RTE_RING_SYNC_MT;
+		break;
+	case RING_F_SP_ENQ:
+		*prod_st = RTE_RING_SYNC_ST;
+		break;
+	case RING_F_MP_RTS_ENQ:
+		*prod_st = RTE_RING_SYNC_MT_RTS;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	switch (flags & cons_st_flags) {
+	case 0:
+		*cons_st = RTE_RING_SYNC_MT;
+		break;
+	case RING_F_SC_DEQ:
+		*cons_st = RTE_RING_SYNC_ST;
+		break;
+	case RING_F_MC_RTS_DEQ:
+		*cons_st = RTE_RING_SYNC_MT_RTS;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
 }
 
 int
@@ -100,16 +151,20 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
 
+	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+		offsetof(struct rte_ring_rts_headtail, sync_type));
+	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+		offsetof(struct rte_ring_rts_headtail, tail.val.pos));
+
 	/* init the ring structure */
 	memset(r, 0, sizeof(*r));
 	ret = strlcpy(r->name, name, sizeof(r->name));
 	if (ret < 0 || ret >= (int)sizeof(r->name))
 		return -ENAMETOOLONG;
 	r->flags = flags;
-	r->prod.sync_type = (flags & RING_F_SP_ENQ) ?
-		RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
-	r->cons.sync_type = (flags & RING_F_SC_DEQ) ?
-		RTE_RING_SYNC_ST : RTE_RING_SYNC_MT;
+	ret = get_sync_type(flags, &r->prod.sync_type, &r->cons.sync_type);
+	if (ret != 0)
+		return ret;
 
 	if (flags & RING_F_EXACT_SZ) {
 		r->size = rte_align32pow2(count + 1);
@@ -126,8 +181,12 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 		r->mask = count - 1;
 		r->capacity = r->mask;
 	}
-	r->prod.head = r->cons.head = 0;
-	r->prod.tail = r->cons.tail = 0;
+
+	/* set default values for head-tail distance */
+	if (flags & RING_F_MP_RTS_ENQ)
+		rte_ring_set_prod_htd_max(r, r->capacity / HTD_MAX_DEF);
+	if (flags & RING_F_MC_RTS_DEQ)
+		rte_ring_set_cons_htd_max(r, r->capacity / HTD_MAX_DEF);
 
 	return 0;
 }
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index aa8c628eb..a130aeb9d 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -65,13 +65,15 @@ enum rte_ring_queue_behavior {
 enum {
 	RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
 	RTE_RING_SYNC_ST,     /**< single thread only */
+	RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
 };
 
 /**
  * structure to hold a pair of head/tail values and other metadata.
  * used by RTE_RING_SYNC_MT, RTE_RING_SYNC_ST sync types.
- * Depending on sync_type format of that structure might be different,
- * but offset for *sync_type* and *tail* values should remain the same.
+ * Depending on sync_type format of that structure might differ
+ * depending on the sync mechanism selelcted, but offsets for
+ * *sync_type* and *tail* values should always remain the same.
  */
 struct rte_ring_headtail {
 	uint32_t sync_type;                      /**< sync type of prod/cons */
@@ -79,6 +81,21 @@ struct rte_ring_headtail {
 	volatile uint32_t head;                  /**< prod/consumer head. */
 };
 
+union rte_ring_ht_poscnt {
+	uint64_t raw;
+	struct {
+		uint32_t pos; /**< head/tail position */
+		uint32_t cnt; /**< head/tail reference counter */
+	} val;
+};
+
+struct rte_ring_rts_headtail {
+	uint32_t sync_type; /**< sync type of prod/cons */
+	uint32_t htd_max;   /**< max allowed distance between head/tail */
+	volatile union rte_ring_ht_poscnt tail;
+	volatile union rte_ring_ht_poscnt head;
+};
+
 /**
  * An RTE ring structure.
  *
@@ -106,11 +123,21 @@ struct rte_ring {
 	char pad0 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring producer status. */
-	struct rte_ring_headtail prod __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail prod;
+		struct rte_ring_rts_headtail rts_prod;
+	}  __rte_cache_aligned;
+
 	char pad1 __rte_cache_aligned; /**< empty cache line */
 
 	/** Ring consumer status. */
-	struct rte_ring_headtail cons __rte_cache_aligned;
+	RTE_STD_C11
+	union {
+		struct rte_ring_headtail cons;
+		struct rte_ring_rts_headtail rts_cons;
+	}  __rte_cache_aligned;
+
 	char pad2 __rte_cache_aligned; /**< empty cache line */
 };
 
@@ -127,6 +154,9 @@ struct rte_ring {
 #define RING_F_EXACT_SZ 0x0004
 #define RTE_RING_SZ_MASK  (0x7fffffffU) /**< Ring size mask */
 
+#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
+#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
+
 #define __IS_SP RTE_RING_SYNC_ST
 #define __IS_MP RTE_RING_SYNC_MT
 #define __IS_SC RTE_RING_SYNC_ST
@@ -407,6 +437,82 @@ __rte_ring_do_dequeue(struct rte_ring *r, void **obj_table,
 	return n;
 }
 
+#include <rte_ring_rts_generic.h>
+
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue(struct rte_ring *r, void * const *obj_table,
+		uint32_t n, enum rte_ring_queue_behavior behavior,
+		uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		unsigned int *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+	if (n != 0) {
+		DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
@@ -456,6 +562,29 @@ rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 			RTE_RING_SYNC_ST, free_space);
 }
 
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ *   if non-NULL, returns the amount of space in the ring after the
+ *   enqueue operation has finished.
+ * @return
+ *   The number of objects enqueued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	return __rte_ring_do_rts_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+			free_space);
+}
+
 /**
  * Enqueue several objects on a ring.
  *
@@ -479,8 +608,18 @@ static __rte_always_inline unsigned int
 rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 		      unsigned int n, unsigned int *free_space)
 {
-	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-			r->prod.sync_type, free_space);
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_MT:
+		return rte_ring_mp_enqueue_bulk(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_ST:
+		return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_MT_RTS:
+		return rte_ring_rts_enqueue_bulk(r, obj_table, n, free_space);
+	}
+
+	/* valid ring should never reach this point */
+	RTE_ASSERT(0);
+	return 0;
 }
 
 /**
@@ -591,6 +730,29 @@ rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table,
 			RTE_RING_SYNC_ST, available);
 }
 
+/**
+ * Dequeue several objects from an RTS ring (multi-consumers safe).
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ *   The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ *   If non-NULL, returns the number of remaining ring entries after the
+ *   dequeue has finished.
+ * @return
+ *   The number of objects dequeued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	return __rte_ring_do_rts_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
+			available);
+}
+
 /**
  * Dequeue several objects from a ring.
  *
@@ -614,8 +776,18 @@ static __rte_always_inline unsigned int
 rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
 		unsigned int *available)
 {
-	return __rte_ring_do_dequeue(r, obj_table, n, RTE_RING_QUEUE_FIXED,
-				r->cons.sync_type, available);
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_MT:
+		return rte_ring_mc_dequeue_bulk(r, obj_table, n, available);
+	case RTE_RING_SYNC_ST:
+		return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
+	case RTE_RING_SYNC_MT_RTS:
+		return rte_ring_rts_dequeue_bulk(r, obj_table, n, available);
+	}
+
+	/* valid ring should never reach this point */
+	RTE_ASSERT(0);
+	return 0;
 }
 
 /**
@@ -811,6 +983,42 @@ rte_ring_cons_single(const struct rte_ring *r)
 	return (rte_ring_get_cons_sync_type(r) == RTE_RING_SYNC_ST);
 }
 
+static inline uint32_t
+rte_ring_get_prod_htd_max(const struct rte_ring *r)
+{
+	if (r->prod.sync_type == RTE_RING_SYNC_MT_RTS)
+		return r->rts_prod.htd_max;
+	return UINT32_MAX;
+}
+
+static inline int
+rte_ring_set_prod_htd_max(struct rte_ring *r, uint32_t v)
+{
+	if (r->prod.sync_type != RTE_RING_SYNC_MT_RTS)
+		return -ENOTSUP;
+
+	r->rts_prod.htd_max = v;
+	return 0;
+}
+
+static inline uint32_t
+rte_ring_get_cons_htd_max(const struct rte_ring *r)
+{
+	if (r->cons.sync_type == RTE_RING_SYNC_MT_RTS)
+		return r->rts_prod.htd_max;
+	return UINT32_MAX;
+}
+
+static inline int
+rte_ring_set_cons_htd_max(struct rte_ring *r, uint32_t v)
+{
+	if (r->cons.sync_type != RTE_RING_SYNC_MT_RTS)
+		return -ENOTSUP;
+
+	r->rts_cons.htd_max = v;
+	return 0;
+}
+
 /**
  * Dump the status of all rings on the console
  *
@@ -880,6 +1088,29 @@ rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, free_space);
 }
 
+/**
+ * Enqueue several objects on the RTS ring (multi-producers safe).
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ *   if non-NULL, returns the amount of space in the ring after the
+ *   enqueue operation has finished.
+ * @return
+ *   - n: Actual number of objects enqueued.
+ */
+static __rte_always_inline unsigned
+rte_ring_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	return __rte_ring_do_rts_enqueue(r, obj_table, n,
+			RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
 /**
  * Enqueue several objects on a ring.
  *
@@ -903,8 +1134,18 @@ static __rte_always_inline unsigned
 rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 		      unsigned int n, unsigned int *free_space)
 {
-	return __rte_ring_do_enqueue(r, obj_table, n, RTE_RING_QUEUE_VARIABLE,
-			r->prod.sync_type, free_space);
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_MT:
+		return rte_ring_mp_enqueue_burst(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_ST:
+		return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_MT_RTS:
+		return rte_ring_rts_enqueue_burst(r, obj_table, n, free_space);
+	}
+
+	/* valid ring should never reach this point */
+	RTE_ASSERT(0);
+	return 0;
 }
 
 /**
@@ -960,6 +1201,30 @@ rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table,
 			RTE_RING_QUEUE_VARIABLE, RTE_RING_SYNC_ST, available);
 }
 
+/**
+ * Dequeue several objects from an RTS  ring (multi-consumers safe).
+ * When the requested objects are more than the available objects,
+ * only dequeue the actual number of objects.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ *   The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ *   If non-NULL, returns the number of remaining ring entries after the
+ *   dequeue has finished.
+ * @return
+ *   - n: Actual number of objects dequeued, 0 if ring is empty
+ */
+static __rte_always_inline unsigned
+rte_ring_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	return __rte_ring_do_rts_dequeue(r, obj_table, n,
+			RTE_RING_QUEUE_VARIABLE, available);
+}
 /**
  * Dequeue multiple objects from a ring up to a maximum number.
  *
@@ -983,9 +1248,18 @@ static __rte_always_inline unsigned
 rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
 		unsigned int n, unsigned int *available)
 {
-	return __rte_ring_do_dequeue(r, obj_table, n,
-				RTE_RING_QUEUE_VARIABLE,
-				r->cons.sync_type, available);
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_MT:
+		return rte_ring_mc_dequeue_burst(r, obj_table, n, available);
+	case RTE_RING_SYNC_ST:
+		return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
+	case RTE_RING_SYNC_MT_RTS:
+		return rte_ring_rts_dequeue_burst(r, obj_table, n, available);
+	}
+
+	/* valid ring should never reach this point */
+	RTE_ASSERT(0);
+	return 0;
 }
 
 #ifdef __cplusplus
diff --git a/lib/librte_ring/rte_ring_rts_generic.h b/lib/librte_ring/rte_ring_rts_generic.h
new file mode 100644
index 000000000..add8630b2
--- /dev/null
+++ b/lib/librte_ring/rte_ring_rts_generic.h
@@ -0,0 +1,240 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2017 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_RTS_GENERIC_H_
+#define _RTE_RING_RTS_GENERIC_H_
+
+/**
+ * @file rte_ring_rts_generic.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for Relaxed Tail Sync (RTS) ring mode.
+ * The main idea remains the same as for our original MP/MC synchronization
+ * mechanism.
+ * The main difference is that tail value is increased not
+ * by every thread that finished enqueue/dequeue,
+ * but only by the last one doing enqueue/dequeue.
+ * That allows threads to skip spinning on tail value,
+ * leaving actual tail value change to last thread in the update queue.
+ * RTS requires 2 64-bit CAS for each enqueue(/dequeue) operation:
+ * one for head update, second for tail update.
+ * As a gain it allows thread to avoid spinning/waiting on tail value.
+ * In comparision original MP/MC algorithm requires one 32-bit CAS
+ * for head update and waiting/spinning on tail value.
+ *
+ * Brief outline:
+ *  - introduce refcnt for both head and tail.
+ *  - increment head.refcnt for each head.value update
+ *  - write head:value and head:refcnt atomically (64-bit CAS)
+ *  - move tail.value ahead only when tail.refcnt + 1 == head.refcnt
+ *  - increment tail.refcnt when each enqueue/dequeue op finishes
+ *    (no matter is tail:value going to change or not)
+ *  - write tail.value and tail.recnt atomically (64-bit CAS)
+ *
+ * To avoid head/tail starvation:
+ *  - limit max allowed distance between head and tail value (HTD_MAX).
+ *    I.E. thread is allowed to proceed with changing head.value,
+ *    only when:  head.value - tail.value <= HTD_MAX
+ * HTD_MAX is an optional parameter.
+ * With HTD_MAX == 0 we'll have ring with fully synchronized
+ * head/tail (like HTS).
+ * With HTD_MAX == UINT32_MAX - no limitation.
+ * By default HTD_MAX == ring.capacity / 8.
+ */
+
+/**
+ * @internal This function updates tail values.
+ */
+static __rte_always_inline void
+__rte_ring_rts_update_tail(struct rte_ring_rts_headtail *ht)
+{
+	union rte_ring_ht_poscnt h, ot, nt;
+
+	/*
+	 * If there are other enqueues/dequeues in progress that
+	 * might preceded us, then don't update tail with new value.
+	 */
+
+	do {
+		ot.raw = ht->tail.raw;
+		rte_smp_rmb();
+
+		/* on 32-bit systems we have to do atomic read here */
+		h.raw = rte_atomic64_read((rte_atomic64_t *)
+			(uintptr_t)&ht->head.raw);
+
+		nt.raw = ot.raw;
+		if (++nt.val.cnt == h.val.cnt)
+			nt.val.pos = h.val.pos;
+
+	} while (rte_atomic64_cmpset(&ht->tail.raw, ot.raw, nt.raw) == 0);
+}
+
+/**
+ * @internal This function waits till head/tail distance wouldn't
+ * exceed pre-defined max value.
+ */
+static __rte_always_inline void
+__rte_ring_rts_head_wait(const struct rte_ring_rts_headtail *ht,
+	union rte_ring_ht_poscnt *h)
+{
+	uint32_t max;
+
+	max = ht->htd_max;
+	h->raw = ht->head.raw;
+	rte_smp_rmb();
+
+	while (h->val.pos - ht->tail.val.pos > max) {
+		rte_pause();
+		h->raw = ht->head.raw;
+		rte_smp_rmb();
+	}
+}
+
+/**
+ * @internal This function updates the producer head for enqueue.
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sp
+ *   Indicates whether multi-producer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ *   Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline uint32_t
+__rte_ring_rts_move_prod_head(struct rte_ring *r, uint32_t num,
+	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+	uint32_t *free_entries)
+{
+	uint32_t n;
+	union rte_ring_ht_poscnt nh, oh;
+
+	const uint32_t capacity = r->capacity;
+
+	do {
+		/* Reset n to the initial burst count */
+		n = num;
+
+		/* read prod head (may spin on prod tail) */
+		__rte_ring_rts_head_wait(&r->rts_prod, &oh);
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/*
+		 *  The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * *old_head > cons_tail). So 'free_entries' is always between 0
+		 * and capacity (which is < size).
+		 */
+		*free_entries = capacity + r->cons.tail - oh.val.pos;
+
+		/* check that we have enough room in ring */
+		if (unlikely(n > *free_entries))
+			n = (behavior == RTE_RING_QUEUE_FIXED) ?
+					0 : *free_entries;
+
+		if (n == 0)
+			return 0;
+
+		nh.val.pos = oh.val.pos + n;
+		nh.val.cnt = oh.val.cnt + 1;
+
+	} while (rte_atomic64_cmpset(&r->rts_prod.head.raw,
+			oh.raw, nh.raw) == 0);
+
+	*old_head = oh.val.pos;
+	return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sc
+ *   Indicates whether multi-consumer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ *   Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
+	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+	uint32_t *entries)
+{
+	uint32_t n;
+	union rte_ring_ht_poscnt nh, oh;
+
+	/* move cons.head atomically */
+	do {
+		/* Restore n as it may change every loop */
+		n = num;
+
+		/* read cons head (may spin on cons tail) */
+		__rte_ring_rts_head_wait(&r->rts_cons, &oh);
+
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/* The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * cons_head > prod_tail). So 'entries' is always between 0
+		 * and size(ring)-1.
+		 */
+		*entries = r->prod.tail - oh.val.pos;
+
+		/* Set the actual entries for dequeue */
+		if (n > *entries)
+			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+		if (unlikely(n == 0))
+			return 0;
+
+		nh.val.pos = oh.val.pos + n;
+		nh.val.cnt = oh.val.cnt + 1;
+
+	} while (rte_atomic64_cmpset(&r->rts_cons.head.raw,
+			oh.raw, nh.raw) == 0);
+
+	*old_head = oh.val.pos;
+	return n;
+}
+
+#endif /* _RTE_RING_RTS_GENERIC_H_ */
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
                   ` (2 preceding siblings ...)
  2020-02-24 11:35 ` [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Introduce new test case to test RTS ring mode under contention.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/Makefile               |  1 +
 app/test/meson.build            |  1 +
 app/test/test_ring_rts_stress.c | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)
 create mode 100644 app/test/test_ring_rts_stress.c

diff --git a/app/test/Makefile b/app/test/Makefile
index 4f586d95f..d22b9f702 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -78,6 +78,7 @@ SRCS-y += test_rand_perf.c
 
 SRCS-y += test_ring.c
 SRCS-y += test_ring_perf.c
+SRCS-y += test_ring_rts_stress.c
 SRCS-y += test_ring_stress.c
 SRCS-y += test_pmd_perf.c
 
diff --git a/app/test/meson.build b/app/test/meson.build
index 84dde28ad..fa4fb4b51 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files('commands.c',
 	'test_rib6.c',
 	'test_ring.c',
 	'test_ring_perf.c',
+	'test_ring_rts_stress.c',
 	'test_ring_stress.c',
 	'test_rwlock.c',
 	'test_sched.c',
diff --git a/app/test/test_ring_rts_stress.c b/app/test/test_ring_rts_stress.c
new file mode 100644
index 000000000..525d222b2
--- /dev/null
+++ b/app/test/test_ring_rts_stress.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "test_ring_stress.h"
+
+static inline uint32_t
+_st_ring_dequeue_bulk(struct rte_ring *r, void **obj, uint32_t n,
+	uint32_t *avail)
+{
+	return rte_ring_rts_dequeue_bulk(r, obj, n, avail);
+}
+
+static inline uint32_t
+_st_ring_enqueue_bulk(struct rte_ring *r, void * const *obj, uint32_t n,
+	uint32_t *free)
+{
+	return rte_ring_rts_enqueue_bulk(r, obj, n, free);
+}
+
+static int
+_st_ring_init(struct rte_ring *r, const char *name, uint32_t num)
+{
+	return rte_ring_init(r, name, num,
+		RING_F_MP_RTS_ENQ | RING_F_MC_RTS_DEQ);
+}
+
+REGISTER_TEST_COMMAND(ring_stress_rts_autotest, test_ring_stress);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
                   ` (3 preceding siblings ...)
  2020-02-24 11:35 ` [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-03-25 20:44   ` Honnappa Nagarahalli
  2020-02-24 11:35 ` [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring Konstantin Ananyev
                   ` (2 subsequent siblings)
  7 siblings, 1 reply; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Introduce head/tail sync mode for MT ring synchronization.
In that mode enqueue/dequeue operation is fully serialized:
only one thread at a time is allowed to perform given op.
Suppose to reduce stall times in case when ring is used on
overcommitted cpus (multiple active threads on the same cpu).
As another enhancement provide ability to split enqueue/dequeue
operation into two phases:
  - enqueue/dequeue start
  - enqueue/dequeue finish
That allows user to inspect objects in the ring without removing
them from it (aka MT safe peek).

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 lib/librte_ring/Makefile               |   1 +
 lib/librte_ring/meson.build            |   1 +
 lib/librte_ring/rte_ring.c             |  15 +-
 lib/librte_ring/rte_ring.h             | 259 ++++++++++++++++++++++++-
 lib/librte_ring/rte_ring_hts_generic.h | 228 ++++++++++++++++++++++
 5 files changed, 500 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_ring/rte_ring_hts_generic.h

diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile
index 4f90344f4..0c7f8f918 100644
--- a/lib/librte_ring/Makefile
+++ b/lib/librte_ring/Makefile
@@ -19,6 +19,7 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
 					rte_ring_elem.h \
 					rte_ring_generic.h \
 					rte_ring_c11_mem.h \
+					rte_ring_hts_generic.h \
 					rte_ring_rts_generic.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
index dc8d7dbea..5aa673199 100644
--- a/lib/librte_ring/meson.build
+++ b/lib/librte_ring/meson.build
@@ -6,6 +6,7 @@ headers = files('rte_ring.h',
 		'rte_ring_elem.h',
 		'rte_ring_c11_mem.h',
 		'rte_ring_generic.h',
+		'rte_ring_hts_generic.h',
 		'rte_ring_rts_generic.h')
 
 # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c
index 1ce0af3e5..d3b948667 100644
--- a/lib/librte_ring/rte_ring.c
+++ b/lib/librte_ring/rte_ring.c
@@ -102,9 +102,9 @@ static int
 get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)
 {
 	static const uint32_t prod_st_flags =
-		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
+		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ | RING_F_MP_HTS_ENQ);
 	static const uint32_t cons_st_flags =
-		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
+		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ | RING_F_MC_HTS_DEQ);
 
 	switch (flags & prod_st_flags) {
 	case 0:
@@ -116,6 +116,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)
 	case RING_F_MP_RTS_ENQ:
 		*prod_st = RTE_RING_SYNC_MT_RTS;
 		break;
+	case RING_F_MP_HTS_ENQ:
+		*prod_st = RTE_RING_SYNC_MT_HTS;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -130,6 +133,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)
 	case RING_F_MC_RTS_DEQ:
 		*cons_st = RTE_RING_SYNC_MT_RTS;
 		break;
+	case RING_F_MC_HTS_DEQ:
+		*cons_st = RTE_RING_SYNC_MT_HTS;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -151,6 +157,11 @@ rte_ring_init(struct rte_ring *r, const char *name, unsigned count,
 	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
 			  RTE_CACHE_LINE_MASK) != 0);
 
+	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
+		offsetof(struct rte_ring_hts_headtail, sync_type));
+	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
+		offsetof(struct rte_ring_hts_headtail, ht.pos.tail));
+
 	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
 		offsetof(struct rte_ring_rts_headtail, sync_type));
 	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
diff --git a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h
index a130aeb9d..52edcea11 100644
--- a/lib/librte_ring/rte_ring.h
+++ b/lib/librte_ring/rte_ring.h
@@ -66,11 +66,11 @@ enum {
 	RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
 	RTE_RING_SYNC_ST,     /**< single thread only */
 	RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
+	RTE_RING_SYNC_MT_HTS, /**< multi-thread head/tail sync */
 };
 
 /**
- * structure to hold a pair of head/tail values and other metadata.
- * used by RTE_RING_SYNC_MT, RTE_RING_SYNC_ST sync types.
+ * Structure to hold a pair of head/tail values and other metadata.
  * Depending on sync_type format of that structure might differ
  * depending on the sync mechanism selelcted, but offsets for
  * *sync_type* and *tail* values should always remain the same.
@@ -96,6 +96,19 @@ struct rte_ring_rts_headtail {
 	volatile union rte_ring_ht_poscnt head;
 };
 
+union rte_ring_ht_pos {
+	uint64_t raw;
+	struct {
+		uint32_t tail; /**< tail position */
+		uint32_t head; /**< head position */
+	} pos;
+};
+
+struct rte_ring_hts_headtail {
+	uint32_t sync_type; /**< sync type of prod/cons */
+	volatile union rte_ring_ht_pos ht __rte_aligned(8);
+};
+
 /**
  * An RTE ring structure.
  *
@@ -126,6 +139,7 @@ struct rte_ring {
 	RTE_STD_C11
 	union {
 		struct rte_ring_headtail prod;
+		struct rte_ring_hts_headtail hts_prod;
 		struct rte_ring_rts_headtail rts_prod;
 	}  __rte_cache_aligned;
 
@@ -135,6 +149,7 @@ struct rte_ring {
 	RTE_STD_C11
 	union {
 		struct rte_ring_headtail cons;
+		struct rte_ring_hts_headtail hts_cons;
 		struct rte_ring_rts_headtail rts_cons;
 	}  __rte_cache_aligned;
 
@@ -157,6 +172,9 @@ struct rte_ring {
 #define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
 #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */
 
+#define RING_F_MP_HTS_ENQ 0x0020 /**< The default enqueue is "MP HTS". */
+#define RING_F_MC_HTS_DEQ 0x0040 /**< The default dequeue is "MC HTS". */
+
 #define __IS_SP RTE_RING_SYNC_ST
 #define __IS_MP RTE_RING_SYNC_MT
 #define __IS_SC RTE_RING_SYNC_ST
@@ -513,6 +531,82 @@ __rte_ring_do_rts_dequeue(struct rte_ring *r, void **obj_table,
 	return n;
 }
 
+#include <rte_ring_hts_generic.h>
+
+/**
+ * @internal Start to enqueue several objects on the HTS ring.
+ * Note that user has to call appropriate enqueue_finish()
+ * to complete given enqueue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_start(struct rte_ring *r, void * const *obj_table,
+		uint32_t n, enum rte_ring_queue_behavior behavior,
+		uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0)
+		ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Start to dequeue several objects from the HTS ring.
+ * Note that user has to call appropriate dequeue_finish()
+ * to complete given dequeue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_start(struct rte_ring *r, void **obj_table,
+		unsigned int n, enum rte_ring_queue_behavior behavior,
+		unsigned int *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries);
+
+	if (n != 0)
+		DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
@@ -585,6 +679,47 @@ rte_ring_rts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 			free_space);
 }
 
+/**
+ * Start to enqueue several objects on the HTS ring (multi-producers safe).
+ * Note that user has to call appropriate dequeue_finish()
+ * to complete given dequeue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ *   if non-NULL, returns the amount of space in the ring after the
+ *   enqueue operation has finished.
+ * @return
+ *   The number of objects enqueued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_hts_enqueue_bulk_start(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
+		RTE_RING_QUEUE_FIXED, free_space);
+}
+
+static __rte_always_inline void
+rte_ring_hts_enqueue_finish(struct rte_ring *r, unsigned int n)
+{
+	__rte_ring_hts_update_tail(&r->hts_prod, n, 1);
+}
+
+static __rte_always_inline unsigned int
+rte_ring_hts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	n = rte_ring_hts_enqueue_bulk_start(r, obj_table, n, free_space);
+	if (n != 0)
+		rte_ring_hts_enqueue_finish(r, n);
+	return n;
+}
+
 /**
  * Enqueue several objects on a ring.
  *
@@ -615,6 +750,8 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
 		return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
 	case RTE_RING_SYNC_MT_RTS:
 		return rte_ring_rts_enqueue_bulk(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_MT_HTS:
+		return rte_ring_hts_enqueue_bulk(r, obj_table, n, free_space);
 	}
 
 	/* valid ring should never reach this point */
@@ -753,6 +890,47 @@ rte_ring_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
 			available);
 }
 
+/**
+ * Start to dequeue several objects from an HTS ring (multi-consumers safe).
+ * Note that user has to call appropriate dequeue_finish()
+ * to complete given dequeue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ *   The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ *   If non-NULL, returns the number of remaining ring entries after the
+ *   dequeue has finished.
+ * @return
+ *   The number of objects dequeued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_hts_dequeue_bulk_start(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
+		RTE_RING_QUEUE_FIXED, available);
+}
+
+static __rte_always_inline void
+rte_ring_hts_dequeue_finish(struct rte_ring *r, unsigned int n)
+{
+	__rte_ring_hts_update_tail(&r->hts_cons, n, 0);
+}
+
+static __rte_always_inline unsigned int
+rte_ring_hts_dequeue_bulk(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	n = rte_ring_hts_dequeue_bulk_start(r, obj_table, n, available);
+	if (n != 0)
+		rte_ring_hts_dequeue_finish(r, n);
+	return n;
+}
+
 /**
  * Dequeue several objects from a ring.
  *
@@ -783,6 +961,8 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n,
 		return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
 	case RTE_RING_SYNC_MT_RTS:
 		return rte_ring_rts_dequeue_bulk(r, obj_table, n, available);
+	case RTE_RING_SYNC_MT_HTS:
+		return rte_ring_hts_dequeue_bulk(r, obj_table, n, available);
 	}
 
 	/* valid ring should never reach this point */
@@ -1111,6 +1291,41 @@ rte_ring_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 			RTE_RING_QUEUE_VARIABLE, free_space);
 }
 
+/**
+ * Start to enqueue several objects on the HTS ring (multi-producers safe).
+ * Note that user has to call appropriate dequeue_finish()
+ * to complete given dequeue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects).
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param free_space
+ *   if non-NULL, returns the amount of space in the ring after the
+ *   enqueue operation has finished.
+ * @return
+ *   The number of objects enqueued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_hts_enqueue_burst_start(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
+		RTE_RING_QUEUE_VARIABLE, free_space);
+}
+
+static __rte_always_inline unsigned int
+rte_ring_hts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
+			 unsigned int n, unsigned int *free_space)
+{
+	n = rte_ring_hts_enqueue_burst_start(r, obj_table, n, free_space);
+	if (n != 0)
+		rte_ring_hts_enqueue_finish(r, n);
+	return n;
+}
+
 /**
  * Enqueue several objects on a ring.
  *
@@ -1141,6 +1356,8 @@ rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table,
 		return rte_ring_sp_enqueue_burst(r, obj_table, n, free_space);
 	case RTE_RING_SYNC_MT_RTS:
 		return rte_ring_rts_enqueue_burst(r, obj_table, n, free_space);
+	case RTE_RING_SYNC_MT_HTS:
+		return rte_ring_hts_enqueue_burst(r, obj_table, n, free_space);
 	}
 
 	/* valid ring should never reach this point */
@@ -1225,6 +1442,42 @@ rte_ring_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
 	return __rte_ring_do_rts_dequeue(r, obj_table, n,
 			RTE_RING_QUEUE_VARIABLE, available);
 }
+
+/**
+ * Start to dequeue several objects from an HTS ring (multi-consumers safe).
+ * Note that user has to call appropriate dequeue_finish()
+ * to complete given dequeue operation.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of void * pointers (objects) that will be filled.
+ * @param n
+ *   The number of objects to dequeue from the ring to the obj_table.
+ * @param available
+ *   If non-NULL, returns the number of remaining ring entries after the
+ *   dequeue has finished.
+ * @return
+ *   The number of objects dequeued, either 0 or n
+ */
+static __rte_always_inline unsigned int
+rte_ring_hts_dequeue_burst_start(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
+		RTE_RING_QUEUE_VARIABLE, available);
+}
+
+static __rte_always_inline unsigned int
+rte_ring_hts_dequeue_burst(struct rte_ring *r, void **obj_table,
+		unsigned int n, unsigned int *available)
+{
+	n = rte_ring_hts_dequeue_burst_start(r, obj_table, n, available);
+	if (n != 0)
+		rte_ring_hts_dequeue_finish(r, n);
+	return n;
+}
+
 /**
  * Dequeue multiple objects from a ring up to a maximum number.
  *
@@ -1255,6 +1508,8 @@ rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table,
 		return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
 	case RTE_RING_SYNC_MT_RTS:
 		return rte_ring_rts_dequeue_burst(r, obj_table, n, available);
+	case RTE_RING_SYNC_MT_HTS:
+		return rte_ring_hts_dequeue_burst(r, obj_table, n, available);
 	}
 
 	/* valid ring should never reach this point */
diff --git a/lib/librte_ring/rte_ring_hts_generic.h b/lib/librte_ring/rte_ring_hts_generic.h
new file mode 100644
index 000000000..7e447e30b
--- /dev/null
+++ b/lib/librte_ring/rte_ring_hts_generic.h
@@ -0,0 +1,228 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2010-2020 Intel Corporation
+ * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RTE_RING_HTS_GENERIC_H_
+#define _RTE_RING_HTS_GENERIC_H_
+
+/**
+ * @file rte_ring_hts_generic.h
+ * It is not recommended to include this file directly,
+ * include <rte_ring.h> instead.
+ * Contains internal helper functions for head/tail sync (HTS) ring mode.
+ * In that mode enqueue/dequeue operation is fully serialized:
+ * only one thread at a time is allowed to perform given op.
+ * This is achieved by thread is allowed to proceed with changing head.value
+ * only when head.value == tail.value.
+ * Both head and tail values are updated atomically (as one 64-bit value).
+ * As another enhancement that provides ability to split enqueue/dequeue
+ * operation into two phases:
+ * - enqueue/dequeue start
+ * - enqueue/dequeue finish
+ * That allows user to inspect objects in the ring without removing
+ * them from it (aka MT safe peek).
+ * As an example:
+ * // read 1 elem from the ring:
+ * n = rte_ring_hts_dequeue_bulk_start(ring, &obj, 1, NULL);
+ * if (n != 0) {
+ *    //examined object
+ *    if (object_examine(obj) == KEEP)
+ *       //decided to keep it in the ring.
+ *       rte_ring_hts_dequeue_finish(ring, 0);
+ *    else
+ *       //decided to remove it in the ring.
+ *       rte_ring_hts_dequeue_finish(ring, n);
+ * }
+ * Note that between _start_ and _finish_ the ring is sort of locked -
+ * none other thread can proceed with enqueue(/dequeue) operation till
+ * _finish_ will complete.
+ */
+
+static __rte_always_inline void
+__rte_ring_hts_update_tail(struct rte_ring_hts_headtail *ht, uint32_t num,
+	uint32_t enqueue)
+{
+	uint32_t n;
+	union rte_ring_ht_pos p;
+
+	if (enqueue)
+		rte_smp_wmb();
+	else
+		rte_smp_rmb();
+
+	p.raw = rte_atomic64_read((rte_atomic64_t *)(uintptr_t)&ht->ht.raw);
+
+	n = p.pos.head - p.pos.tail;
+	RTE_ASSERT(n >= num);
+	RTE_SET_USED(n);
+
+	p.pos.head = p.pos.tail + num;
+	p.pos.tail = p.pos.head;
+
+	rte_atomic64_set((rte_atomic64_t *)(uintptr_t)&ht->ht.raw, p.raw);
+}
+
+/**
+ * @internal waits till tail will become equal to head.
+ * Means no writer/reader is active for that ring.
+ * Suppose to work as serialization point.
+ */
+static __rte_always_inline void
+__rte_ring_hts_head_wait(const struct rte_ring_hts_headtail *ht,
+		union rte_ring_ht_pos *p)
+{
+	p->raw = rte_atomic64_read((rte_atomic64_t *)
+			(uintptr_t)&ht->ht.raw);
+
+	while (p->pos.head != p->pos.tail) {
+		rte_pause();
+		p->raw = rte_atomic64_read((rte_atomic64_t *)
+				(uintptr_t)&ht->ht.raw);
+	}
+}
+
+/**
+ * @internal This function updates the producer head for enqueue
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sp
+ *   Indicates whether multi-producer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where enqueue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where enqueue finishes
+ * @param free_entries
+ *   Returns the amount of free space in the ring BEFORE head was moved
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num,
+	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+	uint32_t *free_entries)
+{
+	uint32_t n;
+	union rte_ring_ht_pos np, op;
+
+	const uint32_t capacity = r->capacity;
+
+	do {
+		/* Reset n to the initial burst count */
+		n = num;
+
+		/* wait for tail to be equal to head */
+		__rte_ring_hts_head_wait(&r->hts_prod, &op);
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/*
+		 *  The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * *old_head > cons_tail). So 'free_entries' is always between 0
+		 * and capacity (which is < size).
+		 */
+		*free_entries = capacity + r->cons.tail - op.pos.head;
+
+		/* check that we have enough room in ring */
+		if (unlikely(n > *free_entries))
+			n = (behavior == RTE_RING_QUEUE_FIXED) ?
+					0 : *free_entries;
+
+		if (n == 0)
+			return 0;
+
+		np.pos.tail = op.pos.tail;
+		np.pos.head = op.pos.head + n;
+
+	} while (rte_atomic64_cmpset(&r->hts_prod.ht.raw,
+			op.raw, np.raw) == 0);
+
+	*old_head = op.pos.head;
+	return n;
+}
+
+/**
+ * @internal This function updates the consumer head for dequeue
+ *
+ * @param r
+ *   A pointer to the ring structure
+ * @param is_sc
+ *   Indicates whether multi-consumer path is needed or not
+ * @param n
+ *   The number of elements we will want to enqueue, i.e. how far should the
+ *   head be moved
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param old_head
+ *   Returns head value as it was before the move, i.e. where dequeue starts
+ * @param new_head
+ *   Returns the current/new head value i.e. where dequeue finishes
+ * @param entries
+ *   Returns the number of entries in the ring BEFORE head was moved
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,
+	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
+	uint32_t *entries)
+{
+	uint32_t n;
+	union rte_ring_ht_pos np, op;
+
+	/* move cons.head atomically */
+	do {
+		/* Restore n as it may change every loop */
+		n = num;
+
+		/* wait for tail to be equal to head */
+		__rte_ring_hts_head_wait(&r->hts_cons, &op);
+
+		/* add rmb barrier to avoid load/load reorder in weak
+		 * memory model. It is noop on x86
+		 */
+		rte_smp_rmb();
+
+		/* The subtraction is done between two unsigned 32bits value
+		 * (the result is always modulo 32 bits even if we have
+		 * cons_head > prod_tail). So 'entries' is always between 0
+		 * and size(ring)-1.
+		 */
+		*entries = r->prod.tail - op.pos.head;
+
+		/* Set the actual entries for dequeue */
+		if (n > *entries)
+			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
+
+		if (unlikely(n == 0))
+			return 0;
+
+		np.pos.tail = op.pos.tail;
+		np.pos.head = op.pos.head + n;
+
+	} while (rte_atomic64_cmpset(&r->hts_cons.ht.raw,
+			op.raw, np.raw) == 0);
+
+	*old_head = op.pos.head;
+	return n;
+}
+
+#endif /* _RTE_RING_HTS_GENERIC_H_ */
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
                   ` (4 preceding siblings ...)
  2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
@ 2020-02-24 11:35 ` Konstantin Ananyev
  2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
  2020-03-25 20:43 ` Honnappa Nagarahalli
  7 siblings, 0 replies; 23+ messages in thread
From: Konstantin Ananyev @ 2020-02-24 11:35 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, Konstantin Ananyev

Introduce new test case to test HTS ring mode under contention.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 app/test/Makefile               |  1 +
 app/test/meson.build            |  1 +
 app/test/test_ring_hts_stress.c | 28 ++++++++++++++++++++++++++++
 3 files changed, 30 insertions(+)
 create mode 100644 app/test/test_ring_hts_stress.c

diff --git a/app/test/Makefile b/app/test/Makefile
index d22b9f702..ff151cd55 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -77,6 +77,7 @@ SRCS-y += test_external_mem.c
 SRCS-y += test_rand_perf.c
 
 SRCS-y += test_ring.c
+SRCS-y += test_ring_hts_stress.c
 SRCS-y += test_ring_perf.c
 SRCS-y += test_ring_rts_stress.c
 SRCS-y += test_ring_stress.c
diff --git a/app/test/meson.build b/app/test/meson.build
index fa4fb4b51..7e58fa999 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -100,6 +100,7 @@ test_sources = files('commands.c',
 	'test_rib.c',
 	'test_rib6.c',
 	'test_ring.c',
+	'test_ring_hts_stress.c',
 	'test_ring_perf.c',
 	'test_ring_rts_stress.c',
 	'test_ring_stress.c',
diff --git a/app/test/test_ring_hts_stress.c b/app/test/test_ring_hts_stress.c
new file mode 100644
index 000000000..b7f2d21fc
--- /dev/null
+++ b/app/test/test_ring_hts_stress.c
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#include "test_ring_stress.h"
+
+static inline uint32_t
+_st_ring_dequeue_bulk(struct rte_ring *r, void **obj, uint32_t n,
+	uint32_t *avail)
+{
+	return rte_ring_hts_dequeue_bulk(r, obj, n, avail);
+}
+
+static inline uint32_t
+_st_ring_enqueue_bulk(struct rte_ring *r, void * const *obj, uint32_t n,
+	uint32_t *free)
+{
+	return rte_ring_hts_enqueue_bulk(r, obj, n, free);
+}
+
+static int
+_st_ring_init(struct rte_ring *r, const char *name, uint32_t num)
+{
+	return rte_ring_init(r, name, num,
+		RING_F_MP_HTS_ENQ | RING_F_MC_HTS_DEQ);
+}
+
+REGISTER_TEST_COMMAND(ring_stress_hts_autotest, test_ring_stress);
-- 
2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
                   ` (5 preceding siblings ...)
  2020-02-24 11:35 ` [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring Konstantin Ananyev
@ 2020-02-24 16:59 ` Stephen Hemminger
  2020-02-24 17:59   ` Jerin Jacob
  2020-03-25 20:43 ` Honnappa Nagarahalli
  7 siblings, 1 reply; 23+ messages in thread
From: Stephen Hemminger @ 2020-02-24 16:59 UTC (permalink / raw)
  To: Konstantin Ananyev; +Cc: dev, olivier.matz

On Mon, 24 Feb 2020 11:35:09 +0000
Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:

> Upfront note - that RFC is not a complete patch.
> It introduces an ABI breakage, plus it doesn't update ring_elem
> code properly, etc.
> I plan to deal with all these things in later versions.
> Right now I seek an initial feedback about proposed ideas.
> Would also ask people to repeat performance tests (see below)
> on their platforms to confirm the impact.
> 
> More and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives
> (spin-locks, etc.) on overcommitted systems.
> The situation gets much worse when some sort of
> fair-locking technique is used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling
> order matters a lot.
> This is a well-known problem for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of
> un-fair locking, waiting on tail is very similar to ticket lock schema -
> tail has to be updated in particular order.
> That makes current rte_ring implementation to perform
> really pure on some overcommited scenarios.

Rather than reform rte_ring to fit this scenario, it would make
more sense to me to introduce another primitive. The current lockless
ring performs very well for the isolated thread model that DPDK
was built around. This looks like a case of customers violating
the usage model of the DPDK and then being surprised at the fallout.


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
@ 2020-02-24 17:59   ` Jerin Jacob
  2020-02-24 19:35     ` Stephen Hemminger
  2020-02-25  0:58     ` Honnappa Nagarahalli
  0 siblings, 2 replies; 23+ messages in thread
From: Jerin Jacob @ 2020-02-24 17:59 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz

On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 24 Feb 2020 11:35:09 +0000
> Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
>
> > Upfront note - that RFC is not a complete patch.
> > It introduces an ABI breakage, plus it doesn't update ring_elem
> > code properly, etc.
> > I plan to deal with all these things in later versions.
> > Right now I seek an initial feedback about proposed ideas.
> > Would also ask people to repeat performance tests (see below)
> > on their platforms to confirm the impact.
> >
> > More and more customers use(/try to use) DPDK based apps within
> > overcommitted systems (multiple acttive threads over same pysical cores):
> > VM, container deployments, etc.
> > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > LHP is quite a common problem for spin-based sync primitives
> > (spin-locks, etc.) on overcommitted systems.
> > The situation gets much worse when some sort of
> > fair-locking technique is used (ticket-lock, etc.).
> > As now not only lock-owner but also lock-waiters scheduling
> > order matters a lot.
> > This is a well-known problem for kernel within VMs:
> > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > The problem with rte_ring is that while head accusion is sort of
> > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > tail has to be updated in particular order.
> > That makes current rte_ring implementation to perform
> > really pure on some overcommited scenarios.
>
> Rather than reform rte_ring to fit this scenario, it would make
> more sense to me to introduce another primitive. The current lockless
> ring performs very well for the isolated thread model that DPDK
> was built around. This looks like a case of customers violating
> the usage model of the DPDK and then being surprised at the fallout.

I agree with Stephen here.

I think, adding more runtime check in the enqueue() and dequeue() will
have a bad effect on the low-end cores too.
But I agree with the problem statement that in the virtualization use
case, It may be possible to have N virtual cores runs on a physical
core.

IMO, The best solution would be keeping the ring API same and have a
different flavor in "compile-time". Something like
liburcu did for accommodating different flavors.

i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
application can simply include ONE header file in a C file based on
the flavor.
If need both at runtime. Need to have function pointer or so in the
application and define the function in different c file by including
the approaite flavor in C file.

#include <urcu-qsbr.h> /* QSBR RCU flavor */
#include <urcu-bp.h> /* Bulletproof RCU flavor */













>

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 17:59   ` Jerin Jacob
@ 2020-02-24 19:35     ` Stephen Hemminger
  2020-02-24 20:52       ` Honnappa Nagarahalli
  2020-02-25 13:41       ` Ananyev, Konstantin
  2020-02-25  0:58     ` Honnappa Nagarahalli
  1 sibling, 2 replies; 23+ messages in thread
From: Stephen Hemminger @ 2020-02-24 19:35 UTC (permalink / raw)
  To: Jerin Jacob; +Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz

On Mon, 24 Feb 2020 23:29:57 +0530
Jerin Jacob <jerinjacobk@gmail.com> wrote:

> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >  
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > code properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below)
> > > on their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of
> > > fair-locking technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling
> > > order matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform
> > > really pure on some overcommited scenarios.  
> >
> > Rather than reform rte_ring to fit this scenario, it would make
> > more sense to me to introduce another primitive. The current lockless
> > ring performs very well for the isolated thread model that DPDK
> > was built around. This looks like a case of customers violating
> > the usage model of the DPDK and then being surprised at the fallout.  
> 
> I agree with Stephen here.
> 
> I think, adding more runtime check in the enqueue() and dequeue() will
> have a bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use
> case, It may be possible to have N virtual cores runs on a physical
> core.
> 
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like
> liburcu did for accommodating different flavors.
> 
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> application can simply include ONE header file in a C file based on
> the flavor.
> If need both at runtime. Need to have function pointer or so in the
> application and define the function in different c file by including
> the approaite flavor in C file.

This would also be a good time to consider the tradeoffs of the
heavy use of inlining that is done in rte_ring vs the impact that
has on API/ABI stability.



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 19:35     ` Stephen Hemminger
@ 2020-02-24 20:52       ` Honnappa Nagarahalli
  2020-02-25 11:45         ` Ananyev, Konstantin
  2020-02-25 13:41       ` Ananyev, Konstantin
  1 sibling, 1 reply; 23+ messages in thread
From: Honnappa Nagarahalli @ 2020-02-24 20:52 UTC (permalink / raw)
  To: Stephen Hemminger, Jerin Jacob
  Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd



> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> Sent: Monday, February 24, 2020 1:35 PM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> 
> On Mon, 24 Feb 2020 23:29:57 +0530
> Jerin Jacob <jerinjacobk@gmail.com> wrote:
> 
> > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > >
> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical
> cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure
> > > > on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless
> > > ring performs very well for the isolated thread model that DPDK was
> > > built around. This looks like a case of customers violating the
> > > usage model of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.
> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.
> 
> This would also be a good time to consider the tradeoffs of the heavy use of
> inlining that is done in rte_ring vs the impact that has on API/ABI stability.
> 
I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/. 

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 17:59   ` Jerin Jacob
  2020-02-24 19:35     ` Stephen Hemminger
@ 2020-02-25  0:58     ` Honnappa Nagarahalli
  2020-02-25 15:14       ` Ananyev, Konstantin
  1 sibling, 1 reply; 23+ messages in thread
From: Honnappa Nagarahalli @ 2020-02-25  0:58 UTC (permalink / raw)
  To: Jerin Jacob, Stephen Hemminger
  Cc: Konstantin Ananyev, dpdk-dev, Olivier Matz, Honnappa Nagarahalli, nd, nd

<snip>

> 
> On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 24 Feb 2020 11:35:09 +0000
> > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> >
> > > Upfront note - that RFC is not a complete patch.
> > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > properly, etc.
> > > I plan to deal with all these things in later versions.
> > > Right now I seek an initial feedback about proposed ideas.
> > > Would also ask people to repeat performance tests (see below) on
> > > their platforms to confirm the impact.
> > >
> > > More and more customers use(/try to use) DPDK based apps within
> > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > VM, container deployments, etc.
> > > One quite common problem they hit: Lock-Holder-Preemption with
> rte_ring.
> > > LHP is quite a common problem for spin-based sync primitives
> > > (spin-locks, etc.) on overcommitted systems.
> > > The situation gets much worse when some sort of fair-locking
> > > technique is used (ticket-lock, etc.).
> > > As now not only lock-owner but also lock-waiters scheduling order
> > > matters a lot.
> > > This is a well-known problem for kernel within VMs:
> > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?

> > > The problem with rte_ring is that while head accusion is sort of
> > > un-fair locking, waiting on tail is very similar to ticket lock
> > > schema - tail has to be updated in particular order.
> > > That makes current rte_ring implementation to perform really pure on
> > > some overcommited scenarios.
> >
> > Rather than reform rte_ring to fit this scenario, it would make more
> > sense to me to introduce another primitive. The current lockless ring
> > performs very well for the isolated thread model that DPDK was built
> > around. This looks like a case of customers violating the usage model
> > of the DPDK and then being surprised at the fallout.
> 
> I agree with Stephen here.
> 
> I think, adding more runtime check in the enqueue() and dequeue() will have a
> bad effect on the low-end cores too.
> But I agree with the problem statement that in the virtualization use case, It
> may be possible to have N virtual cores runs on a physical core.
It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?

> 
> IMO, The best solution would be keeping the ring API same and have a
> different flavor in "compile-time". Something like liburcu did for
> accommodating different flavors.
> 
> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> can simply include ONE header file in a C file based on the flavor.
> If need both at runtime. Need to have function pointer or so in the application
> and define the function in different c file by including the approaite flavor in C
> file.
> 
> #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> Bulletproof RCU flavor */
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> >

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 20:52       ` Honnappa Nagarahalli
@ 2020-02-25 11:45         ` Ananyev, Konstantin
  0 siblings, 0 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-02-25 11:45 UTC (permalink / raw)
  To: Honnappa Nagarahalli, Stephen Hemminger, Jerin Jacob
  Cc: dpdk-dev, Olivier Matz, nd, nd


> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Stephen Hemminger
> > Sent: Monday, February 24, 2020 1:35 PM
> > To: Jerin Jacob <jerinjacobk@gmail.com>
> > Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>; dpdk-dev
> > <dev@dpdk.org>; Olivier Matz <olivier.matz@6wind.com>
> > Subject: Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
> >
> > On Mon, 24 Feb 2020 23:29:57 +0530
> > Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > > On Mon, Feb 24, 2020 at 10:29 PM Stephen Hemminger
> > > <stephen@networkplumber.org> wrote:
> > > >
> > > > On Mon, 24 Feb 2020 11:35:09 +0000
> > > > Konstantin Ananyev <konstantin.ananyev@intel.com> wrote:
> > > >
> > > > > Upfront note - that RFC is not a complete patch.
> > > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > > code properly, etc.
> > > > > I plan to deal with all these things in later versions.
> > > > > Right now I seek an initial feedback about proposed ideas.
> > > > > Would also ask people to repeat performance tests (see below) on
> > > > > their platforms to confirm the impact.
> > > > >
> > > > > More and more customers use(/try to use) DPDK based apps within
> > > > > overcommitted systems (multiple acttive threads over same pysical
> > cores):
> > > > > VM, container deployments, etc.
> > > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > > LHP is quite a common problem for spin-based sync primitives
> > > > > (spin-locks, etc.) on overcommitted systems.
> > > > > The situation gets much worse when some sort of fair-locking
> > > > > technique is used (ticket-lock, etc.).
> > > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > > matters a lot.
> > > > > This is a well-known problem for kernel within VMs:
> > > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > > The problem with rte_ring is that while head accusion is sort of
> > > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > > schema - tail has to be updated in particular order.
> > > > > That makes current rte_ring implementation to perform really pure
> > > > > on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make more
> > > > sense to me to introduce another primitive. The current lockless
> > > > ring performs very well for the isolated thread model that DPDK was
> > > > built around. This looks like a case of customers violating the
> > > > usage model of the DPDK and then being surprised at the fallout.
> > >
> > > I agree with Stephen here.
> > >
> > > I think, adding more runtime check in the enqueue() and dequeue() will
> > > have a bad effect on the low-end cores too.
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like liburcu did for
> > > accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
> > > If need both at runtime. Need to have function pointer or so in the
> > > application and define the function in different c file by including
> > > the approaite flavor in C file.
> >
> > This would also be a good time to consider the tradeoffs of the heavy use of
> > inlining that is done in rte_ring vs the impact that has on API/ABI stability.
> >
> I was working on few requirements in rte_ring library for RCU defer APIs. RFC is at https://patchwork.dpdk.org/cover/66020/.

Yep, noticed your patch, seems we sort of collided here.
As I understand you patch aims to provide functionality similar to my HTS one.
Will try to look at yours one in next fee days, hopefully we can end-up with
some common denominator.
Konstantin




^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 19:35     ` Stephen Hemminger
  2020-02-24 20:52       ` Honnappa Nagarahalli
@ 2020-02-25 13:41       ` Ananyev, Konstantin
  2020-02-26 16:53         ` Morten Brørup
  2020-02-27 10:31         ` Jerin Jacob
  1 sibling, 2 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-02-25 13:41 UTC (permalink / raw)
  To: Stephen Hemminger, Jerin Jacob; +Cc: dpdk-dev, Olivier Matz, drc

> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below)
> > > > on their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of
> > > > fair-locking technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling
> > > > order matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > > tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform
> > > > really pure on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make
> > > more sense to me to introduce another primitive. 

I don't see much advantages it will bring us.
As a disadvantages, for developers and maintainers - code duplication,
for end users - extra code churn and removed ability to mix and match
different sync modes in one ring.

> The current lockless
> > > ring performs very well for the isolated thread model that DPDK
> > > was built around. This looks like a case of customers violating
> > > the usage model of the DPDK and then being surprised at the fallout.

For customers using isolated thread model - nothing should change
(both in terms of API and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
From other side I don't see why we should ignore customers that want to use
their DPDK apps in different deployment scenarios.

> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.

We do have a run-time check in our current enqueue()/dequeue implementation.
In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
where sync behaviour is determined at runtime by value of prod(/cons).single.
Or user can call  rte_ring_(mp/sp)_enqueue_* functions directly.
This RFC follows exactly the same paradigm:
rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
behaviour is determined at runtime, by value of prod(/cons).sync_type.
Or user can call enqueue/dequeue with particular sync mode directly:
rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
The only thing that changed:
 Format of prod/cons now could differ depending on mode selected at _init_.
 So you can't create a ring for let say SP mode and then in the middle of data-path
 change your mind and start using MP_RTS mode.
 For existing modes (SP/MP, SC/MC)  format remains the same and user can still
 use them interchangeably, though of course that is an error prone practice.   

> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like
> > liburcu did for accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.

I don't think it is a flexible enough approach.
In one app user might need to have several rings with different sync modes.
Or even user might need a ring with different sync modes for enqueue/dequeue.

> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.

Big issue with function pointers here would be DPDK MP model.
AFAIK,  rte_ring is quite popular mechanism for IPC between DPDK apps.
To support such model, we'll need to split rte_ring data into 'shared'
and 'private' and initialize private one for every process that is going to use it.
That sounds like a massive change, and I am not sure the required effort will worth it. 
BTW, if user just calls API functions without trying to access structure internals directly,
I don't think it would be a big difference for him what is inside:
indirect function call or inlined switch(...) {}.  

> This would also be a good time to consider the tradeoffs of the
> heavy use of inlining that is done in rte_ring vs the impact that
> has on API/ABI stability.

Yes, hiding rte_ring implementation inside .c would help a lot
in terms of ABI maintenance and would make our future life easier.
The question is what is the price for it in terms of performance,
and are we ready to pay it. Not to mention that it would cause
changes in many other libs/apps...
So I think it should be a subject for a separate discussion.
But, agree it would be good at least to measure the performance
impact of such change.
If I'll have some spare cycles, will give it a try.
Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
on their HW? Before continuing discussion would probably be good to know
does the suggested patch work as expected across different platforms.
Thanks
Konstantin   

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-25  0:58     ` Honnappa Nagarahalli
@ 2020-02-25 15:14       ` Ananyev, Konstantin
  0 siblings, 0 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-02-25 15:14 UTC (permalink / raw)
  To: Honnappa Nagarahalli, Jerin Jacob, Stephen Hemminger
  Cc: dpdk-dev, Olivier Matz, nd, nd, wang.yong19

> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem code
> > > > properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below) on
> > > > their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with
> > rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of fair-locking
> > > > technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling order
> > > > matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> These slides seem to indicate that the problems are mitigated through the Hypervisor configuration. Do we still need to address the issues?

I am not really an expert here, but AFAIK current mitigations deal mostly with guest kernel:
linux implements PV version of spinlocks (unfair and/or based on hypercall availability),
hypervisor might make decision itself based on is guest in user/kernel mode,
plus on some special cpu instructions. 
We do spin in user-space mode.
Might be hypervisors became smarter these days, but so far,
I heard about few different customers that hit such problem.
As an example, NA DPDK summit presentation:
https://dpdkna2019.sched.com/event/WYBG/dpdk-containers-challenges-solutions-wang-yong-zte
page 16 (problem #4) describes same issue.

> 
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock
> > > > schema - tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform really pure on
> > > > some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make more
> > > sense to me to introduce another primitive. The current lockless ring
> > > performs very well for the isolated thread model that DPDK was built
> > > around. This looks like a case of customers violating the usage model
> > > of the DPDK and then being surprised at the fallout.
> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will have a
> > bad effect on the low-end cores too.
> > But I agree with the problem statement that in the virtualization use case, It
> > may be possible to have N virtual cores runs on a physical core.
> It is hard to imagine that there are data plane applications deployed in such environments. Wouldn't this affect the performance terribly?

It wouldn't reach same performance as isolated threads, 
but for some tasks it might be enough. 
AFAIK, one quite common scenario - few isolated threads(/processes) doing
actual IO and then spread packets over dozens(/hundreds) non-isolated
consumers. 

> 
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like liburcu did for
> > accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The application
> > can simply include ONE header file in a C file based on the flavor.
> > If need both at runtime. Need to have function pointer or so in the application
> > and define the function in different c file by including the approaite flavor in C
> > file.
> >
> > #include <urcu-qsbr.h> /* QSBR RCU flavor */ #include <urcu-bp.h> /*
> > Bulletproof RCU flavor */

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-25 13:41       ` Ananyev, Konstantin
@ 2020-02-26 16:53         ` Morten Brørup
  2020-02-27 10:31         ` Jerin Jacob
  1 sibling, 0 replies; 23+ messages in thread
From: Morten Brørup @ 2020-02-26 16:53 UTC (permalink / raw)
  To: Ananyev, Konstantin, Stephen Hemminger, Jerin Jacob
  Cc: dpdk-dev, Olivier Matz, drc

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev,
> Konstantin
> 

<snip>

> > > > > More and more customers use(/try to use) DPDK based apps within
> > > > > overcommitted systems (multiple acttive threads over same
> pysical cores):
> > > > > VM, container deployments, etc.

<snip>

> > > > > That makes current rte_ring implementation to perform
> > > > > really pure on some overcommited scenarios.
> > > >
> > > > Rather than reform rte_ring to fit this scenario, it would make
> > > > more sense to me to introduce another primitive.
> 
> I don't see much advantages it will bring us.
> As a disadvantages, for developers and maintainers - code duplication,
> for end users - extra code churn and removed ability to mix and match
> different sync modes in one ring.
> 

I strongly agree with Konstantin on this.

Please consider this discussion at a higher abstraction level:

As DPDK applications grow in number and popularity, people will deploy them in overcommitted systems. In this scenario, I consider it extremely unlikely that the user is able to dedicate N physical cores to N specific lcores of the M total lcores of the DPDK application. In the typical hypervisor scenario, the user will be able to assign a number of virtual CPUs to the virtual machine running the DPDK application, and these vCPUs will either be dedicated to the virtual machine or shared with other virtual machines.

DPDK is currently designed for dedicated CPUs only. If a user runs it on shared vCPUs, the user is violating an important DPDK precondition, and DPDK's behavior is undefined!

Adding the ability to run DPDK applications on shared vCPUs would be a great improvement.

I prefer that support for this is as painless as possible for the DPDK application developer.

Perhaps run-time detection during the EAL initialization could be a solution. The EAL would then configure relevant libraries to use the appropriate synchronization primitives.


Med venlig hilsen / kind regards
- Morten Brørup




^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-25 13:41       ` Ananyev, Konstantin
  2020-02-26 16:53         ` Morten Brørup
@ 2020-02-27 10:31         ` Jerin Jacob
  2020-02-28  0:17           ` David Christensen
  1 sibling, 1 reply; 23+ messages in thread
From: Jerin Jacob @ 2020-02-27 10:31 UTC (permalink / raw)
  To: Ananyev, Konstantin; +Cc: Stephen Hemminger, dpdk-dev, Olivier Matz, drc

On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:

> We do have a run-time check in our current enqueue()/dequeue implementation.
> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
> where sync behaviour is determined at runtime by value of prod(/cons).single.
> Or user can call  rte_ring_(mp/sp)_enqueue_* functions directly.
> This RFC follows exactly the same paradigm:
> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
> behaviour is determined at runtime, by value of prod(/cons).sync_type.
> Or user can call enqueue/dequeue with particular sync mode directly:
> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
> The only thing that changed:
>  Format of prod/cons now could differ depending on mode selected at _init_.
>  So you can't create a ring for let say SP mode and then in the middle of data-path
>  change your mind and start using MP_RTS mode.
>  For existing modes (SP/MP, SC/MC)  format remains the same and user can still
>  use them interchangeably, though of course that is an error prone practice.

Makes sense.


>
> > > But I agree with the problem statement that in the virtualization use
> > > case, It may be possible to have N virtual cores runs on a physical
> > > core.
> > >
> > > IMO, The best solution would be keeping the ring API same and have a
> > > different flavor in "compile-time". Something like
> > > liburcu did for accommodating different flavors.
> > >
> > > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > > application can simply include ONE header file in a C file based on
> > > the flavor.
>
> I don't think it is a flexible enough approach.
> In one app user might need to have several rings with different sync modes.
> Or even user might need a ring with different sync modes for enqueue/dequeue.

Ack.


> Yes, hiding rte_ring implementation inside .c would help a lot
> in terms of ABI maintenance and would make our future life easier.
> The question is what is the price for it in terms of performance,
> and are we ready to pay it. Not to mention that it would cause
> changes in many other libs/apps...
> So I think it should be a subject for a separate discussion.
> But, agree it would be good at least to measure the performance
> impact of such change.
> If I'll have some spare cycles, will give it a try.
> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
> on their HW? Before continuing discussion would probably be good to know
> does the suggested patch work as expected across different platforms.


I tested on an arm64 HW. The former section is without the
patch(20.02) and later one with this patch.
I agree with Konstantin that getting more platform tests will be good
early so that we can focus on the approach
to avoid back and forth latter.


RTE>>ring_perf_autotest // without path

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 516.20

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 312.88
legacy APIs: SP/SC: burst (size: 32): 426.72
legacy APIs: MP/MC: burst (size: 8): 510.95
legacy APIs: MP/MC: burst (size: 32): 702.01

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 306.74
legacy APIs: SP/SC: bulk (size: 32): 411.56
legacy APIs: MP/MC: bulk (size: 8): 501.32
legacy APIs: MP/MC: bulk (size: 32): 693.07

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.36
legacy APIs: MP/MC: bulk (size: 8): 110.18
legacy APIs: SP/SC: bulk (size: 32): 23.04
legacy APIs: MP/MC: bulk (size: 32): 32.29

### Testing using all slave nodes ##
Bulk enq/dequeue count on size 8
Core [8] count = 293741
Core [9] count = 293741
Total count (size: 8): 587482

Bulk enq/dequeue count on size 32
Core [8] count = 244909
Core [9] count = 244909
Total count (size: 32): 1077300

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 255.37
elem APIs: element size 16B: MP/MC: single: 456.68

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [8] count = 308724
Core [9] count = 308723
Total count (size: 8): 617447

Bulk enq/dequeue count on size 32
Core [8] count = 214269
Core [9] count = 214269
Total count (size: 32): 1045985

RTE>>ring_perf_autotest // with patch

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 289.78
legacy APIs: MP/MC: single: 475.76

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 323.91
legacy APIs: SP/SC: burst (size: 32): 424.60
legacy APIs: MP/MC: burst (size: 8): 523.00
legacy APIs: MP/MC: burst (size: 32): 717.09

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 317.74
legacy APIs: SP/SC: bulk (size: 32): 413.57
legacy APIs: MP/MC: bulk (size: 8): 512.89
legacy APIs: MP/MC: bulk (size: 32): 712.45

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.00
legacy APIs: MP/MC: bulk (size: 8): 7.00

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 74.82
legacy APIs: MP/MC: bulk (size: 8): 96.45
legacy APIs: SP/SC: bulk (size: 32): 22.97
legacy APIs: MP/MC: bulk (size: 32): 32.52

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [8] count = 283928
Core [9] count = 283927
Total count (size: 8): 567855

Bulk enq/dequeue count on size 32
Core [8] count = 223916
Core [9] count = 223915
Total count (size: 32): 1015686

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 267.65
elem APIs: element size 16B: MP/MC: single: 439.06

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [8] count = 304223
Core [9] count = 304221
Total count (size: 8): 608444

Bulk enq/dequeue count on size 32
Core [8] count = 214856
Core [9] count = 214855
Total count (size: 32): 1038155
Test OK
RTE>>quit









> Thanks
> Konstantin

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-27 10:31         ` Jerin Jacob
@ 2020-02-28  0:17           ` David Christensen
  2020-03-20 16:45             ` Ananyev, Konstantin
  0 siblings, 1 reply; 23+ messages in thread
From: David Christensen @ 2020-02-28  0:17 UTC (permalink / raw)
  To: Jerin Jacob, Ananyev, Konstantin
  Cc: Stephen Hemminger, dpdk-dev, Olivier Matz

> On Tue, Feb 25, 2020 at 7:11 PM Ananyev, Konstantin
> <konstantin.ananyev@intel.com> wrote:
> 
>> We do have a run-time check in our current enqueue()/dequeue implementation.
>> In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
>> where sync behaviour is determined at runtime by value of prod(/cons).single.
>> Or user can call  rte_ring_(mp/sp)_enqueue_* functions directly.
>> This RFC follows exactly the same paradigm:
>> rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
>> behaviour is determined at runtime, by value of prod(/cons).sync_type.
>> Or user can call enqueue/dequeue with particular sync mode directly:
>> rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
>> The only thing that changed:
>>   Format of prod/cons now could differ depending on mode selected at _init_.
>>   So you can't create a ring for let say SP mode and then in the middle of data-path
>>   change your mind and start using MP_RTS mode.
>>   For existing modes (SP/MP, SC/MC)  format remains the same and user can still
>>   use them interchangeably, though of course that is an error prone practice.
> 
> Makes sense.
> 
> 
>>
>>>> But I agree with the problem statement that in the virtualization use
>>>> case, It may be possible to have N virtual cores runs on a physical
>>>> core.
>>>>
>>>> IMO, The best solution would be keeping the ring API same and have a
>>>> different flavor in "compile-time". Something like
>>>> liburcu did for accommodating different flavors.
>>>>
>>>> i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
>>>> application can simply include ONE header file in a C file based on
>>>> the flavor.
>>
>> I don't think it is a flexible enough approach.
>> In one app user might need to have several rings with different sync modes.
>> Or even user might need a ring with different sync modes for enqueue/dequeue.
> 
> Ack.
> 
> 
>> Yes, hiding rte_ring implementation inside .c would help a lot
>> in terms of ABI maintenance and would make our future life easier.
>> The question is what is the price for it in terms of performance,
>> and are we ready to pay it. Not to mention that it would cause
>> changes in many other libs/apps...
>> So I think it should be a subject for a separate discussion.
>> But, agree it would be good at least to measure the performance
>> impact of such change.
>> If I'll have some spare cycles, will give it a try.
>> Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
>> on their HW? Before continuing discussion would probably be good to know
>> does the suggested patch work as expected across different platforms.
> 
> 
> I tested on an arm64 HW. The former section is without the
> patch(20.02) and later one with this patch.
> I agree with Konstantin that getting more platform tests will be good
> early so that we can focus on the approach
> to avoid back and forth latter.
> 
> 
> RTE>>ring_perf_autotest // without path
> 
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 516.20
> 
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 312.88
> legacy APIs: SP/SC: burst (size: 32): 426.72
> legacy APIs: MP/MC: burst (size: 8): 510.95
> legacy APIs: MP/MC: burst (size: 32): 702.01
> 
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 306.74
> legacy APIs: SP/SC: bulk (size: 32): 411.56
> legacy APIs: MP/MC: bulk (size: 8): 501.32
> legacy APIs: MP/MC: bulk (size: 32): 693.07
> 
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
> 
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.36
> legacy APIs: MP/MC: bulk (size: 8): 110.18
> legacy APIs: SP/SC: bulk (size: 32): 23.04
> legacy APIs: MP/MC: bulk (size: 32): 32.29
> 
> ### Testing using all slave nodes ##
> Bulk enq/dequeue count on size 8
> Core [8] count = 293741
> Core [9] count = 293741
> Total count (size: 8): 587482
> 
> Bulk enq/dequeue count on size 32
> Core [8] count = 244909
> Core [9] count = 244909
> Total count (size: 32): 1077300
> 
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 255.37
> elem APIs: element size 16B: MP/MC: single: 456.68
> 
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
> elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
> elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
> elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
> 
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
> 
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
> 
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
> 
> ### Testing using all slave nodes ###
> 
> Bulk enq/dequeue count on size 8
> Core [8] count = 308724
> Core [9] count = 308723
> Total count (size: 8): 617447
> 
> Bulk enq/dequeue count on size 32
> Core [8] count = 214269
> Core [9] count = 214269
> Total count (size: 32): 1045985
> 
> RTE>>ring_perf_autotest // with patch
> 
> ### Testing single element enq/deq ###
> legacy APIs: SP/SC: single: 289.78
> legacy APIs: MP/MC: single: 475.76
> 
> ### Testing burst enq/deq ###
> legacy APIs: SP/SC: burst (size: 8): 323.91
> legacy APIs: SP/SC: burst (size: 32): 424.60
> legacy APIs: MP/MC: burst (size: 8): 523.00
> legacy APIs: MP/MC: burst (size: 32): 717.09
> 
> ### Testing bulk enq/deq ###
> legacy APIs: SP/SC: bulk (size: 8): 317.74
> legacy APIs: SP/SC: bulk (size: 32): 413.57
> legacy APIs: MP/MC: bulk (size: 8): 512.89
> legacy APIs: MP/MC: bulk (size: 32): 712.45
> 
> ### Testing empty bulk deq ###
> legacy APIs: SP/SC: bulk (size: 8): 7.00
> legacy APIs: MP/MC: bulk (size: 8): 7.00
> 
> ### Testing using two physical cores ###
> legacy APIs: SP/SC: bulk (size: 8): 74.82
> legacy APIs: MP/MC: bulk (size: 8): 96.45
> legacy APIs: SP/SC: bulk (size: 32): 22.97
> legacy APIs: MP/MC: bulk (size: 32): 32.52
> 
> ### Testing using all slave nodes ###
> 
> Bulk enq/dequeue count on size 8
> Core [8] count = 283928
> Core [9] count = 283927
> Total count (size: 8): 567855
> 
> Bulk enq/dequeue count on size 32
> Core [8] count = 223916
> Core [9] count = 223915
> Total count (size: 32): 1015686
> 
> ### Testing single element enq/deq ###
> elem APIs: element size 16B: SP/SC: single: 267.65
> elem APIs: element size 16B: MP/MC: single: 439.06
> 
> ### Testing burst enq/deq ###
> elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
> elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
> elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
> elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
> 
> ### Testing bulk enq/deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
> 
> ### Testing empty bulk deq ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
> 
> ### Testing using two physical cores ###
> elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
> elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
> elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
> elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
> 
> ### Testing using all slave nodes ###
> 
> Bulk enq/dequeue count on size 8
> Core [8] count = 304223
> Core [9] count = 304221
> Total count (size: 8): 608444
> 
> Bulk enq/dequeue count on size 32
> Core [8] count = 214856
> Core [9] count = 214855
> Total count (size: 32): 1038155
> Test OK
> RTE>>quit
> 
> 

Encountered a couple of different build errors with these patches on my 
Power 9 system:

In file included from ../lib/librte_ring/rte_ring.h:534,
                  from ../drivers/mempool/ring/rte_mempool_ring.c:9:
../lib/librte_ring/rte_ring_hts_generic.h: In function 
‘__rte_ring_hts_update_tail’:
../lib/librte_ring/rte_ring_hts_generic.h:61:2: warning: implicit 
declaration of function ‘RTE_ASSERT’; did you mean ‘RTE_STR’? 
[-Wimplicit-function-declaration]
   RTE_ASSERT(n >= num);
   ^~~~~~~~~~
   RTE_STR

Fixed by adding "#include <rte_debug.h>" to rte_ring.h.

Also encountered:

In file included from ../app/test/test_ring_hts_stress.c:5:
../app/test/test_ring_stress.h: In function ‘check_updt_elem’:
../app/test/test_ring_stress.h:162:9: error: unknown type name 
‘rte_spinlock_t’
   static rte_spinlock_t dump_lock;
          ^~~~~~~~~~~~~~
../app/test/test_ring_stress.h:166:4: warning: implicit declaration of 
function ‘rte_spinlock_lock’; did you mean ‘rte_calloc_socket’? 
[-Wimplicit-function-declaration]
     rte_spinlock_lock(&dump_lock);
     ^~~~~~~~~~~~~~~~~
     rte_calloc_socket
../app/test/test_ring_stress.h:166:4: warning: nested extern declaration 
of ‘rte_spinlock_lock’ [-Wnested-externs]
../app/test/test_ring_stress.h:172:4: warning: implicit declaration of 
function ‘rte_spinlock_unlock’; did you mean ‘pthread_rwlock_unlock’? 
[-Wimplicit-function-declaration]
     rte_spinlock_unlock(&dump_lock);
     ^~~~~~~~~~~~~~~~~~~
     pthread_rwlock_unlock
../app/test/test_ring_stress.h:172:4: warning: nested extern declaration 
of ‘rte_spinlock_unlock’ [-Wnested-externs]

Fixed by adding "#include <rte_spinlock.h>" to test_ring_stress.h.

Autoperf test results
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.14
legacy APIs: MP/MC: single: 56.26

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.59
legacy APIs: SP/SC: burst (size: 32): 49.87
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.68

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.85
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.60

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16

### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.46
legacy APIs: MP/MC: bulk (size: 8): 16.20
legacy APIs: SP/SC: bulk (size: 32): 3.21
legacy APIs: MP/MC: bulk (size: 32): 3.73

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.34
legacy APIs: MP/MC: bulk (size: 8): 37.99
legacy APIs: SP/SC: bulk (size: 32): 10.19
legacy APIs: MP/MC: bulk (size: 32): 11.90

### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.50
legacy APIs: MP/MC: bulk (size: 8): 63.65
legacy APIs: SP/SC: bulk (size: 32): 12.49
legacy APIs: MP/MC: bulk (size: 32): 23.53

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [4] count = 5604
Core [5] count = 5563
Core [6] count = 5576
Core [7] count = 5630
Core [8] count = 5643
Core [9] count = 5727
Core [10] count = 5698
Core [11] count = 5711
Core [64] count = 5259
Core [65] count = 5322
Core [66] count = 5321
Core [67] count = 5310
Core [68] count = 4350
Core [69] count = 4455
Core [70] count = 4546
Core [71] count = 4475
Total count (size: 8): 84190

Bulk enq/dequeue count on size 32
Core [4] count = 5543
Core [5] count = 5555
Core [6] count = 5596
Core [7] count = 5584
Core [8] count = 5613
Core [9] count = 5686
Core [10] count = 5689
Core [11] count = 5677
Core [64] count = 5228
Core [65] count = 5389
Core [66] count = 5406
Core [67] count = 5359
Core [68] count = 4554
Core [69] count = 4673
Core [70] count = 4675
Core [71] count = 4644
Total count (size: 32): 169061

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.84
elem APIs: element size 16B: MP/MC: single: 56.77

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.98
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.16
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.58
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.75

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 45.01
elem APIs: element size 16B: SP/SC: bulk (size: 32): 59.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.58
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.76

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16

### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.44
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.22
elem APIs: element size 16B: MP/MC: bulk (size: 32): 3.97

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.07
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.50
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 11.73

### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.55
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.10
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.33
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.10

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [4] count = 5489
Core [5] count = 5559
Core [6] count = 5566
Core [7] count = 5577
Core [8] count = 5645
Core [9] count = 5699
Core [10] count = 5695
Core [11] count = 5733
Core [64] count = 5202
Core [65] count = 5284
Core [66] count = 5319
Core [67] count = 5349
Core [68] count = 4331
Core [69] count = 4465
Core [70] count = 4484
Core [71] count = 4439
Total count (size: 8): 83836

Bulk enq/dequeue count on size 32
Core [4] count = 5567
Core [5] count = 5492
Core [6] count = 5517
Core [7] count = 5515
Core [8] count = 5593
Core [9] count = 5650
Core [10] count = 5694
Core [11] count = 5665
Core [64] count = 5236
Core [65] count = 5319
Core [66] count = 5333
Core [67] count = 5304
Core [68] count = 4608
Core [69] count = 4669
Core [70] count = 4690
Core [71] count = 4654
Total count (size: 32): 168342
Test OK
---------------------
RTE>>ring_perf_autotest // DPDK 20.02, without patch

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 42.18
legacy APIs: MP/MC: single: 56.26

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 43.60
legacy APIs: SP/SC: burst (size: 32): 49.86
legacy APIs: MP/MC: burst (size: 8): 58.43
legacy APIs: MP/MC: burst (size: 32): 65.67

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 43.59
legacy APIs: SP/SC: bulk (size: 32): 49.86
legacy APIs: MP/MC: bulk (size: 8): 58.43
legacy APIs: MP/MC: bulk (size: 32): 65.63

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 7.16
legacy APIs: MP/MC: bulk (size: 8): 7.16

### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 12.07
legacy APIs: MP/MC: bulk (size: 8): 16.24
legacy APIs: SP/SC: bulk (size: 32): 3.20
legacy APIs: MP/MC: bulk (size: 32): 3.72

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 33.41
legacy APIs: MP/MC: bulk (size: 8): 38.01
legacy APIs: SP/SC: bulk (size: 32): 10.23
legacy APIs: MP/MC: bulk (size: 32): 11.90

### Testing using two NUMA nodes ###
legacy APIs: SP/SC: bulk (size: 8): 49.27
legacy APIs: MP/MC: bulk (size: 8): 64.80
legacy APIs: SP/SC: bulk (size: 32): 12.45
legacy APIs: MP/MC: bulk (size: 32): 23.11

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [4] count = 5637
Core [5] count = 5599
Core [6] count = 5623
Core [7] count = 5627
Core [8] count = 5723
Core [9] count = 5758
Core [10] count = 5714
Core [11] count = 5724
Core [64] count = 5310
Core [65] count = 5438
Core [66] count = 5448
Core [67] count = 5374
Core [68] count = 4441
Core [69] count = 4550
Core [70] count = 4550
Core [71] count = 4558
Total count (size: 8): 85074

Bulk enq/dequeue count on size 32
Core [4] count = 5608
Core [5] count = 5623
Core [6] count = 5590
Core [7] count = 5658
Core [8] count = 5680
Core [9] count = 5738
Core [10] count = 5692
Core [11] count = 5712
Core [64] count = 5273
Core [65] count = 5363
Core [66] count = 5341
Core [67] count = 5349
Core [68] count = 4591
Core [69] count = 4673
Core [70] count = 4698
Core [71] count = 4687
Total count (size: 32): 170350

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 42.82
elem APIs: element size 16B: MP/MC: single: 56.79

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 44.99
elem APIs: element size 16B: SP/SC: burst (size: 32): 59.00
elem APIs: element size 16B: MP/MC: burst (size: 8): 60.59
elem APIs: element size 16B: MP/MC: burst (size: 32): 74.78

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 44.97
elem APIs: element size 16B: SP/SC: bulk (size: 32): 58.91
elem APIs: element size 16B: MP/MC: bulk (size: 8): 60.60
elem APIs: element size 16B: MP/MC: bulk (size: 32): 74.61

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.16
elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.16

### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.18
elem APIs: element size 16B: MP/MC: bulk (size: 8): 15.41
elem APIs: element size 16B: SP/SC: bulk (size: 32): 3.19
elem APIs: element size 16B: MP/MC: bulk (size: 32): 4.06

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 42.08
elem APIs: element size 16B: MP/MC: bulk (size: 8): 44.52
elem APIs: element size 16B: SP/SC: bulk (size: 32): 10.73
elem APIs: element size 16B: MP/MC: bulk (size: 32): 12.39

### Testing using two NUMA nodes ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 49.65
elem APIs: element size 16B: MP/MC: bulk (size: 8): 93.27
elem APIs: element size 16B: SP/SC: bulk (size: 32): 12.38
elem APIs: element size 16B: MP/MC: bulk (size: 32): 27.19

### Testing using all slave nodes ###

Bulk enq/dequeue count on size 8
Core [4] count = 5629
Core [5] count = 5585
Core [6] count = 5676
Core [7] count = 5604
Core [8] count = 5639
Core [9] count = 5731
Core [10] count = 5694
Core [11] count = 5707
Core [64] count = 5254
Core [65] count = 5331
Core [66] count = 5340
Core [67] count = 5355
Core [68] count = 4339
Core [69] count = 4481
Core [70] count = 4504
Core [71] count = 4507
Total count (size: 8): 84376

Bulk enq/dequeue count on size 32
Core [4] count = 5518
Core [5] count = 5493
Core [6] count = 5559
Core [7] count = 5484
Core [8] count = 5623
Core [9] count = 5669
Core [10] count = 5661
Core [11] count = 5658
Core [64] count = 5207
Core [65] count = 5305
Core [66] count = 5273
Core [67] count = 5303
Core [68] count = 4542
Core [69] count = 4682
Core [70] count = 4672
Core [71] count = 4609
Total count (size: 32): 168634
Test OK


Dave

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-28  0:17           ` David Christensen
@ 2020-03-20 16:45             ` Ananyev, Konstantin
  0 siblings, 0 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-03-20 16:45 UTC (permalink / raw)
  To: David Christensen, Jerin Jacob; +Cc: Stephen Hemminger, dpdk-dev, Olivier Matz


> >
> >
> > I tested on an arm64 HW. The former section is without the
> > patch(20.02) and later one with this patch.
> > I agree with Konstantin that getting more platform tests will be good
> > early so that we can focus on the approach
> > to avoid back and forth latter.
> >
> >
> > RTE>>ring_perf_autotest // without path
> >
> > ### Testing single element enq/deq ###
> > legacy APIs: SP/SC: single: 289.78
> > legacy APIs: MP/MC: single: 516.20
> >
> > ### Testing burst enq/deq ###
> > legacy APIs: SP/SC: burst (size: 8): 312.88
> > legacy APIs: SP/SC: burst (size: 32): 426.72
> > legacy APIs: MP/MC: burst (size: 8): 510.95
> > legacy APIs: MP/MC: burst (size: 32): 702.01
> >
> > ### Testing bulk enq/deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 306.74
> > legacy APIs: SP/SC: bulk (size: 32): 411.56
> > legacy APIs: MP/MC: bulk (size: 8): 501.32
> > legacy APIs: MP/MC: bulk (size: 32): 693.07
> >
> > ### Testing empty bulk deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 7.00
> > legacy APIs: MP/MC: bulk (size: 8): 7.00
> >
> > ### Testing using two physical cores ###
> > legacy APIs: SP/SC: bulk (size: 8): 74.36
> > legacy APIs: MP/MC: bulk (size: 8): 110.18
> > legacy APIs: SP/SC: bulk (size: 32): 23.04
> > legacy APIs: MP/MC: bulk (size: 32): 32.29
> >
> > ### Testing using all slave nodes ##
> > Bulk enq/dequeue count on size 8
> > Core [8] count = 293741
> > Core [9] count = 293741
> > Total count (size: 8): 587482
> >
> > Bulk enq/dequeue count on size 32
> > Core [8] count = 244909
> > Core [9] count = 244909
> > Total count (size: 32): 1077300
> >
> > ### Testing single element enq/deq ###
> > elem APIs: element size 16B: SP/SC: single: 255.37
> > elem APIs: element size 16B: MP/MC: single: 456.68
> >
> > ### Testing burst enq/deq ###
> > elem APIs: element size 16B: SP/SC: burst (size: 8): 291.99
> > elem APIs: element size 16B: SP/SC: burst (size: 32): 456.25
> > elem APIs: element size 16B: MP/MC: burst (size: 8): 497.77
> > elem APIs: element size 16B: MP/MC: burst (size: 32): 680.87
> >
> > ### Testing bulk enq/deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 284.40
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 453.17
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 485.77
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 675.08
> >
> > ### Testing empty bulk deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 8.00
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.00
> >
> > ### Testing using two physical cores ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 74.45
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 105.91
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 22.92
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.55
> >
> > ### Testing using all slave nodes ###
> >
> > Bulk enq/dequeue count on size 8
> > Core [8] count = 308724
> > Core [9] count = 308723
> > Total count (size: 8): 617447
> >
> > Bulk enq/dequeue count on size 32
> > Core [8] count = 214269
> > Core [9] count = 214269
> > Total count (size: 32): 1045985
> >
> > RTE>>ring_perf_autotest // with patch
> >
> > ### Testing single element enq/deq ###
> > legacy APIs: SP/SC: single: 289.78
> > legacy APIs: MP/MC: single: 475.76
> >
> > ### Testing burst enq/deq ###
> > legacy APIs: SP/SC: burst (size: 8): 323.91
> > legacy APIs: SP/SC: burst (size: 32): 424.60
> > legacy APIs: MP/MC: burst (size: 8): 523.00
> > legacy APIs: MP/MC: burst (size: 32): 717.09
> >
> > ### Testing bulk enq/deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 317.74
> > legacy APIs: SP/SC: bulk (size: 32): 413.57
> > legacy APIs: MP/MC: bulk (size: 8): 512.89
> > legacy APIs: MP/MC: bulk (size: 32): 712.45
> >
> > ### Testing empty bulk deq ###
> > legacy APIs: SP/SC: bulk (size: 8): 7.00
> > legacy APIs: MP/MC: bulk (size: 8): 7.00
> >
> > ### Testing using two physical cores ###
> > legacy APIs: SP/SC: bulk (size: 8): 74.82
> > legacy APIs: MP/MC: bulk (size: 8): 96.45
> > legacy APIs: SP/SC: bulk (size: 32): 22.97
> > legacy APIs: MP/MC: bulk (size: 32): 32.52
> >
> > ### Testing using all slave nodes ###
> >
> > Bulk enq/dequeue count on size 8
> > Core [8] count = 283928
> > Core [9] count = 283927
> > Total count (size: 8): 567855
> >
> > Bulk enq/dequeue count on size 32
> > Core [8] count = 223916
> > Core [9] count = 223915
> > Total count (size: 32): 1015686
> >
> > ### Testing single element enq/deq ###
> > elem APIs: element size 16B: SP/SC: single: 267.65
> > elem APIs: element size 16B: MP/MC: single: 439.06
> >
> > ### Testing burst enq/deq ###
> > elem APIs: element size 16B: SP/SC: burst (size: 8): 302.44
> > elem APIs: element size 16B: SP/SC: burst (size: 32): 466.31
> > elem APIs: element size 16B: MP/MC: burst (size: 8): 502.51
> > elem APIs: element size 16B: MP/MC: burst (size: 32): 695.81
> >
> > ### Testing bulk enq/deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 295.15
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 462.77
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 496.89
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 690.46
> >
> > ### Testing empty bulk deq ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 7.50
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 7.44
> >
> > ### Testing using two physical cores ###
> > elem APIs: element size 16B: SP/SC: bulk (size: 8): 65.85
> > elem APIs: element size 16B: MP/MC: bulk (size: 8): 103.80
> > elem APIs: element size 16B: SP/SC: bulk (size: 32): 23.27
> > elem APIs: element size 16B: MP/MC: bulk (size: 32): 31.17
> >
> > ### Testing using all slave nodes ###
> >
> > Bulk enq/dequeue count on size 8
> > Core [8] count = 304223
> > Core [9] count = 304221
> > Total count (size: 8): 608444
> >
> > Bulk enq/dequeue count on size 32
> > Core [8] count = 214856
> > Core [9] count = 214855
> > Total count (size: 32): 1038155
> > Test OK
> > RTE>>quit
> >
> >
> 
> Encountered a couple of different build errors with these patches on my
> Power 9 system:
> 
> In file included from ../lib/librte_ring/rte_ring.h:534,
>                   from ../drivers/mempool/ring/rte_mempool_ring.c:9:
> ../lib/librte_ring/rte_ring_hts_generic.h: In function
> ‘__rte_ring_hts_update_tail’:
> ../lib/librte_ring/rte_ring_hts_generic.h:61:2: warning: implicit
> declaration of function ‘RTE_ASSERT’; did you mean ‘RTE_STR’?
> [-Wimplicit-function-declaration]
>    RTE_ASSERT(n >= num);
>    ^~~~~~~~~~
>    RTE_STR
> 
> Fixed by adding "#include <rte_debug.h>" to rte_ring.h.
> 
> Also encountered:
> 
> In file included from ../app/test/test_ring_hts_stress.c:5:
> ../app/test/test_ring_stress.h: In function ‘check_updt_elem’:
> ../app/test/test_ring_stress.h:162:9: error: unknown type name
> ‘rte_spinlock_t’
>    static rte_spinlock_t dump_lock;
>           ^~~~~~~~~~~~~~
> ../app/test/test_ring_stress.h:166:4: warning: implicit declaration of
> function ‘rte_spinlock_lock’; did you mean ‘rte_calloc_socket’?
> [-Wimplicit-function-declaration]
>      rte_spinlock_lock(&dump_lock);
>      ^~~~~~~~~~~~~~~~~
>      rte_calloc_socket
> ../app/test/test_ring_stress.h:166:4: warning: nested extern declaration
> of ‘rte_spinlock_lock’ [-Wnested-externs]
> ../app/test/test_ring_stress.h:172:4: warning: implicit declaration of
> function ‘rte_spinlock_unlock’; did you mean ‘pthread_rwlock_unlock’?
> [-Wimplicit-function-declaration]
>      rte_spinlock_unlock(&dump_lock);
>      ^~~~~~~~~~~~~~~~~~~
>      pthread_rwlock_unlock
> ../app/test/test_ring_stress.h:172:4: warning: nested extern declaration
> of ‘rte_spinlock_unlock’ [-Wnested-externs]
> 
> Fixed by adding "#include <rte_spinlock.h>" to test_ring_stress.h.

Thanks a lot for trying it guys. 
About compilation issues - will  to address it in V1.
Konstantin


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
                   ` (6 preceding siblings ...)
  2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
@ 2020-03-25 20:43 ` Honnappa Nagarahalli
  2020-03-26  1:50   ` Ananyev, Konstantin
  7 siblings, 1 reply; 23+ messages in thread
From: Honnappa Nagarahalli @ 2020-03-25 20:43 UTC (permalink / raw)
  To: Konstantin Ananyev, dev; +Cc: olivier.matz, nd, Honnappa Nagarahalli, nd

<snip>

> Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> 
> Upfront note - that RFC is not a complete patch.
> It introduces an ABI breakage, plus it doesn't update ring_elem code properly,
As per the current rules, these changes (in the current form) will be accepted only for 20.11 release. How do we address this for immediate requirements like RCU defer APIs? 
I suggest that we move forward with my RFC (taking into consideration your feedback) to make progress on RCU APIs.

> etc.
> I plan to deal with all these things in later versions.
> Right now I seek an initial feedback about proposed ideas.
> Would also ask people to repeat performance tests (see below) on their
> platforms to confirm the impact.
> 
> More and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives (spin-locks, etc.)
> on overcommitted systems.
> The situation gets much worse when some sort of fair-locking technique is
> used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling order matters a
> lot.
> This is a well-known problem for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of un-fair locking,
> waiting on tail is very similar to ticket lock schema - tail has to be updated in
> particular order.
> That makes current rte_ring implementation to perform really pure on some
> overcommited scenarios.
> While it is probably not possible to completely resolve this problem in
> userspace only (without some kernel communication/intervention), removing
> fairness in tail update can mitigate it significantly.
> So this RFC proposes two new optional ring synchronization modes:
> 1) Head/Tail Sync (HTS) mode
> In that mode enqueue/dequeue operation is fully serialized:
>     only one thread at a time is allowed to perform given op.
>     As another enhancement provide ability to split enqueue/dequeue
>     operation into two phases:
>       - enqueue/dequeue start
>       - enqueue/dequeue finish
>     That allows user to inspect objects in the ring without removing
>     them from it (aka MT safe peek).
IMO, this will not address the problem described above. For ex: when a producer updates the head and gets scheduled out, other producers have to spin. The problem is probably worse as with non-HTS case moving of the head and copying of the ring elements can happen in parallel between the producers (similarly for consumers).
IMO, HTS should not be a configurable flag. In RCU requirement, a MP enqueue and HTS dequeue are required.

> 2) Relaxed Tail Sync (RTS)
> The main difference from original MP/MC algorithm is that tail value is
> increased not by every thread that finished enqueue/dequeue, but only by the
> last one.
> That allows threads to avoid spinning on ring tail value, leaving actual tail value
> change to the last thread in the update queue.
This can be a configurable flag on the ring.
I am not sure how this solves the problem you have stated above completely. Updating the count from all intermediate threads is still required to update the value of the head. But yes, it reduces the severity of the problem by not enforcing the order in which the tail is updated.
I also think it introduces the problem on the other side of the ring because the tail is not updated soon enough (the other side has to wait longer for the elements to become available). It also introduces another configuration parameter (HTD_MAX_DEF) which they have to deal with.
Users have to still implement the current hypervisor related solutions.
IMO, we should run the benchmark for this on an over committed setup to understand the benefits.

> 
> Test results on IA (see below) show significant improvements for average
> enqueue/dequeue op times on overcommitted systems.
> For 'classic' DPDK deployments (one thread per core) original MP/MC
> algorithm still shows best numbers, though for 64-bit target RTS numbers are
> not that far away.
> Numbers were produced by ring_stress_*autotest (first patch in these series).
> 
> X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
> 
>                                                 MP/MC      HTS     RTS
> 1thread@1core(--lcores=6-7)                     8.00       8.15    8.99
> 2thread@2core(--lcores=6-8)                     19.14      19.61   20.35
> 4thread@4core(--lcores=6-10)                    29.43      29.79   31.82
> 8thread@8core(--lcores=6-14)                    110.59     192.81  119.50
> 16thread@16core(--lcores=6-22)                  461.03     813.12  495.59
> 32thread/@32core(--lcores='6-22,55-70')         982.90     1972.38 1160.51
> 
> 2thread@1core(--lcores='6,(10-11)@7'            20140.50   23.58   25.14
> 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8'  153680.60  76.88   80.05
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  280314.32  294.72  318.79
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59  1144.02
> 1175.14 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80
> 4627.48 4892.68
> 
> 8thread@2core(--lcores='6,(10-17)@(7,8))'       321085.98  298.59  307.47
> 16thread@4core(--lcores='6,(20-35)@(7-10))'     1900705.61 575.35  678.29
> 32thread@4core(--lcores='6,(20-51)@(7-10))'     5510445.85 2164.36 2714.12
> 
> i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
> 
>                                                 MP/MC      HTS     RTS
> 1thread@1core(--lcores=6-7)                     7.85       12.13   11.31
> 2thread@2core(--lcores=6-8)                     17.89      24.52   21.86
> 8thread@8core(--lcores=6-14)                    32.58      354.20  54.58
> 32thread/@32core(--lcores='6-22,55-70')         813.77     6072.41 2169.91
> 
> 2thread@1core(--lcores='6,(10-11)@7'            16095.00   36.06   34.74
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  1140354.54 346.61  361.57
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90
> 1416.65
> 
> 8thread@2core(--lcores='6,(10-17)@(7,8))'       594358.61  332.70  357.74
> 32thread@4core(--lcores='6,(20-51)@(7-10))'     5319896.86 2836.44 3028.87
> 
> Konstantin Ananyev (6):
>   test/ring: add contention stress test
>   ring: rework ring layout to allow new sync schemes
>   ring: introduce RTS ring mode
>   test/ring: add contention stress test for RTS ring
>   ring: introduce HTS ring mode
>   test/ring: add contention stress test for HTS ring
> 
>  app/test/Makefile                      |   3 +
>  app/test/meson.build                   |   3 +
>  app/test/test_pdump.c                  |   6 +-
>  app/test/test_ring_hts_stress.c        |  28 ++
>  app/test/test_ring_rts_stress.c        |  28 ++
>  app/test/test_ring_stress.c            |  27 ++
>  app/test/test_ring_stress.h            | 477 +++++++++++++++++++
>  lib/librte_pdump/rte_pdump.c           |   2 +-
>  lib/librte_port/rte_port_ring.c        |  12 +-
>  lib/librte_ring/Makefile               |   4 +-
>  lib/librte_ring/meson.build            |   4 +-
>  lib/librte_ring/rte_ring.c             |  84 +++-
>  lib/librte_ring/rte_ring.h             | 619 +++++++++++++++++++++++--
>  lib/librte_ring/rte_ring_elem.h        |   8 +-
>  lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
> lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
>  16 files changed, 1721 insertions(+), 52 deletions(-)  create mode 100644
> app/test/test_ring_hts_stress.c  create mode 100644
> app/test/test_ring_rts_stress.c  create mode 100644
> app/test/test_ring_stress.c  create mode 100644 app/test/test_ring_stress.h
> create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
>  create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> 
> --
> 2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode
  2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
@ 2020-03-25 20:44   ` Honnappa Nagarahalli
  2020-03-26 12:26     ` Ananyev, Konstantin
  0 siblings, 1 reply; 23+ messages in thread
From: Honnappa Nagarahalli @ 2020-03-25 20:44 UTC (permalink / raw)
  To: Konstantin Ananyev, dev; +Cc: olivier.matz, nd, Honnappa Nagarahalli, nd

<snip>

> 
> Introduce head/tail sync mode for MT ring synchronization.
> In that mode enqueue/dequeue operation is fully serialized:
> only one thread at a time is allowed to perform given op.
> Suppose to reduce stall times in case when ring is used on overcommitted
> cpus (multiple active threads on the same cpu).
> As another enhancement provide ability to split enqueue/dequeue operation
> into two phases:
>   - enqueue/dequeue start
>   - enqueue/dequeue finish
> That allows user to inspect objects in the ring without removing them from it
> (aka MT safe peek).
> 
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
>  lib/librte_ring/Makefile               |   1 +
>  lib/librte_ring/meson.build            |   1 +
>  lib/librte_ring/rte_ring.c             |  15 +-
>  lib/librte_ring/rte_ring.h             | 259 ++++++++++++++++++++++++-
>  lib/librte_ring/rte_ring_hts_generic.h | 228 ++++++++++++++++++++++
>  5 files changed, 500 insertions(+), 4 deletions(-)  create mode 100644
> lib/librte_ring/rte_ring_hts_generic.h
> 
> diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile index
> 4f90344f4..0c7f8f918 100644
> --- a/lib/librte_ring/Makefile
> +++ b/lib/librte_ring/Makefile
> @@ -19,6 +19,7 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include :=
> rte_ring.h \
>  					rte_ring_elem.h \
>  					rte_ring_generic.h \
>  					rte_ring_c11_mem.h \
> +					rte_ring_hts_generic.h \
>  					rte_ring_rts_generic.h
> 
>  include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build index
> dc8d7dbea..5aa673199 100644
> --- a/lib/librte_ring/meson.build
> +++ b/lib/librte_ring/meson.build
> @@ -6,6 +6,7 @@ headers = files('rte_ring.h',
>  		'rte_ring_elem.h',
>  		'rte_ring_c11_mem.h',
>  		'rte_ring_generic.h',
> +		'rte_ring_hts_generic.h',
>  		'rte_ring_rts_generic.h')
> 
>  # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
> diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index
> 1ce0af3e5..d3b948667 100644
> --- a/lib/librte_ring/rte_ring.c
> +++ b/lib/librte_ring/rte_ring.c
> @@ -102,9 +102,9 @@ static int
>  get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)  {
>  	static const uint32_t prod_st_flags =
> -		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> +		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ |
> RING_F_MP_HTS_ENQ);
>  	static const uint32_t cons_st_flags =
> -		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> +		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ |
> RING_F_MC_HTS_DEQ);
> 
>  	switch (flags & prod_st_flags) {
>  	case 0:
> @@ -116,6 +116,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st,
> uint32_t *cons_st)
>  	case RING_F_MP_RTS_ENQ:
>  		*prod_st = RTE_RING_SYNC_MT_RTS;
>  		break;
> +	case RING_F_MP_HTS_ENQ:
> +		*prod_st = RTE_RING_SYNC_MT_HTS;
> +		break;
>  	default:
>  		return -EINVAL;
>  	}
> @@ -130,6 +133,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st,
> uint32_t *cons_st)
>  	case RING_F_MC_RTS_DEQ:
>  		*cons_st = RTE_RING_SYNC_MT_RTS;
>  		break;
> +	case RING_F_MC_HTS_DEQ:
> +		*cons_st = RTE_RING_SYNC_MT_HTS;
> +		break;
>  	default:
>  		return -EINVAL;
>  	}
> @@ -151,6 +157,11 @@ rte_ring_init(struct rte_ring *r, const char *name,
> unsigned count,
>  	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
>  			  RTE_CACHE_LINE_MASK) != 0);
> 
> +	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> +		offsetof(struct rte_ring_hts_headtail, sync_type));
> +	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> +		offsetof(struct rte_ring_hts_headtail, ht.pos.tail));
> +
>  	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
>  		offsetof(struct rte_ring_rts_headtail, sync_type));
>  	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) != diff --git
> a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> a130aeb9d..52edcea11 100644
> --- a/lib/librte_ring/rte_ring.h
> +++ b/lib/librte_ring/rte_ring.h
> @@ -66,11 +66,11 @@ enum {
>  	RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
>  	RTE_RING_SYNC_ST,     /**< single thread only */
>  	RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> +	RTE_RING_SYNC_MT_HTS, /**< multi-thread head/tail sync */
>  };
> 
>  /**
> - * structure to hold a pair of head/tail values and other metadata.
> - * used by RTE_RING_SYNC_MT, RTE_RING_SYNC_ST sync types.
> + * Structure to hold a pair of head/tail values and other metadata.
>   * Depending on sync_type format of that structure might differ
>   * depending on the sync mechanism selelcted, but offsets for
>   * *sync_type* and *tail* values should always remain the same.
> @@ -96,6 +96,19 @@ struct rte_ring_rts_headtail {
>  	volatile union rte_ring_ht_poscnt head;  };
> 
> +union rte_ring_ht_pos {
> +	uint64_t raw;
> +	struct {
> +		uint32_t tail; /**< tail position */
> +		uint32_t head; /**< head position */
> +	} pos;
> +};
> +
> +struct rte_ring_hts_headtail {
> +	uint32_t sync_type; /**< sync type of prod/cons */
> +	volatile union rte_ring_ht_pos ht __rte_aligned(8); };
> +
>  /**
>   * An RTE ring structure.
>   *
> @@ -126,6 +139,7 @@ struct rte_ring {
>  	RTE_STD_C11
>  	union {
>  		struct rte_ring_headtail prod;
> +		struct rte_ring_hts_headtail hts_prod;
>  		struct rte_ring_rts_headtail rts_prod;
>  	}  __rte_cache_aligned;
> 
> @@ -135,6 +149,7 @@ struct rte_ring {
>  	RTE_STD_C11
>  	union {
>  		struct rte_ring_headtail cons;
> +		struct rte_ring_hts_headtail hts_cons;
>  		struct rte_ring_rts_headtail rts_cons;
>  	}  __rte_cache_aligned;
> 
> @@ -157,6 +172,9 @@ struct rte_ring {
>  #define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS".
> */  #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC
> RTS". */
> 
> +#define RING_F_MP_HTS_ENQ 0x0020 /**< The default enqueue is "MP
> HTS".
> +*/ #define RING_F_MC_HTS_DEQ 0x0040 /**< The default dequeue is "MC
> +HTS". */
> +
>  #define __IS_SP RTE_RING_SYNC_ST
>  #define __IS_MP RTE_RING_SYNC_MT
>  #define __IS_SC RTE_RING_SYNC_ST
> @@ -513,6 +531,82 @@ __rte_ring_do_rts_dequeue(struct rte_ring *r, void
> **obj_table,
>  	return n;
>  }
> 
> +#include <rte_ring_hts_generic.h>
> +
> +/**
> + * @internal Start to enqueue several objects on the HTS ring.
> + * Note that user has to call appropriate enqueue_finish()
> + * to complete given enqueue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects).
> + * @param n
> + *   The number of objects to add in the ring from the obj_table.
> + * @param behavior
> + *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
> + *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> ring
> + * @param free_space
> + *   returns the amount of space after the enqueue operation has finished
> + * @return
> + *   Actual number of objects enqueued.
> + *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_hts_enqueue_start(struct rte_ring *r, void * const *obj_table,
> +		uint32_t n, enum rte_ring_queue_behavior behavior,
> +		uint32_t *free_space)
> +{
> +	uint32_t free, head;
> +
> +	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
> +
> +	if (n != 0)
> +		ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> +
> +	if (free_space != NULL)
> +		*free_space = free - n;
> +	return n;
> +}
rte_ring.h is becoming too big. May be we should move these functions to another HTS specific file. But leave the top level API in rte_ring.h. Similarly for RTS.

> +
> +/**
> + * @internal Start to dequeue several objects from the HTS ring.
> + * Note that user has to call appropriate dequeue_finish()
> + * to complete given dequeue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects).
> + * @param n
> + *   The number of objects to pull from the ring.
> + * @param behavior
> + *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
> + *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> ring
> + * @param available
> + *   returns the number of remaining ring entries after the dequeue has
> finished
> + * @return
> + *   - Actual number of objects dequeued.
> + *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_do_hts_dequeue_start(struct rte_ring *r, void **obj_table,
> +		unsigned int n, enum rte_ring_queue_behavior behavior,
> +		unsigned int *available)
> +{
> +	uint32_t entries, head;
> +
> +	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries);
> +
> +	if (n != 0)
> +		DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> +
> +	if (available != NULL)
> +		*available = entries - n;
> +	return n;
> +}
> +
>  /**
>   * Enqueue several objects on the ring (multi-producers safe).
>   *
> @@ -585,6 +679,47 @@ rte_ring_rts_enqueue_bulk(struct rte_ring *r, void *
> const *obj_table,
>  			free_space);
>  }
> 
> +/**
> + * Start to enqueue several objects on the HTS ring (multi-producers safe).
> + * Note that user has to call appropriate dequeue_finish()
> + * to complete given dequeue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects).
> + * @param n
> + *   The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + *   if non-NULL, returns the amount of space in the ring after the
> + *   enqueue operation has finished.
> + * @return
> + *   The number of objects enqueued, either 0 or n
> + */
> +static __rte_always_inline unsigned int
> +rte_ring_hts_enqueue_bulk_start(struct rte_ring *r, void * const *obj_table,
> +			 unsigned int n, unsigned int *free_space) {
> +	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
> +		RTE_RING_QUEUE_FIXED, free_space);
> +}
I do not clearly understand the requirements on the enqueue_start and enqueue_finish in the form they are here.
IMO, only requirement for these APIs is to provide the ability to avoid intermediate memcpys.

> +
> +static __rte_always_inline void
> +rte_ring_hts_enqueue_finish(struct rte_ring *r, unsigned int n) {
> +	__rte_ring_hts_update_tail(&r->hts_prod, n, 1); }
> +
> +static __rte_always_inline unsigned int
> +rte_ring_hts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> +			 unsigned int n, unsigned int *free_space) {
> +	n = rte_ring_hts_enqueue_bulk_start(r, obj_table, n, free_space);
> +	if (n != 0)
> +		rte_ring_hts_enqueue_finish(r, n);
> +	return n;
> +}
> +
>  /**
>   * Enqueue several objects on a ring.
>   *
> @@ -615,6 +750,8 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const
> *obj_table,
>  		return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
>  	case RTE_RING_SYNC_MT_RTS:
>  		return rte_ring_rts_enqueue_bulk(r, obj_table, n, free_space);
> +	case RTE_RING_SYNC_MT_HTS:
> +		return rte_ring_hts_enqueue_bulk(r, obj_table, n,
> free_space);
>  	}
> 
>  	/* valid ring should never reach this point */ @@ -753,6 +890,47 @@
> rte_ring_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
>  			available);
>  }
> 
> +/**
> + * Start to dequeue several objects from an HTS ring (multi-consumers safe).
> + * Note that user has to call appropriate dequeue_finish()
> + * to complete given dequeue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + *   The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + *   If non-NULL, returns the number of remaining ring entries after the
> + *   dequeue has finished.
> + * @return
> + *   The number of objects dequeued, either 0 or n
> + */
> +static __rte_always_inline unsigned int
> +rte_ring_hts_dequeue_bulk_start(struct rte_ring *r, void **obj_table,
> +		unsigned int n, unsigned int *available) {
> +	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
> +		RTE_RING_QUEUE_FIXED, available);
> +}
IMO, we should look to provide the ability to avoid intermediate copies when the data from the ring needs to be distributed to different locations.
My proposal in its form is complicated. But, I am thinking that, if the return values are abstracted in a structure, it might look much simple.

> +
> +static __rte_always_inline void
> +rte_ring_hts_dequeue_finish(struct rte_ring *r, unsigned int n) {
> +	__rte_ring_hts_update_tail(&r->hts_cons, n, 0); }
> +
> +static __rte_always_inline unsigned int
> +rte_ring_hts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> +		unsigned int n, unsigned int *available) {
> +	n = rte_ring_hts_dequeue_bulk_start(r, obj_table, n, available);
> +	if (n != 0)
> +		rte_ring_hts_dequeue_finish(r, n);
> +	return n;
> +}
> +
>  /**
>   * Dequeue several objects from a ring.
>   *
> @@ -783,6 +961,8 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void
> **obj_table, unsigned int n,
>  		return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
>  	case RTE_RING_SYNC_MT_RTS:
>  		return rte_ring_rts_dequeue_bulk(r, obj_table, n, available);
> +	case RTE_RING_SYNC_MT_HTS:
> +		return rte_ring_hts_dequeue_bulk(r, obj_table, n, available);
>  	}
> 
>  	/* valid ring should never reach this point */ @@ -1111,6 +1291,41
> @@ rte_ring_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
>  			RTE_RING_QUEUE_VARIABLE, free_space);  }
> 
> +/**
> + * Start to enqueue several objects on the HTS ring (multi-producers safe).
> + * Note that user has to call appropriate dequeue_finish()
> + * to complete given dequeue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects).
> + * @param n
> + *   The number of objects to add in the ring from the obj_table.
> + * @param free_space
> + *   if non-NULL, returns the amount of space in the ring after the
> + *   enqueue operation has finished.
> + * @return
> + *   The number of objects enqueued, either 0 or n
> + */
> +static __rte_always_inline unsigned int
> +rte_ring_hts_enqueue_burst_start(struct rte_ring *r, void * const *obj_table,
> +			 unsigned int n, unsigned int *free_space) {
> +	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
> +		RTE_RING_QUEUE_VARIABLE, free_space); }
> +
rte_ring_hts_enqueue_burst_finish is not implemented. It requires the 'n' returned from ' rte_ring_hts_enqueue_burst_start' to be passed. We can't completely avoid passing correct information between xxx_start and xxx_finish APIs.

> +static __rte_always_inline unsigned int
> +rte_ring_hts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> +			 unsigned int n, unsigned int *free_space) {
> +	n = rte_ring_hts_enqueue_burst_start(r, obj_table, n, free_space);
> +	if (n != 0)
> +		rte_ring_hts_enqueue_finish(r, n);
> +	return n;
> +}
> +
>  /**
>   * Enqueue several objects on a ring.
>   *
> @@ -1141,6 +1356,8 @@ rte_ring_enqueue_burst(struct rte_ring *r, void *
> const *obj_table,
>  		return rte_ring_sp_enqueue_burst(r, obj_table, n,
> free_space);
>  	case RTE_RING_SYNC_MT_RTS:
>  		return rte_ring_rts_enqueue_burst(r, obj_table, n,
> free_space);
> +	case RTE_RING_SYNC_MT_HTS:
> +		return rte_ring_hts_enqueue_burst(r, obj_table, n,
> free_space);
>  	}
> 
>  	/* valid ring should never reach this point */ @@ -1225,6 +1442,42
> @@ rte_ring_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
>  	return __rte_ring_do_rts_dequeue(r, obj_table, n,
>  			RTE_RING_QUEUE_VARIABLE, available);  }
> +
> +/**
> + * Start to dequeue several objects from an HTS ring (multi-consumers safe).
> + * Note that user has to call appropriate dequeue_finish()
> + * to complete given dequeue operation.
> + *
> + * @param r
> + *   A pointer to the ring structure.
> + * @param obj_table
> + *   A pointer to a table of void * pointers (objects) that will be filled.
> + * @param n
> + *   The number of objects to dequeue from the ring to the obj_table.
> + * @param available
> + *   If non-NULL, returns the number of remaining ring entries after the
> + *   dequeue has finished.
> + * @return
> + *   The number of objects dequeued, either 0 or n
> + */
> +static __rte_always_inline unsigned int
> +rte_ring_hts_dequeue_burst_start(struct rte_ring *r, void **obj_table,
> +		unsigned int n, unsigned int *available) {
> +	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
> +		RTE_RING_QUEUE_VARIABLE, available);
> +}
> +
> +static __rte_always_inline unsigned int
> +rte_ring_hts_dequeue_burst(struct rte_ring *r, void **obj_table,
> +		unsigned int n, unsigned int *available) {
> +	n = rte_ring_hts_dequeue_burst_start(r, obj_table, n, available);
> +	if (n != 0)
> +		rte_ring_hts_dequeue_finish(r, n);
> +	return n;
> +}
> +
>  /**
>   * Dequeue multiple objects from a ring up to a maximum number.
>   *
> @@ -1255,6 +1508,8 @@ rte_ring_dequeue_burst(struct rte_ring *r, void
> **obj_table,
>  		return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
>  	case RTE_RING_SYNC_MT_RTS:
>  		return rte_ring_rts_dequeue_burst(r, obj_table, n, available);
> +	case RTE_RING_SYNC_MT_HTS:
> +		return rte_ring_hts_dequeue_burst(r, obj_table, n, available);
>  	}
> 
>  	/* valid ring should never reach this point */ diff --git
> a/lib/librte_ring/rte_ring_hts_generic.h
> b/lib/librte_ring/rte_ring_hts_generic.h
> new file mode 100644
> index 000000000..7e447e30b
> --- /dev/null
> +++ b/lib/librte_ring/rte_ring_hts_generic.h
> @@ -0,0 +1,228 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * Copyright (c) 2010-2020 Intel Corporation
> + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> + * All rights reserved.
> + * Derived from FreeBSD's bufring.h
> + * Used as BSD-3 Licensed with permission from Kip Macy.
> + */
> +
> +#ifndef _RTE_RING_HTS_GENERIC_H_
> +#define _RTE_RING_HTS_GENERIC_H_
> +
> +/**
> + * @file rte_ring_hts_generic.h
> + * It is not recommended to include this file directly,
> + * include <rte_ring.h> instead.
> + * Contains internal helper functions for head/tail sync (HTS) ring mode.
> + * In that mode enqueue/dequeue operation is fully serialized:
> + * only one thread at a time is allowed to perform given op.
> + * This is achieved by thread is allowed to proceed with changing
> +head.value
> + * only when head.value == tail.value.
> + * Both head and tail values are updated atomically (as one 64-bit value).
> + * As another enhancement that provides ability to split
> +enqueue/dequeue
> + * operation into two phases:
> + * - enqueue/dequeue start
> + * - enqueue/dequeue finish
> + * That allows user to inspect objects in the ring without removing
> + * them from it (aka MT safe peek).
> + * As an example:
> + * // read 1 elem from the ring:
> + * n = rte_ring_hts_dequeue_bulk_start(ring, &obj, 1, NULL);
> + * if (n != 0) {
> + *    //examined object
> + *    if (object_examine(obj) == KEEP)
> + *       //decided to keep it in the ring.
> + *       rte_ring_hts_dequeue_finish(ring, 0);
> + *    else
> + *       //decided to remove it in the ring.
> + *       rte_ring_hts_dequeue_finish(ring, n);
> + * }
> + * Note that between _start_ and _finish_ the ring is sort of locked -
> + * none other thread can proceed with enqueue(/dequeue) operation till
> + * _finish_ will complete.
This means it does not solve the problem for over committed systems. Do you agree?

> + */
> +
> +static __rte_always_inline void
> +__rte_ring_hts_update_tail(struct rte_ring_hts_headtail *ht, uint32_t num,
> +	uint32_t enqueue)
> +{
> +	uint32_t n;
> +	union rte_ring_ht_pos p;
> +
> +	if (enqueue)
> +		rte_smp_wmb();
> +	else
> +		rte_smp_rmb();
> +
> +	p.raw = rte_atomic64_read((rte_atomic64_t *)(uintptr_t)&ht->ht.raw);
> +
> +	n = p.pos.head - p.pos.tail;
> +	RTE_ASSERT(n >= num);
> +	RTE_SET_USED(n);
> +
> +	p.pos.head = p.pos.tail + num;
> +	p.pos.tail = p.pos.head;
> +
> +	rte_atomic64_set((rte_atomic64_t *)(uintptr_t)&ht->ht.raw, p.raw); }
> +
> +/**
> + * @internal waits till tail will become equal to head.
> + * Means no writer/reader is active for that ring.
> + * Suppose to work as serialization point.
> + */
> +static __rte_always_inline void
> +__rte_ring_hts_head_wait(const struct rte_ring_hts_headtail *ht,
> +		union rte_ring_ht_pos *p)
> +{
> +	p->raw = rte_atomic64_read((rte_atomic64_t *)
> +			(uintptr_t)&ht->ht.raw);
> +
> +	while (p->pos.head != p->pos.tail) {
> +		rte_pause();
> +		p->raw = rte_atomic64_read((rte_atomic64_t *)
> +				(uintptr_t)&ht->ht.raw);
> +	}
> +}
> +
> +/**
> + * @internal This function updates the producer head for enqueue
> + *
> + * @param r
> + *   A pointer to the ring structure
> + * @param is_sp
> + *   Indicates whether multi-producer path is needed or not
> + * @param n
> + *   The number of elements we will want to enqueue, i.e. how far should the
> + *   head be moved
> + * @param behavior
> + *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
> + *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> ring
> + * @param old_head
> + *   Returns head value as it was before the move, i.e. where enqueue starts
> + * @param new_head
> + *   Returns the current/new head value i.e. where enqueue finishes
> + * @param free_entries
> + *   Returns the amount of free space in the ring BEFORE head was moved
> + * @return
> + *   Actual number of objects enqueued.
> + *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num,
> +	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> +	uint32_t *free_entries)
> +{
> +	uint32_t n;
> +	union rte_ring_ht_pos np, op;
> +
> +	const uint32_t capacity = r->capacity;
> +
> +	do {
> +		/* Reset n to the initial burst count */
> +		n = num;
> +
> +		/* wait for tail to be equal to head */
> +		__rte_ring_hts_head_wait(&r->hts_prod, &op);
> +
> +		/* add rmb barrier to avoid load/load reorder in weak
> +		 * memory model. It is noop on x86
> +		 */
> +		rte_smp_rmb();
> +
> +		/*
> +		 *  The subtraction is done between two unsigned 32bits value
> +		 * (the result is always modulo 32 bits even if we have
> +		 * *old_head > cons_tail). So 'free_entries' is always between
> 0
> +		 * and capacity (which is < size).
> +		 */
> +		*free_entries = capacity + r->cons.tail - op.pos.head;
> +
> +		/* check that we have enough room in ring */
> +		if (unlikely(n > *free_entries))
> +			n = (behavior == RTE_RING_QUEUE_FIXED) ?
> +					0 : *free_entries;
> +
> +		if (n == 0)
> +			return 0;
> +
> +		np.pos.tail = op.pos.tail;
> +		np.pos.head = op.pos.head + n;
> +
> +	} while (rte_atomic64_cmpset(&r->hts_prod.ht.raw,
> +			op.raw, np.raw) == 0);
> +
> +	*old_head = op.pos.head;
> +	return n;
> +}
> +
> +/**
> + * @internal This function updates the consumer head for dequeue
> + *
> + * @param r
> + *   A pointer to the ring structure
> + * @param is_sc
> + *   Indicates whether multi-consumer path is needed or not
> + * @param n
> + *   The number of elements we will want to enqueue, i.e. how far should the
> + *   head be moved
> + * @param behavior
> + *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
> + *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> ring
> + * @param old_head
> + *   Returns head value as it was before the move, i.e. where dequeue starts
> + * @param new_head
> + *   Returns the current/new head value i.e. where dequeue finishes
> + * @param entries
> + *   Returns the number of entries in the ring BEFORE head was moved
> + * @return
> + *   - Actual number of objects dequeued.
> + *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,
> +	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> +	uint32_t *entries)
> +{
> +	uint32_t n;
> +	union rte_ring_ht_pos np, op;
> +
> +	/* move cons.head atomically */
> +	do {
> +		/* Restore n as it may change every loop */
> +		n = num;
> +
> +		/* wait for tail to be equal to head */
> +		__rte_ring_hts_head_wait(&r->hts_cons, &op);
> +
> +		/* add rmb barrier to avoid load/load reorder in weak
> +		 * memory model. It is noop on x86
> +		 */
> +		rte_smp_rmb();
> +
> +		/* The subtraction is done between two unsigned 32bits value
> +		 * (the result is always modulo 32 bits even if we have
> +		 * cons_head > prod_tail). So 'entries' is always between 0
> +		 * and size(ring)-1.
> +		 */
> +		*entries = r->prod.tail - op.pos.head;
> +
> +		/* Set the actual entries for dequeue */
> +		if (n > *entries)
> +			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
> *entries;
> +
> +		if (unlikely(n == 0))
> +			return 0;
> +
> +		np.pos.tail = op.pos.tail;
> +		np.pos.head = op.pos.head + n;
> +
> +	} while (rte_atomic64_cmpset(&r->hts_cons.ht.raw,
> +			op.raw, np.raw) == 0);
> +
> +	*old_head = op.pos.head;
> +	return n;
> +}
> +
> +#endif /* _RTE_RING_HTS_GENERIC_H_ */
> --
> 2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 0/6] New sync modes for ring
  2020-03-25 20:43 ` Honnappa Nagarahalli
@ 2020-03-26  1:50   ` Ananyev, Konstantin
  0 siblings, 0 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-03-26  1:50 UTC (permalink / raw)
  To: Honnappa Nagarahalli, dev; +Cc: olivier.matz, nd, nd

> 
> <snip>
> 
> > Subject: [dpdk-dev] [RFC 0/6] New sync modes for ring
> >
> > Upfront note - that RFC is not a complete patch.
> > It introduces an ABI breakage, plus it doesn't update ring_elem code properly,
> As per the current rules, these changes (in the current form) will be accepted only for 20.11 release. How do we address this for immediate
> requirements like RCU defer APIs?

I think I found a way to introduce these new modes without API/ABI breakage.
Working on v1 right now. Plan to submit it by end of that week/start of next one.

> I suggest that we move forward with my RFC (taking into consideration your feedback) to make progress on RCU APIs.
> 
> > etc.
> > I plan to deal with all these things in later versions.
> > Right now I seek an initial feedback about proposed ideas.
> > Would also ask people to repeat performance tests (see below) on their
> > platforms to confirm the impact.
> >
> > More and more customers use(/try to use) DPDK based apps within
> > overcommitted systems (multiple acttive threads over same pysical cores):
> > VM, container deployments, etc.
> > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > LHP is quite a common problem for spin-based sync primitives (spin-locks, etc.)
> > on overcommitted systems.
> > The situation gets much worse when some sort of fair-locking technique is
> > used (ticket-lock, etc.).
> > As now not only lock-owner but also lock-waiters scheduling order matters a
> > lot.
> > This is a well-known problem for kernel within VMs:
> > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > The problem with rte_ring is that while head accusion is sort of un-fair locking,
> > waiting on tail is very similar to ticket lock schema - tail has to be updated in
> > particular order.
> > That makes current rte_ring implementation to perform really pure on some
> > overcommited scenarios.
> > While it is probably not possible to completely resolve this problem in
> > userspace only (without some kernel communication/intervention), removing
> > fairness in tail update can mitigate it significantly.
> > So this RFC proposes two new optional ring synchronization modes:
> > 1) Head/Tail Sync (HTS) mode
> > In that mode enqueue/dequeue operation is fully serialized:
> >     only one thread at a time is allowed to perform given op.
> >     As another enhancement provide ability to split enqueue/dequeue
> >     operation into two phases:
> >       - enqueue/dequeue start
> >       - enqueue/dequeue finish
> >     That allows user to inspect objects in the ring without removing
> >     them from it (aka MT safe peek).
> IMO, this will not address the problem described above.

It does, please see the results produced by ring_stress_*autotest below.
Let say for test-case: 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' it shows:
avg number of cycles per object for enqueue /dequeue:
MP/MC: 280314.32
HTS:         294.72
RTS:         318.79

Customer who tried it reported similar level of improvement.
Actually if you have time - would be very interesting to see what numbers will be on ARM boxes.
To reproduce, just:
$cat ring_tests_u4
ring_stress_autotest
ring_stress_hts_autotest
ring_stress_rts_autotest

/app/test/dpdk-test --lcores='6,(10-13)@7,(20-23)@8'  -n 4 < ring_tests_u4 2>&1 | tee res1

Then look at the ' AGGREGATE' stats.
Right now it is a bit too verbose, so probably the easiest thing to extract same numbers quickly:
grep 'cycles/obj'  res1 | grep 'cycles/obj' | cat -n | awk '{if ($(1)%9==0) print $(NF);}'
280314.32
1057833.55
294.72
480.10
318.79
461.52

First 2 numbers will be for MP/MC, next 2 for HTS, last 2 for RTS.

> For ex: when a producer updates the head and gets scheduled out, other producers
> have to spin.

Sure, as I wrote in original cover letter:
" While it is probably not possible to completely resolve this problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly."
Results from the ring_stress_*_autotest confirm that.

> The problem is probably worse as with non-HTS case moving of the head and copying of the ring elements can happen in
> parallel between the producers (similarly for consumers).

Yes as we serialize the ring, we remove possibility of simultaneous copy.
That's why for 'normal' cases (one thread per core) original MP/MC is usually faster.
Though on overcommitted scenarios current MP/MC performance degrades dramatically.
The main problem with current MP/MC implementation is in that tail update
have to be done in strict order (sort of fair locking scheme).
Which means that we have much-much worse LHP manifestation,
then when we use unfair schemes.
With serialized ring (HTS) we remove that ordering completely
(same idea as switch from fair to unfair locking for PV spin-locks).  

> IMO, HTS should not be a configurable flag. 

Why?

> In RCU requirement, a MP enqueue and HTS dequeue are required.

This is supported, user can specify different modes for consumer and producer:
(0 | RING_F_MC_HTS_DEQ).
Then it is up to the user either to call generic rte_ring_enqueue/rte_ring_dequeue,
or specify mode manually by function name:
rte_ring_mp_enqueue_bulk/ rte_ring_hts_dequeue_bulk.

> 
> > 2) Relaxed Tail Sync (RTS)
> > The main difference from original MP/MC algorithm is that tail value is
> > increased not by every thread that finished enqueue/dequeue, but only by the
> > last one.
> > That allows threads to avoid spinning on ring tail value, leaving actual tail value
> > change to the last thread in the update queue.
> This can be a configurable flag on the ring.
> I am not sure how this solves the problem you have stated above completely. Updating the count from all intermediate threads is still
> required to update the value of the head. But yes, it reduces the severity of the problem by not enforcing the order in which the tail is
> updated.

As I said above, main source of slowdown here -
that we have to  update tail in particular order.
So the main objective (same as for HTS) is to remove
that ordering.  

> I also think it introduces the problem on the other side of the ring because the tail is not updated soon enough (the other side has to wait
> longer for the elements to become available).

Yes, producer/consumer starvation.
That's why we need max allowed Head-Tail-Distance (htd_max) -
to limit how far head can go away from tail.

> It also introduces another configuration parameter (HTD_MAX_DEF) which they have to deal
> with.

If user doesn't provide any value, it will be set by default to ring.capacity / 8.
From my measurements works quite well.
Though there possibility for the user to set another value, if needed.

> Users have to still implement the current hypervisor related solutions.

Didn't get what you trying to say with that phrase.

> IMO, we should run the benchmark for this on an over committed setup to understand the benefits.

That's why I created ring_stress_*autotest test-cases and collected numbers provided below.
I suppose they clearly show the problem on overcommitted scenarios,
and how RTS/HTS improve that situation. 
Would appreciate if you repeat these tests on your machines. 

> 
> >
> > Test results on IA (see below) show significant improvements for average
> > enqueue/dequeue op times on overcommitted systems.
> > For 'classic' DPDK deployments (one thread per core) original MP/MC
> > algorithm still shows best numbers, though for 64-bit target RTS numbers are
> > not that far away.
> > Numbers were produced by ring_stress_*autotest (first patch in these series).
> >
> > X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > DEQ+ENQ average cycles/obj
> >
> >                                                 MP/MC      HTS     RTS
> > 1thread@1core(--lcores=6-7)                     8.00       8.15    8.99
> > 2thread@2core(--lcores=6-8)                     19.14      19.61   20.35
> > 4thread@4core(--lcores=6-10)                    29.43      29.79   31.82
> > 8thread@8core(--lcores=6-14)                    110.59     192.81  119.50
> > 16thread@16core(--lcores=6-22)                  461.03     813.12  495.59
> > 32thread/@32core(--lcores='6-22,55-70')         982.90     1972.38 1160.51
> >
> > 2thread@1core(--lcores='6,(10-11)@7'            20140.50   23.58   25.14
> > 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8'  153680.60  76.88   80.05
> > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  280314.32  294.72  318.79
> > 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59  1144.02
> > 1175.14 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80
> > 4627.48 4892.68
> >
> > 8thread@2core(--lcores='6,(10-17)@(7,8))'       321085.98  298.59  307.47
> > 16thread@4core(--lcores='6,(20-35)@(7-10))'     1900705.61 575.35  678.29
> > 32thread@4core(--lcores='6,(20-51)@(7-10))'     5510445.85 2164.36 2714.12
> >
> > i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> > DEQ+ENQ average cycles/obj
> >
> >                                                 MP/MC      HTS     RTS
> > 1thread@1core(--lcores=6-7)                     7.85       12.13   11.31
> > 2thread@2core(--lcores=6-8)                     17.89      24.52   21.86
> > 8thread@8core(--lcores=6-14)                    32.58      354.20  54.58
> > 32thread/@32core(--lcores='6-22,55-70')         813.77     6072.41 2169.91
> >
> > 2thread@1core(--lcores='6,(10-11)@7'            16095.00   36.06   34.74
> > 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  1140354.54 346.61  361.57
> > 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90
> > 1416.65
> >
> > 8thread@2core(--lcores='6,(10-17)@(7,8))'       594358.61  332.70  357.74
> > 32thread@4core(--lcores='6,(20-51)@(7-10))'     5319896.86 2836.44 3028.87
> >
> > Konstantin Ananyev (6):
> >   test/ring: add contention stress test
> >   ring: rework ring layout to allow new sync schemes
> >   ring: introduce RTS ring mode
> >   test/ring: add contention stress test for RTS ring
> >   ring: introduce HTS ring mode
> >   test/ring: add contention stress test for HTS ring
> >
> >  app/test/Makefile                      |   3 +
> >  app/test/meson.build                   |   3 +
> >  app/test/test_pdump.c                  |   6 +-
> >  app/test/test_ring_hts_stress.c        |  28 ++
> >  app/test/test_ring_rts_stress.c        |  28 ++
> >  app/test/test_ring_stress.c            |  27 ++
> >  app/test/test_ring_stress.h            | 477 +++++++++++++++++++
> >  lib/librte_pdump/rte_pdump.c           |   2 +-
> >  lib/librte_port/rte_port_ring.c        |  12 +-
> >  lib/librte_ring/Makefile               |   4 +-
> >  lib/librte_ring/meson.build            |   4 +-
> >  lib/librte_ring/rte_ring.c             |  84 +++-
> >  lib/librte_ring/rte_ring.h             | 619 +++++++++++++++++++++++--
> >  lib/librte_ring/rte_ring_elem.h        |   8 +-
> >  lib/librte_ring/rte_ring_hts_generic.h | 228 +++++++++
> > lib/librte_ring/rte_ring_rts_generic.h | 240 ++++++++++
> >  16 files changed, 1721 insertions(+), 52 deletions(-)  create mode 100644
> > app/test/test_ring_hts_stress.c  create mode 100644
> > app/test/test_ring_rts_stress.c  create mode 100644
> > app/test/test_ring_stress.c  create mode 100644 app/test/test_ring_stress.h
> > create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
> >  create mode 100644 lib/librte_ring/rte_ring_rts_generic.h
> >
> > --
> > 2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode
  2020-03-25 20:44   ` Honnappa Nagarahalli
@ 2020-03-26 12:26     ` Ananyev, Konstantin
  0 siblings, 0 replies; 23+ messages in thread
From: Ananyev, Konstantin @ 2020-03-26 12:26 UTC (permalink / raw)
  To: Honnappa Nagarahalli, dev; +Cc: olivier.matz, nd, nd

> > Introduce head/tail sync mode for MT ring synchronization.
> > In that mode enqueue/dequeue operation is fully serialized:
> > only one thread at a time is allowed to perform given op.
> > Suppose to reduce stall times in case when ring is used on overcommitted
> > cpus (multiple active threads on the same cpu).
> > As another enhancement provide ability to split enqueue/dequeue operation
> > into two phases:
> >   - enqueue/dequeue start
> >   - enqueue/dequeue finish
> > That allows user to inspect objects in the ring without removing them from it
> > (aka MT safe peek).
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > ---
> >  lib/librte_ring/Makefile               |   1 +
> >  lib/librte_ring/meson.build            |   1 +
> >  lib/librte_ring/rte_ring.c             |  15 +-
> >  lib/librte_ring/rte_ring.h             | 259 ++++++++++++++++++++++++-
> >  lib/librte_ring/rte_ring_hts_generic.h | 228 ++++++++++++++++++++++
> >  5 files changed, 500 insertions(+), 4 deletions(-)  create mode 100644
> > lib/librte_ring/rte_ring_hts_generic.h
> >
> > diff --git a/lib/librte_ring/Makefile b/lib/librte_ring/Makefile index
> > 4f90344f4..0c7f8f918 100644
> > --- a/lib/librte_ring/Makefile
> > +++ b/lib/librte_ring/Makefile
> > @@ -19,6 +19,7 @@ SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include :=
> > rte_ring.h \
> >  					rte_ring_elem.h \
> >  					rte_ring_generic.h \
> >  					rte_ring_c11_mem.h \
> > +					rte_ring_hts_generic.h \
> >  					rte_ring_rts_generic.h
> >
> >  include $(RTE_SDK)/mk/rte.lib.mk
> > diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build index
> > dc8d7dbea..5aa673199 100644
> > --- a/lib/librte_ring/meson.build
> > +++ b/lib/librte_ring/meson.build
> > @@ -6,6 +6,7 @@ headers = files('rte_ring.h',
> >  		'rte_ring_elem.h',
> >  		'rte_ring_c11_mem.h',
> >  		'rte_ring_generic.h',
> > +		'rte_ring_hts_generic.h',
> >  		'rte_ring_rts_generic.h')
> >
> >  # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
> > diff --git a/lib/librte_ring/rte_ring.c b/lib/librte_ring/rte_ring.c index
> > 1ce0af3e5..d3b948667 100644
> > --- a/lib/librte_ring/rte_ring.c
> > +++ b/lib/librte_ring/rte_ring.c
> > @@ -102,9 +102,9 @@ static int
> >  get_sync_type(uint32_t flags, uint32_t *prod_st, uint32_t *cons_st)  {
> >  	static const uint32_t prod_st_flags =
> > -		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ);
> > +		(RING_F_SP_ENQ | RING_F_MP_RTS_ENQ |
> > RING_F_MP_HTS_ENQ);
> >  	static const uint32_t cons_st_flags =
> > -		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ);
> > +		(RING_F_SC_DEQ | RING_F_MC_RTS_DEQ |
> > RING_F_MC_HTS_DEQ);
> >
> >  	switch (flags & prod_st_flags) {
> >  	case 0:
> > @@ -116,6 +116,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st,
> > uint32_t *cons_st)
> >  	case RING_F_MP_RTS_ENQ:
> >  		*prod_st = RTE_RING_SYNC_MT_RTS;
> >  		break;
> > +	case RING_F_MP_HTS_ENQ:
> > +		*prod_st = RTE_RING_SYNC_MT_HTS;
> > +		break;
> >  	default:
> >  		return -EINVAL;
> >  	}
> > @@ -130,6 +133,9 @@ get_sync_type(uint32_t flags, uint32_t *prod_st,
> > uint32_t *cons_st)
> >  	case RING_F_MC_RTS_DEQ:
> >  		*cons_st = RTE_RING_SYNC_MT_RTS;
> >  		break;
> > +	case RING_F_MC_HTS_DEQ:
> > +		*cons_st = RTE_RING_SYNC_MT_HTS;
> > +		break;
> >  	default:
> >  		return -EINVAL;
> >  	}
> > @@ -151,6 +157,11 @@ rte_ring_init(struct rte_ring *r, const char *name,
> > unsigned count,
> >  	RTE_BUILD_BUG_ON((offsetof(struct rte_ring, prod) &
> >  			  RTE_CACHE_LINE_MASK) != 0);
> >
> > +	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> > +		offsetof(struct rte_ring_hts_headtail, sync_type));
> > +	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) !=
> > +		offsetof(struct rte_ring_hts_headtail, ht.pos.tail));
> > +
> >  	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, sync_type) !=
> >  		offsetof(struct rte_ring_rts_headtail, sync_type));
> >  	RTE_BUILD_BUG_ON(offsetof(struct rte_ring_headtail, tail) != diff --git
> > a/lib/librte_ring/rte_ring.h b/lib/librte_ring/rte_ring.h index
> > a130aeb9d..52edcea11 100644
> > --- a/lib/librte_ring/rte_ring.h
> > +++ b/lib/librte_ring/rte_ring.h
> > @@ -66,11 +66,11 @@ enum {
> >  	RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
> >  	RTE_RING_SYNC_ST,     /**< single thread only */
> >  	RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
> > +	RTE_RING_SYNC_MT_HTS, /**< multi-thread head/tail sync */
> >  };
> >
> >  /**
> > - * structure to hold a pair of head/tail values and other metadata.
> > - * used by RTE_RING_SYNC_MT, RTE_RING_SYNC_ST sync types.
> > + * Structure to hold a pair of head/tail values and other metadata.
> >   * Depending on sync_type format of that structure might differ
> >   * depending on the sync mechanism selelcted, but offsets for
> >   * *sync_type* and *tail* values should always remain the same.
> > @@ -96,6 +96,19 @@ struct rte_ring_rts_headtail {
> >  	volatile union rte_ring_ht_poscnt head;  };
> >
> > +union rte_ring_ht_pos {
> > +	uint64_t raw;
> > +	struct {
> > +		uint32_t tail; /**< tail position */
> > +		uint32_t head; /**< head position */
> > +	} pos;
> > +};
> > +
> > +struct rte_ring_hts_headtail {
> > +	uint32_t sync_type; /**< sync type of prod/cons */
> > +	volatile union rte_ring_ht_pos ht __rte_aligned(8); };
> > +
> >  /**
> >   * An RTE ring structure.
> >   *
> > @@ -126,6 +139,7 @@ struct rte_ring {
> >  	RTE_STD_C11
> >  	union {
> >  		struct rte_ring_headtail prod;
> > +		struct rte_ring_hts_headtail hts_prod;
> >  		struct rte_ring_rts_headtail rts_prod;
> >  	}  __rte_cache_aligned;
> >
> > @@ -135,6 +149,7 @@ struct rte_ring {
> >  	RTE_STD_C11
> >  	union {
> >  		struct rte_ring_headtail cons;
> > +		struct rte_ring_hts_headtail hts_cons;
> >  		struct rte_ring_rts_headtail rts_cons;
> >  	}  __rte_cache_aligned;
> >
> > @@ -157,6 +172,9 @@ struct rte_ring {
> >  #define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS".
> > */  #define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC
> > RTS". */
> >
> > +#define RING_F_MP_HTS_ENQ 0x0020 /**< The default enqueue is "MP
> > HTS".
> > +*/ #define RING_F_MC_HTS_DEQ 0x0040 /**< The default dequeue is "MC
> > +HTS". */
> > +
> >  #define __IS_SP RTE_RING_SYNC_ST
> >  #define __IS_MP RTE_RING_SYNC_MT
> >  #define __IS_SC RTE_RING_SYNC_ST
> > @@ -513,6 +531,82 @@ __rte_ring_do_rts_dequeue(struct rte_ring *r, void
> > **obj_table,
> >  	return n;
> >  }
> >
> > +#include <rte_ring_hts_generic.h>
> > +
> > +/**
> > + * @internal Start to enqueue several objects on the HTS ring.
> > + * Note that user has to call appropriate enqueue_finish()
> > + * to complete given enqueue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects).
> > + * @param n
> > + *   The number of objects to add in the ring from the obj_table.
> > + * @param behavior
> > + *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
> > + *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> > ring
> > + * @param free_space
> > + *   returns the amount of space after the enqueue operation has finished
> > + * @return
> > + *   Actual number of objects enqueued.
> > + *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_hts_enqueue_start(struct rte_ring *r, void * const *obj_table,
> > +		uint32_t n, enum rte_ring_queue_behavior behavior,
> > +		uint32_t *free_space)
> > +{
> > +	uint32_t free, head;
> > +
> > +	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
> > +
> > +	if (n != 0)
> > +		ENQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> > +
> > +	if (free_space != NULL)
> > +		*free_space = free - n;
> > +	return n;
> > +}
> rte_ring.h is becoming too big. May be we should move these functions to another HTS specific file. But leave the top level API in rte_ring.h.
> Similarly for RTS.

Good point, will try in v1.

> 
> > +
> > +/**
> > + * @internal Start to dequeue several objects from the HTS ring.
> > + * Note that user has to call appropriate dequeue_finish()
> > + * to complete given dequeue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects).
> > + * @param n
> > + *   The number of objects to pull from the ring.
> > + * @param behavior
> > + *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
> > + *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> > ring
> > + * @param available
> > + *   returns the number of remaining ring entries after the dequeue has
> > finished
> > + * @return
> > + *   - Actual number of objects dequeued.
> > + *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_do_hts_dequeue_start(struct rte_ring *r, void **obj_table,
> > +		unsigned int n, enum rte_ring_queue_behavior behavior,
> > +		unsigned int *available)
> > +{
> > +	uint32_t entries, head;
> > +
> > +	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries);
> > +
> > +	if (n != 0)
> > +		DEQUEUE_PTRS(r, &r[1], head, obj_table, n, void *);
> > +
> > +	if (available != NULL)
> > +		*available = entries - n;
> > +	return n;
> > +}
> > +
> >  /**
> >   * Enqueue several objects on the ring (multi-producers safe).
> >   *
> > @@ -585,6 +679,47 @@ rte_ring_rts_enqueue_bulk(struct rte_ring *r, void *
> > const *obj_table,
> >  			free_space);
> >  }
> >
> > +/**
> > + * Start to enqueue several objects on the HTS ring (multi-producers safe).
> > + * Note that user has to call appropriate dequeue_finish()
> > + * to complete given dequeue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects).
> > + * @param n
> > + *   The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + *   if non-NULL, returns the amount of space in the ring after the
> > + *   enqueue operation has finished.
> > + * @return
> > + *   The number of objects enqueued, either 0 or n
> > + */
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_enqueue_bulk_start(struct rte_ring *r, void * const *obj_table,
> > +			 unsigned int n, unsigned int *free_space) {
> > +	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
> > +		RTE_RING_QUEUE_FIXED, free_space);
> > +}
> I do not clearly understand the requirements on the enqueue_start and enqueue_finish in the form they are here.
> IMO, only requirement for these APIs is to provide the ability to avoid intermediate memcpys.

I think the main objective here is to provide 'MT safe peek' functionality.
The requirement is to split let say dequeue operation into two parts:
1. start - copy N elems into provided by user data buffer and guarantee that
    these elems will remain in the ring till finish().
2. finish - remove M(<=N) elems from the ring. 

For enqueue it a mirror:
1. start - reserve space for N elems in the ring.
2. finish - copy M (<=N) to the ring. 

> 
> > +
> > +static __rte_always_inline void
> > +rte_ring_hts_enqueue_finish(struct rte_ring *r, unsigned int n) {
> > +	__rte_ring_hts_update_tail(&r->hts_prod, n, 1); }
> > +
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_enqueue_bulk(struct rte_ring *r, void * const *obj_table,
> > +			 unsigned int n, unsigned int *free_space) {
> > +	n = rte_ring_hts_enqueue_bulk_start(r, obj_table, n, free_space);
> > +	if (n != 0)
> > +		rte_ring_hts_enqueue_finish(r, n);
> > +	return n;
> > +}
> > +
> >  /**
> >   * Enqueue several objects on a ring.
> >   *
> > @@ -615,6 +750,8 @@ rte_ring_enqueue_bulk(struct rte_ring *r, void * const
> > *obj_table,
> >  		return rte_ring_sp_enqueue_bulk(r, obj_table, n, free_space);
> >  	case RTE_RING_SYNC_MT_RTS:
> >  		return rte_ring_rts_enqueue_bulk(r, obj_table, n, free_space);
> > +	case RTE_RING_SYNC_MT_HTS:
> > +		return rte_ring_hts_enqueue_bulk(r, obj_table, n,
> > free_space);
> >  	}
> >
> >  	/* valid ring should never reach this point */ @@ -753,6 +890,47 @@
> > rte_ring_rts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> >  			available);
> >  }
> >
> > +/**
> > + * Start to dequeue several objects from an HTS ring (multi-consumers safe).
> > + * Note that user has to call appropriate dequeue_finish()
> > + * to complete given dequeue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + *   The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + *   If non-NULL, returns the number of remaining ring entries after the
> > + *   dequeue has finished.
> > + * @return
> > + *   The number of objects dequeued, either 0 or n
> > + */
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_dequeue_bulk_start(struct rte_ring *r, void **obj_table,
> > +		unsigned int n, unsigned int *available) {
> > +	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
> > +		RTE_RING_QUEUE_FIXED, available);
> > +}
> IMO, we should look to provide the ability to avoid intermediate copies when the data from the ring needs to be distributed to different
> locations.

As I said in other thread - I am not sure it would provide any gain in terms of performance.
Unless we have a case with bulk transfers and big size elems.
If you still strongly feel SG is needed here, I think it should be an add-on API not the main and only one.

> My proposal in its form is complicated. But, I am thinking that, if the return values are abstracted in a structure, it might look much simple.
> 
> > +
> > +static __rte_always_inline void
> > +rte_ring_hts_dequeue_finish(struct rte_ring *r, unsigned int n) {
> > +	__rte_ring_hts_update_tail(&r->hts_cons, n, 0); }
> > +
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_dequeue_bulk(struct rte_ring *r, void **obj_table,
> > +		unsigned int n, unsigned int *available) {
> > +	n = rte_ring_hts_dequeue_bulk_start(r, obj_table, n, available);
> > +	if (n != 0)
> > +		rte_ring_hts_dequeue_finish(r, n);
> > +	return n;
> > +}
> > +
> >  /**
> >   * Dequeue several objects from a ring.
> >   *
> > @@ -783,6 +961,8 @@ rte_ring_dequeue_bulk(struct rte_ring *r, void
> > **obj_table, unsigned int n,
> >  		return rte_ring_sc_dequeue_bulk(r, obj_table, n, available);
> >  	case RTE_RING_SYNC_MT_RTS:
> >  		return rte_ring_rts_dequeue_bulk(r, obj_table, n, available);
> > +	case RTE_RING_SYNC_MT_HTS:
> > +		return rte_ring_hts_dequeue_bulk(r, obj_table, n, available);
> >  	}
> >
> >  	/* valid ring should never reach this point */ @@ -1111,6 +1291,41
> > @@ rte_ring_rts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> >  			RTE_RING_QUEUE_VARIABLE, free_space);  }
> >
> > +/**
> > + * Start to enqueue several objects on the HTS ring (multi-producers safe).
> > + * Note that user has to call appropriate dequeue_finish()
> > + * to complete given dequeue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects).
> > + * @param n
> > + *   The number of objects to add in the ring from the obj_table.
> > + * @param free_space
> > + *   if non-NULL, returns the amount of space in the ring after the
> > + *   enqueue operation has finished.
> > + * @return
> > + *   The number of objects enqueued, either 0 or n
> > + */
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_enqueue_burst_start(struct rte_ring *r, void * const *obj_table,
> > +			 unsigned int n, unsigned int *free_space) {
> > +	return __rte_ring_do_hts_enqueue_start(r, obj_table, n,
> > +		RTE_RING_QUEUE_VARIABLE, free_space); }
> > +
> rte_ring_hts_enqueue_burst_finish is not implemented.

No need to, finish() is identical for both _bulk and _burst.
That's why we have 2 starts:
rte_ring_hts_enqueue_bulk_start()
rte_ring_hts_enqueue_burst_start()
and one finish:
rte_ring_hts_enqueue_finish().

Same story for dequeue.

> It requires the 'n' returned from ' rte_ring_hts_enqueue_burst_start' to be passed.

Yes, it requires some m <= n to be passed.
That's the whole point of peek - we want to be able to inspect N elems
possibly without retrieving them from the ring.
I.E.: inspect N, retrieve M <=N.
 

> We can't completely avoid passing correct information between xxx_start and xxx_finish APIs.

Yes we can't.
But in that model we can check that provided by finish() value is valid,
plus we don't provide user direct access to the contents of the ring,
and don't require him to specify head/tail values directly. 

 
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_enqueue_burst(struct rte_ring *r, void * const *obj_table,
> > +			 unsigned int n, unsigned int *free_space) {
> > +	n = rte_ring_hts_enqueue_burst_start(r, obj_table, n, free_space);
> > +	if (n != 0)
> > +		rte_ring_hts_enqueue_finish(r, n);
> > +	return n;
> > +}
> > +
> >  /**
> >   * Enqueue several objects on a ring.
> >   *
> > @@ -1141,6 +1356,8 @@ rte_ring_enqueue_burst(struct rte_ring *r, void *
> > const *obj_table,
> >  		return rte_ring_sp_enqueue_burst(r, obj_table, n,
> > free_space);
> >  	case RTE_RING_SYNC_MT_RTS:
> >  		return rte_ring_rts_enqueue_burst(r, obj_table, n,
> > free_space);
> > +	case RTE_RING_SYNC_MT_HTS:
> > +		return rte_ring_hts_enqueue_burst(r, obj_table, n,
> > free_space);
> >  	}
> >
> >  	/* valid ring should never reach this point */ @@ -1225,6 +1442,42
> > @@ rte_ring_rts_dequeue_burst(struct rte_ring *r, void **obj_table,
> >  	return __rte_ring_do_rts_dequeue(r, obj_table, n,
> >  			RTE_RING_QUEUE_VARIABLE, available);  }
> > +
> > +/**
> > + * Start to dequeue several objects from an HTS ring (multi-consumers safe).
> > + * Note that user has to call appropriate dequeue_finish()
> > + * to complete given dequeue operation.
> > + *
> > + * @param r
> > + *   A pointer to the ring structure.
> > + * @param obj_table
> > + *   A pointer to a table of void * pointers (objects) that will be filled.
> > + * @param n
> > + *   The number of objects to dequeue from the ring to the obj_table.
> > + * @param available
> > + *   If non-NULL, returns the number of remaining ring entries after the
> > + *   dequeue has finished.
> > + * @return
> > + *   The number of objects dequeued, either 0 or n
> > + */
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_dequeue_burst_start(struct rte_ring *r, void **obj_table,
> > +		unsigned int n, unsigned int *available) {
> > +	return __rte_ring_do_hts_dequeue_start(r, obj_table, n,
> > +		RTE_RING_QUEUE_VARIABLE, available);
> > +}
> > +
> > +static __rte_always_inline unsigned int
> > +rte_ring_hts_dequeue_burst(struct rte_ring *r, void **obj_table,
> > +		unsigned int n, unsigned int *available) {
> > +	n = rte_ring_hts_dequeue_burst_start(r, obj_table, n, available);
> > +	if (n != 0)
> > +		rte_ring_hts_dequeue_finish(r, n);
> > +	return n;
> > +}
> > +
> >  /**
> >   * Dequeue multiple objects from a ring up to a maximum number.
> >   *
> > @@ -1255,6 +1508,8 @@ rte_ring_dequeue_burst(struct rte_ring *r, void
> > **obj_table,
> >  		return rte_ring_sc_dequeue_burst(r, obj_table, n, available);
> >  	case RTE_RING_SYNC_MT_RTS:
> >  		return rte_ring_rts_dequeue_burst(r, obj_table, n, available);
> > +	case RTE_RING_SYNC_MT_HTS:
> > +		return rte_ring_hts_dequeue_burst(r, obj_table, n, available);
> >  	}
> >
> >  	/* valid ring should never reach this point */ diff --git
> > a/lib/librte_ring/rte_ring_hts_generic.h
> > b/lib/librte_ring/rte_ring_hts_generic.h
> > new file mode 100644
> > index 000000000..7e447e30b
> > --- /dev/null
> > +++ b/lib/librte_ring/rte_ring_hts_generic.h
> > @@ -0,0 +1,228 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + *
> > + * Copyright (c) 2010-2020 Intel Corporation
> > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > + * All rights reserved.
> > + * Derived from FreeBSD's bufring.h
> > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > + */
> > +
> > +#ifndef _RTE_RING_HTS_GENERIC_H_
> > +#define _RTE_RING_HTS_GENERIC_H_
> > +
> > +/**
> > + * @file rte_ring_hts_generic.h
> > + * It is not recommended to include this file directly,
> > + * include <rte_ring.h> instead.
> > + * Contains internal helper functions for head/tail sync (HTS) ring mode.
> > + * In that mode enqueue/dequeue operation is fully serialized:
> > + * only one thread at a time is allowed to perform given op.
> > + * This is achieved by thread is allowed to proceed with changing
> > +head.value
> > + * only when head.value == tail.value.
> > + * Both head and tail values are updated atomically (as one 64-bit value).
> > + * As another enhancement that provides ability to split
> > +enqueue/dequeue
> > + * operation into two phases:
> > + * - enqueue/dequeue start
> > + * - enqueue/dequeue finish
> > + * That allows user to inspect objects in the ring without removing
> > + * them from it (aka MT safe peek).
> > + * As an example:
> > + * // read 1 elem from the ring:
> > + * n = rte_ring_hts_dequeue_bulk_start(ring, &obj, 1, NULL);
> > + * if (n != 0) {
> > + *    //examined object
> > + *    if (object_examine(obj) == KEEP)
> > + *       //decided to keep it in the ring.
> > + *       rte_ring_hts_dequeue_finish(ring, 0);
> > + *    else
> > + *       //decided to remove it in the ring.
> > + *       rte_ring_hts_dequeue_finish(ring, n);
> > + * }
> > + * Note that between _start_ and _finish_ the ring is sort of locked -
> > + * none other thread can proceed with enqueue(/dequeue) operation till
> > + * _finish_ will complete.
> This means it does not solve the problem for over committed systems. Do you agree?

I never stated that serialized ring fixes the problem completely.
I said that current approach mitigates is quite well.
And yes, I still believe that statement is correct.
See other thread for more detailed discussion.

> 
> > + */
> > +
> > +static __rte_always_inline void
> > +__rte_ring_hts_update_tail(struct rte_ring_hts_headtail *ht, uint32_t num,
> > +	uint32_t enqueue)
> > +{
> > +	uint32_t n;
> > +	union rte_ring_ht_pos p;
> > +
> > +	if (enqueue)
> > +		rte_smp_wmb();
> > +	else
> > +		rte_smp_rmb();
> > +
> > +	p.raw = rte_atomic64_read((rte_atomic64_t *)(uintptr_t)&ht->ht.raw);
> > +
> > +	n = p.pos.head - p.pos.tail;
> > +	RTE_ASSERT(n >= num);
> > +	RTE_SET_USED(n);
> > +
> > +	p.pos.head = p.pos.tail + num;
> > +	p.pos.tail = p.pos.head;
> > +
> > +	rte_atomic64_set((rte_atomic64_t *)(uintptr_t)&ht->ht.raw, p.raw); }
> > +
> > +/**
> > + * @internal waits till tail will become equal to head.
> > + * Means no writer/reader is active for that ring.
> > + * Suppose to work as serialization point.
> > + */
> > +static __rte_always_inline void
> > +__rte_ring_hts_head_wait(const struct rte_ring_hts_headtail *ht,
> > +		union rte_ring_ht_pos *p)
> > +{
> > +	p->raw = rte_atomic64_read((rte_atomic64_t *)
> > +			(uintptr_t)&ht->ht.raw);
> > +
> > +	while (p->pos.head != p->pos.tail) {
> > +		rte_pause();
> > +		p->raw = rte_atomic64_read((rte_atomic64_t *)
> > +				(uintptr_t)&ht->ht.raw);
> > +	}
> > +}
> > +
> > +/**
> > + * @internal This function updates the producer head for enqueue
> > + *
> > + * @param r
> > + *   A pointer to the ring structure
> > + * @param is_sp
> > + *   Indicates whether multi-producer path is needed or not
> > + * @param n
> > + *   The number of elements we will want to enqueue, i.e. how far should the
> > + *   head be moved
> > + * @param behavior
> > + *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
> > + *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from
> > ring
> > + * @param old_head
> > + *   Returns head value as it was before the move, i.e. where enqueue starts
> > + * @param new_head
> > + *   Returns the current/new head value i.e. where enqueue finishes
> > + * @param free_entries
> > + *   Returns the amount of free space in the ring BEFORE head was moved
> > + * @return
> > + *   Actual number of objects enqueued.
> > + *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_hts_move_prod_head(struct rte_ring *r, unsigned int num,
> > +	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> > +	uint32_t *free_entries)
> > +{
> > +	uint32_t n;
> > +	union rte_ring_ht_pos np, op;
> > +
> > +	const uint32_t capacity = r->capacity;
> > +
> > +	do {
> > +		/* Reset n to the initial burst count */
> > +		n = num;
> > +
> > +		/* wait for tail to be equal to head */
> > +		__rte_ring_hts_head_wait(&r->hts_prod, &op);
> > +
> > +		/* add rmb barrier to avoid load/load reorder in weak
> > +		 * memory model. It is noop on x86
> > +		 */
> > +		rte_smp_rmb();
> > +
> > +		/*
> > +		 *  The subtraction is done between two unsigned 32bits value
> > +		 * (the result is always modulo 32 bits even if we have
> > +		 * *old_head > cons_tail). So 'free_entries' is always between
> > 0
> > +		 * and capacity (which is < size).
> > +		 */
> > +		*free_entries = capacity + r->cons.tail - op.pos.head;
> > +
> > +		/* check that we have enough room in ring */
> > +		if (unlikely(n > *free_entries))
> > +			n = (behavior == RTE_RING_QUEUE_FIXED) ?
> > +					0 : *free_entries;
> > +
> > +		if (n == 0)
> > +			return 0;
> > +
> > +		np.pos.tail = op.pos.tail;
> > +		np.pos.head = op.pos.head + n;
> > +
> > +	} while (rte_atomic64_cmpset(&r->hts_prod.ht.raw,
> > +			op.raw, np.raw) == 0);
> > +
> > +	*old_head = op.pos.head;
> > +	return n;
> > +}
> > +
> > +/**
> > + * @internal This function updates the consumer head for dequeue
> > + *
> > + * @param r
> > + *   A pointer to the ring structure
> > + * @param is_sc
> > + *   Indicates whether multi-consumer path is needed or not
> > + * @param n
> > + *   The number of elements we will want to enqueue, i.e. how far should the
> > + *   head be moved
> > + * @param behavior
> > + *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
> > + *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from
> > ring
> > + * @param old_head
> > + *   Returns head value as it was before the move, i.e. where dequeue starts
> > + * @param new_head
> > + *   Returns the current/new head value i.e. where dequeue finishes
> > + * @param entries
> > + *   Returns the number of entries in the ring BEFORE head was moved
> > + * @return
> > + *   - Actual number of objects dequeued.
> > + *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
> > + */
> > +static __rte_always_inline unsigned int
> > +__rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,
> > +	enum rte_ring_queue_behavior behavior, uint32_t *old_head,
> > +	uint32_t *entries)
> > +{
> > +	uint32_t n;
> > +	union rte_ring_ht_pos np, op;
> > +
> > +	/* move cons.head atomically */
> > +	do {
> > +		/* Restore n as it may change every loop */
> > +		n = num;
> > +
> > +		/* wait for tail to be equal to head */
> > +		__rte_ring_hts_head_wait(&r->hts_cons, &op);
> > +
> > +		/* add rmb barrier to avoid load/load reorder in weak
> > +		 * memory model. It is noop on x86
> > +		 */
> > +		rte_smp_rmb();
> > +
> > +		/* The subtraction is done between two unsigned 32bits value
> > +		 * (the result is always modulo 32 bits even if we have
> > +		 * cons_head > prod_tail). So 'entries' is always between 0
> > +		 * and size(ring)-1.
> > +		 */
> > +		*entries = r->prod.tail - op.pos.head;
> > +
> > +		/* Set the actual entries for dequeue */
> > +		if (n > *entries)
> > +			n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 :
> > *entries;
> > +
> > +		if (unlikely(n == 0))
> > +			return 0;
> > +
> > +		np.pos.tail = op.pos.tail;
> > +		np.pos.head = op.pos.head + n;
> > +
> > +	} while (rte_atomic64_cmpset(&r->hts_cons.ht.raw,
> > +			op.raw, np.raw) == 0);
> > +
> > +	*old_head = op.pos.head;
> > +	return n;
> > +}
> > +
> > +#endif /* _RTE_RING_HTS_GENERIC_H_ */
> > --
> > 2.17.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, back to index

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-24 11:35 [dpdk-dev] [RFC 0/6] New sync modes for ring Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
2020-03-25 20:44   ` Honnappa Nagarahalli
2020-03-26 12:26     ` Ananyev, Konstantin
2020-02-24 11:35 ` [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
2020-02-24 17:59   ` Jerin Jacob
2020-02-24 19:35     ` Stephen Hemminger
2020-02-24 20:52       ` Honnappa Nagarahalli
2020-02-25 11:45         ` Ananyev, Konstantin
2020-02-25 13:41       ` Ananyev, Konstantin
2020-02-26 16:53         ` Morten Brørup
2020-02-27 10:31         ` Jerin Jacob
2020-02-28  0:17           ` David Christensen
2020-03-20 16:45             ` Ananyev, Konstantin
2020-02-25  0:58     ` Honnappa Nagarahalli
2020-02-25 15:14       ` Ananyev, Konstantin
2020-03-25 20:43 ` Honnappa Nagarahalli
2020-03-26  1:50   ` Ananyev, Konstantin

DPDK patches and discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox