From: Wathsala Vithanage <wathsala.vithanage@arm.com>
To: honnappa.nagarahalli@arm.com, konstantin.v.ananyev@yandex.ru,
thomas@monjalon.net, ruifeng.wang@arm.com
Cc: dev@dpdk.org, nd@arm.com,
Wathsala Vithanage <wathsala.vithanage@arm.com>
Subject: [RFC] ring: further performance improvements with C11
Date: Thu, 15 Jun 2023 20:13:35 +0000 [thread overview]
Message-ID: <20230615201335.919563-2-wathsala.vithanage@arm.com> (raw)
In-Reply-To: <20230615201335.919563-1-wathsala.vithanage@arm.com>
For improved performance over the current C11 based ring implementation
following changes were made.
(1) Replace tail store with RELEASE semantics in __rte_ring_update_tail
with a RELEASE fence. Replace load of the tail with ACQUIRE semantics
in __rte_ring_move_prod_head and __rte_ring_move_cons_head with ACQUIRE
fences.
(2) Remove ACQUIRE fences between load of the old_head and load of the
cons_tail in __rte_ring_move_prod_head and __rte_ring_move_cons_head.
These two fences are not required for the safety of the ring library.
Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
---
.mailmap | 1 +
lib/ring/rte_ring_c11_pvt.h | 35 ++++++++++++++++++++---------------
2 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/.mailmap b/.mailmap
index 4018f0fc47..367115d134 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1430,6 +1430,7 @@ Walter Heymans <walter.heymans@corigine.com>
Wang Sheng-Hui <shhuiw@gmail.com>
Wangyu (Eric) <seven.wangyu@huawei.com>
Waterman Cao <waterman.cao@intel.com>
+Wathsala Vithanage <wathsala.vithanage@arm.com>
Weichun Chen <weichunx.chen@intel.com>
Wei Dai <wei.dai@intel.com>
Weifeng Li <liweifeng96@126.com>
diff --git a/lib/ring/rte_ring_c11_pvt.h b/lib/ring/rte_ring_c11_pvt.h
index f895950df4..63fe58ce9e 100644
--- a/lib/ring/rte_ring_c11_pvt.h
+++ b/lib/ring/rte_ring_c11_pvt.h
@@ -16,6 +16,13 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
uint32_t new_val, uint32_t single, uint32_t enqueue)
{
RTE_SET_USED(enqueue);
+ /*
+ * Updating of ht->tail cannot happen before elements are added to or
+ * removed from the ring, as it could result in data races between
+ * producer and consumer threads. Therefore we need a release
+ * barrier here.
+ */
+ rte_atomic_thread_fence(__ATOMIC_RELEASE);
/*
* If there are other enqueues/dequeues in progress that preceded us,
@@ -24,7 +31,7 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
if (!single)
rte_wait_until_equal_32(&ht->tail, old_val, __ATOMIC_RELAXED);
- __atomic_store_n(&ht->tail, new_val, __ATOMIC_RELEASE);
+ __atomic_store_n(&ht->tail, new_val, __ATOMIC_RELAXED);
}
/**
@@ -66,14 +73,8 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
/* Reset n to the initial burst count */
n = max;
- /* Ensure the head is read before tail */
- __atomic_thread_fence(__ATOMIC_ACQUIRE);
-
- /* load-acquire synchronize with store-release of ht->tail
- * in update_tail.
- */
cons_tail = __atomic_load_n(&r->cons.tail,
- __ATOMIC_ACQUIRE);
+ __ATOMIC_RELAXED);
/* The subtraction is done between two unsigned 32bits value
* (the result is always modulo 32 bits even if we have
@@ -100,6 +101,11 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
0, __ATOMIC_RELAXED,
__ATOMIC_RELAXED);
} while (unlikely(success == 0));
+ /*
+ * Ensure that updates to the ring doesn't rise above
+ * load of the new_head in SP and MP cases.
+ */
+ rte_atomic_thread_fence(__ATOMIC_ACQUIRE);
return n;
}
@@ -142,14 +148,8 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
/* Restore n as it may change every loop */
n = max;
- /* Ensure the head is read before tail */
- __atomic_thread_fence(__ATOMIC_ACQUIRE);
-
- /* this load-acquire synchronize with store-release of ht->tail
- * in update_tail.
- */
prod_tail = __atomic_load_n(&r->prod.tail,
- __ATOMIC_ACQUIRE);
+ __ATOMIC_RELAXED);
/* The subtraction is done between two unsigned 32bits value
* (the result is always modulo 32 bits even if we have
@@ -175,6 +175,11 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
0, __ATOMIC_RELAXED,
__ATOMIC_RELAXED);
} while (unlikely(success == 0));
+ /*
+ * Ensure that updates to the ring doesn't rise above
+ * load of the new_head in SP and MP cases.
+ */
+ rte_atomic_thread_fence(__ATOMIC_ACQUIRE);
return n;
}
--
2.25.1
next prev parent reply other threads:[~2023-06-15 20:14 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-15 20:13 [RFC] ring: Further " Wathsala Vithanage
2023-06-15 20:13 ` Wathsala Vithanage [this message]
2023-07-31 12:31 ` [RFC] ring: further " Thomas Monjalon
2023-08-03 2:56 ` Honnappa Nagarahalli
2023-08-02 9:42 ` Konstantin Ananyev
2023-08-04 22:50 ` Wathsala Wathawana Vithanage
2023-08-09 18:18 ` Konstantin Ananyev
2023-08-15 5:14 ` Honnappa Nagarahalli
2023-08-21 13:27 ` Konstantin Ananyev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230615201335.919563-2-wathsala.vithanage@arm.com \
--to=wathsala.vithanage@arm.com \
--cc=dev@dpdk.org \
--cc=honnappa.nagarahalli@arm.com \
--cc=konstantin.v.ananyev@yandex.ru \
--cc=nd@arm.com \
--cc=ruifeng.wang@arm.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).