* [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
@ 2018-01-19 14:47 Andriy Berestovskyy
2018-01-19 17:31 ` Van Haaren, Harry
0 siblings, 1 reply; 5+ messages in thread
From: Andriy Berestovskyy @ 2018-01-19 14:47 UTC (permalink / raw)
To: dev; +Cc: remy.horton
The __rte_cache_aligned was applied to the whole array,
not the array elements. This leads to a false sharing between
the monitored cores.
Fixes: e70a61ad50ab ("keepalive: export states")
Cc: remy.horton@intel.com
Signed-off-by: Andriy Berestovskyy <aber@semihalf.com>
---
lib/librte_eal/common/rte_keepalive.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/lib/librte_eal/common/rte_keepalive.c b/lib/librte_eal/common/rte_keepalive.c
index 7ddf201..a586e03 100644
--- a/lib/librte_eal/common/rte_keepalive.c
+++ b/lib/librte_eal/common/rte_keepalive.c
@@ -13,8 +13,13 @@
struct rte_keepalive {
/** Core Liveness. */
- enum rte_keepalive_state __rte_cache_aligned state_flags[
- RTE_KEEPALIVE_MAXCORES];
+ struct {
+ /*
+ * Each element of the state_flags table must be cache aligned
+ * to prevent false sharing.
+ */
+ enum rte_keepalive_state s __rte_cache_aligned;
+ } state_flags[RTE_KEEPALIVE_MAXCORES];
/** Last-seen-alive timestamps */
uint64_t last_alive[RTE_KEEPALIVE_MAXCORES];
@@ -67,19 +72,19 @@ rte_keepalive_dispatch_pings(__rte_unused void *ptr_timer,
if (keepcfg->active_cores[idx_core] == 0)
continue;
- switch (keepcfg->state_flags[idx_core]) {
+ switch (keepcfg->state_flags[idx_core].s) {
case RTE_KA_STATE_UNUSED:
break;
case RTE_KA_STATE_ALIVE: /* Alive */
- keepcfg->state_flags[idx_core] = RTE_KA_STATE_MISSING;
+ keepcfg->state_flags[idx_core].s = RTE_KA_STATE_MISSING;
keepcfg->last_alive[idx_core] = rte_rdtsc();
break;
case RTE_KA_STATE_MISSING: /* MIA */
print_trace("Core MIA. ", keepcfg, idx_core);
- keepcfg->state_flags[idx_core] = RTE_KA_STATE_DEAD;
+ keepcfg->state_flags[idx_core].s = RTE_KA_STATE_DEAD;
break;
case RTE_KA_STATE_DEAD: /* Dead */
- keepcfg->state_flags[idx_core] = RTE_KA_STATE_GONE;
+ keepcfg->state_flags[idx_core].s = RTE_KA_STATE_GONE;
print_trace("Core died. ", keepcfg, idx_core);
if (keepcfg->callback)
keepcfg->callback(
@@ -90,7 +95,7 @@ rte_keepalive_dispatch_pings(__rte_unused void *ptr_timer,
case RTE_KA_STATE_GONE: /* Buried */
break;
case RTE_KA_STATE_DOZING: /* Core going idle */
- keepcfg->state_flags[idx_core] = RTE_KA_STATE_SLEEP;
+ keepcfg->state_flags[idx_core].s = RTE_KA_STATE_SLEEP;
keepcfg->last_alive[idx_core] = rte_rdtsc();
break;
case RTE_KA_STATE_SLEEP: /* Idled core */
@@ -100,7 +105,7 @@ rte_keepalive_dispatch_pings(__rte_unused void *ptr_timer,
keepcfg->relay_callback(
keepcfg->relay_callback_data,
idx_core,
- keepcfg->state_flags[idx_core],
+ keepcfg->state_flags[idx_core].s,
keepcfg->last_alive[idx_core]
);
}
@@ -144,11 +149,11 @@ rte_keepalive_register_core(struct rte_keepalive *keepcfg, const int id_core)
void
rte_keepalive_mark_alive(struct rte_keepalive *keepcfg)
{
- keepcfg->state_flags[rte_lcore_id()] = RTE_KA_STATE_ALIVE;
+ keepcfg->state_flags[rte_lcore_id()].s = RTE_KA_STATE_ALIVE;
}
void
rte_keepalive_mark_sleep(struct rte_keepalive *keepcfg)
{
- keepcfg->state_flags[rte_lcore_id()] = RTE_KA_STATE_DOZING;
+ keepcfg->state_flags[rte_lcore_id()].s = RTE_KA_STATE_DOZING;
}
--
2.7.4
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
2018-01-19 14:47 [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment Andriy Berestovskyy
@ 2018-01-19 17:31 ` Van Haaren, Harry
2018-01-22 18:20 ` Andriy Berestovskyy
0 siblings, 1 reply; 5+ messages in thread
From: Van Haaren, Harry @ 2018-01-19 17:31 UTC (permalink / raw)
To: Andriy Berestovskyy, dev; +Cc: Horton, Remy
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andriy Berestovskyy
> Sent: Friday, January 19, 2018 2:48 PM
> To: dev@dpdk.org
> Cc: Horton, Remy <remy.horton@intel.com>
> Subject: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
>
> The __rte_cache_aligned was applied to the whole array,
> not the array elements. This leads to a false sharing between
> the monitored cores.
>
> Fixes: e70a61ad50ab ("keepalive: export states")
> Cc: remy.horton@intel.com
> Signed-off-by: Andriy Berestovskyy <aber@semihalf.com>
> ---
> lib/librte_eal/common/rte_keepalive.c | 25 +++++++++++++++----------
> 1 file changed, 15 insertions(+), 10 deletions(-)
>
> diff --git a/lib/librte_eal/common/rte_keepalive.c
> b/lib/librte_eal/common/rte_keepalive.c
> index 7ddf201..a586e03 100644
> --- a/lib/librte_eal/common/rte_keepalive.c
> +++ b/lib/librte_eal/common/rte_keepalive.c
> @@ -13,8 +13,13 @@
>
> struct rte_keepalive {
> /** Core Liveness. */
> - enum rte_keepalive_state __rte_cache_aligned state_flags[
> - RTE_KEEPALIVE_MAXCORES];
> + struct {
> + /*
> + * Each element of the state_flags table must be cache aligned
> + * to prevent false sharing.
> + */
> + enum rte_keepalive_state s __rte_cache_aligned;
> + } state_flags[RTE_KEEPALIVE_MAXCORES];
By aligning each item in the array, we do reduce false-sharing of the cache lines for all cores, however we pay the cost of increasing the footprint of the rte_keepalive struct. Note that the code iterates the full MAX_CORES in various loops in the monitoring core.
Before
(gdb) p sizeof(struct rte_keepalive)
$1 = 1728 # 27 cache lines
After
(gdb) p sizeof(struct rte_keepalive)
$1 = 9408 # 147 cache lines
These changes do reduce false-sharing however is there actually a performance benefit? A lot of cache space will be taken up if each core requires its own cache line, which will reduce performance again.. it's a tradeoff.
Little fix for a v2: "s" is not a good variable name for the rte_keepalive_state, please use something more descriptive.
<snip>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
2018-01-19 17:31 ` Van Haaren, Harry
@ 2018-01-22 18:20 ` Andriy Berestovskyy
2018-01-23 10:16 ` Remy Horton
0 siblings, 1 reply; 5+ messages in thread
From: Andriy Berestovskyy @ 2018-01-22 18:20 UTC (permalink / raw)
To: Van Haaren, Harry; +Cc: dev, Horton, Remy
Hey Harry,
Thanks for the review.
On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry
<harry.van.haaren@intel.com> wrote:
> These changes do reduce false-sharing however is there actually a performance benefit? A lot of cache space will be taken up if each core requires its own cache line, which will reduce performance again.. it's a tradeoff.
1. False sharing is happening in the data path vs loops in control paths.
2. The original code (prior e70a61ad50ab "keepalive: export states")
had each element aligned to the cache line, not the whole array.
> Little fix for a v2: "s" is not a good variable name for the rte_keepalive_state, please use something more descriptive.
Sure, if there are no more comments, I'll change it.
Andriy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
2018-01-22 18:20 ` Andriy Berestovskyy
@ 2018-01-23 10:16 ` Remy Horton
2018-01-23 10:27 ` Van Haaren, Harry
0 siblings, 1 reply; 5+ messages in thread
From: Remy Horton @ 2018-01-23 10:16 UTC (permalink / raw)
To: Andriy Berestovskyy, Van Haaren, Harry; +Cc: dev
On 22/01/2018 18:20, Andriy Berestovskyy wrote:
[..]
> On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry
> <harry.van.haaren@intel.com> wrote:
>> These changes do reduce false-sharing however is there actually a
>> performance benefit? A lot of cache space will be taken up if each
>> core requires its own cache line, which will reduce performance
>> again.. it's a tradeoff.
[..]
> 2. The original code (prior e70a61ad50ab "keepalive: export states")
> had each element aligned to the cache line, not the whole array.
Aligning each flag element was the original intention, so I see no issue
in restoring it. The monitoring core only reads the entries within
state_flags for which the corresponding active_core is set, so
ultimately the trade-off in cache line usage is one made by the
application when it decides which cores need monitoring.
..Remy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
2018-01-23 10:16 ` Remy Horton
@ 2018-01-23 10:27 ` Van Haaren, Harry
0 siblings, 0 replies; 5+ messages in thread
From: Van Haaren, Harry @ 2018-01-23 10:27 UTC (permalink / raw)
To: Horton, Remy, Andriy Berestovskyy; +Cc: dev
> From: Horton, Remy
> Sent: Tuesday, January 23, 2018 10:17 AM
> To: Andriy Berestovskyy <aber@semihalf.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment
>
>
> On 22/01/2018 18:20, Andriy Berestovskyy wrote:
> [..]
> > On Fri, Jan 19, 2018 at 6:31 PM, Van Haaren, Harry
> > <harry.van.haaren@intel.com> wrote:
> >> These changes do reduce false-sharing however is there actually a
> >> performance benefit? A lot of cache space will be taken up if each
> >> core requires its own cache line, which will reduce performance
> >> again.. it's a tradeoff.
> [..]
> > 2. The original code (prior e70a61ad50ab "keepalive: export states")
> > had each element aligned to the cache line, not the whole array.
>
> Aligning each flag element was the original intention, so I see no issue
> in restoring it. The monitoring core only reads the entries within
> state_flags for which the corresponding active_core is set, so
> ultimately the trade-off in cache line usage is one made by the
> application when it decides which cores need monitoring.
>
> ..Remy
No objection here - just making sure the change was intentional
and the effects are considered.
@Remy as you're the Keepalive maintainer I'll leave Acking to you :)
-Harry
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-01-23 10:27 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-19 14:47 [dpdk-dev] [PATCH] keepalive: fix keepalive state alignment Andriy Berestovskyy
2018-01-19 17:31 ` Van Haaren, Harry
2018-01-22 18:20 ` Andriy Berestovskyy
2018-01-23 10:16 ` Remy Horton
2018-01-23 10:27 ` Van Haaren, Harry
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).