From: David Marchand <david.marchand@redhat.com>
To: Konstantin Ananyev <konstantin.ananyev@intel.com>,
Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Cc: dev <dev@dpdk.org>,
jielong.zjl@antfin.com,
Pavan Nikhilesh <pbhagavatula@marvell.com>,
Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
Thomas Monjalon <thomas@monjalon.net>
Subject: Re: [dpdk-dev] [PATCH v7 00/10] New sync modes for ring
Date: Tue, 21 Apr 2020 13:31:39 +0200 [thread overview]
Message-ID: <CAJFAV8zAXxzP4HhxsjcoqgPVCDDRtJyyHccYLbyh8BSmqBY=7A@mail.gmail.com> (raw)
In-Reply-To: <20200420122831.16973-1-konstantin.ananyev@intel.com>
On Mon, Apr 20, 2020 at 2:28 PM Konstantin Ananyev
<konstantin.ananyev@intel.com> wrote:
> These days more and more customers use(/try to use) DPDK based apps within
> overcommitted systems (multiple acttive threads over same pysical cores):
> VM, container deployments, etc.
> One quite common problem they hit:
> Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
> LHP is quite a common problem for spin-based sync primitives
> (spin-locks, etc.) on overcommitted systems.
> The situation gets much worse when some sort of
> fair-locking technique is used (ticket-lock, etc.).
> As now not only lock-owner but also lock-waiters scheduling
> order matters a lot (LWP).
> These two problems are well-known for kernel within VMs:
> http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> The problem with rte_ring is that while head accusion is sort of
> un-fair locking, waiting on tail is very similar to ticket lock schema -
> tail has to be updated in particular order.
> That makes current rte_ring implementation to perform
> really pure on some overcommited scenarios.
> It is probably not possible to completely resolve LHP problem in
> userspace only (without some kernel communication/intervention).
> But removing fairness at tail update helps to avoid LWP and
> can mitigate the situation significantly.
> This patch proposes two new optional ring synchronization modes:
> 1) Head/Tail Sync (HTS) mode
> In that mode enqueue/dequeue operation is fully serialized:
> only one thread at a time is allowed to perform given op.
> As another enhancement provide ability to split enqueue/dequeue
> operation into two phases:
> - enqueue/dequeue start
> - enqueue/dequeue finish
> That allows user to inspect objects in the ring without removing
> them from it (aka MT safe peek).
> 2) Relaxed Tail Sync (RTS)
> The main difference from original MP/MC algorithm is that
> tail value is increased not by every thread that finished enqueue/dequeue,
> but only by the last one.
> That allows threads to avoid spinning on ring tail value,
> leaving actual tail value change to the last thread in the update queue.
>
> Note that these new sync modes are optional.
> For current rte_ring users nothing should change
> (both in terms of API/ABI and performance).
> Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
> way (via flags and _init_), and MP/MC remains as default one.
> The only thing that changed:
> Format of prod/cons now could differ depending on mode selected at _init_.
> So user has to stick with one sync model through whole ring lifetime.
> In other words, user can't create a ring for let say SP mode and then
> in the middle of data-path change his mind and start using MP_RTS mode.
> For existing modes (SP/MP, SC/MC) format remains the same and
> user can still use them interchangeably, though of course it is an
> error prone practice.
>
> Test results on IA (see below) show significant improvements
> for average enqueue/dequeue op times on overcommitted systems.
> For 'classic' DPDK deployments (one thread per core) original MP/MC
> algorithm still shows best numbers, though for 64-bit target
> RTS numbers are not that far away.
> Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
> echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'
>
> X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
> MP/MC HTS RTS
> 1thread@1core(--lcores=6-7) 8.00 8.15 8.99
> 2thread@2core(--lcores=6-8) 19.14 19.61 20.35
> 4thread@4core(--lcores=6-10) 29.43 29.79 31.82
> 8thread@8core(--lcores=6-14) 110.59 192.81 119.50
> 16thread@16core(--lcores=6-22) 461.03 813.12 495.59
> 32thread/@32core(--lcores='6-22,55-70') 982.90 1972.38 1160.51
>
> 2thread@1core(--lcores='6,(10-11)@7' 20140.50 23.58 25.14
> 4thread@2core(--lcores='6,(10-11)@7,(20-21)@8' 153680.60 76.88 80.05
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 280314.32 294.72 318.79
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59 1144.02 1175.14
> 32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68
>
> 8thread@2core(--lcores='6,(10-17)@(7,8))' 321085.98 298.59 307.47
> 16thread@4core(--lcores='6,(20-35)@(7-10))' 1900705.61 575.35 678.29
> 32thread@4core(--lcores='6,(20-51)@(7-10))' 5510445.85 2164.36 2714.12
>
> i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
> DEQ+ENQ average cycles/obj
> MP/MC HTS RTS
> 1thread@1core(--lcores=6-7) 7.85 12.13 11.31
> 2thread@2core(--lcores=6-8) 17.89 24.52 21.86
> 8thread@8core(--lcores=6-14) 32.58 354.20 54.58
> 32thread/@32core(--lcores='6-22,55-70') 813.77 6072.41 2169.91
>
> 2thread@1core(--lcores='6,(10-11)@7' 16095.00 36.06 34.74
> 8thread@2core(--lcores='6,(10-13)@7,(20-23)@8' 1140354.54 346.61 361.57
> 16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65
>
> 8thread@2core(--lcores='6,(10-17)@(7,8))' 594358.61 332.70 357.74
> 32thread@4core(--lcores='6,(20-51)@(7-10))' 5319896.86 2836.44 3028.87
I fixed a couple of typos and split the doc updates.
Series applied with the patch from Pavan.
Thanks for the work Konstantin, Honnappa.
--
David Marchand
prev parent reply other threads:[~2020-04-21 11:31 UTC|newest]
Thread overview: 146+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-24 11:35 [dpdk-dev] [RFC 0/6] " Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
2020-03-25 20:44 ` Honnappa Nagarahalli
2020-03-26 12:26 ` Ananyev, Konstantin
2020-02-24 11:35 ` [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
2020-02-24 17:59 ` Jerin Jacob
2020-02-24 19:35 ` Stephen Hemminger
2020-02-24 20:52 ` Honnappa Nagarahalli
2020-02-25 11:45 ` Ananyev, Konstantin
2020-02-25 13:41 ` Ananyev, Konstantin
2020-02-26 16:53 ` Morten Brørup
2020-02-27 10:31 ` Jerin Jacob
2020-02-28 0:17 ` David Christensen
2020-03-20 16:45 ` Ananyev, Konstantin
2020-02-25 0:58 ` Honnappa Nagarahalli
2020-02-25 15:14 ` Ananyev, Konstantin
2020-03-25 20:43 ` Honnappa Nagarahalli
2020-03-26 1:50 ` Ananyev, Konstantin
2020-03-30 21:29 ` Honnappa Nagarahalli
2020-03-30 23:37 ` Honnappa Nagarahalli
2020-03-31 17:21 ` Ananyev, Konstantin
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 0/8] " Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 1/8] test/ring: add contention stress test Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 2/8] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 3/8] ring: introduce RTS ring mode Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 4/8] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 5/8] ring: introduce HTS ring mode Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 6/8] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 7/8] ring: introduce peek style API Konstantin Ananyev
2020-03-31 16:43 ` [dpdk-dev] [PATCH v1 8/8] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 0/9] New sync modes for ring Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-02 22:09 ` [dpdk-dev] [PATCH v2 9/9] ring: add C11 memory model for new sync modes Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-08 4:59 ` Honnappa Nagarahalli
2020-04-09 12:36 ` Ananyev, Konstantin
2020-04-09 13:00 ` Ananyev, Konstantin
2020-04-10 18:01 ` Honnappa Nagarahalli
2020-04-10 16:59 ` Honnappa Nagarahalli
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-08 4:59 ` Honnappa Nagarahalli
2020-04-09 13:39 ` Ananyev, Konstantin
2020-04-10 20:15 ` Honnappa Nagarahalli
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-04 17:27 ` Wang, Haiyue
2020-04-08 5:00 ` Honnappa Nagarahalli
2020-04-09 14:52 ` Ananyev, Konstantin
2020-04-10 23:10 ` Honnappa Nagarahalli
2020-04-13 14:29 ` David Marchand
2020-04-13 16:42 ` Honnappa Nagarahalli
2020-04-14 13:47 ` David Marchand
2020-04-14 15:57 ` Honnappa Nagarahalli
2020-04-14 16:21 ` Ananyev, Konstantin
2020-04-14 13:18 ` Ananyev, Konstantin
2020-04-14 15:58 ` Honnappa Nagarahalli
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-13 23:27 ` Honnappa Nagarahalli
2020-04-14 16:12 ` Ananyev, Konstantin
2020-04-14 17:06 ` Honnappa Nagarahalli
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-14 3:45 ` Honnappa Nagarahalli
2020-04-14 16:47 ` Ananyev, Konstantin
2020-04-14 17:30 ` Honnappa Nagarahalli
2020-04-14 22:24 ` Ananyev, Konstantin
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-03 17:42 ` [dpdk-dev] [PATCH v3 9/9] ring: add C11 memory model for new sync modes Konstantin Ananyev
2020-04-04 14:16 ` [dpdk-dev] 回复:[PATCH " 周介龙
2020-04-14 4:28 ` [dpdk-dev] [PATCH " Honnappa Nagarahalli
2020-04-14 18:29 ` Ananyev, Konstantin
2020-04-15 20:28 ` Ananyev, Konstantin
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-17 13:36 ` [dpdk-dev] [PATCH v4 9/9] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-19 2:30 ` Honnappa Nagarahalli
2020-04-19 8:03 ` David Marchand
2020-04-19 11:47 ` Ananyev, Konstantin
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-19 2:31 ` Honnappa Nagarahalli
2020-04-19 18:32 ` Ananyev, Konstantin
2020-04-19 19:12 ` Ananyev, Konstantin
2020-04-19 21:14 ` Honnappa Nagarahalli
2020-04-19 22:41 ` Ananyev, Konstantin
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-19 2:32 ` Honnappa Nagarahalli
2020-04-18 16:32 ` [dpdk-dev] [PATCH v5 9/9] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-19 2:32 ` Honnappa Nagarahalli
2020-04-19 2:32 ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Honnappa Nagarahalli
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 00/10] " Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 01/10] test/ring: add contention stress test Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 02/10] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 03/10] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 04/10] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 05/10] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 06/10] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 07/10] ring: introduce peek style API Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 08/10] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 09/10] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-20 12:11 ` [dpdk-dev] [PATCH v6 10/10] doc: update ring guide Konstantin Ananyev
2020-04-20 13:47 ` David Marchand
2020-04-20 14:07 ` Ananyev, Konstantin
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 00/10] New sync modes for ring Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 01/10] test/ring: add contention stress test Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 02/10] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 03/10] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 04/10] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 05/10] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 06/10] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 07/10] ring: introduce peek style API Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 08/10] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 09/10] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-20 12:28 ` [dpdk-dev] [PATCH v7 10/10] doc: update ring guide Konstantin Ananyev
2020-04-21 11:31 ` David Marchand [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJFAV8zAXxzP4HhxsjcoqgPVCDDRtJyyHccYLbyh8BSmqBY=7A@mail.gmail.com' \
--to=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=honnappa.nagarahalli@arm.com \
--cc=jerinj@marvell.com \
--cc=jielong.zjl@antfin.com \
--cc=konstantin.ananyev@intel.com \
--cc=pbhagavatula@marvell.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).