DPDK patches and discussions
 help / color / mirror / Atom feed
From: Konstantin Ananyev <konstantin.ananyev@intel.com>
To: dev@dpdk.org
Cc: honnappa.nagarahalli@arm.com,
	Konstantin Ananyev <konstantin.ananyev@intel.com>
Subject: [dpdk-dev] [PATCH v1 0/8] New sync modes for ring
Date: Tue, 31 Mar 2020 17:43:22 +0100
Message-ID: <20200331164330.28854-1-konstantin.ananyev@intel.com> (raw)
In-Reply-To: <20200224113515.1744-1-konstantin.ananyev@intel.com>

RFC - V1 changes:
1. remove ABI brekage (at least I hope I did)
2. Add support for ring_elem
3. Rework peek related API a bit
4. Rework test to make it less verbose and unite all test-cases
   in one command
5. Add new test-case for MT peek API

TODO list:
1. Add C11 atomics support
2. Update docs

These days many customers use(/try to use) DPDK based apps within
overcommitted systems (multiple acttive threads over same pysical cores):
VM, container deployments, etc.
One quite common problem they hit:
Lock-Holder-Preemption/Lock-Waiter-Preemption with rte_ring.
LHP/LWP are quite a common problem for spin-based sync primitives
(spin-locks, etc.) on overcommitted systems.
The situation gets much worse when some sort of
fair-locking technique is used (ticket-lock, etc.).
As now not only lock-owner but also lock-waiters scheduling
order matters a lot.
This is a well-known problem for kernel within VMs:
http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
The problem with rte_ring is that while head accusion is sort of
un-fair locking, waiting on tail is very similar to ticket lock schema -
tail has to be updated in particular order.
That makes current rte_ring implementation to perform
really pure on some overcommited scenarios.
While it is probably not possible to completely resolve LHP problem in
userspace only (without some kernel communication/intervention),
removing fairness in tail update can mitigate it significantly.
So this RFC proposes two new optional ring synchronization modes:
1) Head/Tail Sync (HTS) mode
In that mode enqueue/dequeue operation is fully serialized:
    only one thread at a time is allowed to perform given op.
    As another enhancement provide ability to split enqueue/dequeue
    operation into two phases:
      - enqueue/dequeue start
      - enqueue/dequeue finish
    That allows user to inspect objects in the ring without removing
    them from it (aka MT safe peek).
2) Relaxed Tail Sync (RTS)
The main difference from original MP/MC algorithm is that
tail value is increased not by every thread that finished enqueue/dequeue,
but only by the last one.
That allows threads to avoid spinning on ring tail value,
leaving actual tail value change to the last thread in the update queue.

Note that these new sync modes are optional.
For current rte_ring users nothing should change
(both in terms of API/ABI and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
The only thing that changed:
Format of prod/cons now could differ depending on mode selected at _init_.
So user has to stick with one sync model through whole ring lifetime.
In other words, user can't create a ring for let say SP mode and then
in the middle of data-path change his mind and start using MP_RTS mode.
For existing modes (SP/MP, SC/MC) format remains the same and
user can still use them interchangeably, though of course it is an
error prone practice. 

Test results on IA (see below) show significant improvements
for average enqueue/dequeue op times on overcommitted systems.
For 'classic' DPDK deployments (one thread per core) original MP/MC
algorithm still shows best numbers, though for 64-bit target
RTS numbers are not that far away.
Numbers were produced by new UT test-case: ring_stress_autotest, i.e.:
echo ring_stress_autotest | ./dpdk-test -n 4 --lcores='...'

X86_64 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj

                                                MP/MC      HTS     RTS
1thread@1core(--lcores=6-7)                     8.00       8.15    8.99
2thread@2core(--lcores=6-8)                     19.14      19.61   20.35
4thread@4core(--lcores=6-10)                    29.43      29.79   31.82
8thread@8core(--lcores=6-14)                    110.59     192.81  119.50
16thread@16core(--lcores=6-22)                  461.03     813.12  495.59
32thread/@32core(--lcores='6-22,55-70')         982.90     1972.38 1160.51

2thread@1core(--lcores='6,(10-11)@7'            20140.50   23.58   25.14
4thread@2core(--lcores='6,(10-11)@7,(20-21)@8'  153680.60  76.88   80.05
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  280314.32  294.72  318.79
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 643176.59  1144.02 1175.14
32thread@2core(--lcores='6,(10-25)@7,(30-45)@8' 4264238.80 4627.48 4892.68

8thread@2core(--lcores='6,(10-17)@(7,8))'       321085.98  298.59  307.47
16thread@4core(--lcores='6,(20-35)@(7-10))'     1900705.61 575.35  678.29
32thread@4core(--lcores='6,(20-51)@(7-10))'     5510445.85 2164.36 2714.12

i686 @ Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
DEQ+ENQ average cycles/obj

                                                MP/MC      HTS     RTS
1thread@1core(--lcores=6-7)                     7.85       12.13   11.31
2thread@2core(--lcores=6-8)                     17.89      24.52   21.86
8thread@8core(--lcores=6-14)                    32.58      354.20  54.58
32thread/@32core(--lcores='6-22,55-70')         813.77     6072.41 2169.91

2thread@1core(--lcores='6,(10-11)@7'            16095.00   36.06   34.74
8thread@2core(--lcores='6,(10-13)@7,(20-23)@8'  1140354.54 346.61  361.57
16thread@2core(--lcores='6,(10-17)@7,(20-27)@8' 1920417.86 1314.90 1416.65

8thread@2core(--lcores='6,(10-17)@(7,8))'       594358.61  332.70  357.74
32thread@4core(--lcores='6,(20-51)@(7-10))'     5319896.86 2836.44 3028.87

Konstantin Ananyev (8):
  test/ring: add contention stress test
  ring: prepare ring to allow new sync schemes
  ring: introduce RTS ring mode
  test/ring: add contention stress test for RTS ring
  ring: introduce HTS ring mode
  test/ring: add contention stress test for HTS ring
  ring: introduce peek style API
  test/ring: add stress test for MT peek API

 app/test/Makefile                      |   5 +
 app/test/meson.build                   |   5 +
 app/test/test_pdump.c                  |   6 +-
 app/test/test_ring_hts_stress.c        |  32 ++
 app/test/test_ring_mpmc_stress.c       |  31 ++
 app/test/test_ring_peek_stress.c       |  43 +++
 app/test/test_ring_rts_stress.c        |  32 ++
 app/test/test_ring_stress.c            |  57 ++++
 app/test/test_ring_stress.h            |  37 +++
 app/test/test_ring_stress_impl.h       | 436 +++++++++++++++++++++++++
 lib/librte_pdump/rte_pdump.c           |   2 +-
 lib/librte_port/rte_port_ring.c        |  12 +-
 lib/librte_ring/Makefile               |   9 +-
 lib/librte_ring/meson.build            |   9 +-
 lib/librte_ring/rte_ring.c             | 114 ++++++-
 lib/librte_ring/rte_ring.h             | 243 ++++++++++++--
 lib/librte_ring/rte_ring_c11_mem.h     |  44 +++
 lib/librte_ring/rte_ring_elem.h        | 105 +++++-
 lib/librte_ring/rte_ring_generic.h     |  48 +++
 lib/librte_ring/rte_ring_hts.h         | 210 ++++++++++++
 lib/librte_ring/rte_ring_hts_elem.h    | 205 ++++++++++++
 lib/librte_ring/rte_ring_hts_generic.h | 235 +++++++++++++
 lib/librte_ring/rte_ring_peek.h        | 379 +++++++++++++++++++++
 lib/librte_ring/rte_ring_rts.h         | 316 ++++++++++++++++++
 lib/librte_ring/rte_ring_rts_elem.h    | 205 ++++++++++++
 lib/librte_ring/rte_ring_rts_generic.h | 210 ++++++++++++
 26 files changed, 2974 insertions(+), 56 deletions(-)
 create mode 100644 app/test/test_ring_hts_stress.c
 create mode 100644 app/test/test_ring_mpmc_stress.c
 create mode 100644 app/test/test_ring_peek_stress.c
 create mode 100644 app/test/test_ring_rts_stress.c
 create mode 100644 app/test/test_ring_stress.c
 create mode 100644 app/test/test_ring_stress.h
 create mode 100644 app/test/test_ring_stress_impl.h
 create mode 100644 lib/librte_ring/rte_ring_hts.h
 create mode 100644 lib/librte_ring/rte_ring_hts_elem.h
 create mode 100644 lib/librte_ring/rte_ring_hts_generic.h
 create mode 100644 lib/librte_ring/rte_ring_peek.h
 create mode 100644 lib/librte_ring/rte_ring_rts.h
 create mode 100644 lib/librte_ring/rte_ring_rts_elem.h
 create mode 100644 lib/librte_ring/rte_ring_rts_generic.h

-- 
2.17.1


  parent reply	other threads:[~2020-03-31 16:43 UTC|newest]

Thread overview: 146+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-24 11:35 [dpdk-dev] [RFC 0/6] " Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 1/6] test/ring: add contention stress test Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 2/6] ring: rework ring layout to allow new sync schemes Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 3/6] ring: introduce RTS ring mode Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 4/6] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-02-24 11:35 ` [dpdk-dev] [RFC 5/6] ring: introduce HTS ring mode Konstantin Ananyev
2020-03-25 20:44   ` Honnappa Nagarahalli
2020-03-26 12:26     ` Ananyev, Konstantin
2020-02-24 11:35 ` [dpdk-dev] [RFC 6/6] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-02-24 16:59 ` [dpdk-dev] [RFC 0/6] New sync modes for ring Stephen Hemminger
2020-02-24 17:59   ` Jerin Jacob
2020-02-24 19:35     ` Stephen Hemminger
2020-02-24 20:52       ` Honnappa Nagarahalli
2020-02-25 11:45         ` Ananyev, Konstantin
2020-02-25 13:41       ` Ananyev, Konstantin
2020-02-26 16:53         ` Morten Brørup
2020-02-27 10:31         ` Jerin Jacob
2020-02-28  0:17           ` David Christensen
2020-03-20 16:45             ` Ananyev, Konstantin
2020-02-25  0:58     ` Honnappa Nagarahalli
2020-02-25 15:14       ` Ananyev, Konstantin
2020-03-25 20:43 ` Honnappa Nagarahalli
2020-03-26  1:50   ` Ananyev, Konstantin
2020-03-30 21:29     ` Honnappa Nagarahalli
2020-03-30 23:37       ` Honnappa Nagarahalli
2020-03-31 17:21         ` Ananyev, Konstantin
2020-03-31 16:43 ` Konstantin Ananyev [this message]
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 1/8] test/ring: add contention stress test Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 2/8] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 3/8] ring: introduce RTS ring mode Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 4/8] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 5/8] ring: introduce HTS ring mode Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 6/8] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 7/8] ring: introduce peek style API Konstantin Ananyev
2020-03-31 16:43   ` [dpdk-dev] [PATCH v1 8/8] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-02 22:09   ` [dpdk-dev] [PATCH v2 0/9] New sync modes for ring Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-02 22:09     ` [dpdk-dev] [PATCH v2 9/9] ring: add C11 memory model for new sync modes Konstantin Ananyev
2020-04-03 17:42     ` [dpdk-dev] [PATCH v3 0/9] New sync modes for ring Konstantin Ananyev
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-08  4:59         ` Honnappa Nagarahalli
2020-04-09 12:36           ` Ananyev, Konstantin
2020-04-09 13:00             ` Ananyev, Konstantin
2020-04-10 18:01               ` Honnappa Nagarahalli
2020-04-10 16:59             ` Honnappa Nagarahalli
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-08  4:59         ` Honnappa Nagarahalli
2020-04-09 13:39           ` Ananyev, Konstantin
2020-04-10 20:15             ` Honnappa Nagarahalli
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-04 17:27         ` Wang, Haiyue
2020-04-08  5:00         ` Honnappa Nagarahalli
2020-04-09 14:52           ` Ananyev, Konstantin
2020-04-10 23:10             ` Honnappa Nagarahalli
2020-04-13 14:29               ` David Marchand
2020-04-13 16:42                 ` Honnappa Nagarahalli
2020-04-14 13:47                   ` David Marchand
2020-04-14 15:57                     ` Honnappa Nagarahalli
2020-04-14 16:21                       ` Ananyev, Konstantin
2020-04-14 13:18               ` Ananyev, Konstantin
2020-04-14 15:58                 ` Honnappa Nagarahalli
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-13 23:27         ` Honnappa Nagarahalli
2020-04-14 16:12           ` Ananyev, Konstantin
2020-04-14 17:06             ` Honnappa Nagarahalli
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-14  3:45         ` Honnappa Nagarahalli
2020-04-14 16:47           ` Ananyev, Konstantin
2020-04-14 17:30             ` Honnappa Nagarahalli
2020-04-14 22:24               ` Ananyev, Konstantin
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-03 17:42       ` [dpdk-dev] [PATCH v3 9/9] ring: add C11 memory model for new sync modes Konstantin Ananyev
2020-04-04 14:16         ` [dpdk-dev] 回复:[PATCH " 周介龙
2020-04-14  4:28         ` [dpdk-dev] [PATCH " Honnappa Nagarahalli
2020-04-14 18:29           ` Ananyev, Konstantin
2020-04-15 20:28           ` Ananyev, Konstantin
2020-04-17 13:36       ` [dpdk-dev] [PATCH v4 0/9] New sync modes for ring Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-17 13:36         ` [dpdk-dev] [PATCH v4 9/9] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-18 16:32         ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Konstantin Ananyev
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 1/9] test/ring: add contention stress test Konstantin Ananyev
2020-04-19  2:30             ` Honnappa Nagarahalli
2020-04-19  8:03               ` David Marchand
2020-04-19 11:47                 ` Ananyev, Konstantin
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 2/9] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 3/9] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 4/9] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 5/9] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 6/9] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 7/9] ring: introduce peek style API Konstantin Ananyev
2020-04-19  2:31             ` Honnappa Nagarahalli
2020-04-19 18:32               ` Ananyev, Konstantin
2020-04-19 19:12                 ` Ananyev, Konstantin
2020-04-19 21:14                   ` Honnappa Nagarahalli
2020-04-19 22:41                     ` Ananyev, Konstantin
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 8/9] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-19  2:32             ` Honnappa Nagarahalli
2020-04-18 16:32           ` [dpdk-dev] [PATCH v5 9/9] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-19  2:32             ` Honnappa Nagarahalli
2020-04-19  2:32           ` [dpdk-dev] [PATCH v5 0/9] New sync modes for ring Honnappa Nagarahalli
2020-04-20 12:11           ` [dpdk-dev] [PATCH v6 00/10] " Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 01/10] test/ring: add contention stress test Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 02/10] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 03/10] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 04/10] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 05/10] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 06/10] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 07/10] ring: introduce peek style API Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 08/10] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 09/10] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-20 12:11             ` [dpdk-dev] [PATCH v6 10/10] doc: update ring guide Konstantin Ananyev
2020-04-20 13:47               ` David Marchand
2020-04-20 14:07                 ` Ananyev, Konstantin
2020-04-20 12:28             ` [dpdk-dev] [PATCH v7 00/10] New sync modes for ring Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 01/10] test/ring: add contention stress test Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 02/10] ring: prepare ring to allow new sync schemes Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 03/10] ring: introduce RTS ring mode Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 04/10] test/ring: add contention stress test for RTS ring Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 05/10] ring: introduce HTS ring mode Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 06/10] test/ring: add contention stress test for HTS ring Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 07/10] ring: introduce peek style API Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 08/10] test/ring: add stress test for MT peek API Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 09/10] test/ring: add functional tests for new sync modes Konstantin Ananyev
2020-04-20 12:28               ` [dpdk-dev] [PATCH v7 10/10] doc: update ring guide Konstantin Ananyev
2020-04-21 11:31               ` [dpdk-dev] [PATCH v7 00/10] New sync modes for ring David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200331164330.28854-1-konstantin.ananyev@intel.com \
    --to=konstantin.ananyev@intel.com \
    --cc=dev@dpdk.org \
    --cc=honnappa.nagarahalli@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git