From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Stephen Hemminger" <stephen@networkplumber.org>
Cc: "Bruce Richardson" <bruce.richardson@intel.com>,
"Ferruh Yigit" <ferruh.yigit@amd.com>,
"Mattias Rönnblom" <hofors@lysator.liu.se>,
"John W. Linville" <linville@tuxdriver.com>,
"Thomas Monjalon" <thomas@monjalon.net>,
dev@dpdk.org, "Mattias Rönnblom" <mattias.ronnblom@ericsson.com>
Subject: RE: [RFC v3] net/af_packet: make stats reset reliable
Date: Fri, 10 May 2024 11:14:25 +0200 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35E9F434@smartserver.smartshare.dk> (raw)
In-Reply-To: <20240509215602.7e03f34d@hermes.local>
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 10 May 2024 06.56
>
> On Thu, 9 May 2024 16:19:08 +0200
> Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > > From: Morten Brørup [mailto:mb@smartsharesystems.com]
> > > Sent: Thursday, 9 May 2024 13.37
> > >
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Thursday, 9 May 2024 11.30
> > > >
> > > > On Thu, May 09, 2024 at 09:43:16AM +0200, Morten Brørup wrote:
> > > > > > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > > > > > Sent: Wednesday, 8 May 2024 22.54
> > > > > >
> > > > > > On Wed, 8 May 2024 20:48:06 +0100
> > > > > > Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> > > > > >
> > > > > > > >
> > > > > > > > The idea of load tearing is crazy talk of integral types.
> It
> > > > would
> > > > > > break so many things.
> > > > > > > > It is the kind of stupid compiler thing that would send
> Linus
> > > on
> > > > a
> > > > > > rant and get
> > > > > > > > the GCC compiler writers in trouble.
> > > > > > > >
> > > > > > > > The DPDK has always favored performance over strict safety
> > > guard
> > > > > > rails everywhere.
> > > > > > > > Switching to making every statistic an atomic operation is
> not
> > > > in
> > > > > > the spirit of
> > > > > > > > what is required. There is no strict guarantee necessary
> here.
> > > > > > > >
> > > > > > >
> > > > > > > I kind of agree with Stephen.
> > > > > > >
> > > > > > > Thanks Mattias, Morten & Stephen, it was informative
> discussion.
> > > > But
> > > > > > for
> > > > > > > *SW drivers* stats update and reset is not core
> functionality
> > > and
> > > > I
> > > > > > > think we can be OK to get hit on corner cases, instead of
> > > > > > > over-engineering or making code more complex.
> > > > > >
> > > > > >
> > > > > > I forgot the case of 64 bit values on 32 bit platforms!
> > > > > > Mostly because haven't cared about 32 bit for years...
> > > > > >
> > > > > > The Linux kernel uses some wrappers to handle this.
> > > > > > On 64 bit platforms they become noop.
> > > > > > On 32 bit platform, they are protected by a seqlock and
> updates
> > > are
> > > > > > wrapped by the sequence count.
> > > > > >
> > > > > > If we go this way, then doing similar Noop on 64 bit and
> atomic or
> > > > > > seqlock
> > > > > > on 32 bit should be done, but in common helper.
> > > > > >
> > > > > > Looking inside FreeBSD, it looks like that has changed over
> the
> > > > years as
> > > > > > well.
> > > > > >
> > > > > > if_inc_counter
> > > > > > counter_u64_add
> > > > > > atomic_add_64
> > > > > > But the counters are always per-cpu in this case. So although
> it
> > > > does
> > > > > > use
> > > > > > locked operation, will always be uncontended.
> > > > > >
> > > > > >
> > > > > > PS: Does DPDK still actually support 32 bit on x86? Can it be
> > > > dropped
> > > > > > this cycle?
> > > > >
> > > > > We cannot drop 32 bit architecture support altogether.
> > > > >
> > > > > But, unlike the Linux kernel, DPDK doesn't need to support
> ancient
> > > 32
> > > > bit architectures.
> > > > > If the few 32 bit architectures supported by DPDK provide non-
> > > tearing
> > > > 64 bit loads/stores, we don't need locks (in the fast path) for 64
> bit
> > > > counters.
> > > > >
> > > > > In addition to 32 bit x86, DPDK supports ARMv7-A (a 32 bit
> > > > architecture) and 32 bit ARMv8.
> > > > > I don't think DPDK support any other 32 bit architectures.
> > > > >
> > > > >
> > > > > As Mattias mentioned, 32 bit x86 can use xmm registers to
> provide 64
> > > > bit non-tearing load/store.
> > > > >
> > > >
> > > > Testing this a little in godbolt, I see gcc using xmm registers on
> 32-
> > > > bit
> > > > when updating 64-bit counters, but clang doesn't seem to do so,
> but
> > > > instead
> > > > does 2 stores when writing back the 64 value. (I tried with both
> > > > volatile
> > > > and non-volatile 64-bit values, just to see if volatile would
> > > encourage
> > > > clang to do a single store).
> > > >
> > > > GCC: https://godbolt.org/z/9eqKfT3hz
> > > > Clang: https://godbolt.org/z/PT5EqKn4c
> > >
> > > Interesting.
> > > I guess this can be fixed by manually implementing what GCC does.
> > >
> > > I'm more concerned about finding a high-performance (in the fast
> path)
> > > 64 bit counter solution for 32 bit ARM.
> >
> > Reading up on the topic, and continuing Bruce's experiment on Godbolt,
> it is possible on 32 bit ARMv7-A too, using LDRD/STRD (Load/Store
> Register Dual) instructions, which load/store 64 bit from memory into
> two registers at once.
> >
> > Clang is emits more efficient code without volatile.
> > GCC requires volatile to use STRD.
> >
> > Clang: https://godbolt.org/z/WjdTq6EKh
> > GCC: https://godbolt.org/z/qq9j7d4Ea
> >
> > Summing it up, it is possible to implement non-tearing 64 bit high-
> performance (lockless, barrier-free) counters on the 32 bit
> architectures supported by DPDK.
> >
> > But the implementation is both architecture and compiler specific.
> > So it seems a "64 bit counters" library would be handy. (Or a "non-
> tearing 64 bit integers" library, for support of the signed variant too;
> but I don't think we need that.)
> > We can use uint64_t as the underlying type and type cast in the
> library (where needed by the specific compiler/architecture), or
> introduce a new rte_ctr64_t type to ensure that accessor functions are
> always used and the developer doesn't have to worry about tearing on 32
> bit architectures.
> >
> > The most simple variant of such a library only provides load and store
> functions. The API would look like this:
> >
> > uint64_t inline
> > rte_ctr64_get(const rte_ctr64_t *const ctr);
> >
> > void inline
> > rte_ctr64_set(rte_ctr64_t *const ctr, const uint64_t value);
> >
> > And if some CPU offers special instructions for increment or addition,
> faster (regarding performance) and/or more compact (regarding
> instruction memory) than a sequence of load-add-store instructions:
> >
> > void inline
> > rte_ctr64_inc(rte_ctr64_t *const ctr);
> >
> > void inline
> > rte_ctr64_add(rte_ctr64_t *const ctr, const uint64_t value);
Note: 32 bit architectures might achieve higher performance if the "value" parameter to rte_ctr64_add() is unsigned long (or unsigned int) instead of uint64_t.
> >
> > <feature creep>
> > And perhaps atomic variants of all these functions, with explicit
> and/or relaxed memory ordering, for counters shared by multiple threads.
> > </feature creep>
> >
>
>
> This kind of what I am experimenting with but...
Excellent!
> Intend to keep the details of the counters read and update in one file
> and not as inlines.
Please note that traffic management applications maintain many counters (e.g. per-flow, per-subscriber and per-link packet and byte counters, some also per QoS class), so rte_ctr64_add() must have the highest performance technically possible.
For reference, the packet scheduling code in our application updates 28 statistics counters per burst of packets. (In addition to internal state variables for e.g. queue lenghts.)
Furthermore, our application processes and displays live statistics with one second granularity, using a separate thread. Although statistics processing is not part of the fast path, the sheer number of counters processed requires high performance read access to those counters.
<more feature creep>
Some groups of counters are maintained locally in the inner loop, and then added in bulk to the "public" statistics afterwards. Conceptually:
struct stats_per_prio {
uint64_t forwarded_bytes;
uint64_t forwarded_packets;
uint64_t marked_bytes;
uint64_t marked_packets;
uint64_t dropped_bytes;
uint64_t dropped_packets;
};
If this is a common design pattern in DPDK (drivers, libraries and/or applications), perhaps also provide a performance optimized function for architectures offering vector instructions:
void inline
rte_ctr64_add_bulk(
rte_ctr64_t *const ctrs,
const unsigned long *const values,
const unsigned int n /* compile time constant */);
This slightly resembles the Linux kernel's design pattern, where counters are updated in bulk, protected by a common lock for the bulk update. (However, DPDK has no lock, so the motivation for optimizing for this design pattern is only "nice to have".)
PS:
Many DPDK applications are 64 bit only, including the SmartShare appliances, and can easily manage 64 bit counters without all this.
However, if the DPDK project still considers 32 bit architectures first class citizens, 64 bit counters should have high performance on 32 bit architectures too.
next prev parent reply other threads:[~2024-05-10 9:14 UTC|newest]
Thread overview: 178+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-25 17:46 [RFC] " Ferruh Yigit
2024-04-26 11:33 ` Morten Brørup
2024-04-26 13:37 ` Ferruh Yigit
2024-04-26 14:56 ` Morten Brørup
2024-04-28 15:42 ` Mattias Rönnblom
2024-04-26 14:38 ` [RFC v2] " Ferruh Yigit
2024-04-26 14:47 ` Morten Brørup
2024-04-28 15:11 ` Mattias Rönnblom
2024-05-01 16:19 ` Ferruh Yigit
2024-05-02 5:51 ` Mattias Rönnblom
2024-05-02 14:22 ` Ferruh Yigit
2024-05-02 15:59 ` Stephen Hemminger
2024-05-02 18:20 ` Ferruh Yigit
2024-05-02 17:37 ` Mattias Rönnblom
2024-05-02 18:26 ` Stephen Hemminger
2024-05-02 21:26 ` Mattias Rönnblom
2024-05-02 21:46 ` Stephen Hemminger
2024-05-07 7:23 ` Mattias Rönnblom
2024-05-07 13:49 ` Ferruh Yigit
2024-05-07 14:51 ` Stephen Hemminger
2024-05-07 16:00 ` Morten Brørup
2024-05-07 16:54 ` Ferruh Yigit
2024-05-07 18:47 ` Stephen Hemminger
2024-05-08 7:48 ` Mattias Rönnblom
2024-05-08 6:28 ` Mattias Rönnblom
2024-05-08 6:25 ` Mattias Rönnblom
2024-05-07 19:19 ` Morten Brørup
2024-05-08 6:34 ` Mattias Rönnblom
2024-05-08 7:10 ` Morten Brørup
2024-05-08 7:23 ` Mattias Rönnblom
2024-04-26 21:28 ` [RFC] " Patrick Robb
2024-05-03 15:45 ` [RFC v3] " Ferruh Yigit
2024-05-03 22:00 ` Stephen Hemminger
2024-05-07 13:48 ` Ferruh Yigit
2024-05-07 14:52 ` Stephen Hemminger
2024-05-07 17:27 ` Ferruh Yigit
2024-05-08 7:19 ` Mattias Rönnblom
2024-05-08 15:23 ` Stephen Hemminger
2024-05-08 19:48 ` Ferruh Yigit
2024-05-08 20:54 ` Stephen Hemminger
2024-05-09 7:43 ` Morten Brørup
2024-05-09 9:29 ` Bruce Richardson
2024-05-09 11:37 ` Morten Brørup
2024-05-09 14:19 ` Morten Brørup
2024-05-10 4:56 ` Stephen Hemminger
2024-05-10 9:14 ` Morten Brørup [this message]
2024-05-26 7:10 ` Mattias Rönnblom
2024-05-26 7:07 ` Mattias Rönnblom
2024-05-26 7:03 ` Mattias Rönnblom
2024-05-26 7:21 ` Mattias Rönnblom
2024-05-07 15:27 ` Morten Brørup
2024-05-07 17:40 ` Ferruh Yigit
2024-05-10 5:01 ` [RFC 0/3] generic sw counters Stephen Hemminger
2024-05-10 5:01 ` [RFC 1/3] ethdev: add internal helper of SW driver statistics Stephen Hemminger
2024-05-10 5:01 ` [RFC 2/3] net/af_packet: use SW stats helper Stephen Hemminger
2024-05-10 5:01 ` [RFC 3/3] net/tap: use generic SW stats Stephen Hemminger
2024-05-10 17:29 ` [RFC 0/3] generic sw counters Morten Brørup
2024-05-10 19:30 ` Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 0/7] generic SW counters Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 1/7] eal: generic 64 bit counter Stephen Hemminger
2024-05-13 19:36 ` Morten Brørup
2024-05-13 18:52 ` [RFC v2 2/7] ethdev: add internal helper of SW driver statistics Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 3/7] net/af_packet: use SW stats helper Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 4/7] net/tap: use generic SW stats Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 5/7] net/pcap: " Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 6/7] net/af_xdp: " Stephen Hemminger
2024-05-13 18:52 ` [RFC v2 7/7] net/ring: " Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 0/7] Generic SW counters Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 1/7] eal: generic 64 bit counter Stephen Hemminger
2024-05-15 9:30 ` Morten Brørup
2024-05-15 15:03 ` Stephen Hemminger
2024-05-15 16:18 ` Morten Brørup
2024-05-26 7:34 ` Mattias Rönnblom
2024-05-26 6:45 ` Mattias Rönnblom
2024-05-14 15:35 ` [PATCH v3 2/7] ethdev: add internal helper of SW driver statistics Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 3/7] net/af_packet: use SW stats helper Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 4/7] net/af_xdp: use generic SW stats Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 5/7] net/pcap: " Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 6/7] net/ring: " Stephen Hemminger
2024-05-14 15:35 ` [PATCH v3 7/7] net/tap: " Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 0/8] Generic 64 bit counters for SW PMD's Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 1/8] eal: generic 64 bit counter Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 2/8] ethdev: add common counters for statistics Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 3/8] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 4/8] net/af_xdp: " Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 5/8] net/pcap: " Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 6/8] net/ring: " Stephen Hemminger
2024-05-15 23:40 ` [PATCH v4 7/8] net/tap: " Stephen Hemminger
2024-05-15 23:41 ` [PATCH v4 8/8] net/null: " Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 0/9] Generic 64 bit counters Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 1/9] eal: generic 64 bit counter Stephen Hemminger
2024-05-16 18:22 ` Wathsala Wathawana Vithanage
2024-05-16 21:42 ` Stephen Hemminger
2024-05-17 2:39 ` Honnappa Nagarahalli
2024-05-17 3:29 ` Stephen Hemminger
2024-05-17 4:39 ` Honnappa Nagarahalli
2024-05-16 15:40 ` [PATCH v5 2/9] ethdev: add common counters for statistics Stephen Hemminger
2024-05-16 18:30 ` Wathsala Wathawana Vithanage
2024-05-17 0:19 ` Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 3/9] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 4/9] net/af_xdp: " Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 5/9] net/pcap: " Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 6/9] test/pmd_ring: initialize mbufs Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 7/9] net/ring: use generic SW stats Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 8/9] net/tap: " Stephen Hemminger
2024-05-16 15:40 ` [PATCH v5 9/9] net/null: " Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 0/9] Generic 64 bit counters for SW drivers Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 1/9] eal: generic 64 bit counter Stephen Hemminger
2024-05-17 2:45 ` Honnappa Nagarahalli
2024-05-17 3:30 ` Stephen Hemminger
2024-05-17 4:26 ` Honnappa Nagarahalli
2024-05-17 6:44 ` Morten Brørup
2024-05-17 15:05 ` Stephen Hemminger
2024-05-17 16:18 ` Stephen Hemminger
2024-05-18 14:00 ` Morten Brørup
2024-05-19 15:13 ` Stephen Hemminger
2024-05-19 17:10 ` Morten Brørup
2024-05-19 22:49 ` Stephen Hemminger
2024-05-20 7:57 ` Morten Brørup
2024-05-17 15:07 ` Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 2/9] ethdev: add common counters for statistics Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 3/9] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 4/9] net/af_xdp: " Stephen Hemminger
2024-05-17 13:34 ` Loftus, Ciara
2024-05-17 14:54 ` Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 5/9] net/pcap: " Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 6/9] test/pmd_ring: initialize mbufs Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 7/9] net/ring: use generic SW stats Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 8/9] net/tap: " Stephen Hemminger
2024-05-17 0:12 ` [PATCH v6 9/9] net/null: " Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 0/9] Use weak atomic operations for SW PMD counters Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 1/9] eal: generic 64 bit counter Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 2/9] ethdev: add common counters for statistics Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 3/9] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 4/9] net/af_xdp: " Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 5/9] net/pcap: " Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 6/9] test/pmd_ring: initialize mbufs Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 7/9] net/ring: use generic SW stats Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 8/9] net/tap: " Stephen Hemminger
2024-05-17 17:35 ` [PATCH v7 9/9] net/null: " Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 0/8] Common statistics routines for SW based PMD's Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 1/8] eal: generic 64 bit counter Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 2/8] ethdev: add common counters for statistics Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 3/8] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 4/8] net/af_xdp: " Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 5/8] net/pcap: " Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 6/8] net/ring: " Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 7/8] net/tap: " Stephen Hemminger
2024-05-21 17:00 ` [PATCH v8 8/8] net/null: " Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 0/8] Common statistics for SW PMD's Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 1/8] eal: generic 64 bit counter Stephen Hemminger
2024-05-22 8:31 ` Morten Brørup
2024-05-22 15:33 ` Stephen Hemminger
2024-05-22 18:09 ` Morten Brørup
2024-05-22 19:53 ` Stephen Hemminger
2024-05-22 20:56 ` Morten Brørup
2024-05-22 15:37 ` Stephen Hemminger
2024-05-22 17:57 ` Morten Brørup
2024-05-22 19:01 ` Tyler Retzlaff
2024-05-22 19:51 ` Stephen Hemminger
2024-05-26 14:46 ` Mattias Rönnblom
2024-05-26 14:39 ` Mattias Rönnblom
2024-05-21 20:16 ` [PATCH v9 2/8] ethdev: add common counters for statistics Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 3/8] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 4/8] net/af_xdp: " Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 5/8] net/pcap: " Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 6/8] net/ring: " Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 7/8] net/tap: " Stephen Hemminger
2024-05-21 20:16 ` [PATCH v9 8/8] net/null: " Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 0/8] Common statistics for software PMD's Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 1/8] eal: generic 64 bit counter Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 2/8] ethdev: add common counters for statistics Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 3/8] net/af_packet: use generic SW stats Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 4/8] net/af_xdp: " Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 5/8] net/pcap: " Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 6/8] net/ring: " Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 7/8] net/tap: " Stephen Hemminger
2024-05-22 16:12 ` [PATCH v10 8/8] net/null: " Stephen Hemminger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35E9F434@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@amd.com \
--cc=hofors@lysator.liu.se \
--cc=linville@tuxdriver.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).