DPDK patches and discussions
 help / color / mirror / Atom feed
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	"olivier.matz@6wind.com" <olivier.matz@6wind.com>,
	"sthemmin@microsoft.com" <sthemmin@microsoft.com>,
	"jerinj@marvell.com" <jerinj@marvell.com>,
	"Richardson, Bruce" <bruce.richardson@intel.com>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"pbhagavatula@marvell.com" <pbhagavatula@marvell.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	Dharmik Thakkar <Dharmik.Thakkar@arm.com>,
	 Ruifeng Wang <Ruifeng.Wang@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
	nd <nd@arm.com>,
	Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
	nd <nd@arm.com>
Subject: Re: [dpdk-dev] [PATCH v7 02/17] lib/ring: apis to support configurable element size
Date: Wed, 8 Jan 2020 23:40:05 +0000	[thread overview]
Message-ID: <VE1PR08MB514964E8E611BAE63EF90C99983E0@VE1PR08MB5149.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <SN6PR11MB2558806861F0DE439863544F9A3E0@SN6PR11MB2558.namprd11.prod.outlook.com>

<snip>
> > > > > > > > > +
> > > > > > > > > +static __rte_always_inline void
> > > > > > > > > +enqueue_elems_128(struct rte_ring *r, uint32_t
> > > > > > > > > +prod_head, const void *obj_table, uint32_t n) {
> > > > > > > > > +unsigned int i; const uint32_t size =
> > > > > > > > > +r->size; uint32_t idx = prod_head & r->mask;
> > > > > > > > > +r->__uint128_t
> > > > > > > > > +*ring = (__uint128_t *)&r[1]; const __uint128_t *obj =
> > > > > > > > > +(const __uint128_t *)obj_table; if (likely(idx + n <
> > > > > > > > > +size)) { for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
> > > > > > > > > +{ ring[idx] = obj[i]; ring[idx + 1] = obj[i + 1];
> > > > > > > >
> > > > > > > >
> > > > > > > > AFAIK, that implies 16B aligned obj_table...
> > > > > > > > Would it always be the case?
> > > > > > > I am not sure from the compiler perspective.
> > > > > > > At least on Arm architecture, unaligned access (address that
> > > > > > > is accessed is not aligned to the size of the data element
> > > > > > > being
> > > > > > > accessed) will result in faults or require additional cycles.
> > > > > > > So, aligning on
> > > > > 16B should be fine.
> > > > > > Further, I would be changing this to use 'rte_int128_t' as
> > > > > > '__uint128_t' is
> > > > > not defined on 32b systems.
> > > > >
> > > > > What I am trying to say: with this code we imply new requirement
> > > > > for elems
> > > > The only existing use case in DPDK for 16B is the event ring. The
> > > > event ring
> > > already does similar kind of copy (using 'struct rte_event').
> > > > So, there is no change in expectations for event ring.
> > > > For future code, I think this expectation should be fine since it
> > > > allows for
> > > optimal code.
> > > >
> > > > > in the ring: when sizeof(elem)==16 it's alignment also has to be
> > > > > at least
> > > 16.
> > > > > Which from my perspective is not ideal.
> > > > Any reasoning?
> > >
> > > New implicit requirement and inconsistency.
> > > Code like that:
> > >
> > > struct ring_elem {uint64_t a, b;};
> > > ....
> > > struct ring_elem elem;
> > > rte_ring_dequeue_elem(ring, &elem, sizeof(elem));
> > >
> > > might cause a crash.
> > The alignment here is 8B. Assuming that instructions generated will
> > require 16B alignment, it will result in a crash, if configured to generate
> exception.
> > But, these instructions are not atomic instructions. At least on
> > aarch64, unaligned access will not result in an exception for non-atomic
> loads/stores. I believe it is the same behavior for x86 as well.
> 
> On IA, there are 2 types of 16B load/store instructions: aligned and unaligned.
> Aligned are a bit faster, but will cause an exception if used on non 16B aligned
> address.
> As you using uint128_t * compiler will assume that both src and dst are 16B
> aligned and might generate code with aligned instructions.
Ok, looking at few articles, I read that if the address is aligned, the unaligned instructions do not incur the penalty. Is this understanding correct?

I see 2 solutions here:
1) We can switch this copy to use uint32_t pointer. It would still allow the compiler to generate (unaligned) instructions for up to 256b load/store. The 2 multiplications (to normalize the index and the size of copy) can use shifts. This should make it safer. If one wants performance, they can align the obj table to 16B (the ring itself is already aligned on the cache line boundary).

2) Considering that performance is paramount, we could document that the obj table needs to be aligned on 16B boundary. This would affect event dev (if we go ahead with replacing the event ring implementation) significantly.

Note that we have to do the same thing for 64b elements as well.

> 
> >
> > > While exactly the same code with:
> > >
> > > struct ring_elem {uint64_t a, b, c;}; OR struct ring_elem {uint64_t
> > > a, b, c, d;};
> > >
> > > will work ok.
> > The alignment for these structures is still 8B. Are you saying this
> > will work because these will be copied using pointer to uint32_t (whose
> alignment is 4B)?
> 
> Yes, as we doing uint32_t copies, compiler can't assume the data will be 16B
> aligned and will use unaligned instructions.
> 
> >
> > >
> > > >
> > > > > Note that for elem sizes > 16 (24, 32), there is no such constraint.
> > > > The rest of them need to be aligned on 4B boundary. However, this
> > > > should
> > > not affect the existing code.
> > > > The code for 8B and 16B is kept as is to ensure the performance is
> > > > not
> > > affected for the existing code.
> > <snip>


  reply	other threads:[~2020-01-08 23:40 UTC|newest]

Thread overview: 173+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-28 14:46 [dpdk-dev] [PATCH 0/5] lib/ring: templates to support custom " Honnappa Nagarahalli
2019-08-28 14:46 ` [dpdk-dev] [PATCH 1/5] lib/ring: apis to support configurable " Honnappa Nagarahalli
2019-08-28 14:46 ` [dpdk-dev] [PATCH 2/5] lib/ring: add template to support different element sizes Honnappa Nagarahalli
2019-10-01 11:47   ` Ananyev, Konstantin
2019-10-02  4:21     ` Honnappa Nagarahalli
2019-10-02  8:39       ` Ananyev, Konstantin
2019-10-03  3:33         ` Honnappa Nagarahalli
2019-10-03 11:51           ` Ananyev, Konstantin
2019-10-03 12:27             ` Ananyev, Konstantin
2019-10-03 22:49               ` Honnappa Nagarahalli
2019-08-28 14:46 ` [dpdk-dev] [PATCH 3/5] tools/checkpatch: relax constraints on __rte_experimental Honnappa Nagarahalli
2019-08-28 14:46 ` [dpdk-dev] [PATCH 4/5] lib/ring: add ring APIs to support 32b ring elements Honnappa Nagarahalli
2019-08-28 14:46 ` [dpdk-dev] [PATCH 5/5] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2019-08-28 15:12 ` [dpdk-dev] [PATCH 0/5] lib/ring: templates to support custom element size Jerin Jacob Kollanukkaran
2019-08-28 15:16 ` Pavan Nikhilesh Bhagavatula
2019-08-28 22:59   ` Honnappa Nagarahalli
2019-09-06 19:05 ` [dpdk-dev] [PATCH v2 0/6] " Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 1/6] lib/ring: apis to support configurable " Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 2/6] lib/ring: add template to support different element sizes Honnappa Nagarahalli
2019-09-08 19:44     ` Stephen Hemminger
2019-09-09  9:01       ` Bruce Richardson
2019-09-09 22:33         ` Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 3/6] tools/checkpatch: relax constraints on __rte_experimental Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 4/6] lib/ring: add ring APIs to support 32b ring elements Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 5/6] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2019-09-06 19:05   ` [dpdk-dev] [PATCH v2 6/6] lib/eventdev: use ring templates for event rings Honnappa Nagarahalli
2019-09-09 13:04   ` [dpdk-dev] [PATCH v2 0/6] lib/ring: templates to support custom element size Aaron Conole
2019-10-07 13:49   ` David Marchand
2019-10-08 19:19   ` [dpdk-dev] [PATCH v3 0/2] lib/ring: APIs " Honnappa Nagarahalli
2019-10-08 19:19     ` [dpdk-dev] [PATCH v3 1/2] lib/ring: apis to support configurable " Honnappa Nagarahalli
2019-10-08 19:19     ` [dpdk-dev] [PATCH v3 2/2] test/ring: add test cases for configurable element size ring Honnappa Nagarahalli
2019-10-09  2:47   ` [dpdk-dev] [PATCH v3 0/2] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2019-10-09  2:47     ` [dpdk-dev] [PATCH v4 1/2] lib/ring: apis to support configurable " Honnappa Nagarahalli
2019-10-11 19:21       ` Honnappa Nagarahalli
2019-10-14 19:41         ` Ananyev, Konstantin
2019-10-14 23:56           ` Honnappa Nagarahalli
2019-10-15  9:34             ` Ananyev, Konstantin
2019-10-17  4:46               ` Honnappa Nagarahalli
2019-10-17 11:51                 ` Ananyev, Konstantin
2019-10-17 20:16                   ` Honnappa Nagarahalli
2019-10-17 23:17                     ` David Christensen
2019-10-18  3:18                       ` Honnappa Nagarahalli
2019-10-18  8:04                         ` Jerin Jacob
2019-10-18 16:11                           ` Jerin Jacob
2019-10-21  0:27                             ` Honnappa Nagarahalli
2019-10-18 16:44                           ` Ananyev, Konstantin
2019-10-18 19:03                             ` Honnappa Nagarahalli
2019-10-21  0:36                             ` Honnappa Nagarahalli
2019-10-21  9:04                               ` Ananyev, Konstantin
2019-10-22 15:59                                 ` Ananyev, Konstantin
2019-10-22 17:57                                   ` Ananyev, Konstantin
2019-10-23 18:58                                     ` Honnappa Nagarahalli
2019-10-18 17:23                         ` David Christensen
2019-10-09  2:47     ` [dpdk-dev] [PATCH v4 2/2] test/ring: add test cases for configurable element size ring Honnappa Nagarahalli
2019-10-17 20:08   ` [dpdk-dev] [PATCH v5 0/3] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2019-10-17 20:08     ` [dpdk-dev] [PATCH v5 1/3] lib/ring: apis to support configurable " Honnappa Nagarahalli
2019-10-17 20:39       ` Stephen Hemminger
2019-10-17 20:40       ` Stephen Hemminger
2019-10-17 20:08     ` [dpdk-dev] [PATCH v5 2/3] test/ring: add test cases for configurable element size ring Honnappa Nagarahalli
2019-10-17 20:08     ` [dpdk-dev] [PATCH v5 3/3] lib/ring: copy ring elements using memcpy partially Honnappa Nagarahalli
2019-10-21  0:22   ` [dpdk-dev] [RFC v6 0/6] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2019-10-21  0:22     ` [dpdk-dev] [RFC v6 1/6] test/ring: use division for cycle count calculation Honnappa Nagarahalli
2019-10-23  9:49       ` Olivier Matz
2019-10-21  0:22     ` [dpdk-dev] [RFC v6 2/6] lib/ring: apis to support configurable element size Honnappa Nagarahalli
2019-10-23  9:59       ` Olivier Matz
2019-10-23 19:12         ` Honnappa Nagarahalli
2019-10-21  0:22     ` [dpdk-dev] [RFC v6 3/6] test/ring: add functional tests for configurable element size ring Honnappa Nagarahalli
2019-10-23 10:01       ` Olivier Matz
2019-10-23 11:12         ` Ananyev, Konstantin
2019-10-21  0:22     ` [dpdk-dev] [RFC v6 4/6] test/ring: add perf " Honnappa Nagarahalli
2019-10-23 10:02       ` Olivier Matz
2019-10-21  0:22     ` [dpdk-dev] [RFC v6 5/6] lib/ring: copy ring elements using memcpy partially Honnappa Nagarahalli
2019-10-21  0:23     ` [dpdk-dev] [RFC v6 6/6] lib/ring: improved copy function to copy ring elements Honnappa Nagarahalli
2019-10-23 10:05       ` Olivier Matz
2019-10-23  9:48     ` [dpdk-dev] [RFC v6 0/6] lib/ring: APIs to support custom element size Olivier Matz
2019-12-20  4:45   ` [dpdk-dev] [PATCH v7 00/17] " Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 01/17] test/ring: use division for cycle count calculation Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 02/17] lib/ring: apis to support configurable element size Honnappa Nagarahalli
2020-01-02 16:42       ` Ananyev, Konstantin
2020-01-07  5:35         ` Honnappa Nagarahalli
2020-01-07  6:00           ` Honnappa Nagarahalli
2020-01-07 10:21             ` Ananyev, Konstantin
2020-01-07 15:21               ` Honnappa Nagarahalli
2020-01-07 15:41                 ` Ananyev, Konstantin
2020-01-08  6:17                   ` Honnappa Nagarahalli
2020-01-08 10:05                     ` Ananyev, Konstantin
2020-01-08 23:40                       ` Honnappa Nagarahalli [this message]
2020-01-09  0:48                         ` Ananyev, Konstantin
2020-01-09 16:06                           ` Honnappa Nagarahalli
2020-01-13 11:53                             ` Ananyev, Konstantin
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 03/17] test/ring: add functional tests for rte_ring_xxx_elem APIs Honnappa Nagarahalli
2020-01-02 16:31       ` Ananyev, Konstantin
2020-01-07  5:13         ` Honnappa Nagarahalli
2020-01-07 16:03           ` Ananyev, Konstantin
2020-01-09  5:15             ` Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 04/17] test/ring: test burst APIs with random empty-full test case Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 05/17] test/ring: add default, single element test cases Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 06/17] test/ring: rte_ring_xxx_elem test cases for exact size ring Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 07/17] test/ring: negative test cases for rte_ring_xxx_elem APIs Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 08/17] test/ring: remove duplicate test cases Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 09/17] test/ring: removed unused variable synchro Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 10/17] test/ring: modify single element enq/deq perf test cases Honnappa Nagarahalli
2020-01-02 17:03       ` Ananyev, Konstantin
2020-01-07  5:54         ` Honnappa Nagarahalli
2020-01-07 16:13           ` Ananyev, Konstantin
2020-01-07 22:33             ` Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 11/17] test/ring: modify burst " Honnappa Nagarahalli
2020-01-02 16:57       ` Ananyev, Konstantin
2020-01-07  5:42         ` Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 12/17] test/ring: modify bulk " Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 13/17] test/ring: modify bulk empty deq " Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 14/17] test/ring: modify multi-lcore " Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 15/17] test/ring: adjust run-on-all-cores " Honnappa Nagarahalli
2020-01-02 17:00       ` Ananyev, Konstantin
2020-01-07  5:42         ` Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 16/17] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2019-12-20  4:45     ` [dpdk-dev] [PATCH v7 17/17] lib/eventdev: use custom element size ring for event rings Honnappa Nagarahalli
2020-01-13 17:25   ` [dpdk-dev] [PATCH v8 0/6] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 1/6] test/ring: use division for cycle count calculation Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 2/6] lib/ring: apis to support configurable element size Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 3/6] test/ring: add functional tests for rte_ring_xxx_elem APIs Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 4/6] test/ring: modify perf test cases to use " Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 5/6] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2020-01-13 17:25     ` [dpdk-dev] [PATCH v8 6/6] lib/eventdev: use custom element size ring for event rings Honnappa Nagarahalli
     [not found]       ` <1578977880-13011-1-git-send-email-robot@bytheb.org>
     [not found]         ` <VE1PR08MB5149BE79083CD66A41CBD6D198340@VE1PR08MB5149.eurprd08.prod.outlook.com>
2020-01-14 15:12           ` [dpdk-dev] FW: || pw64572 " Aaron Conole
2020-01-14 16:51             ` Aaron Conole
2020-01-14 19:35               ` Honnappa Nagarahalli
2020-01-14 20:44                 ` Aaron Conole
2020-01-15  0:55                   ` Honnappa Nagarahalli
2020-01-15  4:43                   ` Honnappa Nagarahalli
2020-01-15  5:05                     ` Honnappa Nagarahalli
2020-01-15 18:22                       ` Aaron Conole
2020-01-15 18:38                         ` Honnappa Nagarahalli
2020-01-16  5:27                           ` Honnappa Nagarahalli
2020-01-16  5:25   ` [dpdk-dev] [PATCH v9 0/6] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 1/6] test/ring: use division for cycle count calculation Honnappa Nagarahalli
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 2/6] lib/ring: apis to support configurable element size Honnappa Nagarahalli
2020-01-17 16:34       ` Olivier Matz
2020-01-17 16:45         ` Honnappa Nagarahalli
2020-01-17 18:10           ` David Christensen
2020-01-18 12:32           ` Ananyev, Konstantin
2020-01-18 15:01             ` Honnappa Nagarahalli
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 3/6] test/ring: add functional tests for rte_ring_xxx_elem APIs Honnappa Nagarahalli
2020-01-17 17:03       ` Olivier Matz
2020-01-18 16:27         ` Honnappa Nagarahalli
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 4/6] test/ring: modify perf test cases to use " Honnappa Nagarahalli
2020-01-17 17:12       ` Olivier Matz
2020-01-18 16:28         ` Honnappa Nagarahalli
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 5/6] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2020-01-17 20:27       ` David Marchand
2020-01-17 20:54         ` Honnappa Nagarahalli
2020-01-17 21:07           ` David Marchand
2020-01-17 22:24             ` Wang, Yipeng1
2020-01-16  5:25     ` [dpdk-dev] [PATCH v9 6/6] lib/eventdev: use custom element size ring for event rings Honnappa Nagarahalli
2020-01-17 14:41       ` Jerin Jacob
2020-01-17 16:12         ` David Marchand
2020-01-16 16:36     ` [dpdk-dev] [PATCH v9 0/6] lib/ring: APIs to support custom element size Honnappa Nagarahalli
2020-01-17 12:14       ` David Marchand
2020-01-17 13:34         ` Jerin Jacob
2020-01-17 16:37           ` Mattias Rönnblom
2020-01-17 14:28         ` Honnappa Nagarahalli
2020-01-17 14:36           ` Honnappa Nagarahalli
2020-01-17 16:15           ` David Marchand
2020-01-17 16:32             ` Honnappa Nagarahalli
2020-01-17 17:15     ` Olivier Matz
2020-01-18 19:32   ` [dpdk-dev] [PATCH v10 " Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 1/6] test/ring: use division for cycle count calculation Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 2/6] lib/ring: apis to support configurable element size Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 3/6] test/ring: add functional tests for rte_ring_xxx_elem APIs Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 4/6] test/ring: modify perf test cases to use " Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 5/6] lib/hash: use ring with 32b element size to save memory Honnappa Nagarahalli
2020-01-18 19:32     ` [dpdk-dev] [PATCH v10 6/6] eventdev: use custom element size ring for event rings Honnappa Nagarahalli
2020-01-19 19:31     ` [dpdk-dev] [PATCH v10 0/6] lib/ring: APIs to support custom element size David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=VE1PR08MB514964E8E611BAE63EF90C99983E0@VE1PR08MB5149.eurprd08.prod.outlook.com \
    --to=honnappa.nagarahalli@arm.com \
    --cc=Dharmik.Thakkar@arm.com \
    --cc=Gavin.Hu@arm.com \
    --cc=Ruifeng.Wang@arm.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=nd@arm.com \
    --cc=olivier.matz@6wind.com \
    --cc=pbhagavatula@marvell.com \
    --cc=sthemmin@microsoft.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).