From: Bruce Richardson <bruce.richardson@intel.com>
To: "Mattias Rönnblom" <hofors@lysator.liu.se>
Cc: <dev@dpdk.org>, <jerinj@marvell.com>,
<mattias.ronnblom@ericsson.com>, <abdullah.sevincer@intel.com>,
<sachin.saxena@oss.nxp.com>, <hemant.agrawal@nxp.com>,
<pbhagavatula@marvell.com>, <pravin.pathak@intel.com>
Subject: Re: [PATCH v2 11/11] eventdev: RFC clarify docs on event object fields
Date: Thu, 1 Feb 2024 16:59:26 +0000 [thread overview]
Message-ID: <ZbvN7l8he2O5btYa@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <211a59b9-095d-418e-be82-b49f4e5d1d00@lysator.liu.se>
On Wed, Jan 24, 2024 at 12:34:50PM +0100, Mattias Rönnblom wrote:
> On 2024-01-19 18:43, Bruce Richardson wrote:
> > Clarify the meaning of the NEW, FORWARD and RELEASE event types.
> > For the fields in "rte_event" struct, enhance the comments on each to
> > clarify the field's use, and whether it is preserved between enqueue and
> > dequeue, and it's role, if any, in scheduling.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >
> > As with the previous patch, please review this patch to ensure that the
> > expected semantics of the various event types and event fields have not
> > changed in an unexpected way.
> > ---
> > lib/eventdev/rte_eventdev.h | 105 ++++++++++++++++++++++++++----------
> > 1 file changed, 77 insertions(+), 28 deletions(-)
> >
> > diff --git a/lib/eventdev/rte_eventdev.h b/lib/eventdev/rte_eventdev.h
> > index cb13602ffb..4eff1c4958 100644
> > --- a/lib/eventdev/rte_eventdev.h
> > +++ b/lib/eventdev/rte_eventdev.h
> > @@ -1416,21 +1416,25 @@ struct rte_event_vector {
> >
> > /* Event enqueue operations */
> > #define RTE_EVENT_OP_NEW 0
> > -/**< The event producers use this operation to inject a new event to the
> > +/**< The @ref rte_event.op field should be set to this type to inject a new event to the
> > * event device.
> > */
>
> "type" -> "value"
>
> "to" -> "into"?
>
> You could also say "to mark the event as new".
>
> What is new? Maybe "new (as opposed to a forwarded) event." or "new (i.e.,
> not previously dequeued).".
>
Using this latter suggested wording in V3.
> "The application sets the @ref rte_event.op field of an enqueued event to
> this value to mark the event as new (i.e., not previously dequeued)."
>
> > #define RTE_EVENT_OP_FORWARD 1
> > -/**< The CPU use this operation to forward the event to different event queue or
> > - * change to new application specific flow or schedule type to enable
> > - * pipelining.
> > +/**< SW should set the @ref rte_event.op filed to this type to return a
> > + * previously dequeued event to the event device for further processing.
>
> "filed" -> "field"
>
> "SW" -> "The application"
>
> "to be scheduled for further processing (or transmission)"
>
> The wording should otherwise be the same as NEW, whatever you choose there.
>
Ack.
> > *
> > - * This operation must only be enqueued to the same port that the
> > + * This event *must* be enqueued to the same port that the
> > * event to be forwarded was dequeued from.
>
> OK, so you "should" mark a new event RTE_EVENT_OP_FORWARD but you "*must*"
> enqueue it to the same port.
>
> I think you "must" do both.
>
Ack
> > + *
> > + * The event's fields, including (but not limited to) flow_id, scheduling type,
> > + * destination queue, and event payload e.g. mbuf pointer, may all be updated as
> > + * desired by software, but the @ref rte_event.impl_opaque field must
>
> "software" -> "application"
>
Ack
> > + * be kept to the same value as was present when the event was dequeued.
> > */
> > #define RTE_EVENT_OP_RELEASE 2
> > /**< Release the flow context associated with the schedule type.
> > *
> > - * If current flow's scheduler type method is *RTE_SCHED_TYPE_ATOMIC*
> > + * If current flow's scheduler type method is @ref RTE_SCHED_TYPE_ATOMIC
> > * then this function hints the scheduler that the user has completed critical
> > * section processing in the current atomic context.
> > * The scheduler is now allowed to schedule events from the same flow from
> > @@ -1442,21 +1446,19 @@ struct rte_event_vector {
> > * performance, but the user needs to design carefully the split into critical
> > * vs non-critical sections.
> > *
> > - * If current flow's scheduler type method is *RTE_SCHED_TYPE_ORDERED*
> > - * then this function hints the scheduler that the user has done all that need
> > - * to maintain event order in the current ordered context.
> > - * The scheduler is allowed to release the ordered context of this port and
> > - * avoid reordering any following enqueues.
> > - *
> > - * Early ordered context release may increase parallelism and thus system
> > - * performance.
> > + * If current flow's scheduler type method is @ref RTE_SCHED_TYPE_ORDERED
>
> Isn't a missing "or @ref RTE_SCHED_TYPE_ATOMIC" just an oversight (in the
> original API wording)?
>
No, I don't think so, because ATOMIC is described above.
> > + * then this function informs the scheduler that the current event has
> > + * completed processing and will not be returned to the scheduler, i.e.
> > + * it has been dropped, and so the reordering context for that event
> > + * should be considered filled.
> > *
> > - * If current flow's scheduler type method is *RTE_SCHED_TYPE_PARALLEL*
> > + * If current flow's scheduler type method is @ref RTE_SCHED_TYPE_PARALLEL
> > * or no scheduling context is held then this function may be an NOOP,
> > * depending on the implementation.
>
> Maybe you can also fix this "function" -> "operation". I suggest you delete
> that sentence, because it makes no sense.
>
> What is says in a somewhat vague manner is that you tread into the realm of
> undefined behavior if you release PARALLEL events.
>
Agree. Just deleting.
> > *
> > * This operation must only be enqueued to the same port that the
> > - * event to be released was dequeued from.
> > + * event to be released was dequeued from. The @ref rte_event.impl_opaque
> > + * field in the release event must match that in the original dequeued event.
>
> I would say "same value" rather than "match".
>
> "The @ref rte_event.impl_opaque field in the release event have the same
> value as in the original dequeued event."
>
Ack.
> > */
> >
> > /**
> > @@ -1473,53 +1475,100 @@ struct rte_event {
> > /**< Targeted flow identifier for the enqueue and
> > * dequeue operation.
> > * The value must be in the range of
> > - * [0, nb_event_queue_flows - 1] which
> > + * [0, @ref rte_event_dev_config.nb_event_queue_flows - 1] which
>
> The same comment as I had before about ranges for unsigned types.
>
Ack.
> > * previously supplied to rte_event_dev_configure().
> > + *
> > + * For @ref RTE_SCHED_TYPE_ATOMIC, this field is used to identify a
> > + * flow context for atomicity, such that events from each individual flow
> > + * will only be scheduled to one port at a time.
>
> flow_id alone doesn't identify an atomic flow. It's queue_id + flow_id. I'm
> not sure I think "flow context" adds much, unless it's defined somewhere.
> Sounds like some assumed implementation detail.
>
Removing the word context, and adding that it identifies a flow "within a
queue and priority level", to make it clear that it's just not the flow_id
involved here, as you say.
> > + *
> > + * This field is preserved between enqueue and dequeue when
> > + * a device reports the @ref RTE_EVENT_DEV_CAP_CARRY_FLOW_ID
> > + * capability. Otherwise the value is implementation dependent
> > + * on dequeue > */
> > uint32_t sub_event_type:8;
> > /**< Sub-event types based on the event source.
> > + *
> > + * This field is preserved between enqueue and dequeue.
> > + * This field is for SW or event adapter use,
>
> "SW" -> "application"
>
Ack.
> > + * and is unused in scheduling decisions.
> > + *
> > * @see RTE_EVENT_TYPE_CPU
> > */
> > uint32_t event_type:4;
> > - /**< Event type to classify the event source.
> > - * @see RTE_EVENT_TYPE_ETHDEV, (RTE_EVENT_TYPE_*)
> > + /**< Event type to classify the event source. (RTE_EVENT_TYPE_*)
> > + *
> > + * This field is preserved between enqueue and dequeue
> > + * This field is for SW or event adapter use,
> > + * and is unused in scheduling decisions.
>
> "unused" -> "is not considered"?
>
Ack.
> > */
> > uint8_t op:2;
> > - /**< The type of event enqueue operation - new/forward/
> > - * etc.This field is not preserved across an instance
> > + /**< The type of event enqueue operation - new/forward/ etc.
> > + *
> > + * This field is *not* preserved across an instance
> > * and is undefined on dequeue.
>
> Maybe you should use "undefined" rather than "implementation dependent", or
> change this instance of undefined to implementation dependent. Now two
> different terms are used for the same thing.
>
Using implementation dependent.
Ideally, I think we should update all drivers to set this to "FORWARD" by
default on dequeue, but for now it's "implementation dependent".
> > - * @see RTE_EVENT_OP_NEW, (RTE_EVENT_OP_*)
> > + *
> > + * @see RTE_EVENT_OP_NEW
> > + * @see RTE_EVENT_OP_FORWARD
> > + * @see RTE_EVENT_OP_RELEASE
> > */
> > uint8_t rsvd:4;
> > - /**< Reserved for future use */
> > + /**< Reserved for future use.
> > + *
> > + * Should be set to zero on enqueue. Zero on dequeue.
> > + */
>
> Why say they should be zero on dequeue? Doesn't this defeat the purpose of
> having reserved bits, for future use? With you suggested wording, you can't
> use these bits without breaking the ABI.
Good point. Removing the dequeue value bit.
>
> > uint8_t sched_type:2;
> > /**< Scheduler synchronization type (RTE_SCHED_TYPE_*)
> > * associated with flow id on a given event queue
> > * for the enqueue and dequeue operation.
> > + *
> > + * This field is used to determine the scheduling type
> > + * for events sent to queues where @ref RTE_EVENT_QUEUE_CFG_ALL_TYPES
> > + * is supported.
>
> "supported" -> "configured"
>
Ack.
> > + * For queues where only a single scheduling type is available,
> > + * this field must be set to match the configured scheduling type.
> > + *
>
> Why is the API/event device asking this of the application?
>
Historical reasons. I agree that it shouldn't, this should just be marked
as ignored on fixed-type queues, but the spec up till now says it must
match and some drivers do check this. Once we update the drivers to stop
checking then we can change the spec without affecting apps.
> > + * This field is preserved between enqueue and
> > dequeue. + * + * @see
> > RTE_SCHED_TYPE_ORDERED + * @see
> > RTE_SCHED_TYPE_ATOMIC + * @see
> > RTE_SCHED_TYPE_PARALLEL */ uint8_t queue_id; /**< Targeted event queue
> > identifier for the enqueue or * dequeue operation. * The value must be
> > in the range of - * [0, nb_event_queues - 1] which
> > previously supplied to - *
> > rte_event_dev_configure(). + * [0, @ref
> > rte_event_dev_config.nb_event_queues - 1] which was +
> > * previously supplied to rte_event_dev_configure(). +
> > * + * This field is preserved between enqueue on
> > dequeue. */ uint8_t priority; /**< Event priority relative to other
> > events in the * event queue. The requested priority should in the -
> > * range of [RTE_EVENT_DEV_PRIORITY_HIGHEST, - *
> > RTE_EVENT_DEV_PRIORITY_LOWEST]. + * range of [@ref
> > RTE_EVENT_DEV_PRIORITY_HIGHEST, + * @ref
> > RTE_EVENT_DEV_PRIORITY_LOWEST]. * The implementation shall normalize
> > the requested * priority to supported priority value. +
> > * * Valid when the device has - *
> > RTE_EVENT_DEV_CAP_EVENT_QOS capability. + * @ref
> > RTE_EVENT_DEV_CAP_EVENT_QOS capability. + * Ignored
> > otherwise. + * + * This
> > field is preserved between enqueue and dequeue.
>
> Is the normalized or unnormalized value that is preserved?
>
Very good point. It's the normalized & then denormalised version that is
guaranteed to be preserved, I suspect. SW eventdevs probably preserve
as-is, but HW eventdevs may lose precision. Rather than making this
"implementation defined" or "not preserved" which would be annoying for
apps, I think, I'm going to document this as "preserved, but with possible
loss of precision".
> > */
> > uint8_t impl_opaque;
> > /**< Implementation specific opaque value.
>
> Maybe you can also fix "implementation" here to be something more specific.
> Implementation, of what?
>
> "Event device implementation" or just "event device".
>
"Opaque field for event device use"
> > + *
> > * An implementation may use this field to hold
> > * implementation specific value to share between
> > * dequeue and enqueue operation.
> > + *
> > * The application should not modify this field.
> > + * Its value is implementation dependent on dequeue,
> > + * and must be returned unmodified on enqueue when
> > + * op type is @ref RTE_EVENT_OP_FORWARD or @ref RTE_EVENT_OP_RELEASE
>
> Should it be mentioned that impl_opaque is ignored by the event device for
> NEW events?
>
Added in V3.
> > */
> > };
> > };
> > --
> > 2.40.1
> >
next prev parent reply other threads:[~2024-02-01 16:59 UTC|newest]
Thread overview: 123+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-18 13:45 [PATCH v1 0/7] improve eventdev API specification/documentation Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 1/7] eventdev: improve doxygen introduction text Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 2/7] eventdev: move text on driver internals to proper section Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 3/7] eventdev: update documentation on device capability flags Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 4/7] eventdev: cleanup doxygen comments on info structure Bruce Richardson
2024-01-18 13:49 ` Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 5/7] eventdev: improve function documentation for query fns Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 6/7] eventdev: improve doxygen comments on configure struct Bruce Richardson
2024-01-18 13:45 ` [PATCH v1 7/7] eventdev: fix documentation for counting single-link ports Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 00/11] improve eventdev API specification/documentation Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 01/11] eventdev: improve doxygen introduction text Bruce Richardson
2024-01-23 8:57 ` Mattias Rönnblom
2024-01-23 9:06 ` Bruce Richardson
2024-01-24 11:37 ` Mattias Rönnblom
2024-01-31 13:45 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 02/11] eventdev: move text on driver internals to proper section Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 03/11] eventdev: update documentation on device capability flags Bruce Richardson
2024-01-23 9:18 ` Mattias Rönnblom
2024-01-23 9:34 ` Bruce Richardson
2024-01-31 14:09 ` Bruce Richardson
2024-02-02 8:58 ` Mattias Rönnblom
2024-02-02 11:20 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 04/11] eventdev: cleanup doxygen comments on info structure Bruce Richardson
2024-01-23 9:35 ` Mattias Rönnblom
2024-01-23 9:43 ` Bruce Richardson
2024-01-24 11:51 ` Mattias Rönnblom
2024-01-31 14:37 ` Bruce Richardson
2024-02-02 9:24 ` Mattias Rönnblom
2024-02-02 10:30 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 05/11] eventdev: improve function documentation for query fns Bruce Richardson
2024-01-23 9:40 ` Mattias Rönnblom
2024-01-19 17:43 ` [PATCH v2 06/11] eventdev: improve doxygen comments on configure struct Bruce Richardson
2024-01-23 9:46 ` Mattias Rönnblom
2024-01-31 16:15 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 07/11] eventdev: fix documentation for counting single-link ports Bruce Richardson
2024-01-23 9:48 ` Mattias Rönnblom
2024-01-23 9:56 ` Bruce Richardson
2024-01-31 16:18 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 08/11] eventdev: improve doxygen comments on config fns Bruce Richardson
2024-01-23 10:00 ` Mattias Rönnblom
2024-01-23 10:07 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 09/11] eventdev: improve doxygen comments for control APIs Bruce Richardson
2024-01-23 10:10 ` Mattias Rönnblom
2024-01-19 17:43 ` [PATCH v2 10/11] eventdev: RFC clarify comments on scheduling types Bruce Richardson
2024-01-23 16:19 ` Mattias Rönnblom
2024-01-24 11:21 ` Bruce Richardson
2024-01-31 17:54 ` Bruce Richardson
2024-01-19 17:43 ` [PATCH v2 11/11] eventdev: RFC clarify docs on event object fields Bruce Richardson
2024-01-24 11:34 ` Mattias Rönnblom
2024-02-01 16:59 ` Bruce Richardson [this message]
2024-02-02 9:38 ` Mattias Rönnblom
2024-02-02 11:33 ` Bruce Richardson
2024-02-02 12:02 ` Bruce Richardson
2024-02-01 17:02 ` Bruce Richardson
2024-02-02 9:14 ` Bruce Richardson
2024-02-02 9:22 ` Jerin Jacob
2024-02-02 9:36 ` Bruce Richardson
2024-02-02 9:45 ` Mattias Rönnblom
2024-02-02 10:32 ` Bruce Richardson
2024-02-01 9:35 ` Bruce Richardson
2024-02-01 15:00 ` Jerin Jacob
2024-02-01 15:24 ` Bruce Richardson
2024-02-01 16:20 ` Jerin Jacob
2024-02-02 12:39 ` [PATCH v3 00/11] improve eventdev API specification/documentation Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 01/11] eventdev: improve doxygen introduction text Bruce Richardson
2024-02-07 10:14 ` Jerin Jacob
2024-02-08 9:50 ` Mattias Rönnblom
2024-02-09 8:43 ` Jerin Jacob
2024-02-10 7:24 ` Mattias Rönnblom
2024-02-20 16:28 ` Bruce Richardson
2024-02-20 16:33 ` Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 02/11] eventdev: move text on driver internals to proper section Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 03/11] eventdev: update documentation on device capability flags Bruce Richardson
2024-02-07 10:30 ` Jerin Jacob
2024-02-20 16:42 ` Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 04/11] eventdev: cleanup doxygen comments on info structure Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 05/11] eventdev: improve function documentation for query fns Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 06/11] eventdev: improve doxygen comments on configure struct Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 07/11] eventdev: improve doxygen comments on config fns Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 08/11] eventdev: improve doxygen comments for control APIs Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 09/11] eventdev: improve comments on scheduling types Bruce Richardson
2024-02-08 9:18 ` Jerin Jacob
2024-02-08 10:04 ` Mattias Rönnblom
2024-02-20 17:23 ` Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 10/11] eventdev: clarify docs on event object fields and op types Bruce Richardson
2024-02-09 9:14 ` Jerin Jacob
2024-02-20 17:39 ` Bruce Richardson
2024-02-21 9:31 ` Jerin Jacob
2024-02-21 10:28 ` Bruce Richardson
2024-02-20 17:50 ` Bruce Richardson
2024-02-20 18:03 ` Bruce Richardson
2024-02-02 12:39 ` [PATCH v3 11/11] eventdev: drop comment for anon union from doxygen Bruce Richardson
2024-02-21 10:32 ` [PATCH v4 00/12] improve eventdev API specification/documentation Bruce Richardson
2024-02-21 10:32 ` [PATCH v4 01/12] eventdev: improve doxygen introduction text Bruce Richardson
2024-02-26 4:51 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-26 9:59 ` Bruce Richardson
2024-02-29 16:13 ` Jerin Jacob
2024-02-21 10:32 ` [PATCH v4 02/12] eventdev: move text on driver internals to proper section Bruce Richardson
2024-02-26 5:01 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 03/12] eventdev: update documentation on device capability flags Bruce Richardson
2024-02-26 5:07 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 04/12] eventdev: cleanup doxygen comments on info structure Bruce Richardson
2024-02-26 5:18 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 05/12] eventdev: improve function documentation for query fns Bruce Richardson
2024-02-26 5:18 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 06/12] eventdev: improve doxygen comments on configure struct Bruce Richardson
2024-02-26 6:36 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 07/12] eventdev: improve doxygen comments on config fns Bruce Richardson
2024-02-26 6:43 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-26 6:44 ` Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 08/12] eventdev: improve doxygen comments for control APIs Bruce Richardson
2024-02-26 6:44 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 09/12] eventdev: improve comments on scheduling types Bruce Richardson
2024-02-26 6:49 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 10/12] eventdev: clarify docs on event object fields and op types Bruce Richardson
2024-02-26 6:52 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 11/12] eventdev: drop comment for anon union from doxygen Bruce Richardson
2024-02-26 6:52 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-02-21 10:32 ` [PATCH v4 12/12] eventdev: fix doxygen processing of event vector struct Bruce Richardson
2024-02-26 6:53 ` [EXT] " Pavan Nikhilesh Bhagavatula
2024-03-04 15:35 ` Thomas Monjalon
2024-03-04 15:49 ` Bruce Richardson
2024-02-23 12:36 ` [PATCH v4 00/12] improve eventdev API specification/documentation Jerin Jacob
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZbvN7l8he2O5btYa@bricha3-MOBL.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=abdullah.sevincer@intel.com \
--cc=dev@dpdk.org \
--cc=hemant.agrawal@nxp.com \
--cc=hofors@lysator.liu.se \
--cc=jerinj@marvell.com \
--cc=mattias.ronnblom@ericsson.com \
--cc=pbhagavatula@marvell.com \
--cc=pravin.pathak@intel.com \
--cc=sachin.saxena@oss.nxp.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).