DPDK patches and discussions
 help / color / mirror / Atom feed
From: Olivier MATZ <olivier.matz@6wind.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
Date: Mon, 30 Mar 2015 21:55:50 +0200	[thread overview]
Message-ID: <5519AA46.1090409@6wind.com> (raw)
In-Reply-To: <2601191342CEEE43887BDE71AB9772582140FB21@irsmsx105.ger.corp.intel.com>

Hi Konstantin,

On 03/30/2015 02:34 PM, Ananyev, Konstantin wrote:
> Hi Olivier,
> 
>> -----Original Message-----
>> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
>> Sent: Friday, March 27, 2015 3:17 PM
>> To: Ananyev, Konstantin; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
>>
>> Hi Konstantin,
>>
>> On 03/27/2015 03:25 PM, Ananyev, Konstantin wrote:
>>> Hi Olivier,
>>>
>>>> -----Original Message-----
>>>> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
>>>> Sent: Friday, March 27, 2015 1:56 PM
>>>> To: Ananyev, Konstantin; dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
>>>>
>>>> Hi Konstantin,
>>>>
>>>> On 03/27/2015 10:07 AM, Olivier MATZ wrote:
>>>>>> I think that to support ability to setup priv_size on a mempool basis,
>>>>>> and reserve private space between struct rte_mbuf and rte_mbuf. buf_addr,
>>>>>> we need to:
>>>>>>
>>>>>> 1. Store priv_size both inside the mempool and inside the mbuf.
>>>>>>
>>>>>> 2. rte_pktmbuf_attach() should change the value of priv_size to the priv_size of the direct mbuf we are going to attach to:
>>>>>> rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md) {... mi->priv_size = md->priv_size; ...}
>>>>>>
>>>>>> 3. rte_pktmbuf_detach() should restore original value of mbuf's priv_size:
>>>>>> rte_pktmbuf_detach(struct rte_mbuf *m)
>>>>>> {
>>>>>>   ...
>>>>>>    m->priv_size = rte_mempool_get_privsize(m->pool);
>>>>>>    m->buf _addr= rte_mbuf_to_baddr(m);
>>>>>>    ...
>>>>>> }
>>>>>>
>>>>>> Also I think we need to provide a way to specify priv_size for all mbufs of the mepool at init time:
>>>>>> - either force people to specify it at rte_mempool_create() time (probably use init_arg for that),
>>>>>> - or provide separate function that could be called straight after rte_mempool_create() , that
>>>>>> would setup priv_size for the  pool and for all its mbufs.
>>>>>> - or some sort of combination of these 2 approaches - introduce a wrapper function
>>>>>> (rte_mbuf_pool_create() or something) that would take priv_size as parameter,
>>>>>> create a new mempool and then setup priv_size.
>>>>
>>>> I though a bit more about this solution, and I realized that doing
>>>> mi->priv_size = md->priv_size in rte_pktmbuf_attach() is probably not
>>>> a good idea, as there is no garantee that the size of mi is large enough
>>>> to store the priv of md.
>>>>
>>>> Having the same priv_size for mi and md is maybe a good constraint.
>>>> I can add this in the API comments and assertions in the code to
>>>> check this condition, what do you think?
>>>
>>> Probably we have a different concepts of what is mbuf's  private space in mind.
>>> From my point, even indirect buffer should use it's own private space and
>>> leave contents of direct mbuf it attached to unmodified.
>>> After attach() operation, only contents of the buffer are shared between mbufs,
>>> but not the mbuf's metadata.
>>
>> Sorry if it was not clear in my previous messages, but I agree
>> with your description. When attaching a mbuf, only data, not
>> metadata should be shared.
>>
>> In the solution you are suggesting (quoted above), you say we need
>> to set mi->priv_size to md->priv_size in rte_pktmbuf_attach(). I felt
>> this was not possible, but it depends on the meaning we give to
>> priv_size:
>>
>> 1. If the meaning is "the size of the private data embedded in this
>>    mbuf", which is the most logical meaning, we cannot do this
>>    affectation
>>
>> 2. If the meaning is "the size of the private data embedded in the
>>    mbuf the buf_addr is pointing to" (which is harder to get), the
>>    affectation makes sense.
>>
>> From what I understand, you feel we should use 2/ as priv_size
>> definition. Is it correct?
> 
> Yes, I meant 2.
> From my point priv_size inside mbuf is more like 'buf_ofs'.
> It's main purpose is for internal use - to help our mbuf manipulation routinies
> (attach/detach/free) to work correctly.
> If the user wants to query size of private space for the mbuf, he probably should
> use the value from mempool.

Agree.


>> In my previous message, the definition of m->priv_size was 1/,
>> so that's why I felt assigning mi->priv_size to md->priv_size was
>> not possible.
>>
>> I agree 2/ is probably a good choice, as it would allow to attach
>> to a mbuf with a different priv_size. It may require some additional
>> comments above the field in the structure to explain that.
>>
>>
>>> Otherwise on detach(), you'll have to copy contents of private space back, from direct to indirect mbuf?
>>> Again how to deal with the case, when 2 or more mbufs will attach to the same direct one?
>>>
>>> So let say, if we'll have a macro:
>>>
>>> #define RTE_MBUF_PRIV_PTR(mb)	((void *)((struct rte_mbuf *)(mb)) + 1))
>>>
>>> No matter is mb  a direct or indirect mbuf.
>>> Do you have something else in mind here?
>>
>> I completely agree with this macro. We should consider the private data
>> as an extension of the mbuf structure.
>>
>>
>>>>> Introducing rte_mbuf_pool_create() seems a good idea to me, it
>>>>> would hide 'rte_pktmbuf_pool_private' from the user and force
>>>>> to initialize all the required fields (mbuf_data_room_size only
>>>>> today, and maybe mbuf_priv_size).
>>>>>
>>>>> The API would be:
>>>>>
>>>>> struct rte_mempool *
>>>>> rte_mbuf_pool_create(const char *name, unsigned n, unsigned elt_size,
>>>>> 	unsigned cache_size, size_t mbuf_priv_size,
>>>>> 	rte_mempool_obj_ctor_t *obj_init, void *obj_init_arg,
>>>>> 	int socket_id, unsigned flags)
>>>>>
>>>>> I can give it a try and send a patch for this.
>>>>
>>>> About this, it is not required anymore for this patch series if we
>>>> agree with my comment above.
>>>
>>> I still think we need some way to setup priv_size on a per-mempool basis.
>>> Doing that in rte_mbuf_pool_create() seems like a good thing to me.
>>> Not sure, why you decided to drop it?
>>
>> I think we can already do it without changing the API by providing
>> our own rte_pktmbuf_init and rte_pktmbuf_pool_init.
>>
>> rte_pktmbuf_init() has to set:
>>   m->buf_len = mp->elt_size - sizeof(struct mbuf);
>>   m->priv_size = sizeof(struct mbuf) - sizeof(struct rte_mbuf);
>>
>> rte_pktmbuf_pool_init() has to set:
>>   /* we can use the default function */
>>   mbp_priv->mbuf_data_room_size = MBUF_RXDATA_SIZE +
>>   	RTE_PKTMBUF_HEADROOM;
> 
> Yeh, when  arg==NULL for rte_pktmbuf_pool_init() we always set up
> mbuf_data_room_size to the hardcoded value.
> Which always looked strange to me.
> I think it should be set to:
> mp->elt_size - sizeof(struct rte_mbuf) - RTE_PKTMBUF_HEADROOM;
> for that case.

Yes, that would make more sense instead of the hardcoded value.
But I'm not sure it should be part of this series as the clone
patches won't change this behavior. I would prefer to have it in
another series that reworks mbuf pool initialization. I can also
work on it.

On the other hand if you really feel this patch is needed in
this series, it's not a problem as it's a one-liner.

> 
>>
>> In this case, it is possible to calculate the mbuf_priv_size only
>> from the pool object:
>>
>>   mbuf_priv_size = pool->elt_size - RTE_PKTMBUF_HEADROOM -
>> 	pool_private->mbuf_data_room_size -
>> 	sizeof(rte_mbuf)
>>
> 
> So if I understand your idea correctly:
> If  second parameter for rte_pktmbuf_pool_init() is NULL, then 
> we setup
> 
> mbp_priv->mbuf_data_room_size = mp->elt_size - sizeof(struct rte_mbuf) - RTE_PKTMBUF_HEADROOM;
> 
> Which means that  priv_size ==0  for all  mbufs in the pool 
> Otherwise we setup:
> 
>  mbp_priv->mbuf_data_room_size = opaque_arg;
> 
> As we are doing now, and priv_size for all mbufs in the pool will be:
> pool->elt_size - pool_private->mbuf_data_room_size - sizeof(rte_mbuf);
> 
> And in that case, all users have to do to specify priv_size for mempool is to pass a proper argument
> for rte_pktmbuf_pool_init() at  mempool_create().
> Correct? 

Correct.



> 
>>
>> I agree it's not ideal, but I think the mbuf pool initialization
>> is another problem. That's why I suggested to change this in a
>> separate series that will add rte_mbuf_pool_create() with the
>> API described above. Thoughts?
>>
> 
> I also put answers to another mail below.
> Just to keep all discussion in one place.
> 
>>> Though, I still think that the better approach is to reserve private space before struct rte_mbuf, not after.
>>> In that case, user got his private space, and we keep buf_addr straight after  rte_mbuf, without any whole.
>>> So we don't need steps 2 and 3, above,
>>> plus we don't need rte_mbuf_to_baddr() and rte_mbuf_from_indirect() -
>>> RTE_MBUF_TO_BADDR/ RTE_MBUF_FROM_BADDR would keep working correctly.
>>> In fact, with this scheme - we don't even need priv_size for mbuf management (attach/detach/free).
>>>
>>> Wonder did you try that approach?
>>
>> Yes, I though about this approach too. But I think it would require
>> more changes. When an application or a driver allocate a mbuf with
>> mempool_get(), it expects that the returned pointer is the rte_mbuf *.
> 
> Yep, sure it will still get the pointer to the rte_mbuf *.
> Though later, if upper layer would like to convert from  rte_mbuf* to app_specific_mbuf *,
> it would have to use a macro:
> 
> #define RTE_MBUF_TO_PRIV(m)	((void *)((uintptr_t)(m) - (m)->priv_size)) 
> 
>>
>> With this solution, there are 2 options:
>> - no mempool modification, so each application/driver has to add
>>   priv_size bytes to the object to get the mbuf pointer. This does not
>>   look feasible.
>> - change the mempool to support private area before each object. I
>>   think mempool API is already quite complex, and I think it's not
>>   the role of the mempool library to provide such features.
> 
> 
> In fact, I think the changes would be minimal.
> All we have to do, is to make changes in rte_pktmbuf_init():
> 
> void
> rte_pktmbuf_init(struct rte_mempool *mp,
>                  __attribute__((unused)) void *opaque_arg,
>                  void *_m,
>                  __attribute__((unused)) unsigned i)
> {
> 
>      //extract priv_size from mempool   (discussed above).
>       uint16_t priv_size = rte_mbufpool_get_priv_size(mp); 
> 
>       struct rte_mbuf *m = _m + priv_size;
>       uint32_t buf_len = mp->elt_size - sizeof(struct rte_mbuf) - priv_size;
> 
> ....
> 
> 
> With that way priv_size inside mbuf would always be the size its own private space,
> for both direct and indirect mbus., so we don't require to set priv_size at attach/detach.
> Again RTE_MBUF_TO_BADDR() and RTE_MBUF_FROM_BADDR() would just work without any modifications.

I'm not sure I'm getting it. The argument '_m' of your
rte_pktmbuf_init() is the pointer to the priv data, right?
So it means that the mbuf is located priv_size bytes after.

The rte_pktmbuf_init() function is called by mempool_create(),
and the _m parameter is a pointer to a mempool object. So
in my understanding, mempool_get() would not return a rte_mbuf
but a pointer to the application private data.


Regards,
Olivier



> 
> Konstantin
>  
> 

  reply	other threads:[~2015-03-30 19:56 UTC|newest]

Thread overview: 101+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-03-25 17:00 [dpdk-dev] [PATCH 0/5] mbuf: enhancements of mbuf clones Olivier Matz
2015-03-25 17:00 ` [dpdk-dev] [PATCH 1/5] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-03-26 13:35   ` Bruce Richardson
2015-03-26 15:30     ` Olivier MATZ
2015-03-25 17:00 ` [dpdk-dev] [PATCH 2/5] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-03-25 17:00 ` [dpdk-dev] [PATCH 3/5] test/mbuf: rename mc variable in m Olivier Matz
2015-03-25 17:00 ` [dpdk-dev] [PATCH 4/5] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-03-25 17:00 ` [dpdk-dev] [PATCH 5/5] test/mbuf: verify that cloning a clone works properly Olivier Matz
2015-03-26  8:48   ` Olivier MATZ
2015-03-26 15:59 ` [dpdk-dev] [PATCH v2 0/5] mbuf: enhancements of mbuf clones Olivier Matz
2015-03-26 15:59   ` [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-03-26 17:13     ` Zoltan Kiss
2015-03-27  0:24     ` Ananyev, Konstantin
2015-03-27  9:07       ` Olivier MATZ
2015-03-27 13:56         ` Olivier MATZ
2015-03-27 14:25           ` Ananyev, Konstantin
2015-03-27 15:17             ` Olivier MATZ
2015-03-27 18:11               ` Zoltan Kiss
2015-03-28 21:19                 ` Olivier MATZ
2015-03-30 12:34               ` Ananyev, Konstantin
2015-03-30 19:55                 ` Olivier MATZ [this message]
2015-03-30 23:17                   ` Ananyev, Konstantin
2015-03-31 19:01                     ` Olivier MATZ
2015-04-01 13:48                       ` Ananyev, Konstantin
2015-04-01 15:18                         ` Olivier MATZ
2015-03-26 15:59   ` [dpdk-dev] [PATCH v2 2/5] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-03-26 15:59   ` [dpdk-dev] [PATCH v2 3/5] test/mbuf: rename mc variable in m Olivier Matz
2015-03-26 15:59   ` [dpdk-dev] [PATCH v2 4/5] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-03-26 15:59   ` [dpdk-dev] [PATCH v2 5/5] test/mbuf: verify that cloning a clone works properly Olivier Matz
2015-03-31 19:22   ` [dpdk-dev] [PATCH v3 0/5] mbuf: enhancements of mbuf clones Olivier Matz
2015-03-31 19:23     ` [dpdk-dev] [PATCH v3 1/5] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-04-02 14:32       ` Zoltan Kiss
2015-04-02 17:21       ` Ananyev, Konstantin
2015-04-06 21:49         ` Olivier MATZ
2015-04-07 12:40           ` Ananyev, Konstantin
2015-04-07 15:45             ` Olivier MATZ
2015-04-07 17:17               ` Ananyev, Konstantin
2015-04-08  9:44                 ` Olivier MATZ
2015-04-08 13:45                   ` Ananyev, Konstantin
2015-04-09 13:06                     ` Olivier MATZ
2015-04-20 15:41       ` [dpdk-dev] [PATCH v4 00/12] mbuf: enhancements of mbuf clones Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 01/12] mbuf: fix mbuf data room size calculation rte_pktmbuf_pool_init Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 02/12] examples: always initialize mbuf_pool private area Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 03/12] mbuf: add accessors to get data room size and private size Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 04/12] mbuf: fix rte_pktmbuf_init when mbuf private size is not zero Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 05/12] testpmd: use standard functions to initialize mbufs and mbuf pool Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 06/12] mbuf: introduce a new helper to create a " Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 07/12] apps: use rte_pktmbuf_pool_create to create mbuf pools Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 08/12] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 09/12] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 10/12] test/mbuf: rename mc variable in m Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 11/12] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-04-20 15:41         ` [dpdk-dev] [PATCH v4 12/12] test/mbuf: verify that cloning a clone works properly Olivier Matz
2015-04-20 16:53         ` [dpdk-dev] [PATCH v4 00/12] mbuf: enhancements of mbuf clones Neil Horman
2015-04-20 17:07           ` Olivier MATZ
2015-04-20 17:21             ` Neil Horman
2015-04-20 18:24               ` Olivier MATZ
2015-04-21  9:55         ` [dpdk-dev] [PATCH v5 " Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 01/12] mbuf: fix mbuf data room size calculation rte_pktmbuf_pool_init Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 02/12] examples: always initialize mbuf_pool private area Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 03/12] mbuf: add accessors to get data room size and private size Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 04/12] mbuf: fix rte_pktmbuf_init when mbuf private size is not zero Olivier Matz
2015-04-21 15:07             ` Ananyev, Konstantin
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 05/12] testpmd: use standard functions to initialize mbufs and mbuf pool Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 06/12] mbuf: introduce a new helper to create a " Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 07/12] apps: use rte_pktmbuf_pool_create to create mbuf pools Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 08/12] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-04-21 15:01             ` Ananyev, Konstantin
2015-04-21 15:26               ` Olivier MATZ
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 09/12] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 10/12] test/mbuf: rename mc variable in m Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 11/12] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-04-21  9:55           ` [dpdk-dev] [PATCH v5 12/12] test/mbuf: verify that cloning a clone works properly Olivier Matz
2015-04-21 11:50           ` [dpdk-dev] [PATCH v5 00/12] mbuf: enhancements of mbuf clones Neil Horman
2015-04-22  9:57           ` [dpdk-dev] [PATCH v6 00/13] " Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 01/13] mbuf: fix mbuf data room size calculation rte_pktmbuf_pool_init Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 02/13] examples: always initialize mbuf_pool private area Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 03/13] mbuf: add accessors to get data room size and private size Olivier Matz
2015-04-28  9:15               ` Thomas Monjalon
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 04/13] mbuf: fix rte_pktmbuf_init when mbuf private size is not zero Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 05/13] testpmd: use standard functions to initialize mbufs and mbuf pool Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 06/13] mbuf: introduce a new helper to create a " Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 07/13] apps: use rte_pktmbuf_pool_create to create mbuf pools Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 08/13] mbuf: fix clone support when application uses private mbuf data Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 09/13] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 10/13] test/mbuf: rename mc variable in m Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 11/13] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 12/13] test/mbuf: verify that cloning a clone works properly Olivier Matz
2015-04-22  9:57             ` [dpdk-dev] [PATCH v6 13/13] test/mbuf: add a test case for clone with different priv size Olivier Matz
2015-04-22 11:59             ` [dpdk-dev] [PATCH v6 00/13] mbuf: enhancements of mbuf clones Ananyev, Konstantin
2015-04-24 10:38               ` Zoltan Kiss
2015-04-27 17:38                 ` Thomas Monjalon
2015-04-28 11:15                   ` Zoltan Kiss
2015-05-07  1:57                 ` Xu, HuilongX
2015-05-07  7:32                   ` Olivier MATZ
2015-05-07  9:39                     ` Ananyev, Konstantin
2015-04-28  9:50               ` Thomas Monjalon
2015-03-31 19:23     ` [dpdk-dev] [PATCH v3 2/5] mbuf: allow to clone an indirect mbuf Olivier Matz
2015-03-31 19:23     ` [dpdk-dev] [PATCH v3 3/5] test/mbuf: rename mc variable in m Olivier Matz
2015-03-31 19:23     ` [dpdk-dev] [PATCH v3 4/5] test/mbuf: enhance mbuf refcnt test Olivier Matz
2015-03-31 19:23     ` [dpdk-dev] [PATCH v3 5/5] test/mbuf: verify that cloning a clone works properly Olivier Matz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5519AA46.1090409@6wind.com \
    --to=olivier.matz@6wind.com \
    --cc=dev@dpdk.org \
    --cc=konstantin.ananyev@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).