From: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
To: dev@dpdk.org
Cc: shahafs@mellanox.com
Subject: [dpdk-dev] [PATCH v2 00/14] net/mlx5: add support for multiport IB devices
Date: Mon, 25 Mar 2019 17:03:21 +0000 [thread overview]
Message-ID: <1553533414-9911-1-git-send-email-viacheslavo@mellanox.com> (raw)
In-Reply-To: <1553155888-27498-1-git-send-email-viacheslavo@mellanox.com>
The Mellanox NICs support SR-IOV and have E-Switch feature.
When SR-IOV is set up in switchdev mode and E-Switch is enabled
we have so called VF representors in the system. All representors
belonging to the same E-Switch are created on the basis of the
single PCI function and with current implementation each representor
has its own dedicated Infiniband device and operates within its
own Infiniband context. It is proposed to provide representors
as ports of the single Infiniband device and operate on the
shared Infiniband context.
mlx5_pci_probe() routine is planned to be extended to probe
the ports of found Infiniband devices. All active ports,
belonging to the same Infiniband device will use the signle
shared Infiniband context of that device:
- QP are created within shared context, with assigning to
specified port
- Verb flows are also created with specifying port index
- Protection Domain is shared by all representors
- Global Memory Regions Cache is shared by all representors,
local ones are not (not in this patchset)
- Shared Asynchronous Event Handler distributes events
among the device ports (not in this patchset)
- UAR mapping to user space is also shared (not in patchset)
Each representor has its own attach network interface device,
interface name and index can be retrieved via IB device and
port index, so the link control is performed via Netlink
in the same way as in existing implementation.
This patchset also superseedes http://patches.dpdk.org/patch/50486/
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
v2:
- clarifying comments added
- asserts added
- secondary process does not probe IB device at all and
does not create shared context
- minor bugs (like ++ in assert) fixed
- representor_id contains corresponding VF vport index,
no corrections +1 is needed anymore. vport index is
calculated in the only place in code and if indexing
schema is changed it would be easy to modify
v1:
http://patches.dpdk.org/cover/51451/
Viacheslav Ovsiienko (13):
net/mlx5: add representor recognition on kernels 5.x
net/mlx5: modify get ifindex routine for multiport IB
net/mlx5: add getting IB ports number for multiport IB
net/mlx5: add multiport IB device support to probing
net/mlx5: add IB shared context alloc/free functions
net/mlx5: switch to the names in the shared IB context
net/mlx5: switch to the shared Protection Domain
net/mlx5: switch to the shared context IB attributes
net/mlx5: switch to the shared IB device context
net/mlx5: provide IB port for the object being created
net/mlx5: update install/uninstall int handler routines
net/mlx5: update event handler for multiport IB devices
net/mlx5: add source vport match to the ingress rules
drivers/net/mlx5/Makefile | 10 +
drivers/net/mlx5/meson.build | 4 +
drivers/net/mlx5/mlx5.c | 644 ++++++++++++++++++++++++++-----------
drivers/net/mlx5/mlx5.h | 44 ++-
drivers/net/mlx5/mlx5_ethdev.c | 242 +++++++++-----
drivers/net/mlx5/mlx5_flow.c | 2 +
drivers/net/mlx5/mlx5_flow_dv.c | 45 ++-
drivers/net/mlx5/mlx5_flow_verbs.c | 7 +-
drivers/net/mlx5/mlx5_mr.c | 4 +-
drivers/net/mlx5/mlx5_nl.c | 102 +++++-
drivers/net/mlx5/mlx5_rxq.c | 41 +--
drivers/net/mlx5/mlx5_socket.c | 4 +-
drivers/net/mlx5/mlx5_stats.c | 22 +-
drivers/net/mlx5/mlx5_trigger.c | 2 +-
drivers/net/mlx5/mlx5_txq.c | 24 +-
15 files changed, 859 insertions(+), 338 deletions(-)
--
1.8.3.1
next prev parent reply other threads:[~2019-03-25 17:03 UTC|newest]
Thread overview: 217+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-28 18:02 [dpdk-dev] [RFC 00/10] " Viacheslav Ovsiienko
2019-02-28 18:02 ` [dpdk-dev] [RFC 01/10] net/mlx5: add multiport IB device port structure Viacheslav Ovsiienko
2019-02-28 18:02 ` [dpdk-dev] [RFC 02/10] net/mlx5: modify get ifindex routine for multiport IB Viacheslav Ovsiienko
2019-02-28 18:02 ` [dpdk-dev] [RFC 03/10] net/mlx5: add getting IB ports number " Viacheslav Ovsiienko
2019-02-28 18:02 ` [dpdk-dev] [RFC 04/10] net/mlx5: add multiport IB device support to probing Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 05/10] net/mlx5: add IB shared context alloc/free functions Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 06/10] net/mlx5: switch to the names in the shared IB context Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 07/10] net/mlx5: switch to the shared Protection Domain Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 08/10] net/mlx5: switch to the shared context IB attributes Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 09/10] net/mlx5: switch to the shared IB device context Viacheslav Ovsiienko
2019-02-28 18:03 ` [dpdk-dev] [RFC 10/10] net/mlx5: provide IB port for the object being created Viacheslav Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 00/14] net/mlx5: add support for multiport IB devices Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 01/14] net/mlx5: add representor recognition on kernels 5.x Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:13 ` Shahaf Shuler
2019-03-21 12:13 ` Shahaf Shuler
2019-03-21 15:08 ` Stephen Hemminger
2019-03-21 15:08 ` Stephen Hemminger
2019-03-21 15:31 ` Slava Ovsiienko
2019-03-21 15:31 ` Slava Ovsiienko
2019-03-21 19:08 ` Stephen Hemminger
2019-03-21 19:08 ` Stephen Hemminger
2019-03-22 8:15 ` Slava Ovsiienko
2019-03-22 8:15 ` Slava Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 02/14] net/mlx5: introduce multiport IB device shared structure Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 03/14] net/mlx5: modify get ifindex routine for multiport IB Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:58 ` Slava Ovsiienko
2019-03-21 12:58 ` Slava Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 04/14] net/mlx5: add getting IB ports number " Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 05/14] net/mlx5: add multiport IB device support to probing Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:54 ` Slava Ovsiienko
2019-03-21 12:54 ` Slava Ovsiienko
2019-03-21 12:57 ` Slava Ovsiienko
2019-03-21 12:57 ` Slava Ovsiienko
2019-03-24 9:00 ` Shahaf Shuler
2019-03-24 9:00 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 06/14] net/mlx5: add IB shared context alloc/free functions Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 07/14] net/mlx5: switch to the names in the shared IB context Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 08/14] net/mlx5: switch to the shared Protection Domain Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 09/14] net/mlx5: switch to the shared context IB attributes Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 10/14] net/mlx5: switch to the shared IB device context Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 12:14 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 11/14] net/mlx5: provide IB port for the object being created Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 12/14] net/mlx5: update install/uninstall int handler routines Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 14:01 ` Slava Ovsiienko
2019-03-21 14:01 ` Slava Ovsiienko
2019-03-24 9:07 ` Shahaf Shuler
2019-03-24 9:07 ` Shahaf Shuler
2019-03-21 8:11 ` [dpdk-dev] [PATCH 13/14] net/mlx5: update event handler for multiport IB devices Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 14:08 ` Slava Ovsiienko
2019-03-21 14:08 ` Slava Ovsiienko
2019-03-21 8:11 ` [dpdk-dev] [PATCH 14/14] net/mlx5: add source vport match to the ingress rules Viacheslav Ovsiienko
2019-03-21 8:11 ` Viacheslav Ovsiienko
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 12:15 ` Shahaf Shuler
2019-03-21 14:11 ` Slava Ovsiienko
2019-03-21 14:11 ` Slava Ovsiienko
2019-03-24 9:13 ` Shahaf Shuler
2019-03-24 9:13 ` Shahaf Shuler
2019-03-25 7:44 ` Slava Ovsiienko
2019-03-25 7:44 ` Slava Ovsiienko
2019-03-21 12:13 ` [dpdk-dev] [PATCH 00/14] net/mlx5: add support for multiport IB devices Shahaf Shuler
2019-03-21 12:13 ` Shahaf Shuler
2019-03-21 12:58 ` Slava Ovsiienko
2019-03-21 12:58 ` Slava Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko [this message]
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 " Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 01/13] net/mlx5: add representor recognition on kernels 5.x Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 18:06 ` Stephen Hemminger
2019-03-25 18:06 ` Stephen Hemminger
2019-03-25 18:07 ` Stephen Hemminger
2019-03-25 18:07 ` Stephen Hemminger
2019-03-26 7:33 ` Slava Ovsiienko
2019-03-26 7:33 ` Slava Ovsiienko
2019-03-26 12:20 ` Shahaf Shuler
2019-03-26 12:20 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 02/13] net/mlx5: modify get ifindex routine for multiport IB Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 11:47 ` Shahaf Shuler
2019-03-26 11:47 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 03/13] net/mlx5: add getting IB ports number " Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 04/13] net/mlx5: add multiport IB device support to probing Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 12:02 ` Shahaf Shuler
2019-03-26 12:02 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 05/13] net/mlx5: add IB shared context alloc/free functions Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 12:10 ` Shahaf Shuler
2019-03-26 12:10 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 06/13] net/mlx5: switch to the names in the shared IB context Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 07/13] net/mlx5: switch to the shared Protection Domain Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 08/13] net/mlx5: switch to the shared context IB attributes Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 09/13] net/mlx5: switch to the shared IB device context Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 10/13] net/mlx5: provide IB port for the object being created Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 11/13] net/mlx5: update install/uninstall int handler routines Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 12:14 ` Shahaf Shuler
2019-03-26 12:14 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 12/13] net/mlx5: update event handler for multiport IB devices Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 12:16 ` Shahaf Shuler
2019-03-26 12:16 ` Shahaf Shuler
2019-03-25 17:03 ` [dpdk-dev] [PATCH v2 13/13] net/mlx5: add source vport match to the ingress rules Viacheslav Ovsiienko
2019-03-25 17:03 ` Viacheslav Ovsiienko
2019-03-26 12:21 ` Shahaf Shuler
2019-03-26 12:21 ` Shahaf Shuler
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 00/14] net/mlx5: add support for multiport IB devices Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 01/13] net/mlx5: add representor recognition on kernels 5.x Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 19:37 ` Shahaf Shuler
2019-03-26 19:37 ` Shahaf Shuler
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 02/13] net/mlx5: modify get ifindex routine for multiport IB Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 03/13] net/mlx5: add getting IB ports number " Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 04/13] net/mlx5: add multiport IB device support to probing Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 05/13] net/mlx5: add IB shared context alloc/free functions Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 19:35 ` Shahaf Shuler
2019-03-26 19:35 ` Shahaf Shuler
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 06/13] net/mlx5: switch to the names in the shared IB context Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 07/13] net/mlx5: switch to the shared Protection Domain Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 08/13] net/mlx5: switch to the shared context IB attributes Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 09/13] net/mlx5: switch to the shared IB device context Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 10/13] net/mlx5: provide IB port for the object being created Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 11/13] net/mlx5: update install/uninstall int handler routines Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 12/13] net/mlx5: update event handler for multiport IB devices Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 15:35 ` [dpdk-dev] [PATCH v3 13/13] net/mlx5: add source vport match to the ingress rules Viacheslav Ovsiienko
2019-03-26 15:35 ` Viacheslav Ovsiienko
2019-03-26 19:38 ` Shahaf Shuler
2019-03-26 19:38 ` Shahaf Shuler
2019-03-27 6:00 ` [dpdk-dev] [PATCH v3 00/14] net/mlx5: add support for multiport IB devices Shahaf Shuler
2019-03-27 6:00 ` Shahaf Shuler
2019-03-27 7:31 ` Slava Ovsiienko
2019-03-27 7:31 ` Slava Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 " Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 01/13] net/mlx5: add representor recognition on kernels 5.x Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 02/13] net/mlx5: modify get ifindex routine for multiport IB Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 03/13] net/mlx5: add getting IB ports number " Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 04/13] net/mlx5: add multiport IB device support to probing Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 05/13] net/mlx5: add IB shared context alloc/free functions Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 06/13] net/mlx5: switch to the names in the shared IB context Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 07/13] net/mlx5: switch to the shared Protection Domain Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 08/13] net/mlx5: switch to the shared context IB attributes Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 09/13] net/mlx5: switch to the shared IB device context Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-04-02 4:49 ` Shahaf Shuler
2019-04-02 4:49 ` Shahaf Shuler
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 10/13] net/mlx5: provide IB port for the object being created Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 11/13] net/mlx5: update install/uninstall int handler routines Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 12/13] net/mlx5: update event handler for multiport IB devices Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-27 13:15 ` [dpdk-dev] [PATCH v4 13/13] net/mlx5: add source vport match to the ingress rules Viacheslav Ovsiienko
2019-03-27 13:15 ` Viacheslav Ovsiienko
2019-03-28 9:21 ` [dpdk-dev] [PATCH v4 00/14] net/mlx5: add support for multiport IB devices Shahaf Shuler
2019-03-28 9:21 ` Shahaf Shuler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1553533414-9911-1-git-send-email-viacheslavo@mellanox.com \
--to=viacheslavo@mellanox.com \
--cc=dev@dpdk.org \
--cc=shahafs@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).