From: "Qiu, Michael" <michael.qiu@intel.com>
To: "Tan, Jianfeng" <jianfeng.tan@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "nakajima.yoshihiro@lab.ntt.co.jp"
<nakajima.yoshihiro@lab.ntt.co.jp>,
"mst@redhat.com" <mst@redhat.com>,
"ann.zhuangyanying@huawei.com" <ann.zhuangyanying@huawei.com>
Subject: Re: [dpdk-dev] [PATCH 0/4] virtio support for container
Date: Tue, 26 Jan 2016 06:02:03 +0000 [thread overview]
Message-ID: <533710CFB86FA344BFBF2D6802E6028622F226FD@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <1452426182-86851-1-git-send-email-jianfeng.tan@intel.com>
On 1/11/2016 2:43 AM, Tan, Jianfeng wrote:
> This patchset is to provide high performance networking interface (virtio)
> for container-based DPDK applications. The way of starting DPDK apps in
> containers with ownership of NIC devices exclusively is beyond the scope.
> The basic idea here is to present a new virtual device (named eth_cvio),
> which can be discovered and initialized in container-based DPDK apps using
> rte_eal_init(). To minimize the change, we reuse already-existing virtio
> frontend driver code (driver/net/virtio/).
>
> Compared to QEMU/VM case, virtio device framework (translates I/O port r/w
> operations into unix socket/cuse protocol, which is originally provided in
> QEMU), is integrated in virtio frontend driver. So this converged driver
> actually plays the role of original frontend driver and the role of QEMU
> device framework.
>
> The major difference lies in how to calculate relative address for vhost.
> The principle of virtio is that: based on one or multiple shared memory
> segments, vhost maintains a reference system with the base addresses and
> length for each segment so that an address from VM comes (usually GPA,
> Guest Physical Address) can be translated into vhost-recognizable address
> (named VVA, Vhost Virtual Address). To decrease the overhead of address
> translation, we should maintain as few segments as possible. In VM's case,
> GPA is always locally continuous. In container's case, CVA (Container
> Virtual Address) can be used. Specifically:
> a. when set_base_addr, CVA address is used;
> b. when preparing RX's descriptors, CVA address is used;
> c. when transmitting packets, CVA is filled in TX's descriptors;
> d. in TX and CQ's header, CVA is used.
>
> How to share memory? In VM's case, qemu always shares all physical layout
> to backend. But it's not feasible for a container, as a process, to share
> all virtual memory regions to backend. So only specified virtual memory
> regions (with type of shared) are sent to backend. It's a limitation that
> only addresses in these areas can be used to transmit or receive packets.
>
> Known issues
>
> a. When used with vhost-net, root privilege is required to create tap
> device inside.
> b. Control queue and multi-queue are not supported yet.
> c. When --single-file option is used, socket_id of the memory may be
> wrong. (Use "numactl -N x -m x" to work around this for now)
>
> How to use?
>
> a. Apply this patchset.
>
> b. To compile container apps:
> $: make config RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc
> $: make install RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc
> $: make -C examples/l2fwd RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc
> $: make -C examples/vhost RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc
>
> c. To build a docker image using Dockerfile below.
> $: cat ./Dockerfile
> FROM ubuntu:latest
> WORKDIR /usr/src/dpdk
> COPY . /usr/src/dpdk
> ENV PATH "$PATH:/usr/src/dpdk/examples/l2fwd/build/"
> $: docker build -t dpdk-app-l2fwd .
>
> d. Used with vhost-user
> $: ./examples/vhost/build/vhost-switch -c 3 -n 4 \
> --socket-mem 1024,1024 -- -p 0x1 --stats 1
> $: docker run -i -t -v <path_to_vhost_unix_socket>:/var/run/usvhost \
> -v /dev/hugepages:/dev/hugepages \
> dpdk-app-l2fwd l2fwd -c 0x4 -n 4 -m 1024 --no-pci \
> --vdev=eth_cvio0,path=/var/run/usvhost -- -p 0x1
>
> f. Used with vhost-net
> $: modprobe vhost
> $: modprobe vhost-net
> $: docker run -i -t --privileged \
> -v /dev/vhost-net:/dev/vhost-net \
> -v /dev/net/tun:/dev/net/tun \
> -v /dev/hugepages:/dev/hugepages \
> dpdk-app-l2fwd l2fwd -c 0x4 -n 4 -m 1024 --no-pci \
> --vdev=eth_cvio0,path=/dev/vhost-net -- -p 0x1
We'd better add a ifname, like
--vdev=eth_cvio0,path=/dev/vhost-net,ifname=tap0, so that user could add
the tap to the bridge first.
Thanks,
Michael
>
> By the way, it's not necessary to run in a container.
>
> Signed-off-by: Huawei Xie <huawei.xie@intel.com>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>
> Jianfeng Tan (4):
> mem: add --single-file to create single mem-backed file
> mem: add API to obstain memory-backed file info
> virtio/vdev: add ways to interact with vhost
> virtio/vdev: add a new vdev named eth_cvio
>
> config/common_linuxapp | 5 +
> drivers/net/virtio/Makefile | 4 +
> drivers/net/virtio/vhost.c | 734 +++++++++++++++++++++++++++++
> drivers/net/virtio/vhost.h | 192 ++++++++
> drivers/net/virtio/virtio_ethdev.c | 338 ++++++++++---
> drivers/net/virtio/virtio_ethdev.h | 4 +
> drivers/net/virtio/virtio_pci.h | 52 +-
> drivers/net/virtio/virtio_rxtx.c | 11 +-
> drivers/net/virtio/virtio_rxtx_simple.c | 14 +-
> drivers/net/virtio/virtqueue.h | 13 +-
> lib/librte_eal/common/eal_common_options.c | 17 +
> lib/librte_eal/common/eal_internal_cfg.h | 1 +
> lib/librte_eal/common/eal_options.h | 2 +
> lib/librte_eal/common/include/rte_memory.h | 16 +
> lib/librte_eal/linuxapp/eal/eal_memory.c | 82 +++-
> 15 files changed, 1392 insertions(+), 93 deletions(-)
> create mode 100644 drivers/net/virtio/vhost.c
> create mode 100644 drivers/net/virtio/vhost.h
>
next prev parent reply other threads:[~2016-01-26 6:02 UTC|newest]
Thread overview: 196+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-05 18:31 [dpdk-dev] [RFC 0/5] " Jianfeng Tan
2015-11-05 18:31 ` [dpdk-dev] [RFC 1/5] virtio/container: add handler for ioport rd/wr Jianfeng Tan
2015-11-05 18:31 ` [dpdk-dev] [RFC 2/5] virtio/container: add a new virtual device named eth_cvio Jianfeng Tan
2015-11-05 18:31 ` [dpdk-dev] [RFC 3/5] virtio/container: unify desc->addr assignment Jianfeng Tan
2015-11-05 18:31 ` [dpdk-dev] [RFC 4/5] virtio/container: adjust memory initialization process Jianfeng Tan
2015-11-06 16:21 ` Ananyev, Konstantin
2015-11-08 11:18 ` Tan, Jianfeng
2015-11-09 13:32 ` Ananyev, Konstantin
2015-11-09 14:13 ` Tan, Jianfeng
2015-11-05 18:31 ` [dpdk-dev] [RFC 5/5] vhost/container: change mode of vhost listening socket Jianfeng Tan
2015-11-09 3:54 ` Yuanhan Liu
2015-11-09 5:15 ` Tan, Jianfeng
2015-11-09 5:40 ` Yuanhan Liu
2015-11-09 5:46 ` Tan, Jianfeng
2015-11-24 3:53 ` [dpdk-dev] [RFC 0/5] virtio support for container Zhuangyanying
2015-11-24 6:19 ` Tan, Jianfeng
2016-01-10 11:42 ` [dpdk-dev] [PATCH 0/4] " Jianfeng Tan
2016-01-10 11:42 ` [dpdk-dev] [PATCH 1/4] mem: add --single-file to create single mem-backed file Jianfeng Tan
2016-01-21 1:57 ` Xie, Huawei
2016-01-10 11:43 ` [dpdk-dev] [PATCH 2/4] mem: add API to obstain memory-backed file info Jianfeng Tan
2016-01-11 11:43 ` Pavel Fedin
2016-01-11 20:26 ` Rich Lane
2016-01-12 9:12 ` Tan, Jianfeng
2016-01-12 10:04 ` Pavel Fedin
2016-01-12 10:48 ` Tan, Jianfeng
2016-01-12 11:00 ` Pavel Fedin
2016-01-12 11:07 ` Sergio Gonzalez Monroy
2016-01-12 11:37 ` Pavel Fedin
2016-01-12 12:12 ` Sergio Gonzalez Monroy
2016-01-12 13:57 ` Pavel Fedin
2016-01-12 14:13 ` Sergio Gonzalez Monroy
2016-01-12 11:44 ` Sergio Gonzalez Monroy
2016-01-12 11:22 ` Pavel Fedin
2016-01-12 11:33 ` Sergio Gonzalez Monroy
2016-01-12 12:01 ` Pavel Fedin
2016-01-12 13:39 ` Sergio Gonzalez Monroy
2016-01-10 11:43 ` [dpdk-dev] [PATCH 3/4] virtio/vdev: add ways to interact with vhost Jianfeng Tan
2016-01-11 10:42 ` Pavel Fedin
2016-01-11 14:02 ` Pavel Fedin
2016-01-21 2:18 ` Xie, Huawei
2016-01-10 11:43 ` [dpdk-dev] [PATCH 4/4] virtio/vdev: add a new vdev named eth_cvio Jianfeng Tan
2016-01-12 7:45 ` Pavel Fedin
2016-01-12 7:59 ` Yuanhan Liu
2016-01-12 8:39 ` Tan, Jianfeng
2016-01-12 9:15 ` Tan, Jianfeng
2016-01-27 3:10 ` Qiu, Michael
2016-01-11 14:21 ` [dpdk-dev] [PATCH 0/4] virtio support for container Pavel Fedin
2016-01-11 15:53 ` Tan, Jianfeng
2016-01-12 7:38 ` Pavel Fedin
2016-01-12 8:14 ` Rich Lane
2016-01-12 8:39 ` Pavel Fedin
2016-01-12 8:51 ` Tan, Jianfeng
2016-01-12 10:48 ` Pavel Fedin
2016-01-12 14:45 ` Amit Tomer
2016-01-12 14:50 ` Pavel Fedin
2016-01-12 14:58 ` Amit Tomer
2016-01-12 14:53 ` Tan, Jianfeng
2016-01-12 15:11 ` Amit Tomer
2016-01-12 16:18 ` Tan, Jianfeng
2016-01-13 15:00 ` Amit Tomer
2016-01-13 18:41 ` Tan, Jianfeng
2016-01-14 9:34 ` Amit Tomer
2016-01-14 11:41 ` Tan, Jianfeng
2016-01-14 12:03 ` Amit Tomer
2016-01-15 6:39 ` Tan, Jianfeng
2016-01-20 15:19 ` Amit Tomer
2016-01-22 6:04 ` Tan, Jianfeng
2016-01-12 5:36 ` Tetsuya Mukawa
2016-01-12 5:46 ` Tan, Jianfeng
2016-01-12 6:01 ` Tetsuya Mukawa
2016-01-12 6:14 ` Yuanhan Liu
2016-01-12 6:26 ` Tetsuya Mukawa
2016-01-12 6:29 ` Yuanhan Liu
2016-01-20 3:48 ` Xie, Huawei
2016-01-26 6:02 ` Qiu, Michael [this message]
2016-01-26 6:09 ` Tan, Jianfeng
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 0/5] " Jianfeng Tan
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 1/5] mem: add --single-file to create single mem-backed file Jianfeng Tan
2016-03-07 13:13 ` Yuanhan Liu
2016-03-08 1:55 ` Tan, Jianfeng
2016-03-08 2:44 ` Yuanhan Liu
2016-03-09 14:44 ` Tan, Jianfeng
2016-03-08 8:49 ` Panu Matilainen
2016-03-08 9:04 ` Yuanhan Liu
2016-03-08 10:30 ` Thomas Monjalon
2016-03-08 10:57 ` Burakov, Anatoly
2016-03-14 13:53 ` Traynor, Kevin
2016-03-14 14:45 ` Thomas Monjalon
2016-03-14 18:21 ` Traynor, Kevin
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 2/5] mem: add API to obtain memory-backed file info Jianfeng Tan
2016-03-07 13:22 ` Yuanhan Liu
2016-03-08 2:31 ` Tan, Jianfeng
2016-03-08 2:53 ` Yuanhan Liu
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 3/5] virtio/vdev: add embeded device emulation Jianfeng Tan
2016-02-07 10:47 ` Michael S. Tsirkin
2016-02-08 6:59 ` Tetsuya Mukawa
2016-02-16 2:47 ` Tan, Jianfeng
2016-02-16 2:40 ` Tan, Jianfeng
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 4/5] virtio/vdev: add a new vdev named eth_cvio Jianfeng Tan
2016-02-05 11:20 ` [dpdk-dev] [PATCH v2 5/5] docs: add release note for virtio for container Jianfeng Tan
2016-03-23 19:17 ` [dpdk-dev] [PATCH v2 0/5] virtio support " Neil Horman
2016-03-24 3:10 ` Tan, Jianfeng
2016-03-24 13:45 ` Neil Horman
2016-03-25 1:25 ` Tan, Jianfeng
2016-03-25 11:06 ` Neil Horman
2016-04-13 16:14 ` Thomas Monjalon
2016-04-14 6:08 ` Tan, Jianfeng
2016-04-21 2:56 ` [dpdk-dev] [PATCH v3 0/2] " Jianfeng Tan
2016-04-21 2:56 ` [dpdk-dev] [PATCH v3 1/2] virtio/vdev: add embeded device emulation Jianfeng Tan
2016-04-21 22:01 ` Yuanhan Liu
2016-04-22 10:12 ` Tan, Jianfeng
2016-04-22 10:17 ` Thomas Monjalon
2016-04-22 17:27 ` Yuanhan Liu
2016-04-21 2:56 ` [dpdk-dev] [PATCH v3 2/2] virtio/vdev: add a new vdev named eth_cvio Jianfeng Tan
2016-04-21 8:51 ` David Marchand
2016-04-22 5:15 ` Tan, Jianfeng
2016-04-22 7:36 ` David Marchand
2016-04-22 10:25 ` Tan, Jianfeng
2016-04-21 10:05 ` Thomas Monjalon
2016-04-22 7:26 ` Tan, Jianfeng
2016-04-22 8:30 ` Thomas Monjalon
2016-04-21 22:14 ` Yuanhan Liu
2016-04-22 10:12 ` Tan, Jianfeng
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 0/8] virtio support for container Jianfeng Tan
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 1/8] virtio: hide phys addr check inside pci ops Jianfeng Tan
2016-05-11 23:05 ` Yuanhan Liu
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 2/8] virtio: abstract vring hdr desc init as a method Jianfeng Tan
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 3/8] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 4/8] virtio-user: add vhost adapter layer Jianfeng Tan
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 5/8] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-05-12 1:05 ` Yuanhan Liu
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 6/8] virtio-user: add new virtual pci driver for virtio Jianfeng Tan
2016-05-12 2:12 ` Yuanhan Liu
2016-05-12 7:08 ` Tan, Jianfeng
2016-05-12 16:40 ` Yuanhan Liu
2016-05-13 1:54 ` Tan, Jianfeng
2016-05-13 4:45 ` Yuanhan Liu
2016-05-16 1:48 ` Tan, Jianfeng
2016-05-16 2:51 ` Yuanhan Liu
2016-05-12 17:02 ` Michael S. Tsirkin
2016-05-13 2:00 ` Tan, Jianfeng
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 7/8] virtio-user: add a new virtual device named virtio-user Jianfeng Tan
2016-04-29 1:18 ` [dpdk-dev] [PATCH v4 8/8] doc: update doc for virtio-user Jianfeng Tan
2016-04-29 1:35 ` [dpdk-dev] [PATCH v4 0/8] virtio support for container Tan, Jianfeng
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 " Jianfeng Tan
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 1/8] virtio: hide phys addr check inside pci ops Jianfeng Tan
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 2/8] virtio: clean up virtio_dev_queue_setup Jianfeng Tan
2016-06-01 7:38 ` Yuanhan Liu
2016-06-01 7:44 ` Tan, Jianfeng
2016-06-01 7:58 ` Yuanhan Liu
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 3/8] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-06-01 8:03 ` Yuanhan Liu
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 4/8] virtio-user: add vhost adapter layer Jianfeng Tan
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 5/8] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 6/8] virtio-user: add new virtual pci driver for virtio Jianfeng Tan
2016-06-01 8:21 ` Yuanhan Liu
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 7/8] virtio-user: add a new vdev named virtio-user Jianfeng Tan
2016-06-01 8:26 ` Yuanhan Liu
2016-06-02 1:27 ` Tan, Jianfeng
2016-05-30 10:55 ` [dpdk-dev] [PATCH v5 8/8] doc: update doc for virtio-user Jianfeng Tan
2016-06-01 8:30 ` Yuanhan Liu
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 0/7] virtio support for container Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 1/7] virtio: hide phys addr check inside pci ops Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 2/7] virtio: clean up virtio_dev_queue_setup Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 3/7] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 4/7] virtio-user: add vhost adapter layer Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 5/7] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 6/7] virtio-user: add new virtual pci driver for virtio Jianfeng Tan
2016-06-06 8:01 ` Yuanhan Liu
2016-06-06 8:31 ` Tan, Jianfeng
2016-06-02 9:54 ` [dpdk-dev] [PATCH v6 7/7] virtio-user: add a new vdev named virtio-user Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 0/6] virtio support for container Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 1/6] virtio: hide phys addr check inside pci ops Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 2/6] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 3/6] virtio-user: add vhost adapter layer Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 4/6] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 5/6] virtio-user: add new virtual pci driver for virtio Jianfeng Tan
2016-06-12 0:35 ` [dpdk-dev] [PATCH v7 6/6] virtio-user: add a new vdev named virtio-user Jianfeng Tan
2016-06-13 6:38 ` [dpdk-dev] [PATCH v8 0/6] virtio support for container Jianfeng Tan
2016-06-13 6:38 ` [dpdk-dev] [PATCH v8 1/6] virtio: hide phys addr check inside pci ops Jianfeng Tan
2016-06-13 6:38 ` [dpdk-dev] [PATCH v8 2/6] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-06-13 6:39 ` [dpdk-dev] [PATCH v8 3/6] virtio-user: add vhost user adapter layer Jianfeng Tan
2016-06-13 6:39 ` [dpdk-dev] [PATCH v8 4/6] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-06-13 6:39 ` [dpdk-dev] [PATCH v8 5/6] virtio-user: add new virtual pci driver for virtio Jianfeng Tan
2016-06-13 6:39 ` [dpdk-dev] [PATCH v8 6/6] virtio-user: add a new vdev named virtio-user Jianfeng Tan
2016-06-14 8:34 ` [dpdk-dev] [PATCH v8 0/6] virtio support for container Yuanhan Liu
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 " Jianfeng Tan
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 1/6] virtio: hide phys addr check inside PCI ops Jianfeng Tan
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 2/6] virtio: enable use virtual address to fill desc Jianfeng Tan
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 3/6] virtio-user: add vhost user adapter layer Jianfeng Tan
2016-06-23 9:01 ` Ferruh Yigit
2016-06-24 1:18 ` Tan, Jianfeng
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 4/6] virtio-user: add device emulation layer APIs Jianfeng Tan
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 5/6] virtio-user: add new virtual PCI driver for virtio Jianfeng Tan
2016-06-15 9:03 ` [dpdk-dev] [PATCH v9 6/6] virtio-user: add a new vdev named virtio-user Jianfeng Tan
2016-06-15 9:54 ` [dpdk-dev] [PATCH v9 0/6] virtio support for container Yuanhan Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=533710CFB86FA344BFBF2D6802E6028622F226FD@SHSMSX101.ccr.corp.intel.com \
--to=michael.qiu@intel.com \
--cc=ann.zhuangyanying@huawei.com \
--cc=dev@dpdk.org \
--cc=jianfeng.tan@intel.com \
--cc=mst@redhat.com \
--cc=nakajima.yoshihiro@lab.ntt.co.jp \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).