DPDK patches and discussions
 help / color / mirror / Atom feed
From: Bruce Richardson <bruce.richardson@intel.com>
To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	"Thomas Monjalon" <thomas@monjalon.net>,
	"David Marchand" <david.marchand@redhat.com>,
	"Ferruh Yigit" <ferruh.yigit@intel.com>,
	"Andrew Rybchenko" <arybchenko@solarflare.com>,
	"Ajit Khaparde" <ajit.khaparde@broadcom.com>,
	"Qi Zhang" <qi.z.zhang@intel.com>,
	"Xiaolong Ye" <xiaolong.ye@intel.com>,
	"Raslan Darawsheh" <rasland@mellanox.com>,
	"Maxime Coquelin" <maxime.coquelin@redhat.com>,
	"Tiwei Bie" <tiwei.bie@intel.com>,
	"Akhil Goyal" <akhil.goyal@nxp.com>,
	"Luca Boccassi" <bluca@debian.org>,
	"Kevin Traynor" <ktraynor@redhat.com>,
	"maintainers@dpdk.org" <maintainers@dpdk.org>,
	"John McNamara" <john.mcnamara@intel.com>,
	"Marko Kovacevic" <marko.kovacevic@intel.com>,
	"Ray Kinsella" <mdr@ashroe.eu>,
	"Aaron Conole" <aconole@redhat.com>,
	"Michael Santana" <maicolgabriel@hotmail.com>,
	"Harry van Haaren" <harry.van.haaren@intel.com>,
	"Cristian Dumitrescu" <cristian.dumitrescu@intel.com>,
	"Phil Yang" <phil.yang@arm.com>,
	"Joyce Kong" <joyce.kong@arm.com>,
	"Mattias Rönnblom" <mattias.ronnblom@ericsson.com>,
	"Jan Viktorin" <viktorin@rehivetech.com>,
	"Gavin Hu" <gavin.hu@arm.com>,
	"David Christensen" <drc@linux.vnet.ibm.com>,
	"Konstantin Ananyev" <konstantin.ananyev@intel.com>,
	"Anatoly Burakov" <anatoly.burakov@intel.com>,
	"Harini Ramakrishnan" <harini.ramakrishnan@microsoft.com>,
	"Omar Cardona" <ocardona@microsoft.com>,
	"Anand Rawat" <anand.rawat@intel.com>,
	"Ranjit Menon" <ranjit.menon@intel.com>,
	"Olivier Matz" <olivier.matz@6wind.com>,
	"Gage Eads" <gage.eads@intel.com>,
	"Adrien Mazarguil" <adrien.mazarguil@6wind.com>,
	"Nicolas Chautru" <nicolas.chautru@intel.com>,
	"Declan Doherty" <declan.doherty@intel.com>,
	"Fiona Trahe" <fiona.trahe@intel.com>,
	"Ashish Gupta" <ashishg@marvell.com>,
	"Erik Gabriel Carrillo" <erik.g.carrillo@intel.com>,
	"Abhinandan Gujjar" <abhinandan.gujjar@intel.com>,
	"Shreyansh Jain" <shreyansh.jain@nxp.com>,
	"Hemant Agrawal" <hemant.agrawal@nxp.com>,
	"Artem V. Andreev" <artem.andreev@oktetlabs.ru>,
	"Nithin Kumar Dabilpuram" <ndabilpuram@marvell.com>,
	"Vamsi Krishna Attunuru" <vattunuru@marvell.com>,
	"Rosen Xu" <rosen.xu@intel.com>,
	"Sachin Saxena" <sachin.saxena@nxp.com>,
	"Stephen Hemminger" <sthemmin@microsoft.com>,
	"Chas Williams" <chas3@att.com>,
	"John W. Linville" <linville@tuxdriver.com>,
	"Prasun Kapoor" <pkapoor@marvell.com>,
	"Marcin Wojtas" <mw@semihalf.com>,
	"Michal Krawczyk" <mk@semihalf.com>,
	"Guy Tzalik" <gtzalik@amazon.com>,
	"Evgeny Schemeilin" <evgenys@amazon.com>,
	"Igor Chauskin" <igorch@amazon.com>,
	"Ravi Kumar" <ravi1.kumar@amd.com>,
	"Igor Russkikh" <igor.russkikh@aquantia.com>,
	"Pavel Belous" <pavel.belous@aquantia.com>,
	"Shepard Siegel" <shepard.siegel@atomicrules.com>,
	"Ed Czeck" <ed.czeck@atomicrules.com>,
	"John Miller" <john.miller@atomicrules.com>,
	"Somnath Kotur" <somnath.kotur@broadcom.com>,
	"Maciej Czekaj" <mczekaj@marvell.com>,
	"Shijith Thotton" <sthotton@marvell.com>,
	"Srisivasubramanian Srinivasan" <srinivasan@marvell.com>,
	"Rahul Lakkireddy" <rahul.lakkireddy@chelsio.com>,
	"John Daley" <johndale@cisco.com>,
	"Hyong Youb Kim" <hyonkim@cisco.com>,
	"Wei Hu (Xavier" <xavier.huwei@huawei.com>,
	"Min Hu (Connor" <humin29@huawei.com>,
	"Yisen Zhuang" <yisen.zhuang@huawei.com>,
	"Ziyang Xuan" <xuanziyang2@huawei.com>,
	"Xiaoyun Wang" <cloud.wangxiaoyun@huawei.com>,
	"Guoyang Zhou" <zhouguoyang@huawei.com>,
	"Beilei Xing" <beilei.xing@intel.com>,
	"Xiao Wang" <xiao.w.wang@intel.com>,
	"Jingjing Wu" <jingjing.wu@intel.com>,
	"Wenzhuo Lu" <wenzhuo.lu@intel.com>,
	"Qiming Yang" <qiming.yang@intel.com>,
	"Tomasz Duszynski" <tdu@semihalf.com>,
	"Liron Himi" <lironh@marvell.com>, "Zyta Szpak" <zr@semihalf.com>,
	"Kiran Kumar Kokkilagadda" <kirankumark@marvell.com>,
	"Matan Azrad" <matan@mellanox.com>,
	"Shahaf Shuler" <shahafs@mellanox.com>,
	"Viacheslav Ovsiienko" <viacheslavo@mellanox.com>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	"Haiyang Zhang" <haiyangz@microsoft.com>,
	"Jan Remes" <remes@netcope.com>,
	"Heinrich Kuhn" <heinrich.kuhn@netronome.com>,
	"Jan Gutter" <jan.gutter@netronome.com>,
	"Gagandeep Singh" <g.singh@nxp.com>,
	"Rasesh Mody" <rmody@marvell.com>,
	"Shahed Shaikh" <shshaikh@marvell.com>,
	"Yong Wang" <yongwang@vmware.com>,
	"Zhihong Wang" <zhihong.wang@intel.com>,
	"Steven Webster" <steven.webster@windriver.com>,
	"Matt Peters" <matt.peters@windriver.com>,
	"Keith Wiles" <keith.wiles@intel.com>,
	"Tetsuya Mukawa" <mtetsuyah@gmail.com>,
	"Gaetan Rivet" <gaetan.rivet@6wind.com>,
	"Jasvinder Singh" <jasvinder.singh@intel.com>,
	"Jakub Grajciar" <jgrajcia@cisco.com>,
	"Ruifeng Wang" <ruifeng.wang@arm.com>,
	"Anoob Joseph" <anoobj@marvell.com>,
	"Fan Zhang" <roy.fan.zhang@intel.com>,
	"Pablo de Lara" <pablo.de.lara.guarch@intel.com>,
	"John Griffin" <john.griffin@intel.com>,
	"Deepak Kumar Jain" <deepak.k.jain@intel.com>,
	"Michael Shamis" <michaelsh@marvell.com>,
	"Nagadheeraj Rottela" <rnagadheeraj@marvell.com>,
	"Srikanth Jampala" <jsrikanth@marvell.com>,
	"Ankur Dwivedi" <adwivedi@marvell.com>,
	"Jay Zhou" <jianjay.zhou@huawei.com>,
	"Lee Daly" <lee.daly@intel.com>,
	"Sunila Sahu" <ssahu@marvell.com>,
	"Nipun Gupta" <nipun.gupta@nxp.com>,
	"Liang Ma" <liang.j.ma@intel.com>,
	"Peter Mccarthy" <peter.mccarthy@intel.com>,
	"Tianfei zhang" <tianfei.zhang@intel.com>,
	"Satha Koteswara Rao Kottidi" <skoteshwar@marvell.com>,
	"Xiaoyun Li" <xiaoyun.li@intel.com>,
	"Bernard Iremonger" <bernard.iremonger@intel.com>,
	"Vladimir Medvedkin" <vladimir.medvedkin@intel.com>,
	"David Hunt" <david.hunt@intel.com>,
	"Reshma Pattan" <reshma.pattan@intel.com>,
	"Byron Marohn" <byron.marohn@intel.com>,
	"Sameh Gobriel" <sameh.gobriel@intel.com>,
	"Yipeng Wang" <yipeng1.wang@intel.com>,
	"Honnappa Nagarahalli" <honnappa.nagarahalli@arm.com>,
	"Robert Sanford" <rsanford@akamai.com>,
	"Kevin Laatz" <kevin.laatz@intel.com>,
	"Maryam Tahhan" <maryam.tahhan@intel.com>,
	"Ori Kam" <orika@mellanox.com>,
	"Radu Nicolau" <radu.nicolau@intel.com>,
	"Tomasz Kantecki" <tomasz.kantecki@intel.com>,
	"Sunil Kumar Kori" <skori@marvell.com>,
	"Pavan Nikhilesh Bhagavatula" <pbhagavatula@marvell.com>,
	"Kirill Rybalchenko" <kirill.rybalchenko@intel.com>,
	"Kadam, Pallavi" <pallavi.kadam@intel.com>
Subject: Re: [dpdk-dev] [RFC]  DPDK Trace support
Date: Mon, 13 Jan 2020 13:05:43 +0000	[thread overview]
Message-ID: <20200113130543.GC1645@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <BYAPR18MB2424BF2CAD9AFE5F1C3755F0C8350@BYAPR18MB2424.namprd18.prod.outlook.com>

On Mon, Jan 13, 2020 at 10:40:13AM +0000, Jerin Jacob Kollanukkaran wrote:
> Hi All,
> 
> I would like to add tracing support for DPDK.
> I am planning to add this support in v20.05 release.
> 
> This RFC attempts to get feedback from the community on
> 
> a) Tracing Use cases.
> b) Tracing Requirements.
> b) Implementation choices.
> c) Trace format.
> 
> Use-cases
> ---------
> - Most of the cases, The DPDK provider will not have access to the DPDK customer applications.
> To debug/analyze the slow path and fast path DPDK API usage from the field,
> we need to have integrated trace support in DPDK.
> 
> - Need a low overhead Fast path multi-core PMD driver debugging/analysis
> infrastructure in DPDK to fix the functional and performance issue(s) of PMD.
> 
> - Post trace analysis tools can provide various status across the system such
> as cpu_idle() using the timestamp added in the trace.
> 
> 
> Requirements:
> -------------
> - Support for Linux, FreeBSD and Windows OS
> - Open trace format
> - Multi-platform Open source trace viewer
> - Absolute low overhead trace API for DPDK fast path tracing/debugging.
> - Dynamic enable/disable of trace events
> 
> 
> To enable trace support in DPDK, following items need to work out: 
> 
> a) Add the DPDK trace points in the DPDK source code.
> 
> - This includes updating DPDK functions such as,
> rte_eth_dev_configure(), rte_eth_dev_start(), rte_eth_dev_rx_burst() to emit the trace.
> 
> b) Choosing suitable serialization-format
> 
> - Common Trace Format, CTF, is an open format and language to describe trace formats.
> This enables tool reuse, of which line-textual (babeltrace) and 
> graphical (TraceCompass) variants already exist.
> 
> CTF should look familiar to C programmers but adds stronger typing. 
> See CTF - A Flexible, High-performance Binary Trace Format.
> 
> https://diamon.org/ctf/
> 
> c) Writing the on-target serialization code,
> 
> See the section below.(Lttng CTF trace emitter vs DPDK specific CTF trace emitter)
>  
> d) Deciding on and writing the I/O transport mechanics,
> 
> For performance reasons, it should be backed by a huge-page and write to file IO.
> 
> e) Writing the PC-side deserializer/parser,
> 
> Both the babletrace(CLI tool) and Trace Compass(GUI tool) support CTF.
> See: 
> https://lttng.org/viewers/
> 
> f) Writing tools for filtering and presentation.
> 
> See item (e)
> 
> 
> Lttng CTF trace emitter vs DPDK specific CTF trace emitter
> ----------------------------------------------------------
> 
> I have written a performance evaluation application to measure the overhead
> of Lttng CTF emitter(The fastpath infrastructure used by https://lttng.org/ library to emit the trace)
> 
> https://github.com/jerinjacobk/lttng-overhead
> https://github.com/jerinjacobk/lttng-overhead/blob/master/README
> 
> I could improve the performance by 30% by adding the "DPDK"
> based plugin for get_clock() and get_cpu(),
> Here are the performance numbers after adding the plugin on 
> x86 and various arm64 board that I have access to,
> 
> On high-end x86, it comes around 236 cycles/~100ns @ 2.4GHz (See the last line in the log(ZERO_ARG)) 
> On arm64, it varies from 312 cycles to 1100 cycles(based on the class of CPU).
> In short, Based on the "IPC capabilities", The cost would be around 100ns to 400ns
> for single void trace(a trace without any argument)
> 
> 
> [lttng-overhead-x86] $ sudo ./calibrate/build/app/calibrate -c 0xc0
> make[1]: Entering directory '/export/lttng-overhead-x86/calibrate'
> make[1]: Leaving directory '/export/lttng-overhead-x86/calibrate'
> EAL: Detected 56 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Probing VFIO support...
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> CPU Timer freq is 2600.000000MHz
> NOP: cycles=0.194834 ns=0.074936
> GET_CLOCK: cycles=47.854658 ns=18.405638
> GET_CPU: cycles=30.995892 ns=11.921497
> ZERO_ARG: cycles=236.945113 ns=91.132736
> 
> 
> We will have only 16.75ns to process 59.2 mpps(40Gbps), So IMO, Lttng CTF emitter
> may not fit the DPDK fast path purpose due to the cost associated with generic Lttng features.
> 
> One option could be to have, native CTF emitter in EAL/DPDK to emit the
> trace in a hugepage. I think it would be a handful of cycles if we limit the features
> to the requirements above:
> 
> The upside of using Lttng CTF emitter:
> a) No need to write a new CTF trace emitter(the item (c))
> 
> The downside of Lttng CTF emitter(the item (c))
> a) performance issue(See above)
> b) Lack of Windows OS support. It looks like, it has basic FreeBSD support.
> c) dpdk library dependency to lttng for trace.
> 
> So, Probably it good to have native CTF emitter in DPDK and reuse all
> open-source trace viewer(babeltrace and  TraceCompass) and format(CTF) infrastructure.
> I think, it would be best of both world.
> 
> Any thoughts on this subject? Based on the community feedback, I can work on the patch for v20.05.

Forgive my ignorance of LTTng, but is there the concept of
enabling/disabling the trace points? If so, the overhead you refer to, that
is presumably with the trace enabled?

/Bruce

  parent reply	other threads:[~2020-01-13 13:06 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-13 10:40 Jerin Jacob Kollanukkaran
2020-01-13 11:00 ` Ray Kinsella
2020-01-13 12:04   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2020-01-18 15:14   ` [dpdk-dev] " dave
2020-01-20 16:51     ` Stephen Hemminger
2020-01-13 13:05 ` Bruce Richardson [this message]
2020-01-13 14:46   ` Jerin Jacob
2020-01-13 14:58     ` Bruce Richardson
2020-01-13 15:13       ` Jerin Jacob
2020-01-13 16:12         ` Bruce Richardson
2020-01-17  4:41           ` Jerin Jacob
2020-01-17  8:04             ` David Marchand
2020-01-17  9:52               ` Jerin Jacob
2020-01-17 10:30                 ` Mattias Rönnblom
2020-01-17 10:54                   ` Jerin Jacob
2020-02-15 10:21                     ` Jerin Jacob
2020-02-17  9:35                       ` Mattias Rönnblom
2020-02-17 10:23                         ` Jerin Jacob
2020-01-17 10:43                 ` David Marchand
2020-01-17 11:08                   ` Jerin Jacob
2020-01-27 16:12 ` Aaron Conole
2020-01-27 17:23   ` Jerin Jacob
2020-01-20  4:48 Jerin Jacob Kollanukkaran
2020-01-20 12:08 ` Ray Kinsella

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200113130543.GC1645@bricha3-MOBL.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=abhinandan.gujjar@intel.com \
    --cc=aconole@redhat.com \
    --cc=adrien.mazarguil@6wind.com \
    --cc=adwivedi@marvell.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akhil.goyal@nxp.com \
    --cc=anand.rawat@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=anoobj@marvell.com \
    --cc=artem.andreev@oktetlabs.ru \
    --cc=arybchenko@solarflare.com \
    --cc=ashishg@marvell.com \
    --cc=beilei.xing@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=bluca@debian.org \
    --cc=byron.marohn@intel.com \
    --cc=chas3@att.com \
    --cc=cloud.wangxiaoyun@huawei.com \
    --cc=cristian.dumitrescu@intel.com \
    --cc=david.hunt@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=declan.doherty@intel.com \
    --cc=deepak.k.jain@intel.com \
    --cc=dev@dpdk.org \
    --cc=drc@linux.vnet.ibm.com \
    --cc=ed.czeck@atomicrules.com \
    --cc=erik.g.carrillo@intel.com \
    --cc=evgenys@amazon.com \
    --cc=ferruh.yigit@intel.com \
    --cc=fiona.trahe@intel.com \
    --cc=g.singh@nxp.com \
    --cc=gaetan.rivet@6wind.com \
    --cc=gage.eads@intel.com \
    --cc=gavin.hu@arm.com \
    --cc=gtzalik@amazon.com \
    --cc=haiyangz@microsoft.com \
    --cc=harini.ramakrishnan@microsoft.com \
    --cc=harry.van.haaren@intel.com \
    --cc=heinrich.kuhn@netronome.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=humin29@huawei.com \
    --cc=hyonkim@cisco.com \
    --cc=igor.russkikh@aquantia.com \
    --cc=igorch@amazon.com \
    --cc=jan.gutter@netronome.com \
    --cc=jasvinder.singh@intel.com \
    --cc=jerinj@marvell.com \
    --cc=jgrajcia@cisco.com \
    --cc=jianjay.zhou@huawei.com \
    --cc=jingjing.wu@intel.com \
    --cc=john.griffin@intel.com \
    --cc=john.mcnamara@intel.com \
    --cc=john.miller@atomicrules.com \
    --cc=johndale@cisco.com \
    --cc=joyce.kong@arm.com \
    --cc=jsrikanth@marvell.com \
    --cc=keith.wiles@intel.com \
    --cc=kevin.laatz@intel.com \
    --cc=kirankumark@marvell.com \
    --cc=kirill.rybalchenko@intel.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=kys@microsoft.com \
    --cc=lee.daly@intel.com \
    --cc=liang.j.ma@intel.com \
    --cc=linville@tuxdriver.com \
    --cc=lironh@marvell.com \
    --cc=maicolgabriel@hotmail.com \
    --cc=maintainers@dpdk.org \
    --cc=marko.kovacevic@intel.com \
    --cc=maryam.tahhan@intel.com \
    --cc=matan@mellanox.com \
    --cc=matt.peters@windriver.com \
    --cc=mattias.ronnblom@ericsson.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mczekaj@marvell.com \
    --cc=mdr@ashroe.eu \
    --cc=michaelsh@marvell.com \
    --cc=mk@semihalf.com \
    --cc=mtetsuyah@gmail.com \
    --cc=mw@semihalf.com \
    --cc=ndabilpuram@marvell.com \
    --cc=nicolas.chautru@intel.com \
    --cc=nipun.gupta@nxp.com \
    --cc=ocardona@microsoft.com \
    --cc=olivier.matz@6wind.com \
    --cc=orika@mellanox.com \
    --cc=pablo.de.lara.guarch@intel.com \
    --cc=pallavi.kadam@intel.com \
    --cc=pavel.belous@aquantia.com \
    --cc=pbhagavatula@marvell.com \
    --cc=peter.mccarthy@intel.com \
    --cc=phil.yang@arm.com \
    --cc=pkapoor@marvell.com \
    --cc=qi.z.zhang@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=radu.nicolau@intel.com \
    --cc=rahul.lakkireddy@chelsio.com \
    --cc=ranjit.menon@intel.com \
    --cc=rasland@mellanox.com \
    --cc=ravi1.kumar@amd.com \
    --cc=remes@netcope.com \
    --cc=reshma.pattan@intel.com \
    --cc=rmody@marvell.com \
    --cc=rnagadheeraj@marvell.com \
    --cc=rosen.xu@intel.com \
    --cc=roy.fan.zhang@intel.com \
    --cc=rsanford@akamai.com \
    --cc=ruifeng.wang@arm.com \
    --cc=sachin.saxena@nxp.com \
    --cc=sameh.gobriel@intel.com \
    --cc=shahafs@mellanox.com \
    --cc=shepard.siegel@atomicrules.com \
    --cc=shreyansh.jain@nxp.com \
    --cc=shshaikh@marvell.com \
    --cc=skori@marvell.com \
    --cc=skoteshwar@marvell.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=srinivasan@marvell.com \
    --cc=ssahu@marvell.com \
    --cc=steven.webster@windriver.com \
    --cc=sthemmin@microsoft.com \
    --cc=sthotton@marvell.com \
    --cc=tdu@semihalf.com \
    --cc=thomas@monjalon.net \
    --cc=tianfei.zhang@intel.com \
    --cc=tiwei.bie@intel.com \
    --cc=tomasz.kantecki@intel.com \
    --cc=vattunuru@marvell.com \
    --cc=viacheslavo@mellanox.com \
    --cc=viktorin@rehivetech.com \
    --cc=vladimir.medvedkin@intel.com \
    --cc=wenzhuo.lu@intel.com \
    --cc=xavier.huwei@huawei.com \
    --cc=xiao.w.wang@intel.com \
    --cc=xiaolong.ye@intel.com \
    --cc=xiaoyun.li@intel.com \
    --cc=xuanziyang2@huawei.com \
    --cc=yipeng1.wang@intel.com \
    --cc=yisen.zhuang@huawei.com \
    --cc=yongwang@vmware.com \
    --cc=zhihong.wang@intel.com \
    --cc=zhouguoyang@huawei.com \
    --cc=zr@semihalf.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).