From: "Wang, YuanX" <yuanx.wang@intel.com>
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
"dev@dpdk.org" <dev@dpdk.org>,
Thomas Monjalon <thomas@monjalon.net>,
Ferruh Yigit <ferruh.yigit@amd.com>
Cc: "ferruh.yigit@xilinx.com" <ferruh.yigit@xilinx.com>,
"mdr@ashroe.eu" <mdr@ashroe.eu>,
"Li, Xiaoyun" <xiaoyun.li@intel.com>,
"Singh, Aman Deep" <aman.deep.singh@intel.com>,
"Zhang, Yuying" <yuying.zhang@intel.com>,
"Zhang, Qi Z" <qi.z.zhang@intel.com>,
"Yang, Qiming" <qiming.yang@intel.com>,
"jerinjacobk@gmail.com" <jerinjacobk@gmail.com>,
"viacheslavo@nvidia.com" <viacheslavo@nvidia.com>,
"stephen@networkplumber.org" <stephen@networkplumber.org>,
"Ding, Xuan" <xuan.ding@intel.com>,
"hpothula@marvell.com" <hpothula@marvell.com>,
"Tang, Yaqi" <yaqi.tang@intel.com>
Subject: RE: [PATCH v7 2/4] ethdev: introduce protocol hdr based buffer split
Date: Tue, 4 Oct 2022 02:48:52 +0000 [thread overview]
Message-ID: <PH7PR11MB69537A73E3AF44D24C05842E855A9@PH7PR11MB6953.namprd11.prod.outlook.com> (raw)
In-Reply-To: <29eb4f17-30a1-fff8-7be4-0092dbd79f5c@oktetlabs.ru>
Hi Andrew,
> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Sent: Monday, October 3, 2022 3:47 PM
> To: Wang, YuanX <yuanx.wang@intel.com>; dev@dpdk.org; Thomas
> Monjalon <thomas@monjalon.net>; Ferruh Yigit <ferruh.yigit@amd.com>
> Cc: ferruh.yigit@xilinx.com; mdr@ashroe.eu; Li, Xiaoyun
> <xiaoyun.li@intel.com>; Singh, Aman Deep <aman.deep.singh@intel.com>;
> Zhang, Yuying <yuying.zhang@intel.com>; Zhang, Qi Z
> <qi.z.zhang@intel.com>; Yang, Qiming <qiming.yang@intel.com>;
> jerinjacobk@gmail.com; viacheslavo@nvidia.com;
> stephen@networkplumber.org; Ding, Xuan <xuan.ding@intel.com>;
> hpothula@marvell.com; Tang, Yaqi <yaqi.tang@intel.com>; Wenxuan Wu
> <wenxuanx.wu@intel.com>
> Subject: Re: [PATCH v7 2/4] ethdev: introduce protocol hdr based buffer split
>
> On 10/2/22 00:05, Yuan Wang wrote:
> > Currently, Rx buffer split supports length based split. With Rx queue
> > offload RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT enabled and Rx packet
> segment
> > configured, PMD will be able to split the received packets into
> > multiple segments.
> >
> > However, length based buffer split is not suitable for NICs that do
> > split based on protocol headers. Given an arbitrarily variable length
> > in Rx packet segment, it is almost impossible to pass a fixed protocol
> > header to driver. Besides, the existence of tunneling results in the
> > composition of a packet is various, which makes the situation even worse.
> >
> > This patch extends current buffer split to support protocol header
> > based buffer split. A new proto_hdr field is introduced in the
> > reserved field of rte_eth_rxseg_split structure to specify protocol
> > header. The proto_hdr field defines the split position of packet,
> > splitting will always happen after the protocol header defined in the
> > Rx packet segment. When Rx queue offload
> > RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT is enabled and corresponding
> protocol
> > header is configured, driver will split the ingress packets into multiple
> segments.
> >
> > Examples for proto_hdr field defines:
> > To split after ETH-IPV4-UDP, it should be defined as
> > RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4_EXT_UNKNOWN |
> RTE_PTYPE_L4_UDP
> >
> > For inner ETH-IPV4-UDP, it should be defined as
> > RTE_PTYPE_TUNNEL_GRENAT | RTE_PTYPE_INNER_L2_ETHER |
> > RTE_PTYPE_INNER_L3_IPV4_EXT_UNKNOWN | RTE_PTYPE_INNER_L4_UDP
> >
> > struct rte_eth_rxseg_split {
> > struct rte_mempool *mp; /* memory pools to allocate segment from
> */
> > uint16_t length; /* segment maximal data length,
> > configures split point */
> > uint16_t offset; /* data offset from beginning
> > of mbuf data buffer */
> > /**
> > * Proto_hdr defines a bit mask of the protocol sequence as
> > * RTE_PTYPE_*, configures split point. The last RTE_PTYPE*
> > * in the mask indicates the split position.
> > * For non-tunneling packets, the complete protocol sequence
> > * should be defined.
> > * For tunneling packets, for simplicity, only the tunnel and
> > * inner protocol sequence should be defined.
> > */
> > uint32_t proto_hdr;
> > };
> >
> > If protocol header split can be supported by a PMD, the
> > rte_eth_buffer_split_get_supported_hdr_ptypes function can be use to
> > obtain a list of these protocol headers.
> >
> > For example, let's suppose we configured the Rx queue with the
> > following segments:
> > seg0 - pool0, proto_hdr0=RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
> > off0=2B
> > seg1 - pool1, proto_hdr1=RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4
> > | RTE_PTYPE_L4_UDP, off1=128B
> > seg2 - pool2, off1=0B
> >
> > The packet consists of ETH_IPV4_UDP_PAYLOAD will be split like
> > following:
> > seg0 - ipv4 header @ RTE_PKTMBUF_HEADROOM + 2 in mbuf from
> pool0
> > seg1 - udp header @ 128 in mbuf from pool1
> > seg2 - payload @ 0 in mbuf from pool2
> >
> > Note: NIC will only do split when the packets exactly match all the
> > protocol headers in the segments. For example, if ARP packets received
> > with above config, the NIC won't do split for ARP packets since it
> > does not contains ipv4 header and udp header. These packets will be
> > put into the last valid mempool, with zero offset.
> >
> > Now buffer split can be configured in two modes. For length based
> > buffer split, the mp, length, offset field in Rx packet segment should
> > be configured, while the proto_hdr field will be ignored.
> > For protocol header based buffer split, the mp, offset, proto_hdr
> > field in Rx packet segment should be configured, while the length
> > field will be ignored.
> >
> > The split limitations imposed by underlying driver is reported in the
> > rte_eth_dev_info->rx_seg_capa field. The memory attributes for the
> > split parts may differ either, dpdk memory and external memory,
> respectively.
> >
> > Signed-off-by: Yuan Wang <yuanx.wang@intel.com>
> > Signed-off-by: Xuan Ding <xuan.ding@intel.com>
> > Signed-off-by: Wenxuan Wu <wenxuanx.wu@intel.com>
>
> I apologize for delay with review. Overall LGTM now. See few notes below.
Thanks so much for your time and patience for this patch series.
>
> > ---
> > doc/guides/rel_notes/release_22_11.rst | 7 +++
> > lib/ethdev/rte_ethdev.c | 74 ++++++++++++++++++++++----
> > lib/ethdev/rte_ethdev.h | 29 +++++++++-
> > 3 files changed, 98 insertions(+), 12 deletions(-)
> >
> > diff --git a/doc/guides/rel_notes/release_22_11.rst
> > b/doc/guides/rel_notes/release_22_11.rst
> > index 6a7474a3d6..510869c73a 100644
> > --- a/doc/guides/rel_notes/release_22_11.rst
> > +++ b/doc/guides/rel_notes/release_22_11.rst
> > @@ -101,6 +101,13 @@ New Features
> > * Added ``rte_eth_buffer_split_get_supported_hdr_ptypes()``, to get
> supported
> > header protocols of a PMD to split.
> >
> > +* **Added protocol header based buffer split.**
> > +
> > + * Ethdev: The ``reserved`` field in the ``rte_eth_rxseg_split`` structure is
> > + replaced with ``proto_hdr`` to support protocol header based buffer
> split.
> > + User can choose length or protocol header to configure buffer split
> > + according to NIC's capability.
> > +
>
> It should be grouped together with other ethdev features.
We will send a new version. For the doc changes, the same as patch 1, could you help to adjust the doc?
Thanks very much.
>
> >
> > Removed Items
> > -------------
> > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index
> > 1f0a7f8f3f..27ec19faed 100644
> > --- a/lib/ethdev/rte_ethdev.c
> > +++ b/lib/ethdev/rte_ethdev.c
> > @@ -1649,9 +1649,10 @@ rte_eth_dev_is_removed(uint16_t port_id)
> > }
> >
> > static int
> > -rte_eth_rx_queue_check_split(const struct rte_eth_rxseg_split *rx_seg,
> > - uint16_t n_seg, uint32_t *mbp_buf_size,
> > - const struct rte_eth_dev_info *dev_info)
> > +rte_eth_rx_queue_check_split(uint16_t port_id,
> > + const struct rte_eth_rxseg_split *rx_seg,
> > + uint16_t n_seg, uint32_t *mbp_buf_size,
> > + const struct rte_eth_dev_info *dev_info)
> > {
> > const struct rte_eth_rxseg_capa *seg_capa = &dev_info-
> >rx_seg_capa;
> > struct rte_mempool *mp_first;
> > @@ -1674,6 +1675,7 @@ rte_eth_rx_queue_check_split(const struct
> rte_eth_rxseg_split *rx_seg,
> > struct rte_mempool *mpl = rx_seg[seg_idx].mp;
> > uint32_t length = rx_seg[seg_idx].length;
> > uint32_t offset = rx_seg[seg_idx].offset;
> > + uint32_t proto_hdr = rx_seg[seg_idx].proto_hdr;
> >
> > if (mpl == NULL) {
> > RTE_ETHDEV_LOG(ERR, "null mempool pointer\n");
> @@ -1707,13
> > +1709,63 @@ rte_eth_rx_queue_check_split(const struct
> rte_eth_rxseg_split *rx_seg,
> > }
> > offset += seg_idx != 0 ? 0 : RTE_PKTMBUF_HEADROOM;
> > *mbp_buf_size = rte_pktmbuf_data_room_size(mpl);
> > - length = length != 0 ? length : *mbp_buf_size;
> > - if (*mbp_buf_size < length + offset) {
> > - RTE_ETHDEV_LOG(ERR,
> > - "%s mbuf_data_room_size %u < %u
> (segment length=%u + segment offset=%u)\n",
> > - mpl->name, *mbp_buf_size,
> > - length + offset, length, offset);
> > - return -EINVAL;
> > +
> > + if (proto_hdr > 0) {
> > + /* Split based on protocol headers. */
>
> Isn't safer here to ensure that segment length is set to 0?
> Just to protect agains misusage etc.
It's a reasonable suggestion, I will take it, please see v8.
>
> > +
> > + /* skip the payload */
>
> Sorry, it is confusing. What do you mean here?
Because setting n proto_hdr will generate (n+1) segments. If we want to split the packet into n segments, we only need to check the first (n-1) proto_hdr.
For example, for ETH-IPV4-UDP-PAYLOAD, if we want to split after the UDP header, we only need to set and check the UDP header in the first segment.
Maybe mask is not a good way, so we will use index to filter out the check of proto_hdr inside the last segment.
>
> > + if (proto_hdr == RTE_PTYPE_ALL_MASK)
> > + continue;
> > +
> > + int ptype_cnt;
> > +
> > + ptype_cnt =
> rte_eth_buffer_split_get_supported_hdr_ptypes(port_id, NULL, 0);
> > + if (ptype_cnt <= 0) {
> > + RTE_ETHDEV_LOG(ERR,
> > + "Port %u failed to supported buffer
> split header protocols\n",
> > + port_id);
> > + return -EINVAL;
> > + }
> > +
> > + uint32_t ptypes[ptype_cnt];
> > + int i;
>
> First of all do no mix code and variable declaration.
> It significantly complicates code reading.
Thanks, the code and variable declaration will be separated.
> Second creation of an array on stack based on function return value is very
> dangerours from security point of view - potential stack overflow and
> corresponding vulnerabilities.
The function value is used for defining how much space is needed to store ptypes. Thanks for your correction of stack overflow, we will use heap instead.
>
> > +
> > + ptype_cnt =
> rte_eth_buffer_split_get_supported_hdr_ptypes(port_id,
> > +
> ptypes, ptype_cnt);
> > + if (ptype_cnt < 0) {
> > + RTE_ETHDEV_LOG(ERR,
> > + "Port %u failed to supported buffer
> split header protocols\n",
> > + port_id);
> > + return -EINVAL;
> > + }
> > +
> > + for (i = 0; i < ptype_cnt; i++)
> > + if (ptypes[i] == proto_hdr)
> > + break;
> > + if (i == ptype_cnt) {
> > + RTE_ETHDEV_LOG(ERR,
> > + "Requested Rx split header protocols
> 0x%x is not supported.\n",
> > + proto_hdr);
> > + return -EINVAL;
> > + }
> > +
> > + if (*mbp_buf_size < offset) {
>
> The check is obviously insufficient, but I agree that it should be driver
> reponsibility to do extra checks for required space in mbuf.
>
> > + RTE_ETHDEV_LOG(ERR,
> > + "%s
> mbuf_data_room_size %u < %u segment offset)\n",
> > + mpl->name, *mbp_buf_size,
> > + offset);
> > + return -EINVAL;
> > + }
> > + } else {
> > + /* Split at fixed length. */
> > + length = length != 0 ? length : *mbp_buf_size;
> > + if (*mbp_buf_size < length + offset) {
> > + RTE_ETHDEV_LOG(ERR,
> > + "%s mbuf_data_room_size %u < %u
> (segment length=%u + segment offset=%u)\n",
> > + mpl->name, *mbp_buf_size,
> > + length + offset, length, offset);
> > + return -EINVAL;
> > + }
> > }
> > }
> > return 0;
> > @@ -1793,7 +1845,7 @@ rte_eth_rx_queue_setup(uint16_t port_id,
> uint16_t rx_queue_id,
> > n_seg = rx_conf->rx_nseg;
> >
> > if (rx_conf->offloads & RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT)
> {
> > - ret = rte_eth_rx_queue_check_split(rx_seg, n_seg,
> > + ret = rte_eth_rx_queue_check_split(port_id, rx_seg,
> n_seg,
> > &mbp_buf_size,
> > &dev_info);
> > if (ret != 0)
> > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index
> > cf14e04010..a5f9647bd3 100644
> > --- a/lib/ethdev/rte_ethdev.h
> > +++ b/lib/ethdev/rte_ethdev.h
> > @@ -994,6 +994,9 @@ struct rte_eth_txmode {
> > * specified in the first array element, the second buffer, from the
> > * pool in the second element, and so on.
> > *
> > + * - The proto_hdrs in the elements define the split position of
> > + * received packets.
> > + *
> > * - The offsets from the segment description elements specify
> > * the data offset from the buffer beginning except the first mbuf.
> > * The first segment offset is added with RTE_PKTMBUF_HEADROOM.
> > @@ -1015,12 +1018,36 @@ struct rte_eth_txmode {
> > * - pool from the last valid element
> > * - the buffer size from this pool
> > * - zero offset
> > + *
> > + * - Length based buffer split:
> > + * - mp, length, offset should be configured.
> > + * - The proto_hdr field will be ignored.
>
> Looking at the code above I think proto_hdr must be 0.
>
> > + *
> > + * - Protocol header based buffer split:
> > + * - mp, offset, proto_hdr should be configured.
> > + * - The length field will be ignored.
>
> I'd require length to be 0 to avoid misusage of the API.
Sure, we will fix them in v8.
>
> > + *
> > + * - For Protocol header based buffer split, if the received packets
> > + * don't exactly match all protocol headers in the elements, packets
> > + * will not be split.
> > + * These packets will be put into:
> > + * - pool from the last valid element
> > + * - the buffer size from this pool
> > + * - zero offset
>
> Shoundl't be check that dataroom in the last segment mempool is sufficient
> for up to MTU packet if Rx scatter is disabled?
Yes, we will add this check in the last segment.
Thanks,
Yuan
>
> > */
> > struct rte_eth_rxseg_split {
> > struct rte_mempool *mp; /**< Memory pool to allocate segment
> from. */
> > uint16_t length; /**< Segment data length, configures split point. */
> > uint16_t offset; /**< Data offset from beginning of mbuf data buffer.
> */
> > - uint32_t reserved; /**< Reserved field. */
> > + /**
> > + * Proto_hdr defines a bit mask of the protocol sequence as
> RTE_PTYPE_*,
> > + * configures split point. The last RTE_PTYPE* in the mask indicates
> the
> > + * split position.
> > + * For non-tunneling packets, the complete protocol sequence should
> be defined.
> > + * For tunneling packets, for simplicity, only the tunnel and inner
> > + * protocol sequence should be defined.
> > + */
> > + uint32_t proto_hdr;
> > };
> >
> > /**
next prev parent reply other threads:[~2022-10-04 2:48 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-12 18:15 [PATCH 0/4] support protocol " Yuan Wang
2022-08-12 18:15 ` [PATCH 1/4] ethdev: introduce protocol header API Yuan Wang
2022-08-12 18:15 ` [PATCH 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-08-12 18:15 ` [PATCH 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-08-12 18:15 ` [PATCH 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-01 22:33 ` [PATCH v2 0/4] support protocol based buffer split Yuan Wang
2022-09-01 22:34 ` [PATCH v2 1/4] ethdev: introduce protocol header API Yuan Wang
2022-09-01 22:35 ` [PATCH v2 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-09-01 22:36 ` [PATCH v2 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-09-01 22:37 ` [PATCH v2 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-02 19:10 ` [PATCH v3 0/4] support protocol based buffer split Yuan Wang
2022-09-02 19:10 ` [PATCH v3 1/4] ethdev: introduce protocol header API Yuan Wang
2022-09-12 11:24 ` Andrew Rybchenko
2022-09-16 8:34 ` Wang, YuanX
2022-09-02 19:10 ` [PATCH v3 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-09-12 11:47 ` Andrew Rybchenko
2022-09-16 8:38 ` Wang, YuanX
2022-09-20 5:35 ` Andrew Rybchenko
2022-09-22 3:13 ` Wang, YuanX
2022-09-13 7:56 ` Suanming Mou
2022-09-16 8:39 ` Wang, YuanX
2022-09-02 19:10 ` [PATCH v3 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-09-02 19:10 ` [PATCH v3 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-20 11:12 ` [PATCH v4 0/4] support protocol based buffer split Yuan Wang
2022-09-20 11:12 ` [PATCH v4 1/4] ethdev: introduce protocol header API Yuan Wang
2022-09-20 11:12 ` [PATCH v4 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-09-20 11:12 ` [PATCH v4 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-09-20 11:12 ` [PATCH v4 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-26 9:40 ` [PATCH v5 0/4] support protocol based buffer split Yuan Wang
2022-09-26 9:40 ` [PATCH v5 1/4] ethdev: introduce protocol header API Yuan Wang
2022-09-26 9:40 ` [PATCH v5 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-09-28 15:42 ` Wang, YuanX
2022-09-26 9:40 ` [PATCH v5 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-09-26 9:40 ` [PATCH v5 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-29 18:59 ` [PATCH v6 0/4] support protocol based buffer split Yuan Wang
2022-09-29 18:59 ` [PATCH v6 1/4] ethdev: introduce protocol header API Yuan Wang
2022-09-29 18:59 ` [PATCH v6 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-09-29 18:59 ` [PATCH v6 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-09-29 18:59 ` [PATCH v6 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-09-30 6:45 ` Tang, Yaqi
2022-10-01 21:05 ` [PATCH v7 0/4] support protocol based buffer split Yuan Wang
2022-10-01 21:05 ` [PATCH v7 1/4] ethdev: introduce protocol header API Yuan Wang
2022-10-03 7:04 ` Andrew Rybchenko
2022-10-04 2:21 ` Wang, YuanX
2022-10-04 7:52 ` Andrew Rybchenko
2022-10-04 15:00 ` Wang, YuanX
2022-10-01 21:05 ` [PATCH v7 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-10-02 4:01 ` Wang, YuanX
2022-10-03 7:47 ` Andrew Rybchenko
2022-10-04 2:48 ` Wang, YuanX [this message]
2022-10-04 8:22 ` Andrew Rybchenko
2022-10-04 15:01 ` Wang, YuanX
2022-10-01 21:05 ` [PATCH v7 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-10-01 21:05 ` [PATCH v7 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-10-05 23:18 ` [PATCH v8 0/4] support protocol based buffer split Yuan Wang
2022-10-05 23:18 ` [PATCH v8 1/4] ethdev: introduce protocol header API Yuan Wang
2022-10-06 10:11 ` Andrew Rybchenko
2022-10-05 23:18 ` [PATCH v8 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-10-06 10:11 ` Andrew Rybchenko
2022-10-08 14:30 ` Ding, Xuan
2022-10-05 23:18 ` [PATCH v8 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-10-06 10:12 ` Andrew Rybchenko
2022-10-05 23:18 ` [PATCH v8 4/4] net/ice: support buffer split in Rx path Yuan Wang
2022-10-06 10:12 ` Andrew Rybchenko
2022-10-06 10:13 ` [PATCH v8 0/4] support protocol based buffer split Andrew Rybchenko
2022-10-09 20:25 ` [PATCH v9 " Yuan Wang
2022-10-09 14:58 ` Andrew Rybchenko
2022-10-10 2:45 ` Ding, Xuan
2022-10-09 20:25 ` [PATCH v9 1/4] ethdev: introduce protocol header API Yuan Wang
2022-10-09 20:25 ` [PATCH v9 2/4] ethdev: introduce protocol hdr based buffer split Yuan Wang
2022-10-09 20:25 ` [PATCH v9 3/4] app/testpmd: add rxhdrs commands and parameters Yuan Wang
2022-10-09 20:25 ` [PATCH v9 4/4] net/ice: support buffer split in Rx path Yuan Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=PH7PR11MB69537A73E3AF44D24C05842E855A9@PH7PR11MB6953.namprd11.prod.outlook.com \
--to=yuanx.wang@intel.com \
--cc=aman.deep.singh@intel.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@amd.com \
--cc=ferruh.yigit@xilinx.com \
--cc=hpothula@marvell.com \
--cc=jerinjacobk@gmail.com \
--cc=mdr@ashroe.eu \
--cc=qi.z.zhang@intel.com \
--cc=qiming.yang@intel.com \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
--cc=viacheslavo@nvidia.com \
--cc=xiaoyun.li@intel.com \
--cc=xuan.ding@intel.com \
--cc=yaqi.tang@intel.com \
--cc=yuying.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).