* [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking
@ 2020-04-17 7:22 Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 2/3] net/octeontx2: add tm packet marking cb Nithin Dabilpuram
` (3 more replies)
0 siblings, 4 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-04-17 7:22 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko, orika, cristian.dumitrescu,
anatoly.burakov, John McNamara, Marko Kovacevic, Olivier Matz
Cc: dev, jerinj, kkanas, Nithin Dabilpuram
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
packet marking.
When packet marking feature in Traffic manager is enabled,
application has to the use the three new flags to indicate
to PMD on whether packet marking needs to be enabled on the
specific mbuf or not. By setting the three flags, it is
assumed by PMD that application has already verified the
applicability of marking on that specific packet and
PMD need not perform further checks as per RFC.
Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
doc/guides/nics/features.rst | 14 ++++++++++++++
lib/librte_mbuf/rte_mbuf.c | 6 ++++++
lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
3 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index edd21c4..bc978fb 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
* **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
* **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
+.. _nic_features_traffic_manager_packet_marking_offload:
+
+Traffic Manager Packet marking offload
+--------------------------------------
+
+Supports enabling a packet marking offload specific mbuf.
+
+* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
+ ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
+ ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
+* **[uses] mbuf**: ``mbuf.l2_len``.
+* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
+ ``rte_tm_mark_vlan_dei()``.
+
.. _nic_features_other:
Other dev ops not represented by a Feature
diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
index cd5794d..5c6896d 100644
--- a/lib/librte_mbuf/rte_mbuf.c
+++ b/lib/librte_mbuf/rte_mbuf.c
@@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
+ case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
+ case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
+ case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
default: return NULL;
}
}
@@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
{ PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
{ PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
{ PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
+ { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
+ { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
+ { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
};
const char *name;
unsigned int i;
diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
index b9a59c8..d9f1290 100644
--- a/lib/librte_mbuf/rte_mbuf_core.h
+++ b/lib/librte_mbuf/rte_mbuf_core.h
@@ -187,11 +187,40 @@ extern "C" {
/* add new RX flags here, don't forget to update PKT_FIRST_FREE */
#define PKT_FIRST_FREE (1ULL << 23)
-#define PKT_LAST_FREE (1ULL << 40)
+#define PKT_LAST_FREE (1ULL << 37)
/* add new TX flags here, don't forget to update PKT_LAST_FREE */
/**
+ * Packet marking offload flags. These flags indicated what kind
+ * of packet marking needs to be applied on a given mbuf when
+ * appropriate Traffic Manager configuration is in place.
+ * When user set's these flags on a mbuf, below assumptions are made
+ * 1) When PKT_TX_MARK_VLAN_DEI is set,
+ * a) PMD assumes pkt to be a 802.1q packet.
+ * b) Application should also set mbuf.l2_len where 802.1Q header is
+ * at (mbuf.l2_len - 6) offset.
+ * 2) When PKT_TX_MARK_IP_DSCP is set,
+ * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
+ * to indicate whether if it is IPv4 packet or IPv6 packet
+ * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
+ * IPv4 pkt.
+ * b) Application should also set mbuf.l2_len that indicates
+ * start offset of L3 header.
+ * 3) When PKT_TX_MARK_IP_ECN is set,
+ * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
+ * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
+ * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
+ * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
+ * can mark the packet for a configured color.
+ * c) Application should also set mbuf.l2_len that indicates
+ * start offset of L3 header.
+ */
+#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
+#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
+#define PKT_TX_MARK_IP_ECN (1ULL << 40)
+
+/**
* Outer UDP checksum offload flag. This flag is used for enabling
* outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
* 1) Enable the following in mbuf,
@@ -384,7 +413,10 @@ extern "C" {
PKT_TX_MACSEC | \
PKT_TX_SEC_OFFLOAD | \
PKT_TX_UDP_SEG | \
- PKT_TX_OUTER_UDP_CKSUM)
+ PKT_TX_OUTER_UDP_CKSUM | \
+ PKT_TX_MARK_VLAN_DEI | \
+ PKT_TX_MARK_IP_DSCP | \
+ PKT_TX_MARK_IP_ECN)
/**
* Mbuf having an external buffer attached. shinfo in mbuf must be filled.
--
2.8.4
^ permalink raw reply [flat|nested] 36+ messages in thread
* [dpdk-dev] [PATCH 2/3] net/octeontx2: add tm packet marking cb
2020-04-17 7:22 [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Nithin Dabilpuram
@ 2020-04-17 7:22 ` Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 3/3] net/octeontx2: add Tx packet marking offload support Nithin Dabilpuram
` (2 subsequent siblings)
3 siblings, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-04-17 7:22 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko, orika, cristian.dumitrescu,
anatoly.burakov, Jerin Jacob, Nithin Dabilpuram, Kiran Kumar K
Cc: dev, kkanas
From: Krzysztof Kanas <kkanas@marvell.com>
Add tm packet marking enable/disable callbacks for
VLAN DEI, IP DSCP and IP ECN marking to support
marking in VLAN, IPv4 and IPv6 packets.
Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
Depends-on:series-9450
drivers/net/octeontx2/otx2_ethdev.c | 12 +-
drivers/net/octeontx2/otx2_ethdev.h | 5 +
drivers/net/octeontx2/otx2_tm.c | 472 +++++++++++++++++++++++++++++++++++-
drivers/net/octeontx2/otx2_tm.h | 31 +++
drivers/net/octeontx2/otx2_tx.h | 9 +-
5 files changed, 514 insertions(+), 15 deletions(-)
diff --git a/drivers/net/octeontx2/otx2_ethdev.c b/drivers/net/octeontx2/otx2_ethdev.c
index 3116e5c..1c7c465 100644
--- a/drivers/net/octeontx2/otx2_ethdev.c
+++ b/drivers/net/octeontx2/otx2_ethdev.c
@@ -655,8 +655,8 @@ nix_rx_offload_flags(struct rte_eth_dev *eth_dev)
return flags;
}
-static uint16_t
-nix_tx_offload_flags(struct rte_eth_dev *eth_dev)
+uint16_t
+otx2_nix_tx_offload_flags(struct rte_eth_dev *eth_dev)
{
struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
uint64_t conf = dev->tx_offloads;
@@ -974,6 +974,10 @@ otx2_nix_form_default_desc(struct otx2_eth_txq *txq)
send_mem->alg = NIX_SENDMEMALG_SETTSTMP;
send_mem->addr = txq->dev->tstamp.tx_tstamp_iova;
}
+
+ /* Update mark format */
+ txq->markfmt_en = txq->dev->tm_markfmt_en;
+
sg = (union nix_send_sg_s *)&txq->cmd[4];
} else {
send_hdr = (struct nix_send_hdr_s *)&txq->cmd[0];
@@ -1663,7 +1667,7 @@ otx2_nix_configure(struct rte_eth_dev *eth_dev)
dev->rx_offloads = rxmode->offloads;
dev->tx_offloads = txmode->offloads;
dev->rx_offload_flags |= nix_rx_offload_flags(eth_dev);
- dev->tx_offload_flags |= nix_tx_offload_flags(eth_dev);
+ dev->tx_offload_flags |= otx2_nix_tx_offload_flags(eth_dev);
dev->rss_info.rss_grps = NIX_RSS_GRPS;
nb_rxq = RTE_MAX(data->nb_rx_queues, 1);
@@ -1829,7 +1833,7 @@ otx2_nix_configure(struct rte_eth_dev *eth_dev)
nix_lf_free(dev);
fail_offloads:
dev->rx_offload_flags &= ~nix_rx_offload_flags(eth_dev);
- dev->tx_offload_flags &= ~nix_tx_offload_flags(eth_dev);
+ dev->tx_offload_flags &= ~otx2_nix_tx_offload_flags(eth_dev);
fail_configure:
dev->configured = 0;
return rc;
diff --git a/drivers/net/octeontx2/otx2_ethdev.h b/drivers/net/octeontx2/otx2_ethdev.h
index 0fbf68b..b34842a 100644
--- a/drivers/net/octeontx2/otx2_ethdev.h
+++ b/drivers/net/octeontx2/otx2_ethdev.h
@@ -304,6 +304,9 @@ struct otx2_eth_dev {
uint16_t txschq_list[NIX_TXSCH_LVL_CNT][MAX_TXSCHQ_PER_FUNC];
/* Contiguous queues */
uint16_t txschq_contig_list[NIX_TXSCH_LVL_CNT][MAX_TXSCHQ_PER_FUNC];
+ uint64_t tm_markfmt_en;
+ uint8_t tm_markfmt_null;
+ uint8_t tm_markfmt[NIX_TM_MARK_MAX][NIX_TM_MARK_COLOR_MAX];
uint16_t otx2_tm_root_lvl;
uint16_t link_cfg_lvl;
uint16_t tm_flags;
@@ -347,6 +350,7 @@ struct otx2_eth_txq {
rte_iova_t fc_iova;
uint16_t sqes_per_sqb_log2;
int16_t nb_sqb_bufs_adj;
+ uint64_t markfmt_en;
RTE_MARKER slow_path_start;
uint16_t nb_sqb_bufs;
uint16_t sq;
@@ -570,6 +574,7 @@ int otx2_ethdev_parse_devargs(struct rte_devargs *devargs,
void otx2_eth_set_rx_function(struct rte_eth_dev *eth_dev);
void otx2_eth_set_tx_function(struct rte_eth_dev *eth_dev);
void otx2_nix_form_default_desc(struct otx2_eth_txq *txq);
+uint16_t otx2_nix_tx_offload_flags(struct rte_eth_dev *eth_dev);
/* Timesync - PTP routines */
int otx2_nix_timesync_enable(struct rte_eth_dev *eth_dev);
diff --git a/drivers/net/octeontx2/otx2_tm.c b/drivers/net/octeontx2/otx2_tm.c
index e94a276..f11b828 100644
--- a/drivers/net/octeontx2/otx2_tm.c
+++ b/drivers/net/octeontx2/otx2_tm.c
@@ -20,6 +20,30 @@ enum otx2_tm_node_level {
OTX2_TM_LVL_MAX,
};
+static const uint8_t y_mask_val[NIX_TM_MARK_MAX][2] = {
+ [NIX_TM_MARK_VLAN_DEI] = { 0x0, 0x8 },
+ [NIX_TM_MARK_IPV4_DSCP] = { 0x1, 0x2 },
+ [NIX_TM_MARK_IPV4_ECN] = { 0x0, 0xc },
+ [NIX_TM_MARK_IPV6_DSCP] = { 0x1, 0x2 },
+ [NIX_TM_MARK_IPV6_ECN] = { 0x0, 0x3 },
+};
+
+static const uint8_t r_mask_val[NIX_TM_MARK_MAX][2] = {
+ [NIX_TM_MARK_VLAN_DEI] = { 0x0, 0x8 },
+ [NIX_TM_MARK_IPV4_DSCP] = { 0x0, 0x3 },
+ [NIX_TM_MARK_IPV4_ECN] = { 0x0, 0xc },
+ [NIX_TM_MARK_IPV6_DSCP] = { 0x0, 0x3 },
+ [NIX_TM_MARK_IPV6_ECN] = { 0x0, 0x3 },
+};
+
+static const uint8_t mark_off[NIX_TM_MARK_MAX] = {
+ [NIX_TM_MARK_VLAN_DEI] = 0x3, /* Byte 14 Bit[4:1] */
+ [NIX_TM_MARK_IPV4_DSCP] = 0x1, /* Byte 1 Bit[6:3] */
+ [NIX_TM_MARK_IPV4_ECN] = 0x6, /* Byte 1 Bit[1:0], Byte 2 Bit[7:6] */
+ [NIX_TM_MARK_IPV6_DSCP] = 0x5, /* Byte 0 Bit[2:0], Byte 1 Bit[7] */
+ [NIX_TM_MARK_IPV6_ECN] = 0x0, /* Byte 1 Bit[7:4] */
+};
+
static inline
uint64_t shaper2regval(struct shaper_params *shaper)
{
@@ -375,10 +399,10 @@ prepare_tm_shaper_reg(struct otx2_nix_tm_node *tm_node,
otx2_tm_dbg("Shaper config node %s(%u) lvl %u id %u, pir %" PRIu64
"(%" PRIu64 "B), cir %" PRIu64 "(%" PRIu64 "B)"
- "adjust 0x%" PRIx64 "(pktmode %u) (%p)",
+ "adjust 0x%" PRIx64 "(pktmode %u) red_alg %x (%p)",
nix_hwlvl2str(tm_node->hw_lvl), schq, tm_node->lvl,
tm_node->id, pir.rate, pir.burst, cir.rate, cir.burst,
- adjust, tm_node->pkt_mode, tm_node);
+ adjust, tm_node->pkt_mode, tm_node->red_algo, tm_node);
switch (tm_node->hw_lvl) {
case NIX_TXSCH_LVL_SMQ:
@@ -478,6 +502,46 @@ prepare_tm_shaper_reg(struct otx2_nix_tm_node *tm_node,
}
static uint8_t
+prepare_tm_shaper_red_algo(struct otx2_nix_tm_node *tm_node,
+ volatile uint64_t *reg, volatile uint64_t *regval,
+ volatile uint64_t *regval_mask)
+{
+ uint32_t schq = tm_node->hw_id;
+ uint8_t k = 0;
+
+ otx2_tm_dbg("Shaper read alg node %s(%u) lvl %u id %u, red_alg %x (%p)",
+ nix_hwlvl2str(tm_node->hw_lvl), schq, tm_node->lvl,
+ tm_node->id, tm_node->red_algo, tm_node);
+
+ /* Configure just RED algo */
+ regval[k] = ((uint64_t)tm_node->red_algo << 9);
+ regval_mask[k] = ~(BIT_ULL(10) | BIT_ULL(9));
+
+ switch (tm_node->hw_lvl) {
+ case NIX_TXSCH_LVL_SMQ:
+ reg[k] = NIX_AF_MDQX_SHAPE(schq);
+ k++;
+ break;
+ case NIX_TXSCH_LVL_TL4:
+ reg[k] = NIX_AF_TL4X_SHAPE(schq);
+ k++;
+ break;
+ case NIX_TXSCH_LVL_TL3:
+ reg[k] = NIX_AF_TL3X_SHAPE(schq);
+ k++;
+ break;
+ case NIX_TXSCH_LVL_TL2:
+ reg[k] = NIX_AF_TL2X_SHAPE(schq);
+ k++;
+ break;
+ default:
+ break;
+ }
+
+ return k;
+}
+
+static uint8_t
prepare_tm_sw_xoff(struct otx2_nix_tm_node *tm_node, bool enable,
volatile uint64_t *reg, volatile uint64_t *regval)
{
@@ -1939,10 +2003,16 @@ otx2_nix_tm_capa_get(struct rte_eth_dev *eth_dev,
RTE_TM_STATS_N_PKTS_RED_DROPPED |
RTE_TM_STATS_N_BYTES_RED_DROPPED;
- for (i = 0; i < RTE_COLORS; i++) {
- cap->mark_vlan_dei_supported[i] = false;
- cap->mark_ip_ecn_tcp_supported[i] = false;
- cap->mark_ip_dscp_supported[i] = false;
+ cap->mark_vlan_dei_supported[RTE_COLOR_GREEN] = false;
+ cap->mark_ip_ecn_tcp_supported[RTE_COLOR_GREEN] = false;
+ cap->mark_ip_ecn_sctp_supported[RTE_COLOR_GREEN] = false;
+ cap->mark_ip_dscp_supported[RTE_COLOR_GREEN] = false;
+
+ for (i = RTE_COLOR_YELLOW; i < RTE_COLORS; i++) {
+ cap->mark_vlan_dei_supported[i] = true;
+ cap->mark_ip_ecn_tcp_supported[i] = true;
+ cap->mark_ip_ecn_sctp_supported[i] = true;
+ cap->mark_ip_dscp_supported[i] = true;
}
return 0;
@@ -2831,6 +2901,317 @@ otx2_nix_tm_node_stats_read(struct rte_eth_dev *eth_dev, uint32_t node_id,
return rc;
}
+static void
+nix_tm_mark_queue_reconfig(struct rte_eth_dev *eth_dev, bool enable)
+{
+ struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
+ struct otx2_eth_txq *txq;
+ int i, nb_txq;
+
+ nb_txq = RTE_MIN(dev->configured_nb_tx_qs, eth_dev->data->nb_tx_queues);
+
+ if (enable) {
+ dev->tx_offload_flags |= NIX_TX_OFFLOAD_MARK_F;
+ /* Enable L3 and OL3 CKSUM for marking IP DSCP and ECN */
+ if (dev->tm_flags & NIX_TM_MARK_IP_ECN_EN ||
+ dev->tm_flags & NIX_TM_MARK_IP_DSCP_EN)
+ dev->tx_offload_flags |=
+ (NIX_TX_OFFLOAD_L3_L4_CSUM_F |
+ NIX_TX_OFFLOAD_OL3_OL4_CSUM_F);
+ otx2_tm_dbg("Enabling TM marking with tx_offload_flags=0x%04x"
+ " markfmt_en=0x%" PRIx64, dev->tx_offload_flags,
+ dev->tm_markfmt_en);
+ } else {
+ /* Reset to old flags excluding MARK */
+ dev->tx_offload_flags = otx2_nix_tx_offload_flags(eth_dev);
+ otx2_tm_dbg("Disabling TM marking with tx_offload_flags=0x%04x"
+ " markfmt_en=0x%" PRIx64, dev->tx_offload_flags,
+ dev->tm_markfmt_en);
+ }
+
+ for (i = 0; i < nb_txq; i++) {
+ txq = eth_dev->data->tx_queues[i];
+ otx2_nix_form_default_desc(txq);
+ }
+ rte_wmb();
+}
+
+/* Only called while device is stopped */
+static int
+nix_tm_update_red_algo(struct otx2_eth_dev *dev, bool red_send,
+ struct rte_tm_error *error)
+{
+ struct otx2_mbox *mbox = dev->mbox;
+ struct otx2_nix_tm_node *tm_node;
+ struct nix_txschq_config *req;
+ uint8_t k;
+ int rc;
+
+ TAILQ_FOREACH(tm_node, &dev->node_list, node) {
+ /* Skip leaf nodes */
+ if (nix_tm_is_leaf(dev, tm_node->lvl))
+ continue;
+
+ if (tm_node->hw_lvl == NIX_TXSCH_LVL_TL1)
+ continue;
+
+ /* Skip if no update of red_algo is needed */
+ if ((red_send && (tm_node->red_algo == NIX_REDALG_SEND)) ||
+ (!red_send && (tm_node->red_algo != NIX_REDALG_SEND)))
+ continue;
+
+ /* Update Red algo */
+ if (red_send)
+ tm_node->red_algo = NIX_REDALG_SEND;
+ else if (tm_node->flags & NIX_TM_NODE_RED_DISCARD)
+ tm_node->red_algo = NIX_REDALG_DISCARD;
+ else
+ tm_node->red_algo = NIX_REDALG_STD;
+
+ /* Update txschq config */
+ req = otx2_mbox_alloc_msg_nix_txschq_cfg(mbox);
+ req->lvl = tm_node->hw_lvl;
+ k = prepare_tm_shaper_red_algo(tm_node, req->reg,
+ req->regval, req->regval_mask);
+ req->num_regs = k;
+
+ rc = send_tm_reqval(mbox, req, error);
+ if (rc)
+ return rc;
+ }
+ return 0;
+}
+
+/* Return's true if queue reconfig is needed */
+static bool
+nix_tm_update_markfmt(struct otx2_eth_dev *dev,
+ enum otx2_nix_tm_mark type,
+ int mark_yellow, int mark_red)
+{
+ uint64_t new_markfmt, old_markfmt;
+ uint8_t *tm_markfmt;
+ uint8_t en_shift;
+ uint64_t mask;
+
+ if (type >= NIX_TM_MARK_MAX)
+ return 0;
+
+ /* Pre-allocated mark formats for type:color combinations */
+ tm_markfmt = dev->tm_markfmt[type];
+
+ if (!mark_yellow && !mark_red) {
+ /* Null format to disable */
+ new_markfmt = dev->tm_markfmt_null;
+ } else {
+ /* Marking enabled with combination of yellow and red */
+ if (mark_yellow && mark_red)
+ new_markfmt = tm_markfmt[NIX_TM_MARK_COLOR_Y_R];
+ else if (mark_yellow)
+ new_markfmt = tm_markfmt[NIX_TM_MARK_COLOR_Y];
+ else
+ new_markfmt = tm_markfmt[NIX_TM_MARK_COLOR_R];
+ }
+
+ mask = 0xFFull;
+ /* Format of fast path markfmt
+ * ipv6_ecn[8]:ipv4_ecn[8]:ipv6_dscp[8]:ipv4_dscp[8]:vlan_dei[16]
+ * fmt[7] = ptr offset for IPv4/IPv6 on l2_len.
+ * fmt[6:0] = markfmt idx.
+ */
+ switch (type) {
+ case NIX_TM_MARK_VLAN_DEI:
+ en_shift = NIX_TM_MARK_VLAN_DEI_SHIFT;
+ mask = 0xFFFFull;
+ new_markfmt |= new_markfmt << 8;
+ break;
+ case NIX_TM_MARK_IPV4_DSCP:
+ new_markfmt |= BIT_ULL(7);
+ en_shift = NIX_TM_MARK_IPV4_DSCP_SHIFT;
+ break;
+ case NIX_TM_MARK_IPV4_ECN:
+ new_markfmt |= BIT_ULL(7);
+ en_shift = NIX_TM_MARK_IPV4_ECN_SHIFT;
+ break;
+ case NIX_TM_MARK_IPV6_DSCP:
+ en_shift = NIX_TM_MARK_IPV6_DSCP_SHIFT;
+ break;
+ case NIX_TM_MARK_IPV6_ECN:
+ new_markfmt |= BIT_ULL(7);
+ en_shift = NIX_TM_MARK_IPV6_ECN_SHIFT;
+ break;
+ default:
+ return 0;
+ }
+
+ /* Skip if same as old config */
+ old_markfmt = (dev->tm_markfmt_en >> en_shift) & mask;
+ if (old_markfmt == new_markfmt)
+ return false;
+
+ /* Need queue reconfig */
+ dev->tm_markfmt_en &= ~(mask << en_shift);
+ dev->tm_markfmt_en |= (new_markfmt << en_shift);
+ return true;
+}
+
+static int
+otx2_nix_tm_mark_vlan_dei(struct rte_eth_dev *eth_dev, int mark_green,
+ int mark_yellow, int mark_red,
+ struct rte_tm_error *error)
+{
+ struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
+ bool need_reconfig, marking;
+ int rc;
+
+ if (mark_green) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "Green VLAN marking not supported";
+ return -EINVAL;
+ }
+
+ if (eth_dev->data->dev_started) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "VLAN DEI mark for running ports not "
+ "supported";
+ return -EBUSY;
+ }
+
+ if (!(dev->tm_flags & NIX_TM_COMMITTED)) {
+ error->type = RTE_TM_ERROR_TYPE_UNSPECIFIED;
+ error->message = "hierarchy doesn't exist";
+ return -EINVAL;
+ }
+
+ otx2_tm_dbg("VLAN DEI marking update yellow=%u red=%u",
+ mark_yellow, mark_red);
+
+ need_reconfig = nix_tm_update_markfmt(dev, NIX_TM_MARK_VLAN_DEI,
+ mark_yellow, mark_red);
+ if (!need_reconfig)
+ return 0;
+
+ if (!mark_yellow && !mark_red)
+ dev->tm_flags &= ~NIX_TM_MARK_VLAN_DEI_EN;
+ else
+ dev->tm_flags |= NIX_TM_MARK_VLAN_DEI_EN;
+
+ /* Update red algo for change in mark_red */
+ rc = nix_tm_update_red_algo(dev, !!mark_red, error);
+ if (rc)
+ return rc;
+
+ /* Update red algo */
+ marking = !!(dev->tm_flags & NIX_TM_MARK_EN_MASK);
+ nix_tm_mark_queue_reconfig(eth_dev, marking);
+ return 0;
+}
+
+static int
+otx2_nix_tm_mark_ip_ecn(struct rte_eth_dev *eth_dev, int mark_green,
+ int mark_yellow, int mark_red,
+ struct rte_tm_error *error)
+{
+ struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
+ bool need_reconfig, marking;
+ int rc;
+
+ if (mark_green) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "Green IP ECN marking not supported";
+ return -EINVAL;
+ }
+
+ if (eth_dev->data->dev_started) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "IP ECN mark for running ports not "
+ "supported";
+ return -EBUSY;
+ }
+
+ if (!(dev->tm_flags & NIX_TM_COMMITTED)) {
+ error->type = RTE_TM_ERROR_TYPE_UNSPECIFIED;
+ error->message = "hierarchy doesn't exist";
+ return -EINVAL;
+ }
+
+ otx2_tm_dbg("IP ECN marking update yellow=%u red=%u",
+ mark_yellow, mark_red);
+
+ need_reconfig = nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV4_ECN,
+ mark_yellow, mark_red);
+ need_reconfig |= nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV6_ECN,
+ mark_yellow, mark_red);
+ if (!need_reconfig)
+ return 0;
+
+ if (!mark_yellow && !mark_red)
+ dev->tm_flags &= ~NIX_TM_MARK_IP_ECN_EN;
+ else
+ dev->tm_flags |= NIX_TM_MARK_IP_ECN_EN;
+
+ /* Update red algo for change in mark_red */
+ rc = nix_tm_update_red_algo(dev, !!mark_red, error);
+ if (rc)
+ return rc;
+
+ marking = !!(dev->tm_flags & NIX_TM_MARK_EN_MASK);
+ nix_tm_mark_queue_reconfig(eth_dev, marking);
+ return 0;
+}
+
+static int
+otx2_nix_tm_mark_ip_dscp(struct rte_eth_dev *eth_dev, int mark_green,
+ int mark_yellow, int mark_red,
+ struct rte_tm_error *error)
+{
+ struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
+ bool need_reconfig, marking;
+ int rc;
+
+ if (mark_green) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "Green IP DSCP marking not supported";
+ return -EINVAL;
+ }
+
+ if (eth_dev->data->dev_started) {
+ error->type = RTE_TM_ERROR_TYPE_CAPABILITIES;
+ error->message = "IP DSCP mark for running ports not "
+ "supported";
+ return -EBUSY;
+ }
+
+ if (!(dev->tm_flags & NIX_TM_COMMITTED)) {
+ error->type = RTE_TM_ERROR_TYPE_UNSPECIFIED;
+ error->message = "hierarchy doesn't exist";
+ return -EINVAL;
+ }
+
+ otx2_tm_dbg("IP DSCP marking update yellow=%u red=%u",
+ mark_yellow, mark_red);
+
+ need_reconfig = nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV4_DSCP,
+ mark_yellow, mark_red);
+ need_reconfig |= nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV6_DSCP,
+ mark_yellow, mark_red);
+ if (!need_reconfig)
+ return 0;
+
+ if (!mark_yellow && !mark_red)
+ dev->tm_flags &= ~NIX_TM_MARK_IP_DSCP_EN;
+ else
+ dev->tm_flags |= NIX_TM_MARK_IP_DSCP_EN;
+
+ /* Update red algo for change in mark_red */
+ rc = nix_tm_update_red_algo(dev, !!mark_red, error);
+ if (rc)
+ return rc;
+
+ marking = !!(dev->tm_flags & NIX_TM_MARK_EN_MASK);
+ nix_tm_mark_queue_reconfig(eth_dev, marking);
+ return 0;
+}
+
const struct rte_tm_ops otx2_tm_ops = {
.node_type_get = otx2_nix_tm_node_type_get,
@@ -2850,9 +3231,80 @@ const struct rte_tm_ops otx2_tm_ops = {
.node_shaper_update = otx2_nix_tm_node_shaper_update,
.node_parent_update = otx2_nix_tm_node_parent_update,
.node_stats_read = otx2_nix_tm_node_stats_read,
+
+ .mark_vlan_dei = otx2_nix_tm_mark_vlan_dei,
+ .mark_ip_ecn = otx2_nix_tm_mark_ip_ecn,
+ .mark_ip_dscp = otx2_nix_tm_mark_ip_dscp,
};
static int
+nix_tm_mark_init(struct otx2_eth_dev *dev)
+{
+ struct nix_mark_format_cfg_rsp *rsp;
+ struct otx2_mbox *mbox = dev->mbox;
+ struct nix_mark_format_cfg *req;
+ int rc, i, j;
+
+ /* Check for supported revisions */
+ if (otx2_dev_is_95xx_Ax(dev) ||
+ otx2_dev_is_96xx_Ax(dev))
+ return 0;
+
+ /* Null mark format */
+ req = otx2_mbox_alloc_msg_nix_mark_format_cfg(mbox);
+ rc = otx2_mbox_process_msg(mbox, (void *)&rsp);
+ if (rc) {
+ otx2_err("TM failed to alloc null mark format, rc=%d", rc);
+ return rc;
+ }
+
+ dev->tm_markfmt_null = rsp->mark_format_idx;
+
+ /* Alloc vlan, dscp, ecn mark formats */
+ for (i = 0; i < NIX_TM_MARK_MAX; i++) {
+ for (j = 0; j < NIX_TM_MARK_COLOR_MAX; j++) {
+ req = otx2_mbox_alloc_msg_nix_mark_format_cfg(mbox);
+ req->offset = mark_off[i];
+
+ switch (j) {
+ case NIX_TM_MARK_COLOR_Y:
+ req->y_mask = y_mask_val[i][0];
+ req->y_val = y_mask_val[i][1];
+ break;
+ case NIX_TM_MARK_COLOR_R:
+ req->r_mask = r_mask_val[i][0];
+ req->r_val = r_mask_val[i][1];
+ break;
+ case NIX_TM_MARK_COLOR_Y_R:
+ req->y_mask = y_mask_val[i][0];
+ req->y_val = y_mask_val[i][1];
+ req->r_mask = r_mask_val[i][0];
+ req->r_val = r_mask_val[i][1];
+ break;
+ }
+
+ rc = otx2_mbox_process_msg(mbox, (void *)&rsp);
+ if (rc) {
+ otx2_err("TM failed to alloc mark fmt "
+ "type %u color %u, rc=%d", i, j, rc);
+ return rc;
+ }
+
+ dev->tm_markfmt[i][j] = rsp->mark_format_idx;
+ otx2_tm_dbg("Mark type: %u, Mark Color:%u, id:%u\n",
+ i, j, dev->tm_markfmt[i][j]);
+ }
+ }
+ /* Update null mark format as default */
+ nix_tm_update_markfmt(dev, NIX_TM_MARK_VLAN_DEI, 0, 0);
+ nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV4_DSCP, 0, 0);
+ nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV4_ECN, 0, 0);
+ nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV6_DSCP, 0, 0);
+ nix_tm_update_markfmt(dev, NIX_TM_MARK_IPV6_ECN, 0, 0);
+ return 0;
+}
+
+static int
nix_tm_prepare_default_tree(struct rte_eth_dev *eth_dev)
{
struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
@@ -2963,7 +3415,7 @@ void otx2_nix_tm_conf_init(struct rte_eth_dev *eth_dev)
int otx2_nix_tm_init_default(struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
- struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
+ struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
uint16_t sq_cnt = eth_dev->data->nb_tx_queues;
int rc;
@@ -2985,6 +3437,10 @@ int otx2_nix_tm_init_default(struct rte_eth_dev *eth_dev)
if (pci_dev->max_vfs)
dev->tm_flags |= NIX_TM_TL1_NO_SP;
+ rc = nix_tm_mark_init(dev);
+ if (rc != 0)
+ return rc;
+
rc = nix_tm_prepare_default_tree(eth_dev);
if (rc != 0)
return rc;
@@ -2992,8 +3448,8 @@ int otx2_nix_tm_init_default(struct rte_eth_dev *eth_dev)
rc = nix_tm_alloc_resources(eth_dev, false);
if (rc != 0)
return rc;
+
dev->tm_leaf_cnt = sq_cnt;
-
return 0;
}
diff --git a/drivers/net/octeontx2/otx2_tm.h b/drivers/net/octeontx2/otx2_tm.h
index db44d48..afc796a 100644
--- a/drivers/net/octeontx2/otx2_tm.h
+++ b/drivers/net/octeontx2/otx2_tm.h
@@ -13,6 +13,37 @@
#define NIX_TM_COMMITTED BIT_ULL(1)
#define NIX_TM_RATE_LIMIT_TREE BIT_ULL(2)
#define NIX_TM_TL1_NO_SP BIT_ULL(3)
+#define NIX_TM_MARK_IP_DSCP_EN BIT_ULL(4)
+#define NIX_TM_MARK_IP_ECN_EN BIT_ULL(5)
+#define NIX_TM_MARK_VLAN_DEI_EN BIT_ULL(6)
+
+
+#define NIX_TM_MARK_EN_MASK (NIX_TM_MARK_IP_DSCP_EN | \
+ NIX_TM_MARK_IP_ECN_EN | \
+ NIX_TM_MARK_VLAN_DEI_EN)
+
+/* Shift positions for struct otx2_eth_dev::tm_markfmt_en field */
+#define NIX_TM_MARK_VLAN_DEI_SHIFT 0 /* Leave 16b for VLAN for FP logic */
+#define NIX_TM_MARK_IPV4_DSCP_SHIFT 16
+#define NIX_TM_MARK_IPV6_DSCP_SHIFT 24
+#define NIX_TM_MARK_IPV4_ECN_SHIFT 32
+#define NIX_TM_MARK_IPV6_ECN_SHIFT 40
+
+enum otx2_nix_tm_mark {
+ NIX_TM_MARK_VLAN_DEI,
+ NIX_TM_MARK_IPV4_DSCP,
+ NIX_TM_MARK_IPV4_ECN,
+ NIX_TM_MARK_IPV6_DSCP,
+ NIX_TM_MARK_IPV6_ECN,
+ NIX_TM_MARK_MAX
+};
+
+enum otx2_nix_tm_mark_color {
+ NIX_TM_MARK_COLOR_Y,
+ NIX_TM_MARK_COLOR_R,
+ NIX_TM_MARK_COLOR_Y_R,
+ NIX_TM_MARK_COLOR_MAX
+};
struct otx2_eth_dev;
diff --git a/drivers/net/octeontx2/otx2_tx.h b/drivers/net/octeontx2/otx2_tx.h
index 3c43170..928e4ea 100644
--- a/drivers/net/octeontx2/otx2_tx.h
+++ b/drivers/net/octeontx2/otx2_tx.h
@@ -13,6 +13,7 @@
#define NIX_TX_OFFLOAD_TSTAMP_F BIT(4)
#define NIX_TX_OFFLOAD_TSO_F BIT(5)
#define NIX_TX_OFFLOAD_SECURITY_F BIT(6)
+#define NIX_TX_OFFLOAD_MARK_F BIT(7)
/* Flags to control xmit_prepare function.
* Defining it from backwards to denote its been
@@ -22,11 +23,12 @@
#define NIX_TX_NEED_SEND_HDR_W1 \
(NIX_TX_OFFLOAD_L3_L4_CSUM_F | NIX_TX_OFFLOAD_OL3_OL4_CSUM_F | \
- NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)
+ NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F | \
+ NIX_TX_OFFLOAD_MARK_F)
#define NIX_TX_NEED_EXT_HDR \
(NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F | \
- NIX_TX_OFFLOAD_TSO_F)
+ NIX_TX_OFFLOAD_TSO_F | NIX_TX_OFFLOAD_MARK_F)
#define NIX_UDP_TUN_BITMASK \
((1ull << (PKT_TX_TUNNEL_VXLAN >> 45)) | \
@@ -42,7 +44,8 @@ static __rte_always_inline int
otx2_nix_tx_ext_subs(const uint16_t flags)
{
return (flags & NIX_TX_OFFLOAD_TSTAMP_F) ? 2 :
- ((flags & (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) ?
+ ((flags & (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F |
+ NIX_TX_OFFLOAD_MARK_F)) ?
1 : 0);
}
--
2.8.4
^ permalink raw reply [flat|nested] 36+ messages in thread
* [dpdk-dev] [PATCH 3/3] net/octeontx2: add Tx packet marking offload support
2020-04-17 7:22 [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 2/3] net/octeontx2: add tm packet marking cb Nithin Dabilpuram
@ 2020-04-17 7:22 ` Nithin Dabilpuram
2020-05-01 11:18 ` [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Jerin Jacob
2023-07-31 12:54 ` [dpdk-dev] " Thomas Monjalon
3 siblings, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-04-17 7:22 UTC (permalink / raw)
To: thomas, ferruh.yigit, arybchenko, orika, cristian.dumitrescu,
anatoly.burakov, Pavan Nikhilesh, Jerin Jacob, Nithin Dabilpuram,
Kiran Kumar K
Cc: dev, kkanas
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
This patch adds Tx packet marking offload support for
VLAN DEI, IP ECN, and IP DSCP.
Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
Depends-on:series-9450
drivers/event/octeontx2/otx2_evdev.c | 28 +-
drivers/event/octeontx2/otx2_evdev.h | 2 +-
drivers/event/octeontx2/otx2_worker.c | 4 +-
drivers/event/octeontx2/otx2_worker.h | 4 +-
drivers/event/octeontx2/otx2_worker_dual.c | 4 +-
drivers/net/octeontx2/otx2_tx.c | 48 ++-
drivers/net/octeontx2/otx2_tx.h | 580 ++++++++++++++++++++++-------
7 files changed, 503 insertions(+), 167 deletions(-)
diff --git a/drivers/event/octeontx2/otx2_evdev.c b/drivers/event/octeontx2/otx2_evdev.c
index d20213d..33cba09 100644
--- a/drivers/event/octeontx2/otx2_evdev.c
+++ b/drivers/event/octeontx2/otx2_evdev.c
@@ -178,36 +178,36 @@ SSO_RX_ADPTR_ENQ_FASTPATH_FUNC
/* Tx modes */
const event_tx_adapter_enqueue
- ssogws_tx_adptr_enq[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = \
+ ssogws_tx_adptr_enq[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = \
otx2_ssogws_tx_adptr_enq_ ## name,
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
};
const event_tx_adapter_enqueue
- ssogws_tx_adptr_enq_seg[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = \
+ ssogws_tx_adptr_enq_seg[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = \
otx2_ssogws_tx_adptr_enq_seg_ ## name,
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
};
const event_tx_adapter_enqueue
- ssogws_dual_tx_adptr_enq[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = \
+ ssogws_dual_tx_adptr_enq[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = \
otx2_ssogws_dual_tx_adptr_enq_ ## name,
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
};
const event_tx_adapter_enqueue
- ssogws_dual_tx_adptr_enq_seg[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = \
+ ssogws_dual_tx_adptr_enq_seg[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = \
otx2_ssogws_dual_tx_adptr_enq_seg_ ## name,
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
@@ -294,6 +294,7 @@ SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
if (dev->tx_offloads & NIX_TX_MULTI_SEG_F) {
/* [SEC] [TSMP] [MBUF_NOFF] [VLAN] [OL3_L4_CSUM] [L3_L4_CSUM] */
event_dev->txa_enqueue = ssogws_tx_adptr_enq_seg
+ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MARK_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_SECURITY_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)]
@@ -303,6 +304,7 @@ SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_L3_L4_CSUM_F)];
} else {
event_dev->txa_enqueue = ssogws_tx_adptr_enq
+ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MARK_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_SECURITY_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)]
@@ -446,6 +448,7 @@ SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
if (dev->tx_offloads & NIX_TX_MULTI_SEG_F) {
/* [SEC] [TSMP] [MBUF_NOFF] [VLAN] [OL3_L4_CSUM] [L3_L4_CSUM] */
event_dev->txa_enqueue = ssogws_dual_tx_adptr_enq_seg
+ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MARK_F)]
[!!(dev->tx_offloads &
NIX_TX_OFFLOAD_SECURITY_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)]
@@ -460,6 +463,7 @@ SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
NIX_TX_OFFLOAD_L3_L4_CSUM_F)];
} else {
event_dev->txa_enqueue = ssogws_dual_tx_adptr_enq
+ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MARK_F)]
[!!(dev->tx_offloads &
NIX_TX_OFFLOAD_SECURITY_F)]
[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)]
diff --git a/drivers/event/octeontx2/otx2_evdev.h b/drivers/event/octeontx2/otx2_evdev.h
index 3b47782..73eee8f 100644
--- a/drivers/event/octeontx2/otx2_evdev.h
+++ b/drivers/event/octeontx2/otx2_evdev.h
@@ -335,7 +335,7 @@ uint16_t otx2_ssogws_dual_deq_seg_timeout_burst_ ##name(void *port, \
SSO_RX_ADPTR_ENQ_FASTPATH_FUNC
#undef R
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
uint16_t otx2_ssogws_tx_adptr_enq_ ## name(void *port, struct rte_event ev[],\
uint16_t nb_events); \
uint16_t otx2_ssogws_tx_adptr_enq_seg_ ## name(void *port, \
diff --git a/drivers/event/octeontx2/otx2_worker.c b/drivers/event/octeontx2/otx2_worker.c
index 8bec59e..cf73fd0 100644
--- a/drivers/event/octeontx2/otx2_worker.c
+++ b/drivers/event/octeontx2/otx2_worker.c
@@ -267,7 +267,7 @@ otx2_ssogws_enq_fwd_burst(void *port, const struct rte_event ev[],
return 1;
}
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
uint16_t __hot \
otx2_ssogws_tx_adptr_enq_ ## name(void *port, struct rte_event ev[], \
uint16_t nb_events) \
@@ -281,7 +281,7 @@ otx2_ssogws_tx_adptr_enq_ ## name(void *port, struct rte_event ev[], \
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
uint16_t __hot \
otx2_ssogws_tx_adptr_enq_seg_ ## name(void *port, struct rte_event ev[],\
uint16_t nb_events) \
diff --git a/drivers/event/octeontx2/otx2_worker.h b/drivers/event/octeontx2/otx2_worker.h
index 5f5aa87..5a59493 100644
--- a/drivers/event/octeontx2/otx2_worker.h
+++ b/drivers/event/octeontx2/otx2_worker.h
@@ -270,8 +270,10 @@ static __rte_always_inline void
otx2_ssogws_prepare_pkt(const struct otx2_eth_txq *txq, struct rte_mbuf *m,
uint64_t *cmd, const uint32_t flags)
{
+ const uint64_t markfmt_en = txq->markfmt_en;
+
otx2_lmt_mov(cmd, txq->cmd, otx2_nix_tx_ext_subs(flags));
- otx2_nix_xmit_prepare(m, cmd, flags);
+ otx2_nix_xmit_prepare(m, cmd, flags, markfmt_en);
}
static __rte_always_inline uint16_t
diff --git a/drivers/event/octeontx2/otx2_worker_dual.c b/drivers/event/octeontx2/otx2_worker_dual.c
index 3cba09c..5f3acc3 100644
--- a/drivers/event/octeontx2/otx2_worker_dual.c
+++ b/drivers/event/octeontx2/otx2_worker_dual.c
@@ -307,7 +307,7 @@ otx2_ssogws_dual_deq_seg_timeout_burst_ ##name(void *port, \
SSO_RX_ADPTR_ENQ_FASTPATH_FUNC
#undef R
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
uint16_t __hot \
otx2_ssogws_dual_tx_adptr_enq_ ## name(void *port, \
struct rte_event ev[], \
@@ -324,7 +324,7 @@ otx2_ssogws_dual_tx_adptr_enq_ ## name(void *port, \
SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
#undef T
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
uint16_t __hot \
otx2_ssogws_dual_tx_adptr_enq_seg_ ## name(void *port, \
struct rte_event ev[], \
diff --git a/drivers/net/octeontx2/otx2_tx.c b/drivers/net/octeontx2/otx2_tx.c
index 4f2036c..638dd9d 100644
--- a/drivers/net/octeontx2/otx2_tx.c
+++ b/drivers/net/octeontx2/otx2_tx.c
@@ -27,6 +27,7 @@ nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
struct otx2_eth_txq *txq = tx_queue; uint16_t i;
const rte_iova_t io_addr = txq->io_addr;
void *lmt_addr = txq->lmt_addr;
+ uint64_t markfmt_en = 0;
NIX_XMIT_FC_OR_RETURN(txq, pkts);
@@ -38,11 +39,14 @@ nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
otx2_nix_xmit_prepare_tso(tx_pkts[i], flags);
}
+ if (flags & NIX_TX_OFFLOAD_MARK_F)
+ markfmt_en = txq->markfmt_en;
+
/* Lets commit any changes in the packet */
rte_cio_wmb();
for (i = 0; i < pkts; i++) {
- otx2_nix_xmit_prepare(tx_pkts[i], cmd, flags);
+ otx2_nix_xmit_prepare(tx_pkts[i], cmd, flags, markfmt_en);
/* Passing no of segdw as 4: HDR + EXT + SG + SMEM */
otx2_nix_xmit_prepare_tstamp(cmd, &txq->cmd[0],
tx_pkts[i]->ol_flags, 4, flags);
@@ -62,6 +66,7 @@ nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts,
struct otx2_eth_txq *txq = tx_queue; uint64_t i;
const rte_iova_t io_addr = txq->io_addr;
void *lmt_addr = txq->lmt_addr;
+ uint64_t markfmt_en = 0;
uint16_t segdw;
NIX_XMIT_FC_OR_RETURN(txq, pkts);
@@ -74,11 +79,14 @@ nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts,
otx2_nix_xmit_prepare_tso(tx_pkts[i], flags);
}
+ if (flags & NIX_TX_OFFLOAD_MARK_F)
+ markfmt_en = txq->markfmt_en;
+
/* Lets commit any changes in the packet */
rte_cio_wmb();
for (i = 0; i < pkts; i++) {
- otx2_nix_xmit_prepare(tx_pkts[i], cmd, flags);
+ otx2_nix_xmit_prepare(tx_pkts[i], cmd, flags, markfmt_en);
segdw = otx2_nix_prepare_mseg(tx_pkts[i], cmd, flags);
otx2_nix_xmit_prepare_tstamp(cmd, &txq->cmd[0],
tx_pkts[i]->ol_flags, segdw,
@@ -946,7 +954,7 @@ nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts,
}
#endif
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
static uint16_t __rte_noinline __hot \
otx2_nix_xmit_pkts_ ## name(void *tx_queue, \
struct rte_mbuf **tx_pkts, uint16_t pkts) \
@@ -963,7 +971,7 @@ otx2_nix_xmit_pkts_ ## name(void *tx_queue, \
NIX_TX_FASTPATH_MODES
#undef T
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
static uint16_t __rte_noinline __hot \
otx2_nix_xmit_pkts_mseg_ ## name(void *tx_queue, \
struct rte_mbuf **tx_pkts, uint16_t pkts) \
@@ -981,17 +989,18 @@ otx2_nix_xmit_pkts_mseg_ ## name(void *tx_queue, \
NIX_TX_FASTPATH_MODES
#undef T
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
static uint16_t __rte_noinline __hot \
otx2_nix_xmit_pkts_vec_ ## name(void *tx_queue, \
struct rte_mbuf **tx_pkts, uint16_t pkts) \
{ \
uint64_t cmd[sz]; \
\
- /* VLAN, TSTMP, TSO is not supported by vec */ \
+ /* VLAN, TSTMP, TSO, MARK is not supported by vec */ \
if ((flags) & NIX_TX_OFFLOAD_VLAN_QINQ_F || \
(flags) & NIX_TX_OFFLOAD_TSTAMP_F || \
- (flags) & NIX_TX_OFFLOAD_TSO_F) \
+ (flags) & NIX_TX_OFFLOAD_TSO_F || \
+ (flags) & NIX_TX_OFFLOAD_MARK_F) \
return 0; \
return nix_xmit_pkts_vector(tx_queue, tx_pkts, pkts, cmd, (flags)); \
}
@@ -1001,12 +1010,13 @@ NIX_TX_FASTPATH_MODES
static inline void
pick_tx_func(struct rte_eth_dev *eth_dev,
- const eth_tx_burst_t tx_burst[2][2][2][2][2][2][2])
+ const eth_tx_burst_t tx_burst[2][2][2][2][2][2][2][2])
{
struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
- /* [SEC] [TSTMP] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */
+ /* [MARK] [SEC] [TSTMP] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */
eth_dev->tx_pkt_burst = tx_burst
+ [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_MARK_F)]
[!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_SECURITY_F)]
[!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSO_F)]
[!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F)]
@@ -1021,25 +1031,25 @@ otx2_eth_set_tx_function(struct rte_eth_dev *eth_dev)
{
struct otx2_eth_dev *dev = otx2_eth_pmd_priv(eth_dev);
- const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_ ## name,
+ const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_ ## name,
NIX_TX_FASTPATH_MODES
#undef T
};
- const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_mseg_ ## name,
+ const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_mseg_ ## name,
NIX_TX_FASTPATH_MODES
#undef T
};
- const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2][2] = {
-#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
- [f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_vec_ ## name,
+ const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2][2][2] = {
+#define T(name, f7, f6, f5, f4, f3, f2, f1, f0, sz, flags) \
+ [f7][f6][f5][f4][f3][f2][f1][f0] = otx2_nix_xmit_pkts_vec_ ## name,
NIX_TX_FASTPATH_MODES
#undef T
@@ -1048,7 +1058,7 @@ NIX_TX_FASTPATH_MODES
if (dev->scalar_ena ||
(dev->tx_offload_flags &
(NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F |
- NIX_TX_OFFLOAD_TSO_F)))
+ NIX_TX_OFFLOAD_TSO_F | NIX_TX_OFFLOAD_MARK_F)))
pick_tx_func(eth_dev, nix_eth_tx_burst);
else
pick_tx_func(eth_dev, nix_eth_tx_vec_burst);
diff --git a/drivers/net/octeontx2/otx2_tx.h b/drivers/net/octeontx2/otx2_tx.h
index 928e4ea..aba0607 100644
--- a/drivers/net/octeontx2/otx2_tx.h
+++ b/drivers/net/octeontx2/otx2_tx.h
@@ -200,7 +200,8 @@ otx2_nix_xmit_prepare_tso(struct rte_mbuf *m, const uint64_t flags)
}
static __rte_always_inline void
-otx2_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags)
+otx2_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd,
+ const uint16_t flags, const uint64_t markfmt_en)
{
struct nix_send_ext_s *send_hdr_ext;
struct nix_send_hdr_s *send_hdr;
@@ -353,6 +354,23 @@ otx2_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags)
}
}
+ if (flags & NIX_TX_OFFLOAD_MARK_F) {
+ /* Pick the first among the following */
+ uint8_t mark_offload = ffs((ol_flags &
+ (PKT_TX_MARK_VLAN_DEI |
+ PKT_TX_MARK_IP_ECN |
+ PKT_TX_MARK_IP_DSCP)) >> 38);
+ uint8_t ipv6 = !!(ol_flags & PKT_TX_IPV6);
+ uint16_t markfmt = (markfmt_en >>
+ ((mark_offload - 1) << 4));
+ markfmt = (markfmt >> (ipv6 << 3)) & 0xFF;
+ send_hdr_ext->w0.mark_en = !!mark_offload;
+ send_hdr_ext->w0.markform = markfmt & 0x7F;
+ send_hdr_ext->w0.markptr = (mark_offload == 1) ?
+ m->l2_len - 4 : /* VLAN DEI */
+ m->l2_len + (markfmt >> 7); /* IPv4v6 */
+ }
+
if (flags & NIX_TX_NEED_SEND_HDR_W1)
send_hdr->w1.u = w1.u;
@@ -475,273 +493,575 @@ otx2_nix_xmit_mseg_one(uint64_t *cmd, void *lmt_addr,
#define TSP_F NIX_TX_OFFLOAD_TSTAMP_F
#define TSO_F NIX_TX_OFFLOAD_TSO_F
#define TX_SEC_F NIX_TX_OFFLOAD_SECURITY_F
+#define TX_MARK_F NIX_TX_OFFLOAD_MARK_F
-/* [SEC] [TSO] [TSTMP] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */
+/* [MARK] [SEC] [TSO] [TSTMP] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */
#define NIX_TX_FASTPATH_MODES \
-T(no_offload, 0, 0, 0, 0, 0, 0, 0, 4, \
+T(no_offload, 0, 0, 0, 0, 0, 0, 0, 0, 4, \
NIX_TX_OFFLOAD_NONE) \
-T(l3l4csum, 0, 0, 0, 0, 0, 0, 1, 4, \
+T(l3l4csum, 0, 0, 0, 0, 0, 0, 0, 1, 4, \
L3L4CSUM_F) \
-T(ol3ol4csum, 0, 0, 0, 0, 0, 1, 0, 4, \
+T(ol3ol4csum, 0, 0, 0, 0, 0, 0, 1, 0, 4, \
OL3OL4CSUM_F) \
-T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 0, 1, 1, 4, \
+T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 0, 0, 1, 1, 4, \
OL3OL4CSUM_F | L3L4CSUM_F) \
-T(vlan, 0, 0, 0, 0, 1, 0, 0, 6, \
+T(vlan, 0, 0, 0, 0, 0, 1, 0, 0, 6, \
VLAN_F) \
-T(vlan_l3l4csum, 0, 0, 0, 0, 1, 0, 1, 6, \
+T(vlan_l3l4csum, 0, 0, 0, 0, 0, 1, 0, 1, 6, \
VLAN_F | L3L4CSUM_F) \
-T(vlan_ol3ol4csum, 0, 0, 0, 0, 1, 1, 0, 6, \
+T(vlan_ol3ol4csum, 0, 0, 0, 0, 0, 1, 1, 0, 6, \
VLAN_F | OL3OL4CSUM_F) \
-T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 1, 6, \
+T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 0, 1, 1, 1, 6, \
VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(noff, 0, 0, 0, 1, 0, 0, 0, 4, \
+T(noff, 0, 0, 0, 0, 1, 0, 0, 0, 4, \
NOFF_F) \
-T(noff_l3l4csum, 0, 0, 0, 1, 0, 0, 1, 4, \
+T(noff_l3l4csum, 0, 0, 0, 0, 1, 0, 0, 1, 4, \
NOFF_F | L3L4CSUM_F) \
-T(noff_ol3ol4csum, 0, 0, 0, 1, 0, 1, 0, 4, \
+T(noff_ol3ol4csum, 0, 0, 0, 0, 1, 0, 1, 0, 4, \
NOFF_F | OL3OL4CSUM_F) \
-T(noff_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 0, 1, 1, 4, \
+T(noff_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 0, 1, 1, 4, \
NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(noff_vlan, 0, 0, 0, 1, 1, 0, 0, 6, \
+T(noff_vlan, 0, 0, 0, 0, 1, 1, 0, 0, 6, \
NOFF_F | VLAN_F) \
-T(noff_vlan_l3l4csum, 0, 0, 0, 1, 1, 0, 1, 6, \
+T(noff_vlan_l3l4csum, 0, 0, 0, 0, 1, 1, 0, 1, 6, \
NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(noff_vlan_ol3ol4csum, 0, 0, 0, 1, 1, 1, 0, 6, \
+T(noff_vlan_ol3ol4csum, 0, 0, 0, 0, 1, 1, 1, 0, 6, \
NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 1, 6, \
+T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 1, 1, 6, \
NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(ts, 0, 0, 1, 0, 0, 0, 0, 8, \
+T(ts, 0, 0, 0, 1, 0, 0, 0, 0, 8, \
TSP_F) \
-T(ts_l3l4csum, 0, 0, 1, 0, 0, 0, 1, 8, \
+T(ts_l3l4csum, 0, 0, 0, 1, 0, 0, 0, 1, 8, \
TSP_F | L3L4CSUM_F) \
-T(ts_ol3ol4csum, 0, 0, 1, 0, 0, 1, 0, 8, \
+T(ts_ol3ol4csum, 0, 0, 0, 1, 0, 0, 1, 0, 8, \
TSP_F | OL3OL4CSUM_F) \
-T(ts_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 0, 1, 1, 8, \
+T(ts_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 0, 0, 1, 1, 8, \
TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(ts_vlan, 0, 0, 1, 0, 1, 0, 0, 8, \
+T(ts_vlan, 0, 0, 0, 1, 0, 1, 0, 0, 8, \
TSP_F | VLAN_F) \
-T(ts_vlan_l3l4csum, 0, 0, 1, 0, 1, 0, 1, 8, \
+T(ts_vlan_l3l4csum, 0, 0, 0, 1, 0, 1, 0, 1, 8, \
TSP_F | VLAN_F | L3L4CSUM_F) \
-T(ts_vlan_ol3ol4csum, 0, 0, 1, 0, 1, 1, 0, 8, \
+T(ts_vlan_ol3ol4csum, 0, 0, 0, 1, 0, 1, 1, 0, 8, \
TSP_F | VLAN_F | OL3OL4CSUM_F) \
-T(ts_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 1, 8, \
+T(ts_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 0, 1, 1, 1, 8, \
TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(ts_noff, 0, 0, 1, 1, 0, 0, 0, 8, \
+T(ts_noff, 0, 0, 0, 1, 1, 0, 0, 0, 8, \
TSP_F | NOFF_F) \
-T(ts_noff_l3l4csum, 0, 0, 1, 1, 0, 0, 1, 8, \
+T(ts_noff_l3l4csum, 0, 0, 0, 1, 1, 0, 0, 1, 8, \
TSP_F | NOFF_F | L3L4CSUM_F) \
-T(ts_noff_ol3ol4csum, 0, 0, 1, 1, 0, 1, 0, 8, \
+T(ts_noff_ol3ol4csum, 0, 0, 0, 1, 1, 0, 1, 0, 8, \
TSP_F | NOFF_F | OL3OL4CSUM_F) \
-T(ts_noff_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 0, 1, 1, 8, \
+T(ts_noff_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 0, 1, 1, 8, \
TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(ts_noff_vlan, 0, 0, 1, 1, 1, 0, 0, 8, \
+T(ts_noff_vlan, 0, 0, 0, 1, 1, 1, 0, 0, 8, \
TSP_F | NOFF_F | VLAN_F) \
-T(ts_noff_vlan_l3l4csum, 0, 0, 1, 1, 1, 0, 1, 8, \
+T(ts_noff_vlan_l3l4csum, 0, 0, 0, 1, 1, 1, 0, 1, 8, \
TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(ts_noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 1, 0, 8, \
+T(ts_noff_vlan_ol3ol4csum, 0, 0, 0, 1, 1, 1, 1, 0, 8, \
TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 1, 8, \
+T(ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 1, 1, 8, \
TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
\
-T(tso, 0, 1, 0, 0, 0, 0, 0, 6, \
+T(tso, 0, 0, 1, 0, 0, 0, 0, 0, 6, \
TSO_F) \
-T(tso_l3l4csum, 0, 1, 0, 0, 0, 0, 1, 6, \
+T(tso_l3l4csum, 0, 0, 1, 0, 0, 0, 0, 1, 6, \
TSO_F | L3L4CSUM_F) \
-T(tso_ol3ol4csum, 0, 1, 0, 0, 0, 1, 0, 6, \
+T(tso_ol3ol4csum, 0, 0, 1, 0, 0, 0, 1, 0, 6, \
TSO_F | OL3OL4CSUM_F) \
-T(tso_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 0, 1, 1, 6, \
+T(tso_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 0, 0, 1, 1, 6, \
TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_vlan, 0, 1, 0, 0, 1, 0, 0, 6, \
+T(tso_vlan, 0, 0, 1, 0, 0, 1, 0, 0, 6, \
TSO_F | VLAN_F) \
-T(tso_vlan_l3l4csum, 0, 1, 0, 0, 1, 0, 1, 6, \
+T(tso_vlan_l3l4csum, 0, 0, 1, 0, 0, 1, 0, 1, 6, \
TSO_F | VLAN_F | L3L4CSUM_F) \
-T(tso_vlan_ol3ol4csum, 0, 1, 0, 0, 1, 1, 0, 6, \
+T(tso_vlan_ol3ol4csum, 0, 0, 1, 0, 0, 1, 1, 0, 6, \
TSO_F | VLAN_F | OL3OL4CSUM_F) \
-T(tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 1, 6, \
+T(tso_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 0, 1, 1, 1, 6, \
TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_noff, 0, 1, 0, 1, 0, 0, 0, 6, \
+T(tso_noff, 0, 0, 1, 0, 1, 0, 0, 0, 6, \
TSO_F | NOFF_F) \
-T(tso_noff_l3l4csum, 0, 1, 0, 1, 0, 0, 1, 6, \
+T(tso_noff_l3l4csum, 0, 0, 1, 0, 1, 0, 0, 1, 6, \
TSO_F | NOFF_F | L3L4CSUM_F) \
-T(tso_noff_ol3ol4csum, 0, 1, 0, 1, 0, 1, 0, 6, \
+T(tso_noff_ol3ol4csum, 0, 0, 1, 0, 1, 0, 1, 0, 6, \
TSO_F | NOFF_F | OL3OL4CSUM_F) \
-T(tso_noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 0, 1, 1, 6, \
+T(tso_noff_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 0, 1, 1, 6, \
TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_noff_vlan, 0, 1, 0, 1, 1, 0, 0, 6, \
+T(tso_noff_vlan, 0, 0, 1, 0, 1, 1, 0, 0, 6, \
TSO_F | NOFF_F | VLAN_F) \
-T(tso_noff_vlan_l3l4csum, 0, 1, 0, 1, 1, 0, 1, 6, \
+T(tso_noff_vlan_l3l4csum, 0, 0, 1, 0, 1, 1, 0, 1, 6, \
TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(tso_noff_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 1, 0, 6, \
+T(tso_noff_vlan_ol3ol4csum, 0, 0, 1, 0, 1, 1, 1, 0, 6, \
TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 1, 6, \
+T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 1, 1, 6, \
TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_ts, 0, 1, 1, 0, 0, 0, 0, 8, \
+T(tso_ts, 0, 0, 1, 1, 0, 0, 0, 0, 8, \
TSO_F | TSP_F) \
-T(tso_ts_l3l4csum, 0, 1, 1, 0, 0, 0, 1, 8, \
+T(tso_ts_l3l4csum, 0, 0, 1, 1, 0, 0, 0, 1, 8, \
TSO_F | TSP_F | L3L4CSUM_F) \
-T(tso_ts_ol3ol4csum, 0, 1, 1, 0, 0, 1, 0, 8, \
+T(tso_ts_ol3ol4csum, 0, 0, 1, 1, 0, 0, 1, 0, 8, \
TSO_F | TSP_F | OL3OL4CSUM_F) \
-T(tso_ts_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 0, 1, 1, 8, \
+T(tso_ts_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 0, 0, 1, 1, 8, \
TSO_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_ts_vlan, 0, 1, 1, 0, 1, 0, 0, 8, \
+T(tso_ts_vlan, 0, 0, 1, 1, 0, 1, 0, 0, 8, \
TSO_F | TSP_F | VLAN_F) \
-T(tso_ts_vlan_l3l4csum, 0, 1, 1, 0, 1, 0, 1, 8, \
+T(tso_ts_vlan_l3l4csum, 0, 0, 1, 1, 0, 1, 0, 1, 8, \
TSO_F | TSP_F | VLAN_F | L3L4CSUM_F) \
-T(tso_ts_vlan_ol3ol4csum, 0, 1, 1, 0, 1, 1, 0, 8, \
+T(tso_ts_vlan_ol3ol4csum, 0, 0, 1, 1, 0, 1, 1, 0, 8, \
TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
-T(tso_ts_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 1, 8, \
+T(tso_ts_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 0, 1, 1, 1, 8, \
TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_ts_noff, 0, 1, 1, 1, 0, 0, 0, 8, \
+T(tso_ts_noff, 0, 0, 1, 1, 1, 0, 0, 0, 8, \
TSO_F | TSP_F | NOFF_F) \
-T(tso_ts_noff_l3l4csum, 0, 1, 1, 1, 0, 0, 1, 8, \
+T(tso_ts_noff_l3l4csum, 0, 0, 1, 1, 1, 0, 0, 1, 8, \
TSO_F | TSP_F | NOFF_F | L3L4CSUM_F) \
-T(tso_ts_noff_ol3ol4csum, 0, 1, 1, 1, 0, 1, 0, 8, \
+T(tso_ts_noff_ol3ol4csum, 0, 0, 1, 1, 1, 0, 1, 0, 8, \
TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
-T(tso_ts_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 0, 1, 1, 8, \
+T(tso_ts_noff_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 0, 1, 1, 8, \
TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(tso_ts_noff_vlan, 0, 1, 1, 1, 1, 0, 0, 8, \
+T(tso_ts_noff_vlan, 0, 0, 1, 1, 1, 1, 0, 0, 8, \
TSO_F | TSP_F | NOFF_F | VLAN_F) \
-T(tso_ts_noff_vlan_l3l4csum, 0, 1, 1, 1, 1, 0, 1, 8, \
+T(tso_ts_noff_vlan_l3l4csum, 0, 0, 1, 1, 1, 1, 0, 1, 8, \
TSO_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(tso_ts_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 1, 0, 8, \
+T(tso_ts_noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 1, 1, 0, 8, \
TSO_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(tso_ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 1, 8, \
+T(tso_ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 1, 1, 8, \
TSO_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
L3L4CSUM_F) \
-T(sec, 1, 0, 0, 0, 0, 0, 0, 8, \
+T(sec, 0, 1, 0, 0, 0, 0, 0, 0, 8, \
TX_SEC_F) \
-T(sec_l3l4csum, 1, 0, 0, 0, 0, 0, 1, 8, \
+T(sec_l3l4csum, 0, 1, 0, 0, 0, 0, 0, 1, 8, \
TX_SEC_F | L3L4CSUM_F) \
-T(sec_ol3ol4csum, 1, 0, 0, 0, 0, 1, 0, 8, \
+T(sec_ol3ol4csum, 0, 1, 0, 0, 0, 0, 1, 0, 8, \
TX_SEC_F | OL3OL4CSUM_F) \
-T(sec_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 0, 1, 1, 8, \
+T(sec_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 0, 0, 1, 1, 8, \
TX_SEC_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_vlan, 1, 0, 0, 0, 1, 0, 0, 8, \
+T(sec_vlan, 0, 1, 0, 0, 0, 1, 0, 0, 8, \
TX_SEC_F | VLAN_F) \
-T(sec_vlan_l3l4csum, 1, 0, 0, 0, 1, 0, 1, 8, \
+T(sec_vlan_l3l4csum, 0, 1, 0, 0, 0, 1, 0, 1, 8, \
TX_SEC_F | VLAN_F | L3L4CSUM_F) \
-T(sec_vlan_ol3ol4csum, 1, 0, 0, 0, 1, 1, 0, 8, \
+T(sec_vlan_ol3ol4csum, 0, 1, 0, 0, 0, 1, 1, 0, 8, \
TX_SEC_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 1, 8, \
+T(sec_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 0, 1, 1, 1, 8, \
TX_SEC_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_noff, 1, 0, 0, 1, 0, 0, 0, 8, \
+T(sec_noff, 0, 1, 0, 0, 1, 0, 0, 0, 8, \
TX_SEC_F | NOFF_F) \
-T(sec_noff_l3l4csum, 1, 0, 0, 1, 0, 0, 1, 8, \
+T(sec_noff_l3l4csum, 0, 1, 0, 0, 1, 0, 0, 1, 8, \
TX_SEC_F | NOFF_F | L3L4CSUM_F) \
-T(sec_noff_ol3ol4csum, 1, 0, 0, 1, 0, 1, 0, 8, \
+T(sec_noff_ol3ol4csum, 0, 1, 0, 0, 1, 0, 1, 0, 8, \
TX_SEC_F | NOFF_F | OL3OL4CSUM_F) \
-T(sec_noff_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 0, 1, 1, 8, \
+T(sec_noff_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 0, 1, 1, 8, \
TX_SEC_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_noff_vlan, 1, 0, 0, 1, 1, 0, 0, 8, \
+T(sec_noff_vlan, 0, 1, 0, 0, 1, 1, 0, 0, 8, \
TX_SEC_F | NOFF_F | VLAN_F) \
-T(sec_noff_vlan_l3l4csum, 1, 0, 0, 1, 1, 0, 1, 8, \
+T(sec_noff_vlan_l3l4csum, 0, 1, 0, 0, 1, 1, 0, 1, 8, \
TX_SEC_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(sec_noff_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 1, 0, 8, \
+T(sec_noff_vlan_ol3ol4csum, 0, 1, 0, 0, 1, 1, 1, 0, 8, \
TX_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 1, 8, \
+T(sec_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 1, 1, 8, \
TX_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_ts, 1, 0, 1, 0, 0, 0, 0, 8, \
+T(sec_ts, 0, 1, 0, 1, 0, 0, 0, 0, 8, \
TX_SEC_F | TSP_F) \
-T(sec_ts_l3l4csum, 1, 0, 1, 0, 0, 0, 1, 8, \
+T(sec_ts_l3l4csum, 0, 1, 0, 1, 0, 0, 0, 1, 8, \
TX_SEC_F | TSP_F | L3L4CSUM_F) \
-T(sec_ts_ol3ol4csum, 1, 0, 1, 0, 0, 1, 0, 8, \
+T(sec_ts_ol3ol4csum, 0, 1, 0, 1, 0, 0, 1, 0, 8, \
TX_SEC_F | TSP_F | OL3OL4CSUM_F) \
-T(sec_ts_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 0, 1, 1, 8, \
+T(sec_ts_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 0, 0, 1, 1, 8, \
TX_SEC_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_ts_vlan, 1, 0, 1, 0, 1, 0, 0, 8, \
+T(sec_ts_vlan, 0, 1, 0, 1, 0, 1, 0, 0, 8, \
TX_SEC_F | TSP_F | VLAN_F) \
-T(sec_ts_vlan_l3l4csum, 1, 0, 1, 0, 1, 0, 1, 8, \
+T(sec_ts_vlan_l3l4csum, 0, 1, 0, 1, 0, 1, 0, 1, 8, \
TX_SEC_F | TSP_F | VLAN_F | L3L4CSUM_F) \
-T(sec_ts_vlan_ol3ol4csum, 1, 0, 1, 0, 1, 1, 0, 8, \
+T(sec_ts_vlan_ol3ol4csum, 0, 1, 0, 1, 0, 1, 1, 0, 8, \
TX_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 1, 8, \
+T(sec_ts_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 0, 1, 1, 1, 8, \
TX_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_ts_noff, 1, 0, 1, 1, 0, 0, 0, 8, \
+T(sec_ts_noff, 0, 1, 0, 1, 1, 0, 0, 0, 8, \
TX_SEC_F | TSP_F | NOFF_F) \
-T(sec_ts_noff_l3l4csum, 1, 0, 1, 1, 0, 0, 1, 8, \
+T(sec_ts_noff_l3l4csum, 0, 1, 0, 1, 1, 0, 0, 1, 8, \
TX_SEC_F | TSP_F | NOFF_F | L3L4CSUM_F) \
-T(sec_ts_noff_ol3ol4csum, 1, 0, 1, 1, 0, 1, 0, 8, \
+T(sec_ts_noff_ol3ol4csum, 0, 1, 0, 1, 1, 0, 1, 0, 8, \
TX_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
-T(sec_ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 0, 1, 1, 8, \
+T(sec_ts_noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 0, 1, 1, 8, \
TX_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_ts_noff_vlan, 1, 0, 1, 1, 1, 0, 0, 8, \
+T(sec_ts_noff_vlan, 0, 1, 0, 1, 1, 1, 0, 0, 8, \
TX_SEC_F | TSP_F | NOFF_F | VLAN_F) \
-T(sec_ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 1, 0, 1, 8, \
+T(sec_ts_noff_vlan_l3l4csum, 0, 1, 0, 1, 1, 1, 0, 1, 8, \
TX_SEC_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(sec_ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 1, 0, 8, \
+T(sec_ts_noff_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 1, 1, 0, 8, \
TX_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 1, 8, \
+T(sec_ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 1, 1, 8, \
TX_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
L3L4CSUM_F) \
-T(sec_tso, 1, 1, 0, 0, 0, 0, 0, 8, \
+T(sec_tso, 0, 1, 1, 0, 0, 0, 0, 0, 8, \
TX_SEC_F | TSO_F) \
-T(sec_tso_l3l4csum, 1, 1, 0, 0, 0, 0, 1, 8, \
+T(sec_tso_l3l4csum, 0, 1, 1, 0, 0, 0, 0, 1, 8, \
TX_SEC_F | TSO_F | L3L4CSUM_F) \
-T(sec_tso_ol3ol4csum, 1, 1, 0, 0, 0, 1, 0, 8, \
+T(sec_tso_ol3ol4csum, 0, 1, 1, 0, 0, 0, 1, 0, 8, \
TX_SEC_F | TSO_F | OL3OL4CSUM_F) \
-T(sec_tso_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 0, 1, 1, 8, \
+T(sec_tso_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 0, 0, 1, 1, 8, \
TX_SEC_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_tso_vlan, 1, 1, 0, 0, 1, 0, 0, 8, \
+T(sec_tso_vlan, 0, 1, 1, 0, 0, 1, 0, 0, 8, \
TX_SEC_F | TSO_F | VLAN_F) \
-T(sec_tso_vlan_l3l4csum, 1, 1, 0, 0, 1, 0, 1, 8, \
+T(sec_tso_vlan_l3l4csum, 0, 1, 1, 0, 0, 1, 0, 1, 8, \
TX_SEC_F | TSO_F | VLAN_F | L3L4CSUM_F) \
-T(sec_tso_vlan_ol3ol4csum, 1, 1, 0, 0, 1, 1, 0, 8, \
+T(sec_tso_vlan_ol3ol4csum, 0, 1, 1, 0, 0, 1, 1, 0, 8, \
TX_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 1, 8, \
+T(sec_tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 0, 1, 1, 1, 8, \
TX_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_tso_noff, 1, 1, 0, 1, 0, 0, 0, 8, \
+T(sec_tso_noff, 0, 1, 1, 0, 1, 0, 0, 0, 8, \
TX_SEC_F | TSO_F | NOFF_F) \
-T(sec_tso_noff_l3l4csum, 1, 1, 0, 1, 0, 0, 1, 8, \
+T(sec_tso_noff_l3l4csum, 0, 1, 1, 0, 1, 0, 0, 1, 8, \
TX_SEC_F | TSO_F | NOFF_F | L3L4CSUM_F) \
-T(sec_tso_noff_ol3ol4csum, 1, 1, 0, 1, 0, 1, 0, 8, \
+T(sec_tso_noff_ol3ol4csum, 0, 1, 1, 0, 1, 0, 1, 0, 8, \
TX_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \
-T(sec_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 0, 1, 1, 8, \
+T(sec_tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 0, 1, 1, 8, \
TX_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_tso_noff_vlan, 1, 1, 0, 1, 1, 0, 0, 8, \
+T(sec_tso_noff_vlan, 0, 1, 1, 0, 1, 1, 0, 0, 8, \
TX_SEC_F | TSO_F | NOFF_F | VLAN_F) \
-T(sec_tso_noff_vlan_l3l4csum, 1, 1, 0, 1, 1, 0, 1, 8, \
+T(sec_tso_noff_vlan_l3l4csum, 0, 1, 1, 0, 1, 1, 0, 1, 8, \
TX_SEC_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
-T(sec_tso_noff_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 1, 0, 8, \
+T(sec_tso_noff_vlan_ol3ol4csum, 0, 1, 1, 0, 1, 1, 1, 0, 8, \
TX_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
T(sec_tso_noff_vlan_ol3ol4csum_l3l4csum, \
- 1, 1, 0, 1, 1, 1, 1, 8, \
+ 0, 1, 1, 0, 1, 1, 1, 1, 8, \
TX_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
L3L4CSUM_F) \
-T(sec_tso_ts, 1, 1, 1, 0, 0, 0, 0, 8, \
+T(sec_tso_ts, 0, 1, 1, 1, 0, 0, 0, 0, 8, \
TX_SEC_F | TSO_F | TSP_F) \
-T(sec_tso_ts_l3l4csum, 1, 1, 1, 0, 0, 0, 1, 8, \
+T(sec_tso_ts_l3l4csum, 0, 1, 1, 1, 0, 0, 0, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | L3L4CSUM_F) \
-T(sec_tso_ts_ol3ol4csum, 1, 1, 1, 0, 0, 1, 0, 8, \
+T(sec_tso_ts_ol3ol4csum, 0, 1, 1, 1, 0, 0, 1, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | OL3OL4CSUM_F) \
-T(sec_tso_ts_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 0, 1, 1, 8, \
+T(sec_tso_ts_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 0, 0, 1, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
-T(sec_tso_ts_vlan, 1, 1, 1, 0, 1, 0, 0, 8, \
+T(sec_tso_ts_vlan, 0, 1, 1, 1, 0, 1, 0, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | VLAN_F) \
-T(sec_tso_ts_vlan_l3l4csum, 1, 1, 1, 0, 1, 0, 1, 8, \
+T(sec_tso_ts_vlan_l3l4csum, 0, 1, 1, 1, 0, 1, 0, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | VLAN_F | L3L4CSUM_F) \
-T(sec_tso_ts_vlan_ol3ol4csum, 1, 1, 1, 0, 1, 1, 0, 8, \
+T(sec_tso_ts_vlan_ol3ol4csum, 0, 1, 1, 1, 0, 1, 1, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
-T(sec_tso_ts_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 1, 8, \
+T(sec_tso_ts_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 0, 1, 1, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F | \
L3L4CSUM_F) \
-T(sec_tso_ts_noff, 1, 1, 1, 1, 0, 0, 0, 8, \
+T(sec_tso_ts_noff, 0, 1, 1, 1, 1, 0, 0, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F) \
-T(sec_tso_ts_noff_l3l4csum, 1, 1, 1, 1, 0, 0, 1, 8, \
+T(sec_tso_ts_noff_l3l4csum, 0, 1, 1, 1, 1, 0, 0, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | L3L4CSUM_F) \
-T(sec_tso_ts_noff_ol3ol4csum, 1, 1, 1, 1, 0, 1, 0, 8, \
+T(sec_tso_ts_noff_ol3ol4csum, 0, 1, 1, 1, 1, 0, 1, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
-T(sec_tso_ts_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 0, 1, 1, 8, \
+T(sec_tso_ts_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 0, 1, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F | \
L3L4CSUM_F) \
-T(sec_tso_ts_noff_vlan, 1, 1, 1, 1, 1, 0, 0, 8, \
+T(sec_tso_ts_noff_vlan, 0, 1, 1, 1, 1, 1, 0, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F) \
-T(sec_tso_ts_noff_vlan_l3l4csum, 1, 1, 1, 1, 1, 0, 1, 8, \
+T(sec_tso_ts_noff_vlan_l3l4csum, 0, 1, 1, 1, 1, 1, 0, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F)\
-T(sec_tso_ts_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 1, 0, 8, \
+T(sec_tso_ts_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 1, 1, 0, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F | \
OL3OL4CSUM_F) \
T(sec_tso_ts_noff_vlan_ol3ol4csum_l3l4csum, \
- 1, 1, 1, 1, 1, 1, 1, 8, \
+ 0, 1, 1, 1, 1, 1, 1, 1, 8, \
TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark, 1, 0, 0, 0, 0, 0, 0, 0, 6, \
+ TX_MARK_F) \
+T(mark_l3l4csum, 1, 0, 0, 0, 0, 0, 0, 1, 6, \
+ TX_MARK_F | L3L4CSUM_F) \
+T(mark_ol3ol4csum, 1, 0, 0, 0, 0, 0, 1, 0, 6, \
+ TX_MARK_F | OL3OL4CSUM_F) \
+T(mark_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 0, 0, 1, 1, 6, \
+ TX_MARK_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_vlan, 1, 0, 0, 0, 0, 1, 0, 0, 6, \
+ TX_MARK_F | VLAN_F) \
+T(mark_vlan_l3l4csum, 1, 0, 0, 0, 0, 1, 0, 1, 6, \
+ TX_MARK_F | VLAN_F | L3L4CSUM_F) \
+T(mark_vlan_ol3ol4csum, 1, 0, 0, 0, 0, 1, 1, 0, 6, \
+ TX_MARK_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 0, 1, 1, 1, 6, \
+ TX_MARK_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_noff, 1, 0, 0, 0, 1, 0, 0, 0, 6, \
+ TX_MARK_F | NOFF_F) \
+T(mark_noff_l3l4csum, 1, 0, 0, 0, 1, 0, 0, 1, 6, \
+ TX_MARK_F | NOFF_F | L3L4CSUM_F) \
+T(mark_noff_ol3ol4csum, 1, 0, 0, 0, 1, 0, 1, 0, 6, \
+ TX_MARK_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_noff_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 0, 1, 1, 6, \
+ TX_MARK_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_noff_vlan, 1, 0, 0, 0, 1, 1, 0, 0, 6, \
+ TX_MARK_F | NOFF_F | VLAN_F) \
+T(mark_noff_vlan_l3l4csum, 1, 0, 0, 0, 1, 1, 0, 1, 6, \
+ TX_MARK_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
+T(mark_noff_vlan_ol3ol4csum, 1, 0, 0, 0, 1, 1, 1, 0, 6, \
+ TX_MARK_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 1, 1, 6, \
+ TX_MARK_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\
+T(mark_ts, 1, 0, 0, 1, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TSP_F) \
+T(mark_ts_l3l4csum, 1, 0, 0, 1, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TSP_F | L3L4CSUM_F) \
+T(mark_ts_ol3ol4csum, 1, 0, 0, 1, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TSP_F | OL3OL4CSUM_F) \
+T(mark_ts_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_ts_vlan, 1, 0, 0, 1, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TSP_F | VLAN_F) \
+T(mark_ts_vlan_l3l4csum, 1, 0, 0, 1, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TSP_F | VLAN_F | L3L4CSUM_F) \
+T(mark_ts_vlan_ol3ol4csum, 1, 0, 0, 1, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_ts_noff, 1, 0, 0, 1, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TSP_F | NOFF_F) \
+T(mark_ts_noff_l3l4csum, 1, 0, 0, 1, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | L3L4CSUM_F) \
+T(mark_ts_noff_ol3ol4csum, 1, 0, 0, 1, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_ts_noff_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_ts_noff_vlan, 1, 0, 0, 1, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | VLAN_F) \
+T(mark_ts_noff_vlan_l3l4csum, 1, 0, 0, 1, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
+T(mark_ts_noff_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_ts_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 0, 0, 1, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_tso, 1, 0, 1, 0, 0, 0, 0, 0, 6, \
+ TX_MARK_F | TSO_F) \
+T(mark_tso_l3l4csum, 1, 0, 1, 0, 0, 0, 0, 1, 6, \
+ TX_MARK_F | TSO_F | L3L4CSUM_F) \
+T(mark_tso_ol3ol4csum, 1, 0, 1, 0, 0, 0, 1, 0, 6, \
+ TX_MARK_F | TSO_F | OL3OL4CSUM_F) \
+T(mark_tso_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 0, 0, 1, 1, 6, \
+ TX_MARK_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_tso_vlan, 1, 0, 1, 0, 0, 1, 0, 0, 6, \
+ TX_MARK_F | TSO_F | VLAN_F) \
+T(mark_tso_vlan_l3l4csum, 1, 0, 1, 0, 0, 1, 0, 1, 6, \
+ TX_MARK_F | TSO_F | VLAN_F | L3L4CSUM_F) \
+T(mark_tso_vlan_ol3ol4csum, 1, 0, 1, 0, 0, 1, 1, 0, 6, \
+ TX_MARK_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_tso_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 0, 1, 1, 1, 6, \
+ TX_MARK_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_tso_noff, 1, 0, 1, 0, 1, 0, 0, 0, 6, \
+ TX_MARK_F | TSO_F | NOFF_F) \
+T(mark_tso_noff_l3l4csum, 1, 0, 1, 0, 1, 0, 0, 1, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | L3L4CSUM_F) \
+T(mark_tso_noff_ol3ol4csum, 1, 0, 1, 0, 1, 0, 1, 0, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_tso_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 0, 1, 1, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_tso_noff_vlan, 1, 0, 1, 0, 1, 1, 0, 0, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | VLAN_F) \
+T(mark_tso_noff_vlan_l3l4csum, 1, 0, 1, 0, 1, 1, 0, 1, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
+T(mark_tso_noff_vlan_ol3ol4csum, 1, 0, 1, 0, 1, 1, 1, 0, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_tso_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 0, 1, 0, 1, 1, 1, 1, 6, \
+ TX_MARK_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_tso_ts, 1, 0, 1, 1, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F) \
+T(mark_tso_ts_l3l4csum, 1, 0, 1, 1, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | L3L4CSUM_F) \
+T(mark_tso_ts_ol3ol4csum, 1, 0, 1, 1, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | OL3OL4CSUM_F) \
+T(mark_tso_ts_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_tso_ts_vlan, 1, 0, 1, 1, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | VLAN_F) \
+T(mark_tso_ts_vlan_l3l4csum, 1, 0, 1, 1, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | VLAN_F | L3L4CSUM_F) \
+T(mark_tso_ts_vlan_ol3ol4csum, 1, 0, 1, 1, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_tso_ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_tso_ts_noff, 1, 0, 1, 1, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F) \
+T(mark_tso_ts_noff_l3l4csum, 1, 0, 1, 1, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | L3L4CSUM_F) \
+T(mark_tso_ts_noff_ol3ol4csum, 1, 0, 1, 1, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_tso_ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_tso_ts_noff_vlan, 1, 0, 1, 1, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | VLAN_F) \
+T(mark_tso_ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | VLAN_F | \
+ L3L4CSUM_F) \
+T(mark_tso_ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F) \
+T(mark_tso_ts_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 0, 1, 1, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TSO_F | TSP_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec, 1, 1, 0, 0, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F) \
+T(mark_sec_l3l4csum, 1, 1, 0, 0, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | L3L4CSUM_F) \
+T(mark_sec_ol3ol4csum, 1, 1, 0, 0, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | OL3OL4CSUM_F) \
+T(mark_sec_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec_vlan, 1, 1, 0, 0, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | VLAN_F) \
+T(mark_sec_vlan_l3l4csum, 1, 1, 0, 0, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | VLAN_F | L3L4CSUM_F) \
+T(mark_sec_vlan_ol3ol4csum, 1, 1, 0, 0, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_sec_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_noff, 1, 1, 0, 0, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F) \
+T(mark_sec_noff_l3l4csum, 1, 1, 0, 0, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | L3L4CSUM_F) \
+T(mark_sec_noff_ol3ol4csum, 1, 1, 0, 0, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_sec_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_noff_vlan, 1, 1, 0, 0, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | VLAN_F) \
+T(mark_sec_noff_vlan_l3l4csum, 1, 1, 0, 0, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | VLAN_F | L3L4CSUM_F) \
+T(mark_sec_noff_vlan_ol3ol4csum, 1, 1, 0, 0, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_sec_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 0, 0, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_ts, 1, 1, 0, 1, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F) \
+T(mark_sec_ts_l3l4csum, 1, 1, 0, 1, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | L3L4CSUM_F) \
+T(mark_sec_ts_ol3ol4csum, 1, 1, 0, 1, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | OL3OL4CSUM_F) \
+T(mark_sec_ts_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_ts_vlan, 1, 1, 0, 1, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | VLAN_F) \
+T(mark_sec_ts_vlan_l3l4csum, 1, 1, 0, 1, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | VLAN_F | L3L4CSUM_F) \
+T(mark_sec_ts_vlan_ol3ol4csum, 1, 1, 0, 1, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_sec_ts_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_ts_noff, 1, 1, 0, 1, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F) \
+T(mark_sec_ts_noff_l3l4csum, 1, 1, 0, 1, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | L3L4CSUM_F) \
+T(mark_sec_ts_noff_ol3ol4csum, 1, 1, 0, 1, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_sec_ts_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_ts_noff_vlan, 1, 1, 0, 1, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | VLAN_F) \
+T(mark_sec_ts_noff_vlan_l3l4csum, 1, 1, 0, 1, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | VLAN_F | \
+ L3L4CSUM_F) \
+T(mark_sec_ts_noff_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F) \
+T(mark_sec_ts_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 0, 1, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSP_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec_tso, 1, 1, 1, 0, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F) \
+T(mark_sec_tso_l3l4csum, 1, 1, 1, 0, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | L3L4CSUM_F) \
+T(mark_sec_tso_ol3ol4csum, 1, 1, 1, 0, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | OL3OL4CSUM_F) \
+T(mark_sec_tso_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_vlan, 1, 1, 1, 0, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | VLAN_F) \
+T(mark_sec_tso_vlan_l3l4csum, 1, 1, 1, 0, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | VLAN_F | L3L4CSUM_F) \
+T(mark_sec_tso_vlan_ol3ol4csum, 1, 1, 1, 0, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \
+T(mark_sec_tso_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 0, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_noff, 1, 1, 1, 0, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F) \
+T(mark_sec_tso_noff_l3l4csum, 1, 1, 1, 0, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | L3L4CSUM_F) \
+T(mark_sec_tso_noff_ol3ol4csum, 1, 1, 1, 0, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \
+T(mark_sec_tso_noff_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 0, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_noff_vlan, 1, 1, 1, 0, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | VLAN_F) \
+T(mark_sec_tso_noff_vlan_l3l4csum, 1, 1, 1, 0, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | VLAN_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 0, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F) \
+T(mark_sec_tso_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 0, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | NOFF_F | VLAN_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec_tso_ts, 1, 1, 1, 1, 0, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F) \
+T(mark_sec_tso_ts_l3l4csum, 1, 1, 1, 1, 0, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | L3L4CSUM_F) \
+T(mark_sec_tso_ts_ol3ol4csum, 1, 1, 1, 1, 0, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | OL3OL4CSUM_F) \
+T(mark_sec_tso_ts_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 0, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | OL3OL4CSUM_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_ts_vlan, 1, 1, 1, 1, 0, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | VLAN_F) \
+T(mark_sec_tso_ts_vlan_l3l4csum, 1, 1, 1, 1, 0, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | VLAN_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_ts_vlan_ol3ol4csum, 1, 1, 1, 1, 0, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | VLAN_F | \
+ OL3OL4CSUM_F) \
+T(mark_sec_tso_ts_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 1, 0, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | VLAN_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec_tso_ts_noff, 1, 1, 1, 1, 1, 0, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F) \
+T(mark_sec_tso_ts_noff_l3l4csum, 1, 1, 1, 1, 1, 0, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | \
+ L3L4CSUM_F) \
+T(mark_sec_tso_ts_noff_ol3ol4csum, 1, 1, 1, 1, 1, 0, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | \
+ OL3OL4CSUM_F) \
+T(mark_sec_tso_ts_noff_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 1, 1, 0, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | \
+ OL3OL4CSUM_F | L3L4CSUM_F) \
+T(mark_sec_tso_ts_noff_vlan, 1, 1, 1, 1, 1, 1, 0, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F) \
+T(mark_sec_tso_ts_noff_vlan_l3l4csum, 1, 1, 1, 1, 1, 1, 0, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F |\
+ L3L4CSUM_F) \
+T(mark_sec_tso_ts_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 1, 1, 0, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F |\
+ OL3OL4CSUM_F) \
+T(mark_sec_tso_ts_noff_vlan_ol3ol4csum_l3l4csum, \
+ 1, 1, 1, 1, 1, 1, 1, 1, 8, \
+ TX_MARK_F | TX_SEC_F | TSO_F | TSP_F | NOFF_F | VLAN_F |\
OL3OL4CSUM_F | L3L4CSUM_F)
+
+
#endif /* __OTX2_TX_H__ */
--
2.8.4
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-04-17 7:22 [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 2/3] net/octeontx2: add tm packet marking cb Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 3/3] net/octeontx2: add Tx packet marking offload support Nithin Dabilpuram
@ 2020-05-01 11:18 ` Jerin Jacob
2020-05-04 8:06 ` Olivier Matz
2023-07-31 12:54 ` [dpdk-dev] " Thomas Monjalon
3 siblings, 1 reply; 36+ messages in thread
From: Jerin Jacob @ 2020-05-01 11:18 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Thomas Monjalon, Ferruh Yigit, Andrew Rybchenko, Ori Kam,
Cristian Dumitrescu, Anatoly Burakov, John McNamara,
Marko Kovacevic, Olivier Matz, dpdk-dev, Jerin Jacob,
Krzysztof Kanas, Nithin Dabilpuram
On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
<nithind1988@gmail.com> wrote:
>
> From: Nithin Dabilpuram <ndabilpuram@marvell.com>
>
> Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> packet marking.
>
> When packet marking feature in Traffic manager is enabled,
> application has to the use the three new flags to indicate
> to PMD on whether packet marking needs to be enabled on the
> specific mbuf or not. By setting the three flags, it is
> assumed by PMD that application has already verified the
> applicability of marking on that specific packet and
> PMD need not perform further checks as per RFC.
>
> Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
None of the ethdev TM driver implementations has supported packet
marking support.
rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> ---
> doc/guides/nics/features.rst | 14 ++++++++++++++
> lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> 3 files changed, 54 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index edd21c4..bc978fb 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
>
> +.. _nic_features_traffic_manager_packet_marking_offload:
> +
> +Traffic Manager Packet marking offload
> +--------------------------------------
> +
> +Supports enabling a packet marking offload specific mbuf.
> +
> +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> +* **[uses] mbuf**: ``mbuf.l2_len``.
> +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> + ``rte_tm_mark_vlan_dei()``.
> +
> .. _nic_features_other:
>
> Other dev ops not represented by a Feature
> diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> index cd5794d..5c6896d 100644
> --- a/lib/librte_mbuf/rte_mbuf.c
> +++ b/lib/librte_mbuf/rte_mbuf.c
> @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> default: return NULL;
> }
> }
> @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> };
> const char *name;
> unsigned int i;
> diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> index b9a59c8..d9f1290 100644
> --- a/lib/librte_mbuf/rte_mbuf_core.h
> +++ b/lib/librte_mbuf/rte_mbuf_core.h
> @@ -187,11 +187,40 @@ extern "C" {
> /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
>
> #define PKT_FIRST_FREE (1ULL << 23)
> -#define PKT_LAST_FREE (1ULL << 40)
> +#define PKT_LAST_FREE (1ULL << 37)
>
> /* add new TX flags here, don't forget to update PKT_LAST_FREE */
>
> /**
> + * Packet marking offload flags. These flags indicated what kind
> + * of packet marking needs to be applied on a given mbuf when
> + * appropriate Traffic Manager configuration is in place.
> + * When user set's these flags on a mbuf, below assumptions are made
> + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> + * a) PMD assumes pkt to be a 802.1q packet.
> + * b) Application should also set mbuf.l2_len where 802.1Q header is
> + * at (mbuf.l2_len - 6) offset.
> + * 2) When PKT_TX_MARK_IP_DSCP is set,
> + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> + * to indicate whether if it is IPv4 packet or IPv6 packet
> + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> + * IPv4 pkt.
> + * b) Application should also set mbuf.l2_len that indicates
> + * start offset of L3 header.
> + * 3) When PKT_TX_MARK_IP_ECN is set,
> + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> + * can mark the packet for a configured color.
> + * c) Application should also set mbuf.l2_len that indicates
> + * start offset of L3 header.
> + */
> +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> +
> +/**
> * Outer UDP checksum offload flag. This flag is used for enabling
> * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> * 1) Enable the following in mbuf,
> @@ -384,7 +413,10 @@ extern "C" {
> PKT_TX_MACSEC | \
> PKT_TX_SEC_OFFLOAD | \
> PKT_TX_UDP_SEG | \
> - PKT_TX_OUTER_UDP_CKSUM)
> + PKT_TX_OUTER_UDP_CKSUM | \
> + PKT_TX_MARK_VLAN_DEI | \
> + PKT_TX_MARK_IP_DSCP | \
> + PKT_TX_MARK_IP_ECN)
>
> /**
> * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> --
> 2.8.4
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-01 11:18 ` [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Jerin Jacob
@ 2020-05-04 8:06 ` Olivier Matz
2020-05-04 8:27 ` [dpdk-dev] [EXT] " Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Olivier Matz @ 2020-05-04 8:06 UTC (permalink / raw)
To: Jerin Jacob
Cc: Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas, Nithin Dabilpuram
Hi,
On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> <nithind1988@gmail.com> wrote:
> >
> > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> >
> > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > packet marking.
> >
> > When packet marking feature in Traffic manager is enabled,
> > application has to the use the three new flags to indicate
> > to PMD on whether packet marking needs to be enabled on the
> > specific mbuf or not. By setting the three flags, it is
> > assumed by PMD that application has already verified the
> > applicability of marking on that specific packet and
> > PMD need not perform further checks as per RFC.
> >
> > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>
> None of the ethdev TM driver implementations has supported packet
> marking support.
> rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
As you know, the number of mbuf flags is limited (only 18 bits are
remaining), so I think we should use them with care, i.e. for features
that are generic enough.
From what I understand, this feature is bound to octeontx2, so using a
mbuf dynamic flag would make more sense here. There are some examples in
dpdk repository, just grep for "dynflag".
Also, I think that the feature availability should be advertised through
an ethdev offload, so an application can know at initialization time
that these flags can be used.
Regards,
Olivier
>
>
> > ---
> > doc/guides/nics/features.rst | 14 ++++++++++++++
> > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > 3 files changed, 54 insertions(+), 2 deletions(-)
> >
> > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > index edd21c4..bc978fb 100644
> > --- a/doc/guides/nics/features.rst
> > +++ b/doc/guides/nics/features.rst
> > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> >
> > +.. _nic_features_traffic_manager_packet_marking_offload:
> > +
> > +Traffic Manager Packet marking offload
> > +--------------------------------------
> > +
> > +Supports enabling a packet marking offload specific mbuf.
> > +
> > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > + ``rte_tm_mark_vlan_dei()``.
> > +
> > .. _nic_features_other:
> >
> > Other dev ops not represented by a Feature
> > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > index cd5794d..5c6896d 100644
> > --- a/lib/librte_mbuf/rte_mbuf.c
> > +++ b/lib/librte_mbuf/rte_mbuf.c
> > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > default: return NULL;
> > }
> > }
> > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > };
> > const char *name;
> > unsigned int i;
> > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > index b9a59c8..d9f1290 100644
> > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > @@ -187,11 +187,40 @@ extern "C" {
> > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> >
> > #define PKT_FIRST_FREE (1ULL << 23)
> > -#define PKT_LAST_FREE (1ULL << 40)
> > +#define PKT_LAST_FREE (1ULL << 37)
> >
> > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> >
> > /**
> > + * Packet marking offload flags. These flags indicated what kind
> > + * of packet marking needs to be applied on a given mbuf when
> > + * appropriate Traffic Manager configuration is in place.
> > + * When user set's these flags on a mbuf, below assumptions are made
> > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > + * a) PMD assumes pkt to be a 802.1q packet.
> > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > + * at (mbuf.l2_len - 6) offset.
> > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > + * IPv4 pkt.
> > + * b) Application should also set mbuf.l2_len that indicates
> > + * start offset of L3 header.
> > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > + * can mark the packet for a configured color.
> > + * c) Application should also set mbuf.l2_len that indicates
> > + * start offset of L3 header.
> > + */
> > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > +
> > +/**
> > * Outer UDP checksum offload flag. This flag is used for enabling
> > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > * 1) Enable the following in mbuf,
> > @@ -384,7 +413,10 @@ extern "C" {
> > PKT_TX_MACSEC | \
> > PKT_TX_SEC_OFFLOAD | \
> > PKT_TX_UDP_SEG | \
> > - PKT_TX_OUTER_UDP_CKSUM)
> > + PKT_TX_OUTER_UDP_CKSUM | \
> > + PKT_TX_MARK_VLAN_DEI | \
> > + PKT_TX_MARK_IP_DSCP | \
> > + PKT_TX_MARK_IP_ECN)
> >
> > /**
> > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > --
> > 2.8.4
> >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-04 8:06 ` Olivier Matz
@ 2020-05-04 8:27 ` Nithin Dabilpuram
2020-05-04 9:16 ` Olivier Matz
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-04 8:27 UTC (permalink / raw)
To: Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
Hi Olivier,
On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> External Email
>
> ----------------------------------------------------------------------
> Hi,
>
> On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > <nithind1988@gmail.com> wrote:
> > >
> > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > >
> > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > packet marking.
> > >
> > > When packet marking feature in Traffic manager is enabled,
> > > application has to the use the three new flags to indicate
> > > to PMD on whether packet marking needs to be enabled on the
> > > specific mbuf or not. By setting the three flags, it is
> > > assumed by PMD that application has already verified the
> > > applicability of marking on that specific packet and
> > > PMD need not perform further checks as per RFC.
> > >
> > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> >
> > None of the ethdev TM driver implementations has supported packet
> > marking support.
> > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
>
> As you know, the number of mbuf flags is limited (only 18 bits are
> remaining), so I think we should use them with care, i.e. for features
> that are generic enough.
I agree, but I believe this is one of the basic flags needed like other
Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
are needed to identify on which packets HW should/can apply packet marking.
>
> From what I understand, this feature is bound to octeontx2, so using a
> mbuf dynamic flag would make more sense here. There are some examples in
> dpdk repository, just grep for "dynflag".
This is not octeontx2 specific flag but any "packet marking feature" enabled
PMD would need these flags to identify on which packets marking needs to be
done. This is the first PMD that supports packet marking feature and
hence it was not exposed earlier.
For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
VLAN header from Byte 12 as there is no gaurantee that ethernet header
always starts at Byte 0 (Custom headers before ethernet hdr).
>
> Also, I think that the feature availability should be advertised through
> an ethdev offload, so an application can know at initialization time
> that these flags can be used.
Feature availablity is already part of TM spec in rte_tm.h
struct rte_tm_capabilities:mark_vlan_dei_supported
struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
struct rte_tm_capabilities:mark_ip_dscp_supported
Thanks
Nithin
>
> Regards,
> Olivier
>
>
>
> >
> >
> > > ---
> > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > index edd21c4..bc978fb 100644
> > > --- a/doc/guides/nics/features.rst
> > > +++ b/doc/guides/nics/features.rst
> > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > >
> > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > +
> > > +Traffic Manager Packet marking offload
> > > +--------------------------------------
> > > +
> > > +Supports enabling a packet marking offload specific mbuf.
> > > +
> > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > + ``rte_tm_mark_vlan_dei()``.
> > > +
> > > .. _nic_features_other:
> > >
> > > Other dev ops not represented by a Feature
> > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > index cd5794d..5c6896d 100644
> > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > default: return NULL;
> > > }
> > > }
> > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > };
> > > const char *name;
> > > unsigned int i;
> > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > index b9a59c8..d9f1290 100644
> > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > @@ -187,11 +187,40 @@ extern "C" {
> > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > >
> > > #define PKT_FIRST_FREE (1ULL << 23)
> > > -#define PKT_LAST_FREE (1ULL << 40)
> > > +#define PKT_LAST_FREE (1ULL << 37)
> > >
> > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > >
> > > /**
> > > + * Packet marking offload flags. These flags indicated what kind
> > > + * of packet marking needs to be applied on a given mbuf when
> > > + * appropriate Traffic Manager configuration is in place.
> > > + * When user set's these flags on a mbuf, below assumptions are made
> > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > + * at (mbuf.l2_len - 6) offset.
> > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > + * IPv4 pkt.
> > > + * b) Application should also set mbuf.l2_len that indicates
> > > + * start offset of L3 header.
> > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > + * can mark the packet for a configured color.
> > > + * c) Application should also set mbuf.l2_len that indicates
> > > + * start offset of L3 header.
> > > + */
> > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > +
> > > +/**
> > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > * 1) Enable the following in mbuf,
> > > @@ -384,7 +413,10 @@ extern "C" {
> > > PKT_TX_MACSEC | \
> > > PKT_TX_SEC_OFFLOAD | \
> > > PKT_TX_UDP_SEG | \
> > > - PKT_TX_OUTER_UDP_CKSUM)
> > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > + PKT_TX_MARK_VLAN_DEI | \
> > > + PKT_TX_MARK_IP_DSCP | \
> > > + PKT_TX_MARK_IP_ECN)
> > >
> > > /**
> > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > --
> > > 2.8.4
> > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-04 8:27 ` [dpdk-dev] [EXT] " Nithin Dabilpuram
@ 2020-05-04 9:16 ` Olivier Matz
2020-05-04 10:04 ` Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Olivier Matz @ 2020-05-04 9:16 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> Hi Olivier,
>
> On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > External Email
> >
> > ----------------------------------------------------------------------
> > Hi,
> >
> > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > <nithind1988@gmail.com> wrote:
> > > >
> > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > >
> > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > packet marking.
> > > >
> > > > When packet marking feature in Traffic manager is enabled,
> > > > application has to the use the three new flags to indicate
> > > > to PMD on whether packet marking needs to be enabled on the
> > > > specific mbuf or not. By setting the three flags, it is
> > > > assumed by PMD that application has already verified the
> > > > applicability of marking on that specific packet and
> > > > PMD need not perform further checks as per RFC.
> > > >
> > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > >
> > > None of the ethdev TM driver implementations has supported packet
> > > marking support.
> > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> >
> > As you know, the number of mbuf flags is limited (only 18 bits are
> > remaining), so I think we should use them with care, i.e. for features
> > that are generic enough.
>
> I agree, but I believe this is one of the basic flags needed like other
> Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> are needed to identify on which packets HW should/can apply packet marking.
PKT_TX_IP_CKSUM tells the hardware to offload the checksum
calculation. This is pretty straightforward and there is no other
dependency than the offload feature advertised by the PMD.
I'm sorry, I have not a lot of experience with rte_tm.h, so it's
difficult for me to have a global view of what is done for instance when
PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
Can you confirm that my understanding below is correct? (or correct me
where I'm wrong)
Before your patch:
- the application enables the port and traffic manager on it
- the application calls rte_tm_mark_vlan_dei() to select which traffic
class must be marked
- when a packet is transmitted, the traffic class is determined by the
hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
bit is set depending on traffic class
The problem is for packets that cannot be recognized by the hardware,
correct?
So your patch is a way to force the hardware to recognize mark set the
VLAN DEI on packets that are not recognized as VLAN packets?
How the is traffic class of the packet determined?
> > From what I understand, this feature is bound to octeontx2, so using a
> > mbuf dynamic flag would make more sense here. There are some examples in
> > dpdk repository, just grep for "dynflag".
>
> This is not octeontx2 specific flag but any "packet marking feature" enabled
> PMD would need these flags to identify on which packets marking needs to be
> done. This is the first PMD that supports packet marking feature and
> hence it was not exposed earlier.
>
> For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> VLAN header from Byte 12 as there is no gaurantee that ethernet header
> always starts at Byte 0 (Custom headers before ethernet hdr).
>
> >
> > Also, I think that the feature availability should be advertised through
> > an ethdev offload, so an application can know at initialization time
> > that these flags can be used.
>
> Feature availablity is already part of TM spec in rte_tm.h
> struct rte_tm_capabilities:mark_vlan_dei_supported
> struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> struct rte_tm_capabilities:mark_ip_dscp_supported
Does this mean that any driver advertising this existing feature flag
has to support the new mbuf flags too? Shouldn't we have a specific
feature for it?
Please also see few comments below.
> > > > ---
> > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > index edd21c4..bc978fb 100644
> > > > --- a/doc/guides/nics/features.rst
> > > > +++ b/doc/guides/nics/features.rst
> > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > >
> > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > +
> > > > +Traffic Manager Packet marking offload
> > > > +--------------------------------------
> > > > +
> > > > +Supports enabling a packet marking offload specific mbuf.
> > > > +
> > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > + ``rte_tm_mark_vlan_dei()``.
> > > > +
> > > > .. _nic_features_other:
> > > >
> > > > Other dev ops not represented by a Feature
> > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > index cd5794d..5c6896d 100644
> > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > default: return NULL;
> > > > }
> > > > }
> > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > };
> > > > const char *name;
> > > > unsigned int i;
> > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > index b9a59c8..d9f1290 100644
> > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > >
> > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > >
> > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > >
> > > > /**
> > > > + * Packet marking offload flags. These flags indicated what kind
> > > > + * of packet marking needs to be applied on a given mbuf when
> > > > + * appropriate Traffic Manager configuration is in place.
> > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > + * a) PMD assumes pkt to be a 802.1q packet.
What does that imply?
> > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > + * at (mbuf.l2_len - 6) offset.
Why mbuf.l2_len - 6 ?
> > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > + * IPv4 pkt.
> > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > + * start offset of L3 header.
> > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > + * can mark the packet for a configured color.
> > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > + * start offset of L3 header.
> > > > + */
> > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
We should have one comment per define.
> > > > +
> > > > +/**
> > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > * 1) Enable the following in mbuf,
> > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > PKT_TX_MACSEC | \
> > > > PKT_TX_SEC_OFFLOAD | \
> > > > PKT_TX_UDP_SEG | \
> > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > + PKT_TX_MARK_IP_DSCP | \
> > > > + PKT_TX_MARK_IP_ECN)
> > > >
> > > > /**
> > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > --
> > > > 2.8.4
> > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-04 9:16 ` Olivier Matz
@ 2020-05-04 10:04 ` Nithin Dabilpuram
2020-05-04 12:27 ` Olivier Matz
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-04 10:04 UTC (permalink / raw)
To: Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > Hi Olivier,
> >
> > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > External Email
> > >
> > > ----------------------------------------------------------------------
> > > Hi,
> > >
> > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > <nithind1988@gmail.com> wrote:
> > > > >
> > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > >
> > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > packet marking.
> > > > >
> > > > > When packet marking feature in Traffic manager is enabled,
> > > > > application has to the use the three new flags to indicate
> > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > specific mbuf or not. By setting the three flags, it is
> > > > > assumed by PMD that application has already verified the
> > > > > applicability of marking on that specific packet and
> > > > > PMD need not perform further checks as per RFC.
> > > > >
> > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > >
> > > > None of the ethdev TM driver implementations has supported packet
> > > > marking support.
> > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > >
> > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > remaining), so I think we should use them with care, i.e. for features
> > > that are generic enough.
> >
> > I agree, but I believe this is one of the basic flags needed like other
> > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > are needed to identify on which packets HW should/can apply packet marking.
>
> PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> calculation. This is pretty straightforward and there is no other
> dependency than the offload feature advertised by the PMD.
>
> I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> difficult for me to have a global view of what is done for instance when
> PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
>
> Can you confirm that my understanding below is correct? (or correct me
> where I'm wrong)
>
> Before your patch:
> - the application enables the port and traffic manager on it
> - the application calls rte_tm_mark_vlan_dei() to select which traffic
> class must be marked
> - when a packet is transmitted, the traffic class is determined by the
> hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> bit is set depending on traffic class
>
> The problem is for packets that cannot be recognized by the hardware,
> correct?
Yes. Octeontx2 HW always depends on application knowledge instead of walking
through all the layers of packet data in Tx to identify what packet it is
and where the l2, l3, l4 headers start for performance reasons.
I believe there are other hardware too that have the same expectation
and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
for identifying the packet and knowing what are its l2,l3,l4 offsets.
>
> So your patch is a way to force the hardware to recognize mark set the
> VLAN DEI on packets that are not recognized as VLAN packets?
>
> How the is traffic class of the packet determined?
Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
and packet color determines traffic class. The exact behavior of
packet color to traffic class mapping is mentioned in TM spec based on
few other RFC's.
[1] https://tools.ietf.org/html/rfc2697
[2] https://tools.ietf.org/html/rfc2698
>
>
> > > From what I understand, this feature is bound to octeontx2, so using a
> > > mbuf dynamic flag would make more sense here. There are some examples in
> > > dpdk repository, just grep for "dynflag".
> >
> > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > PMD would need these flags to identify on which packets marking needs to be
> > done. This is the first PMD that supports packet marking feature and
> > hence it was not exposed earlier.
> >
> > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > always starts at Byte 0 (Custom headers before ethernet hdr).
> >
> > >
> > > Also, I think that the feature availability should be advertised through
> > > an ethdev offload, so an application can know at initialization time
> > > that these flags can be used.
> >
> > Feature availablity is already part of TM spec in rte_tm.h
> > struct rte_tm_capabilities:mark_vlan_dei_supported
> > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > struct rte_tm_capabilities:mark_ip_dscp_supported
>
> Does this mean that any driver advertising this existing feature flag
> has to support the new mbuf flags too? Shouldn't we have a specific
> feature for it?
Yes, I thought PMD's need to support both.
I'm fine adding specific feature flag for the offload flags alone
if you insist or if there are other PMD's which don't need the offload flags
for packet marking. I was not able to find out about other PMD's as
none of the existing PMD's support packet marking.
>
> Please also see few comments below.
>
> > > > > ---
> > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > index edd21c4..bc978fb 100644
> > > > > --- a/doc/guides/nics/features.rst
> > > > > +++ b/doc/guides/nics/features.rst
> > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > >
> > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > +
> > > > > +Traffic Manager Packet marking offload
> > > > > +--------------------------------------
> > > > > +
> > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > +
> > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > +
> > > > > .. _nic_features_other:
> > > > >
> > > > > Other dev ops not represented by a Feature
> > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > index cd5794d..5c6896d 100644
> > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > default: return NULL;
> > > > > }
> > > > > }
> > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > };
> > > > > const char *name;
> > > > > unsigned int i;
> > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > index b9a59c8..d9f1290 100644
> > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > >
> > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > >
> > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > >
> > > > > /**
> > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > + * a) PMD assumes pkt to be a 802.1q packet.
>
> What does that imply?
I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
>
> > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > + * at (mbuf.l2_len - 6) offset.
>
> Why mbuf.l2_len - 6 ?
L2 header when VLAN header is preset will be
{custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
l2_len = X + 12 + 4 + 2
So, VLAN header starts at (l2_len - 6) bytes.
>
> > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > + * IPv4 pkt.
> > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > + * start offset of L3 header.
> > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > + * can mark the packet for a configured color.
> > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > + * start offset of L3 header.
> > > > > + */
> > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
>
> We should have one comment per define.
Ack, will fix in V2.
>
>
> > > > > +
> > > > > +/**
> > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > * 1) Enable the following in mbuf,
> > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > PKT_TX_MACSEC | \
> > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > PKT_TX_UDP_SEG | \
> > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > + PKT_TX_MARK_IP_ECN)
> > > > >
> > > > > /**
> > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > --
> > > > > 2.8.4
> > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-04 10:04 ` Nithin Dabilpuram
@ 2020-05-04 12:27 ` Olivier Matz
2020-05-05 6:19 ` Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Olivier Matz @ 2020-05-04 12:27 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > Hi Olivier,
> > >
> > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > External Email
> > > >
> > > > ----------------------------------------------------------------------
> > > > Hi,
> > > >
> > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > <nithind1988@gmail.com> wrote:
> > > > > >
> > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > >
> > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > packet marking.
> > > > > >
> > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > application has to the use the three new flags to indicate
> > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > assumed by PMD that application has already verified the
> > > > > > applicability of marking on that specific packet and
> > > > > > PMD need not perform further checks as per RFC.
> > > > > >
> > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > >
> > > > > None of the ethdev TM driver implementations has supported packet
> > > > > marking support.
> > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > >
> > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > remaining), so I think we should use them with care, i.e. for features
> > > > that are generic enough.
> > >
> > > I agree, but I believe this is one of the basic flags needed like other
> > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > are needed to identify on which packets HW should/can apply packet marking.
> >
> > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > calculation. This is pretty straightforward and there is no other
> > dependency than the offload feature advertised by the PMD.
> >
> > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > difficult for me to have a global view of what is done for instance when
> > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> >
> > Can you confirm that my understanding below is correct? (or correct me
> > where I'm wrong)
> >
> > Before your patch:
> > - the application enables the port and traffic manager on it
> > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > class must be marked
> > - when a packet is transmitted, the traffic class is determined by the
> > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > bit is set depending on traffic class
> >
> > The problem is for packets that cannot be recognized by the hardware,
> > correct?
>
> Yes. Octeontx2 HW always depends on application knowledge instead of walking
> through all the layers of packet data in Tx to identify what packet it is
> and where the l2, l3, l4 headers start for performance reasons.
>
> I believe there are other hardware too that have the same expectation
> and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
>
> Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> for identifying the packet and knowing what are its l2,l3,l4 offsets.
The objective is to give an indication to the hardware that the packet has:
- an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
- an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
- an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
PKT_TX_MARK_IP_ECN
Just to be sure I'm getting the point, would it also work if with flags
like this:
- an 802.1q header at offset X for PKT_TX_HAS_VLAN
- an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
- a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
PKT_TX_IPv4 or PKT_TX_IPv6)
The underlying question is: do we need the flags to only describe the
content of the packet or do the flag also indicate that an action has to
be done?
> > So your patch is a way to force the hardware to recognize mark set the
> > VLAN DEI on packets that are not recognized as VLAN packets?
> >
> > How the is traffic class of the packet determined?
>
> Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> and packet color determines traffic class. The exact behavior of
> packet color to traffic class mapping is mentioned in TM spec based on
> few other RFC's.
>
> [1] https://tools.ietf.org/html/rfc2697
> [2] https://tools.ietf.org/html/rfc2698
OK, so the traffic class does not depend on the packet type?
> > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > dpdk repository, just grep for "dynflag".
> > >
> > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > PMD would need these flags to identify on which packets marking needs to be
> > > done. This is the first PMD that supports packet marking feature and
> > > hence it was not exposed earlier.
> > >
> > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > >
> > > >
> > > > Also, I think that the feature availability should be advertised through
> > > > an ethdev offload, so an application can know at initialization time
> > > > that these flags can be used.
> > >
> > > Feature availablity is already part of TM spec in rte_tm.h
> > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > struct rte_tm_capabilities:mark_ip_dscp_supported
> >
> > Does this mean that any driver advertising this existing feature flag
> > has to support the new mbuf flags too? Shouldn't we have a specific
> > feature for it?
>
> Yes, I thought PMD's need to support both.
> I'm fine adding specific feature flag for the offload flags alone
> if you insist or if there are other PMD's which don't need the offload flags
> for packet marking. I was not able to find out about other PMD's as
> none of the existing PMD's support packet marking.
Do you suggest that the behavior of the traffic manager marking should
be:
a- the hardware tries to recognize tx packets, and mark them
accordingly. What packets are recognized depend on hardware.
b- if the mbuf has a specific flag, it helps the PMD and hardware to
recognize packets, so it can mark packets.
For an application, a- is difficult to apprehend as it will be dependent
on hardware.
Or do you suggest that packets should only be marked if there is a mbuf
flag? (only b-)
Do you confirm that there is no support at all for this feature today?
I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
Thanks,
Olivier
>
> >
> > Please also see few comments below.
> >
> > > > > > ---
> > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > index edd21c4..bc978fb 100644
> > > > > > --- a/doc/guides/nics/features.rst
> > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > >
> > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > +
> > > > > > +Traffic Manager Packet marking offload
> > > > > > +--------------------------------------
> > > > > > +
> > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > +
> > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > +
> > > > > > .. _nic_features_other:
> > > > > >
> > > > > > Other dev ops not represented by a Feature
> > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > index cd5794d..5c6896d 100644
> > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > default: return NULL;
> > > > > > }
> > > > > > }
> > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > };
> > > > > > const char *name;
> > > > > > unsigned int i;
> > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > index b9a59c8..d9f1290 100644
> > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > >
> > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > >
> > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > >
> > > > > > /**
> > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> >
> > What does that imply?
>
> I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
>
> >
> > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > + * at (mbuf.l2_len - 6) offset.
> >
> > Why mbuf.l2_len - 6 ?
> L2 header when VLAN header is preset will be
> {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> l2_len = X + 12 + 4 + 2
> So, VLAN header starts at (l2_len - 6) bytes.
>
> >
> > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > + * IPv4 pkt.
> > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > + * start offset of L3 header.
> > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > + * can mark the packet for a configured color.
> > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > + * start offset of L3 header.
> > > > > > + */
> > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> >
> > We should have one comment per define.
> Ack, will fix in V2.
>
> >
> >
> > > > > > +
> > > > > > +/**
> > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > * 1) Enable the following in mbuf,
> > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > PKT_TX_MACSEC | \
> > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > PKT_TX_UDP_SEG | \
> > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > >
> > > > > > /**
> > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > --
> > > > > > 2.8.4
> > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-04 12:27 ` Olivier Matz
@ 2020-05-05 6:19 ` Nithin Dabilpuram
2020-05-13 12:28 ` Nithin Dabilpuram
2020-05-14 20:29 ` Olivier Matz
0 siblings, 2 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-05 6:19 UTC (permalink / raw)
To: Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > Hi Olivier,
> > > >
> > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > External Email
> > > > >
> > > > > ----------------------------------------------------------------------
> > > > > Hi,
> > > > >
> > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > <nithind1988@gmail.com> wrote:
> > > > > > >
> > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > >
> > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > packet marking.
> > > > > > >
> > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > application has to the use the three new flags to indicate
> > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > assumed by PMD that application has already verified the
> > > > > > > applicability of marking on that specific packet and
> > > > > > > PMD need not perform further checks as per RFC.
> > > > > > >
> > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > >
> > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > marking support.
> > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > >
> > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > that are generic enough.
> > > >
> > > > I agree, but I believe this is one of the basic flags needed like other
> > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > are needed to identify on which packets HW should/can apply packet marking.
> > >
> > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > calculation. This is pretty straightforward and there is no other
> > > dependency than the offload feature advertised by the PMD.
> > >
> > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > difficult for me to have a global view of what is done for instance when
> > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > >
> > > Can you confirm that my understanding below is correct? (or correct me
> > > where I'm wrong)
> > >
> > > Before your patch:
> > > - the application enables the port and traffic manager on it
> > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > class must be marked
> > > - when a packet is transmitted, the traffic class is determined by the
> > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > bit is set depending on traffic class
> > >
> > > The problem is for packets that cannot be recognized by the hardware,
> > > correct?
> >
> > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > through all the layers of packet data in Tx to identify what packet it is
> > and where the l2, l3, l4 headers start for performance reasons.
> >
> > I believe there are other hardware too that have the same expectation
> > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> >
> > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > for identifying the packet and knowing what are its l2,l3,l4 offsets.
>
> The objective is to give an indication to the hardware that the packet has:
> - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> PKT_TX_MARK_IP_ECN
>
> Just to be sure I'm getting the point, would it also work if with flags
> like this:
>
> - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> PKT_TX_IPv4 or PKT_TX_IPv6)
>
> The underlying question is: do we need the flags to only describe the
> content of the packet or do the flag also indicate that an action has to
> be done?
If we don't have a specific action based flag, then in future it might collide
with other functionality and we will not be able to choose that specific
offload. All the existing features are having specific flags, like TSO,
CSUM.
RFC wise, even when marking in enabled and packet is coloured, not all packets
can be marked.
For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
only with below 12 code points out of 64 code points (6 bits of DSCP).
Class 1 Class 2 Class 3 Class 4
+----------+----------+----------+----------+
Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
High Drop Prec | 001110 | 010110 | 011110 | 100110 |
+----------+----------+----------+----------+
All other combinations of DSCP value can be used for some other purposes
and hence packets with those values shouldn't be marked.
Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
Having PMD or HW to check if the packet falls in the said class and then do
marking will impact performance. Since application actually fills those values
in packet, it will be more easy for them to say.
>
> > > So your patch is a way to force the hardware to recognize mark set the
> > > VLAN DEI on packets that are not recognized as VLAN packets?
> > >
> > > How the is traffic class of the packet determined?
> >
> > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > and packet color determines traffic class. The exact behavior of
> > packet color to traffic class mapping is mentioned in TM spec based on
> > few other RFC's.
> >
> > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
>
> OK, so the traffic class does not depend on the packet type?
Yes it doesn't. But where to update the traffic class is specific to packet
type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
Also ECN marking is only valid for TCP/SCTP packets.
>
>
> > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > dpdk repository, just grep for "dynflag".
> > > >
> > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > PMD would need these flags to identify on which packets marking needs to be
> > > > done. This is the first PMD that supports packet marking feature and
> > > > hence it was not exposed earlier.
> > > >
> > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > >
> > > > >
> > > > > Also, I think that the feature availability should be advertised through
> > > > > an ethdev offload, so an application can know at initialization time
> > > > > that these flags can be used.
> > > >
> > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > >
> > > Does this mean that any driver advertising this existing feature flag
> > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > feature for it?
> >
> > Yes, I thought PMD's need to support both.
> > I'm fine adding specific feature flag for the offload flags alone
> > if you insist or if there are other PMD's which don't need the offload flags
> > for packet marking. I was not able to find out about other PMD's as
> > none of the existing PMD's support packet marking.
>
> Do you suggest that the behavior of the traffic manager marking should
> be:
>
> a- the hardware tries to recognize tx packets, and mark them
> accordingly. What packets are recognized depend on hardware.
> b- if the mbuf has a specific flag, it helps the PMD and hardware to
> recognize packets, so it can mark packets.
>
> For an application, a- is difficult to apprehend as it will be dependent
> on hardware.
>
> Or do you suggest that packets should only be marked if there is a mbuf
> flag? (only b-)
Yes, I believe b- is the right thing.
>
> Do you confirm that there is no support at all for this feature today?
> I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
supposed to be experimental but was mistakenly marked stable.
You can see related discussion in below threads about marking rte_tm.h
experimental again in v20.11.
https://mails.dpdk.org/archives/dev/2020-April/164970.html
https://mails.dpdk.org/archives/dev/2020-May/166221.html
Thanks
Nithin
>
> Thanks,
> Olivier
>
>
> >
> > >
> > > Please also see few comments below.
> > >
> > > > > > > ---
> > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > >
> > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > index edd21c4..bc978fb 100644
> > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > >
> > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > +
> > > > > > > +Traffic Manager Packet marking offload
> > > > > > > +--------------------------------------
> > > > > > > +
> > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > +
> > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > +
> > > > > > > .. _nic_features_other:
> > > > > > >
> > > > > > > Other dev ops not represented by a Feature
> > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > index cd5794d..5c6896d 100644
> > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > default: return NULL;
> > > > > > > }
> > > > > > > }
> > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > };
> > > > > > > const char *name;
> > > > > > > unsigned int i;
> > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > >
> > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > >
> > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > >
> > > > > > > /**
> > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > >
> > > What does that imply?
> >
> > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> >
> > >
> > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > + * at (mbuf.l2_len - 6) offset.
> > >
> > > Why mbuf.l2_len - 6 ?
> > L2 header when VLAN header is preset will be
> > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > l2_len = X + 12 + 4 + 2
> > So, VLAN header starts at (l2_len - 6) bytes.
> >
> > >
> > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > + * IPv4 pkt.
> > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > + * start offset of L3 header.
> > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > + * can mark the packet for a configured color.
> > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > + * start offset of L3 header.
> > > > > > > + */
> > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > >
> > > We should have one comment per define.
> > Ack, will fix in V2.
> >
> > >
> > >
> > > > > > > +
> > > > > > > +/**
> > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > * 1) Enable the following in mbuf,
> > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > PKT_TX_MACSEC | \
> > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > >
> > > > > > > /**
> > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > --
> > > > > > > 2.8.4
> > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-05 6:19 ` Nithin Dabilpuram
@ 2020-05-13 12:28 ` Nithin Dabilpuram
2020-05-14 20:29 ` Olivier Matz
1 sibling, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-13 12:28 UTC (permalink / raw)
To: Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
Hi Olivier,
Any thoughts on this ?
On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > Hi Olivier,
> > > > >
> > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > External Email
> > > > > >
> > > > > > ----------------------------------------------------------------------
> > > > > > Hi,
> > > > > >
> > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > >
> > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > >
> > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > packet marking.
> > > > > > > >
> > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > applicability of marking on that specific packet and
> > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > >
> > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > >
> > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > marking support.
> > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > >
> > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > that are generic enough.
> > > > >
> > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > >
> > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > calculation. This is pretty straightforward and there is no other
> > > > dependency than the offload feature advertised by the PMD.
> > > >
> > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > difficult for me to have a global view of what is done for instance when
> > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > >
> > > > Can you confirm that my understanding below is correct? (or correct me
> > > > where I'm wrong)
> > > >
> > > > Before your patch:
> > > > - the application enables the port and traffic manager on it
> > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > class must be marked
> > > > - when a packet is transmitted, the traffic class is determined by the
> > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > bit is set depending on traffic class
> > > >
> > > > The problem is for packets that cannot be recognized by the hardware,
> > > > correct?
> > >
> > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > through all the layers of packet data in Tx to identify what packet it is
> > > and where the l2, l3, l4 headers start for performance reasons.
> > >
> > > I believe there are other hardware too that have the same expectation
> > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > >
> > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> >
> > The objective is to give an indication to the hardware that the packet has:
> > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > PKT_TX_MARK_IP_ECN
> >
> > Just to be sure I'm getting the point, would it also work if with flags
> > like this:
> >
> > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > PKT_TX_IPv4 or PKT_TX_IPv6)
> >
> > The underlying question is: do we need the flags to only describe the
> > content of the packet or do the flag also indicate that an action has to
> > be done?
>
> If we don't have a specific action based flag, then in future it might collide
> with other functionality and we will not be able to choose that specific
> offload. All the existing features are having specific flags, like TSO,
> CSUM.
>
> RFC wise, even when marking in enabled and packet is coloured, not all packets
> can be marked.
> For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> only with below 12 code points out of 64 code points (6 bits of DSCP).
>
> Class 1 Class 2 Class 3 Class 4
> +----------+----------+----------+----------+
> Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> +----------+----------+----------+----------+
>
> All other combinations of DSCP value can be used for some other purposes
> and hence packets with those values shouldn't be marked.
> Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
>
> Having PMD or HW to check if the packet falls in the said class and then do
> marking will impact performance. Since application actually fills those values
> in packet, it will be more easy for them to say.
>
> >
> > > > So your patch is a way to force the hardware to recognize mark set the
> > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > >
> > > > How the is traffic class of the packet determined?
> > >
> > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > and packet color determines traffic class. The exact behavior of
> > > packet color to traffic class mapping is mentioned in TM spec based on
> > > few other RFC's.
> > >
> > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> >
> > OK, so the traffic class does not depend on the packet type?
> Yes it doesn't. But where to update the traffic class is specific to packet
> type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> Also ECN marking is only valid for TCP/SCTP packets.
>
> >
> >
> > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > dpdk repository, just grep for "dynflag".
> > > > >
> > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > done. This is the first PMD that supports packet marking feature and
> > > > > hence it was not exposed earlier.
> > > > >
> > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > >
> > > > > >
> > > > > > Also, I think that the feature availability should be advertised through
> > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > that these flags can be used.
> > > > >
> > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > >
> > > > Does this mean that any driver advertising this existing feature flag
> > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > feature for it?
> > >
> > > Yes, I thought PMD's need to support both.
> > > I'm fine adding specific feature flag for the offload flags alone
> > > if you insist or if there are other PMD's which don't need the offload flags
> > > for packet marking. I was not able to find out about other PMD's as
> > > none of the existing PMD's support packet marking.
> >
> > Do you suggest that the behavior of the traffic manager marking should
> > be:
> >
> > a- the hardware tries to recognize tx packets, and mark them
> > accordingly. What packets are recognized depend on hardware.
> > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > recognize packets, so it can mark packets.
> >
> > For an application, a- is difficult to apprehend as it will be dependent
> > on hardware.
> >
> > Or do you suggest that packets should only be marked if there is a mbuf
> > flag? (only b-)
> Yes, I believe b- is the right thing.
>
> >
> > Do you confirm that there is no support at all for this feature today?
> > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
>
> Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> supposed to be experimental but was mistakenly marked stable.
> You can see related discussion in below threads about marking rte_tm.h
> experimental again in v20.11.
> https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-2DApril_164970.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=JWDwlSkCAEkWR-4kKuuIGFmhMtr8W10Ns9kEPidDFbQ&s=XESl2bNVKTkGiVmm3qww3zDb0vYu9_XcaqT2CkCViTs&e=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-2DMay_166221.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=JWDwlSkCAEkWR-4kKuuIGFmhMtr8W10Ns9kEPidDFbQ&s=ZGxLxUL_76HZo9YmWhyvDZeYg28uEh3q6od48a3KlbI&e=
>
> Thanks
> Nithin
>
> >
> > Thanks,
> > Olivier
> >
> >
> > >
> > > >
> > > > Please also see few comments below.
> > > >
> > > > > > > > ---
> > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > >
> > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > +
> > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > +--------------------------------------
> > > > > > > > +
> > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > +
> > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > +
> > > > > > > > .. _nic_features_other:
> > > > > > > >
> > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > default: return NULL;
> > > > > > > > }
> > > > > > > > }
> > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > };
> > > > > > > > const char *name;
> > > > > > > > unsigned int i;
> > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > >
> > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > >
> > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > >
> > > > > > > > /**
> > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > >
> > > > What does that imply?
> > >
> > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > >
> > > >
> > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > >
> > > > Why mbuf.l2_len - 6 ?
> > > L2 header when VLAN header is preset will be
> > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > l2_len = X + 12 + 4 + 2
> > > So, VLAN header starts at (l2_len - 6) bytes.
> > >
> > > >
> > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > + * IPv4 pkt.
> > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > + * start offset of L3 header.
> > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > + * start offset of L3 header.
> > > > > > > > + */
> > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > >
> > > > We should have one comment per define.
> > > Ack, will fix in V2.
> > >
> > > >
> > > >
> > > > > > > > +
> > > > > > > > +/**
> > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > >
> > > > > > > > /**
> > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > --
> > > > > > > > 2.8.4
> > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-05 6:19 ` Nithin Dabilpuram
2020-05-13 12:28 ` Nithin Dabilpuram
@ 2020-05-14 20:29 ` Olivier Matz
2020-05-15 10:08 ` Nithin Dabilpuram
1 sibling, 1 reply; 36+ messages in thread
From: Olivier Matz @ 2020-05-14 20:29 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
Hi Nithin,
On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > Hi Olivier,
> > > > >
> > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > External Email
> > > > > >
> > > > > > ----------------------------------------------------------------------
> > > > > > Hi,
> > > > > >
> > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > >
> > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > >
> > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > packet marking.
> > > > > > > >
> > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > applicability of marking on that specific packet and
> > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > >
> > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > >
> > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > marking support.
> > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > >
> > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > that are generic enough.
> > > > >
> > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > >
> > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > calculation. This is pretty straightforward and there is no other
> > > > dependency than the offload feature advertised by the PMD.
> > > >
> > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > difficult for me to have a global view of what is done for instance when
> > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > >
> > > > Can you confirm that my understanding below is correct? (or correct me
> > > > where I'm wrong)
> > > >
> > > > Before your patch:
> > > > - the application enables the port and traffic manager on it
> > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > class must be marked
> > > > - when a packet is transmitted, the traffic class is determined by the
> > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > bit is set depending on traffic class
> > > >
> > > > The problem is for packets that cannot be recognized by the hardware,
> > > > correct?
> > >
> > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > through all the layers of packet data in Tx to identify what packet it is
> > > and where the l2, l3, l4 headers start for performance reasons.
> > >
> > > I believe there are other hardware too that have the same expectation
> > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > >
> > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> >
> > The objective is to give an indication to the hardware that the packet has:
> > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > PKT_TX_MARK_IP_ECN
> >
> > Just to be sure I'm getting the point, would it also work if with flags
> > like this:
> >
> > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > PKT_TX_IPv4 or PKT_TX_IPv6)
> >
> > The underlying question is: do we need the flags to only describe the
> > content of the packet or do the flag also indicate that an action has to
> > be done?
>
> If we don't have a specific action based flag, then in future it might collide
> with other functionality and we will not be able to choose that specific
> offload. All the existing features are having specific flags, like TSO,
> CSUM.
>
> RFC wise, even when marking in enabled and packet is coloured, not all packets
> can be marked.
> For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> only with below 12 code points out of 64 code points (6 bits of DSCP).
>
> Class 1 Class 2 Class 3 Class 4
> +----------+----------+----------+----------+
> Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> +----------+----------+----------+----------+
>
> All other combinations of DSCP value can be used for some other purposes
> and hence packets with those values shouldn't be marked.
> Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
>
> Having PMD or HW to check if the packet falls in the said class and then do
> marking will impact performance. Since application actually fills those values
> in packet, it will be more easy for them to say.
>
> >
> > > > So your patch is a way to force the hardware to recognize mark set the
> > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > >
> > > > How the is traffic class of the packet determined?
> > >
> > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > and packet color determines traffic class. The exact behavior of
> > > packet color to traffic class mapping is mentioned in TM spec based on
> > > few other RFC's.
> > >
> > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> >
> > OK, so the traffic class does not depend on the packet type?
> Yes it doesn't. But where to update the traffic class is specific to packet
> type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> Also ECN marking is only valid for TCP/SCTP packets.
>
> >
> >
> > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > dpdk repository, just grep for "dynflag".
> > > > >
> > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > done. This is the first PMD that supports packet marking feature and
> > > > > hence it was not exposed earlier.
> > > > >
> > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > >
> > > > > >
> > > > > > Also, I think that the feature availability should be advertised through
> > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > that these flags can be used.
> > > > >
> > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > >
> > > > Does this mean that any driver advertising this existing feature flag
> > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > feature for it?
> > >
> > > Yes, I thought PMD's need to support both.
> > > I'm fine adding specific feature flag for the offload flags alone
> > > if you insist or if there are other PMD's which don't need the offload flags
> > > for packet marking. I was not able to find out about other PMD's as
> > > none of the existing PMD's support packet marking.
> >
> > Do you suggest that the behavior of the traffic manager marking should
> > be:
> >
> > a- the hardware tries to recognize tx packets, and mark them
> > accordingly. What packets are recognized depend on hardware.
> > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > recognize packets, so it can mark packets.
> >
> > For an application, a- is difficult to apprehend as it will be dependent
> > on hardware.
> >
> > Or do you suggest that packets should only be marked if there is a mbuf
> > flag? (only b-)
> Yes, I believe b- is the right thing.
>
> >
> > Do you confirm that there is no support at all for this feature today?
> > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
>
> Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> supposed to be experimental but was mistakenly marked stable.
> You can see related discussion in below threads about marking rte_tm.h
> experimental again in v20.11.
> https://mails.dpdk.org/archives/dev/2020-April/164970.html
> https://mails.dpdk.org/archives/dev/2020-May/166221.html
Thank you for the explanations. I also think b- is a better choice.
I don't see any better approach than having a mbuf flag. However, I'm
still not fully convinced that a dynamic flag won't do the job. Taking
3 additional flags (among 18 remaing) for this feature also means that
we have 3 flags less for dynamic flags for all applications, even for
applications that will not use this feature.
Would it be a problem to use a dynamic flag in this case?
Thanks,
Olivier
>
> Thanks
> Nithin
>
> >
> > Thanks,
> > Olivier
> >
> >
> > >
> > > >
> > > > Please also see few comments below.
> > > >
> > > > > > > > ---
> > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > >
> > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > +
> > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > +--------------------------------------
> > > > > > > > +
> > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > +
> > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > +
> > > > > > > > .. _nic_features_other:
> > > > > > > >
> > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > default: return NULL;
> > > > > > > > }
> > > > > > > > }
> > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > };
> > > > > > > > const char *name;
> > > > > > > > unsigned int i;
> > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > >
> > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > >
> > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > >
> > > > > > > > /**
> > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > >
> > > > What does that imply?
> > >
> > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > >
> > > >
> > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > >
> > > > Why mbuf.l2_len - 6 ?
> > > L2 header when VLAN header is preset will be
> > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > l2_len = X + 12 + 4 + 2
> > > So, VLAN header starts at (l2_len - 6) bytes.
> > >
> > > >
> > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > + * IPv4 pkt.
> > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > + * start offset of L3 header.
> > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > + * start offset of L3 header.
> > > > > > > > + */
> > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > >
> > > > We should have one comment per define.
> > > Ack, will fix in V2.
> > >
> > > >
> > > >
> > > > > > > > +
> > > > > > > > +/**
> > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > >
> > > > > > > > /**
> > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > --
> > > > > > > > 2.8.4
> > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-14 20:29 ` Olivier Matz
@ 2020-05-15 10:08 ` Nithin Dabilpuram
2020-05-15 10:30 ` Ananyev, Konstantin
2020-05-15 13:12 ` Thomas Monjalon
0 siblings, 2 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-15 10:08 UTC (permalink / raw)
To: Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> Hi Nithin,
>
> On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> > On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > > Hi Olivier,
> > > > > >
> > > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > > External Email
> > > > > > >
> > > > > > > ----------------------------------------------------------------------
> > > > > > > Hi,
> > > > > > >
> > > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > >
> > > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > > packet marking.
> > > > > > > > >
> > > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > > applicability of marking on that specific packet and
> > > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > >
> > > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > > marking support.
> > > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > > >
> > > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > > that are generic enough.
> > > > > >
> > > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > > >
> > > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > > calculation. This is pretty straightforward and there is no other
> > > > > dependency than the offload feature advertised by the PMD.
> > > > >
> > > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > > difficult for me to have a global view of what is done for instance when
> > > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > > >
> > > > > Can you confirm that my understanding below is correct? (or correct me
> > > > > where I'm wrong)
> > > > >
> > > > > Before your patch:
> > > > > - the application enables the port and traffic manager on it
> > > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > > class must be marked
> > > > > - when a packet is transmitted, the traffic class is determined by the
> > > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > > bit is set depending on traffic class
> > > > >
> > > > > The problem is for packets that cannot be recognized by the hardware,
> > > > > correct?
> > > >
> > > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > > through all the layers of packet data in Tx to identify what packet it is
> > > > and where the l2, l3, l4 headers start for performance reasons.
> > > >
> > > > I believe there are other hardware too that have the same expectation
> > > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > > >
> > > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> > >
> > > The objective is to give an indication to the hardware that the packet has:
> > > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > > PKT_TX_MARK_IP_ECN
> > >
> > > Just to be sure I'm getting the point, would it also work if with flags
> > > like this:
> > >
> > > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > > PKT_TX_IPv4 or PKT_TX_IPv6)
> > >
> > > The underlying question is: do we need the flags to only describe the
> > > content of the packet or do the flag also indicate that an action has to
> > > be done?
> >
> > If we don't have a specific action based flag, then in future it might collide
> > with other functionality and we will not be able to choose that specific
> > offload. All the existing features are having specific flags, like TSO,
> > CSUM.
> >
> > RFC wise, even when marking in enabled and packet is coloured, not all packets
> > can be marked.
> > For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> > only with below 12 code points out of 64 code points (6 bits of DSCP).
> >
> > Class 1 Class 2 Class 3 Class 4
> > +----------+----------+----------+----------+
> > Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> > Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> > High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> > +----------+----------+----------+----------+
> >
> > All other combinations of DSCP value can be used for some other purposes
> > and hence packets with those values shouldn't be marked.
> > Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
> >
> > Having PMD or HW to check if the packet falls in the said class and then do
> > marking will impact performance. Since application actually fills those values
> > in packet, it will be more easy for them to say.
> >
> > >
> > > > > So your patch is a way to force the hardware to recognize mark set the
> > > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > > >
> > > > > How the is traffic class of the packet determined?
> > > >
> > > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > > and packet color determines traffic class. The exact behavior of
> > > > packet color to traffic class mapping is mentioned in TM spec based on
> > > > few other RFC's.
> > > >
> > > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> > >
> > > OK, so the traffic class does not depend on the packet type?
> > Yes it doesn't. But where to update the traffic class is specific to packet
> > type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> > Also ECN marking is only valid for TCP/SCTP packets.
> >
> > >
> > >
> > > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > > dpdk repository, just grep for "dynflag".
> > > > > >
> > > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > > done. This is the first PMD that supports packet marking feature and
> > > > > > hence it was not exposed earlier.
> > > > > >
> > > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > > >
> > > > > > >
> > > > > > > Also, I think that the feature availability should be advertised through
> > > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > > that these flags can be used.
> > > > > >
> > > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > > >
> > > > > Does this mean that any driver advertising this existing feature flag
> > > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > > feature for it?
> > > >
> > > > Yes, I thought PMD's need to support both.
> > > > I'm fine adding specific feature flag for the offload flags alone
> > > > if you insist or if there are other PMD's which don't need the offload flags
> > > > for packet marking. I was not able to find out about other PMD's as
> > > > none of the existing PMD's support packet marking.
> > >
> > > Do you suggest that the behavior of the traffic manager marking should
> > > be:
> > >
> > > a- the hardware tries to recognize tx packets, and mark them
> > > accordingly. What packets are recognized depend on hardware.
> > > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > > recognize packets, so it can mark packets.
> > >
> > > For an application, a- is difficult to apprehend as it will be dependent
> > > on hardware.
> > >
> > > Or do you suggest that packets should only be marked if there is a mbuf
> > > flag? (only b-)
> > Yes, I believe b- is the right thing.
> >
> > >
> > > Do you confirm that there is no support at all for this feature today?
> > > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
> >
> > Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> > supposed to be experimental but was mistakenly marked stable.
> > You can see related discussion in below threads about marking rte_tm.h
> > experimental again in v20.11.
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-2DApril_164970.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHNjN9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=-o2E-F9aHy3mrQw6xgO__RPXY9t8s3yjJn81X6Ius3k&e=
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-2DMay_166221.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHNjN9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=gTKSzMmlhE75x4TP8IJB7NP5MVO-zxjmNRQ9bZ6MxwI&e=
>
> Thank you for the explanations. I also think b- is a better choice.
>
> I don't see any better approach than having a mbuf flag. However, I'm
> still not fully convinced that a dynamic flag won't do the job. Taking
> 3 additional flags (among 18 remaing) for this feature also means that
> we have 3 flags less for dynamic flags for all applications, even for
> applications that will not use this feature.
>
> Would it be a problem to use a dynamic flag in this case?
Since packet marking feature itself is already part of spec,
if we move the flags to PMD specific dynamic flag, then it creates a confusion.
It is not the case of a custom feature supported by a specific PMD.
I believe when other PMD's implement packet marking, the same flags will
suffice.
>
> Thanks,
> Olivier
>
>
> >
> > Thanks
> > Nithin
> >
> > >
> > > Thanks,
> > > Olivier
> > >
> > >
> > > >
> > > > >
> > > > > Please also see few comments below.
> > > > >
> > > > > > > > > ---
> > > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > > >
> > > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > > +
> > > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > > +--------------------------------------
> > > > > > > > > +
> > > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > > +
> > > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > > +
> > > > > > > > > .. _nic_features_other:
> > > > > > > > >
> > > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > > default: return NULL;
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > > };
> > > > > > > > > const char *name;
> > > > > > > > > unsigned int i;
> > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > > >
> > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > > >
> > > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > > >
> > > > > What does that imply?
> > > >
> > > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > > >
> > > > >
> > > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > > >
> > > > > Why mbuf.l2_len - 6 ?
> > > > L2 header when VLAN header is preset will be
> > > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > > l2_len = X + 12 + 4 + 2
> > > > So, VLAN header starts at (l2_len - 6) bytes.
> > > >
> > > > >
> > > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > > + * IPv4 pkt.
> > > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > > + * start offset of L3 header.
> > > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > > + * start offset of L3 header.
> > > > > > > > > + */
> > > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > > >
> > > > > We should have one comment per define.
> > > > Ack, will fix in V2.
> > > >
> > > > >
> > > > >
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > > --
> > > > > > > > > 2.8.4
> > > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 10:08 ` Nithin Dabilpuram
@ 2020-05-15 10:30 ` Ananyev, Konstantin
2020-05-15 13:57 ` Nithin Dabilpuram
2020-05-15 13:12 ` Thomas Monjalon
1 sibling, 1 reply; 36+ messages in thread
From: Ananyev, Konstantin @ 2020-05-15 10:30 UTC (permalink / raw)
To: Nithin Dabilpuram, Olivier Matz
Cc: Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Dumitrescu, Cristian, Burakov,
Anatoly, Mcnamara, John, Kovacevic, Marko, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
> On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > Hi Nithin,
> >
> > On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> > > On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > > > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > > > Hi Olivier,
> > > > > > >
> > > > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > > > External Email
> > > > > > > >
> > > > > > > > ----------------------------------------------------------------------
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > >
> > > > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > > > packet marking.
> > > > > > > > > >
> > > > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > > > applicability of marking on that specific packet and
> > > > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > >
> > > > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > > > marking support.
> > > > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > > > >
> > > > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > > > that are generic enough.
> > > > > > >
> > > > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > > > >
> > > > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > > > calculation. This is pretty straightforward and there is no other
> > > > > > dependency than the offload feature advertised by the PMD.
> > > > > >
> > > > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > > > difficult for me to have a global view of what is done for instance when
> > > > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > > > >
> > > > > > Can you confirm that my understanding below is correct? (or correct me
> > > > > > where I'm wrong)
> > > > > >
> > > > > > Before your patch:
> > > > > > - the application enables the port and traffic manager on it
> > > > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > > > class must be marked
> > > > > > - when a packet is transmitted, the traffic class is determined by the
> > > > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > > > bit is set depending on traffic class
> > > > > >
> > > > > > The problem is for packets that cannot be recognized by the hardware,
> > > > > > correct?
> > > > >
> > > > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > > > through all the layers of packet data in Tx to identify what packet it is
> > > > > and where the l2, l3, l4 headers start for performance reasons.
> > > > >
> > > > > I believe there are other hardware too that have the same expectation
> > > > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > > > >
> > > > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> > > >
> > > > The objective is to give an indication to the hardware that the packet has:
> > > > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > > > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > > > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > > > PKT_TX_MARK_IP_ECN
> > > >
> > > > Just to be sure I'm getting the point, would it also work if with flags
> > > > like this:
> > > >
> > > > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > > > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > > > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > > > PKT_TX_IPv4 or PKT_TX_IPv6)
> > > >
> > > > The underlying question is: do we need the flags to only describe the
> > > > content of the packet or do the flag also indicate that an action has to
> > > > be done?
> > >
> > > If we don't have a specific action based flag, then in future it might collide
> > > with other functionality and we will not be able to choose that specific
> > > offload. All the existing features are having specific flags, like TSO,
> > > CSUM.
> > >
> > > RFC wise, even when marking in enabled and packet is coloured, not all packets
> > > can be marked.
> > > For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> > > only with below 12 code points out of 64 code points (6 bits of DSCP).
> > >
> > > Class 1 Class 2 Class 3 Class 4
> > > +----------+----------+----------+----------+
> > > Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> > > Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> > > High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> > > +----------+----------+----------+----------+
> > >
> > > All other combinations of DSCP value can be used for some other purposes
> > > and hence packets with those values shouldn't be marked.
> > > Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
> > >
> > > Having PMD or HW to check if the packet falls in the said class and then do
> > > marking will impact performance. Since application actually fills those values
> > > in packet, it will be more easy for them to say.
> > >
> > > >
> > > > > > So your patch is a way to force the hardware to recognize mark set the
> > > > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > > > >
> > > > > > How the is traffic class of the packet determined?
> > > > >
> > > > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > > > and packet color determines traffic class. The exact behavior of
> > > > > packet color to traffic class mapping is mentioned in TM spec based on
> > > > > few other RFC's.
> > > > >
> > > > > [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > > > [2] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> > > >
> > > > OK, so the traffic class does not depend on the packet type?
> > > Yes it doesn't. But where to update the traffic class is specific to packet
> > > type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> > > Also ECN marking is only valid for TCP/SCTP packets.
> > >
> > > >
> > > >
> > > > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > > > dpdk repository, just grep for "dynflag".
> > > > > > >
> > > > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > > > done. This is the first PMD that supports packet marking feature and
> > > > > > > hence it was not exposed earlier.
> > > > > > >
> > > > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > > > >
> > > > > > > >
> > > > > > > > Also, I think that the feature availability should be advertised through
> > > > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > > > that these flags can be used.
> > > > > > >
> > > > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > > > >
> > > > > > Does this mean that any driver advertising this existing feature flag
> > > > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > > > feature for it?
> > > > >
> > > > > Yes, I thought PMD's need to support both.
> > > > > I'm fine adding specific feature flag for the offload flags alone
> > > > > if you insist or if there are other PMD's which don't need the offload flags
> > > > > for packet marking. I was not able to find out about other PMD's as
> > > > > none of the existing PMD's support packet marking.
> > > >
> > > > Do you suggest that the behavior of the traffic manager marking should
> > > > be:
> > > >
> > > > a- the hardware tries to recognize tx packets, and mark them
> > > > accordingly. What packets are recognized depend on hardware.
> > > > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > > > recognize packets, so it can mark packets.
> > > >
> > > > For an application, a- is difficult to apprehend as it will be dependent
> > > > on hardware.
> > > >
> > > > Or do you suggest that packets should only be marked if there is a mbuf
> > > > flag? (only b-)
> > > Yes, I believe b- is the right thing.
> > >
> > > >
> > > > Do you confirm that there is no support at all for this feature today?
> > > > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
> > >
> > > Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> > > supposed to be experimental but was mistakenly marked stable.
> > > You can see related discussion in below threads about marking rte_tm.h
> > > experimental again in v20.11.
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> 2DApril_164970.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHN
> jN9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=-o2E-F9aHy3mrQw6xgO__RPXY9t8s3yjJn81X6Ius3k&e=
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> 2DMay_166221.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHNj
> N9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=gTKSzMmlhE75x4TP8IJB7NP5MVO-zxjmNRQ9bZ6MxwI&e=
> >
> > Thank you for the explanations. I also think b- is a better choice.
> >
> > I don't see any better approach than having a mbuf flag. However, I'm
> > still not fully convinced that a dynamic flag won't do the job. Taking
> > 3 additional flags (among 18 remaing) for this feature also means that
> > we have 3 flags less for dynamic flags for all applications, even for
> > applications that will not use this feature.
I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
Can it probably be squeezed somehow?
Let say we reserve one flag that this information is present or not, and
re-use one of rx-only fields for store additional information (packet_type, or so).
Or might be some other approach.
> >
> > Would it be a problem to use a dynamic flag in this case?
> Since packet marking feature itself is already part of spec,
> if we move the flags to PMD specific dynamic flag, then it creates a confusion.
>
> It is not the case of a custom feature supported by a specific PMD.
> I believe when other PMD's implement packet marking, the same flags will
> suffice.
> >
> > Thanks,
> > Olivier
> >
> >
> > >
> > > Thanks
> > > Nithin
> > >
> > > >
> > > > Thanks,
> > > > Olivier
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > Please also see few comments below.
> > > > > >
> > > > > > > > > > ---
> > > > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > > > >
> > > > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > > > +
> > > > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > > > +--------------------------------------
> > > > > > > > > > +
> > > > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > > > +
> > > > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > > > +
> > > > > > > > > > .. _nic_features_other:
> > > > > > > > > >
> > > > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > > > default: return NULL;
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > > > };
> > > > > > > > > > const char *name;
> > > > > > > > > > unsigned int i;
> > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > > > >
> > > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > > > >
> > > > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > > > >
> > > > > > What does that imply?
> > > > >
> > > > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > > > >
> > > > > >
> > > > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > > > >
> > > > > > Why mbuf.l2_len - 6 ?
> > > > > L2 header when VLAN header is preset will be
> > > > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > > > l2_len = X + 12 + 4 + 2
> > > > > So, VLAN header starts at (l2_len - 6) bytes.
> > > > >
> > > > > >
> > > > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > > > + * IPv4 pkt.
> > > > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > + */
> > > > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > > > >
> > > > > > We should have one comment per define.
> > > > > Ack, will fix in V2.
> > > > >
> > > > > >
> > > > > >
> > > > > > > > > > +
> > > > > > > > > > +/**
> > > > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > > > >
> > > > > > > > > > /**
> > > > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > > > --
> > > > > > > > > > 2.8.4
> > > > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 10:08 ` Nithin Dabilpuram
2020-05-15 10:30 ` Ananyev, Konstantin
@ 2020-05-15 13:12 ` Thomas Monjalon
2020-05-15 13:44 ` Nithin Dabilpuram
1 sibling, 1 reply; 36+ messages in thread
From: Thomas Monjalon @ 2020-05-15 13:12 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Olivier Matz, Jerin Jacob, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
15/05/2020 12:08, Nithin Dabilpuram:
> On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > I don't see any better approach than having a mbuf flag. However, I'm
> > still not fully convinced that a dynamic flag won't do the job. Taking
> > 3 additional flags (among 18 remaing) for this feature also means that
> > we have 3 flags less for dynamic flags for all applications, even for
> > applications that will not use this feature.
> >
> > Would it be a problem to use a dynamic flag in this case?
> Since packet marking feature itself is already part of spec,
> if we move the flags to PMD specific dynamic flag, then it creates a confusion.
>
> It is not the case of a custom feature supported by a specific PMD.
> I believe when other PMD's implement packet marking, the same flags will
> suffice.
A dynamic flag is not necessarily PMD-specific.
It is just avoiding consuming bits if the feature is not used by the application.
We must move more existing flags and fields to be dynamic.
In general, all new flags and fields in mbuf should be dynamic.
And a work must be done to move existing stuff to free more space
for more dynamic features.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 13:12 ` Thomas Monjalon
@ 2020-05-15 13:44 ` Nithin Dabilpuram
2020-05-15 15:10 ` Thomas Monjalon
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-15 13:44 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Olivier Matz, Jerin Jacob, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> 15/05/2020 12:08, Nithin Dabilpuram:
> > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > I don't see any better approach than having a mbuf flag. However, I'm
> > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > 3 additional flags (among 18 remaing) for this feature also means that
> > > we have 3 flags less for dynamic flags for all applications, even for
> > > applications that will not use this feature.
> > >
> > > Would it be a problem to use a dynamic flag in this case?
> > Since packet marking feature itself is already part of spec,
> > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> >
> > It is not the case of a custom feature supported by a specific PMD.
> > I believe when other PMD's implement packet marking, the same flags will
> > suffice.
>
> A dynamic flag is not necessarily PMD-specific.
> It is just avoiding consuming bits if the feature is not used by the application.
> We must move more existing flags and fields to be dynamic.
>
> In general, all new flags and fields in mbuf should be dynamic.
> And a work must be done to move existing stuff to free more space
> for more dynamic features.
My bad, I thought dynamic flags can only be used for PMD specific thing.
There is however a cost of using dynamic flag which I think should be avoided
for DPDK spec defined offloads, though it's fine for PMD specific things.
Dynamic offload flags causes application and PMD to use non constant offset
or shift which are looked up at init, instead of having a constant shift or
offset. This indirection costs some cycles due to extra loads in fast path.
>
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 10:30 ` Ananyev, Konstantin
@ 2020-05-15 13:57 ` Nithin Dabilpuram
2020-05-28 15:43 ` Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-15 13:57 UTC (permalink / raw)
To: Ananyev, Konstantin, Cristian Dumitrescu
Cc: Olivier Matz, Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon,
Yigit, Ferruh, Andrew Rybchenko, Ori Kam, Dumitrescu, Cristian,
Burakov, Anatoly, Mcnamara, John, Kovacevic, Marko, dpdk-dev,
Jerin Jacob, Krzysztof Kanas
On Fri, May 15, 2020 at 10:30:30AM +0000, Ananyev, Konstantin wrote:
>
> > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > Hi Nithin,
> > >
> > > On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> > > > On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > > > > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > > > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > > > > Hi Olivier,
> > > > > > > >
> > > > > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > > > > External Email
> > > > > > > > >
> > > > > > > > > ----------------------------------------------------------------------
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > >
> > > > > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > > > > packet marking.
> > > > > > > > > > >
> > > > > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > > > > applicability of marking on that specific packet and
> > > > > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > >
> > > > > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > > > > marking support.
> > > > > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > > > > >
> > > > > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > > > > that are generic enough.
> > > > > > > >
> > > > > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > > > > >
> > > > > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > > > > calculation. This is pretty straightforward and there is no other
> > > > > > > dependency than the offload feature advertised by the PMD.
> > > > > > >
> > > > > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > > > > difficult for me to have a global view of what is done for instance when
> > > > > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > > > > >
> > > > > > > Can you confirm that my understanding below is correct? (or correct me
> > > > > > > where I'm wrong)
> > > > > > >
> > > > > > > Before your patch:
> > > > > > > - the application enables the port and traffic manager on it
> > > > > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > > > > class must be marked
> > > > > > > - when a packet is transmitted, the traffic class is determined by the
> > > > > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > > > > bit is set depending on traffic class
> > > > > > >
> > > > > > > The problem is for packets that cannot be recognized by the hardware,
> > > > > > > correct?
> > > > > >
> > > > > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > > > > through all the layers of packet data in Tx to identify what packet it is
> > > > > > and where the l2, l3, l4 headers start for performance reasons.
> > > > > >
> > > > > > I believe there are other hardware too that have the same expectation
> > > > > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > > > > >
> > > > > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > > > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> > > > >
> > > > > The objective is to give an indication to the hardware that the packet has:
> > > > > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > > > > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > > > > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > > > > PKT_TX_MARK_IP_ECN
> > > > >
> > > > > Just to be sure I'm getting the point, would it also work if with flags
> > > > > like this:
> > > > >
> > > > > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > > > > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > > > > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > > > > PKT_TX_IPv4 or PKT_TX_IPv6)
> > > > >
> > > > > The underlying question is: do we need the flags to only describe the
> > > > > content of the packet or do the flag also indicate that an action has to
> > > > > be done?
> > > >
> > > > If we don't have a specific action based flag, then in future it might collide
> > > > with other functionality and we will not be able to choose that specific
> > > > offload. All the existing features are having specific flags, like TSO,
> > > > CSUM.
> > > >
> > > > RFC wise, even when marking in enabled and packet is coloured, not all packets
> > > > can be marked.
> > > > For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> > > > only with below 12 code points out of 64 code points (6 bits of DSCP).
> > > >
> > > > Class 1 Class 2 Class 3 Class 4
> > > > +----------+----------+----------+----------+
> > > > Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> > > > Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> > > > High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> > > > +----------+----------+----------+----------+
> > > >
> > > > All other combinations of DSCP value can be used for some other purposes
> > > > and hence packets with those values shouldn't be marked.
> > > > Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
> > > >
> > > > Having PMD or HW to check if the packet falls in the said class and then do
> > > > marking will impact performance. Since application actually fills those values
> > > > in packet, it will be more easy for them to say.
> > > >
> > > > >
> > > > > > > So your patch is a way to force the hardware to recognize mark set the
> > > > > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > > > > >
> > > > > > > How the is traffic class of the packet determined?
> > > > > >
> > > > > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > > > > and packet color determines traffic class. The exact behavior of
> > > > > > packet color to traffic class mapping is mentioned in TM spec based on
> > > > > > few other RFC's.
> > > > > >
> > > > > > [1] https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> > pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > > > > [2] https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> > pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> > > > >
> > > > > OK, so the traffic class does not depend on the packet type?
> > > > Yes it doesn't. But where to update the traffic class is specific to packet
> > > > type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> > > > Also ECN marking is only valid for TCP/SCTP packets.
> > > >
> > > > >
> > > > >
> > > > > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > > > > dpdk repository, just grep for "dynflag".
> > > > > > > >
> > > > > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > > > > done. This is the first PMD that supports packet marking feature and
> > > > > > > > hence it was not exposed earlier.
> > > > > > > >
> > > > > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Also, I think that the feature availability should be advertised through
> > > > > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > > > > that these flags can be used.
> > > > > > > >
> > > > > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > > > > >
> > > > > > > Does this mean that any driver advertising this existing feature flag
> > > > > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > > > > feature for it?
> > > > > >
> > > > > > Yes, I thought PMD's need to support both.
> > > > > > I'm fine adding specific feature flag for the offload flags alone
> > > > > > if you insist or if there are other PMD's which don't need the offload flags
> > > > > > for packet marking. I was not able to find out about other PMD's as
> > > > > > none of the existing PMD's support packet marking.
> > > > >
> > > > > Do you suggest that the behavior of the traffic manager marking should
> > > > > be:
> > > > >
> > > > > a- the hardware tries to recognize tx packets, and mark them
> > > > > accordingly. What packets are recognized depend on hardware.
> > > > > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > > > > recognize packets, so it can mark packets.
> > > > >
> > > > > For an application, a- is difficult to apprehend as it will be dependent
> > > > > on hardware.
> > > > >
> > > > > Or do you suggest that packets should only be marked if there is a mbuf
> > > > > flag? (only b-)
> > > > Yes, I believe b- is the right thing.
> > > >
> > > > >
> > > > > Do you confirm that there is no support at all for this feature today?
> > > > > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
> > > >
> > > > Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> > > > supposed to be experimental but was mistakenly marked stable.
> > > > You can see related discussion in below threads about marking rte_tm.h
> > > > experimental again in v20.11.
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> > 2DApril_164970.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHN
> > jN9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=-o2E-F9aHy3mrQw6xgO__RPXY9t8s3yjJn81X6Ius3k&e=
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> > 2DMay_166221.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHNj
> > N9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=gTKSzMmlhE75x4TP8IJB7NP5MVO-zxjmNRQ9bZ6MxwI&e=
> > >
> > > Thank you for the explanations. I also think b- is a better choice.
> > >
> > > I don't see any better approach than having a mbuf flag. However, I'm
> > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > 3 additional flags (among 18 remaing) for this feature also means that
> > > we have 3 flags less for dynamic flags for all applications, even for
> > > applications that will not use this feature.
>
> I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> Can it probably be squeezed somehow?
> Let say we reserve one flag that this information is present or not, and
> re-use one of rx-only fields for store additional information (packet_type, or so).
> Or might be some other approach.
We are fine with this approach where we define one bit in Tx offloads for pkt
marking and and 3 bits reused from Rx offload flags area.
For example:
@@ -186,10 +186,16 @@ extern "C" {
/* add new RX flags here, don't forget to update PKT_FIRST_FREE */
+/* Reused Rx offload bits for Tx offloads */
+#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
+#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
+#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
+
#define PKT_FIRST_FREE (1ULL << 23)
-#define PKT_LAST_FREE (1ULL << 40)
+#define PKT_LAST_FREE (1ULL << 39)
/* add new TX flags here, don't forget to update PKT_LAST_FREE */
+#define PKT_TX_MARK_EN (1ULL << 40)
Is this fine ?
>
> > >
> > > Would it be a problem to use a dynamic flag in this case?
> > Since packet marking feature itself is already part of spec,
> > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> >
> > It is not the case of a custom feature supported by a specific PMD.
> > I believe when other PMD's implement packet marking, the same flags will
> > suffice.
> > >
> > > Thanks,
> > > Olivier
> > >
> > >
> > > >
> > > > Thanks
> > > > Nithin
> > > >
> > > > >
> > > > > Thanks,
> > > > > Olivier
> > > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > Please also see few comments below.
> > > > > > >
> > > > > > > > > > > ---
> > > > > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > > > > >
> > > > > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > > > > +
> > > > > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > > > > +--------------------------------------
> > > > > > > > > > > +
> > > > > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > > > > +
> > > > > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > > > > +
> > > > > > > > > > > .. _nic_features_other:
> > > > > > > > > > >
> > > > > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > > > > default: return NULL;
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > > > > };
> > > > > > > > > > > const char *name;
> > > > > > > > > > > unsigned int i;
> > > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > > > > >
> > > > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > > > > >
> > > > > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > > > > >
> > > > > > > > > > > /**
> > > > > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > > > > >
> > > > > > > What does that imply?
> > > > > >
> > > > > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > > > > >
> > > > > > >
> > > > > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > > > > >
> > > > > > > Why mbuf.l2_len - 6 ?
> > > > > > L2 header when VLAN header is preset will be
> > > > > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > > > > l2_len = X + 12 + 4 + 2
> > > > > > So, VLAN header starts at (l2_len - 6) bytes.
> > > > > >
> > > > > > >
> > > > > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > > > > + * IPv4 pkt.
> > > > > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > > + */
> > > > > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > > > > >
> > > > > > > We should have one comment per define.
> > > > > > Ack, will fix in V2.
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > > > > +
> > > > > > > > > > > +/**
> > > > > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > > > > >
> > > > > > > > > > > /**
> > > > > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > > > > --
> > > > > > > > > > > 2.8.4
> > > > > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 13:44 ` Nithin Dabilpuram
@ 2020-05-15 15:10 ` Thomas Monjalon
2020-05-15 16:26 ` Jerin Jacob
0 siblings, 1 reply; 36+ messages in thread
From: Thomas Monjalon @ 2020-05-15 15:10 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Olivier Matz, Jerin Jacob, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
15/05/2020 15:44, Nithin Dabilpuram:
> On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> > 15/05/2020 12:08, Nithin Dabilpuram:
> > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > applications that will not use this feature.
> > > >
> > > > Would it be a problem to use a dynamic flag in this case?
> > > Since packet marking feature itself is already part of spec,
> > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > >
> > > It is not the case of a custom feature supported by a specific PMD.
> > > I believe when other PMD's implement packet marking, the same flags will
> > > suffice.
> >
> > A dynamic flag is not necessarily PMD-specific.
> > It is just avoiding consuming bits if the feature is not used by the application.
> > We must move more existing flags and fields to be dynamic.
> >
> > In general, all new flags and fields in mbuf should be dynamic.
> > And a work must be done to move existing stuff to free more space
> > for more dynamic features.
>
> My bad, I thought dynamic flags can only be used for PMD specific thing.
>
> There is however a cost of using dynamic flag which I think should be avoided
> for DPDK spec defined offloads, though it's fine for PMD specific things.
>
> Dynamic offload flags causes application and PMD to use non constant offset
> or shift which are looked up at init, instead of having a constant shift or
> offset. This indirection costs some cycles due to extra loads in fast path.
Yes there is a cost. We described it quite clearly last year.
The default rule is now to add new flags and fields as dynamic.
In case the rule was not clear, I will send a patch to insert some
notes in the code and the doc.
If you disagree with this new rule, you will have to give very good arguments.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 15:10 ` Thomas Monjalon
@ 2020-05-15 16:26 ` Jerin Jacob
2020-05-15 16:52 ` Thomas Monjalon
0 siblings, 1 reply; 36+ messages in thread
From: Jerin Jacob @ 2020-05-15 16:26 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Nithin Dabilpuram, Olivier Matz, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Fri, May 15, 2020 at 8:40 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 15/05/2020 15:44, Nithin Dabilpuram:
> > On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> > > 15/05/2020 12:08, Nithin Dabilpuram:
> > > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > > applications that will not use this feature.
> > > > >
> > > > > Would it be a problem to use a dynamic flag in this case?
> > > > Since packet marking feature itself is already part of spec,
> > > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > > >
> > > > It is not the case of a custom feature supported by a specific PMD.
> > > > I believe when other PMD's implement packet marking, the same flags will
> > > > suffice.
> > >
> > > A dynamic flag is not necessarily PMD-specific.
> > > It is just avoiding consuming bits if the feature is not used by the application.
> > > We must move more existing flags and fields to be dynamic.
> > >
> > > In general, all new flags and fields in mbuf should be dynamic.
> > > And a work must be done to move existing stuff to free more space
> > > for more dynamic features.
> >
> > My bad, I thought dynamic flags can only be used for PMD specific thing.
> >
> > There is however a cost of using dynamic flag which I think should be avoided
> > for DPDK spec defined offloads, though it's fine for PMD specific things.
> >
> > Dynamic offload flags causes application and PMD to use non constant offset
> > or shift which are looked up at init, instead of having a constant shift or
> > offset. This indirection costs some cycles due to extra loads in fast path.
>
> Yes there is a cost. We described it quite clearly last year.
> The default rule is now to add new flags and fields as dynamic.
> In case the rule was not clear, I will send a patch to insert some
> notes in the code and the doc.
Yes. Please send a patch to document the rule. That makes life easy
for everyone to make a boolean decision.
Here is the comment from mbuf: support dynamic fields and flags commit
when accepted this patch.
" The typical use case is a PMD that registers space for an offload
feature, when the application requests to enable this feature. As
the space in mbuf is limited, the space should only be reserved if it
is going to be used (i.e when the application explicitly asks for it).
"
If you are pushing this feature to dynamic mbuf filed then rte_tm
subsystem needs to register dynamic field
not the PMD as the feature is part of rte_tm spec.
>
> If you disagree with this new rule, you will have to give very good arguments.
What would the definition of a good argument? as the same logic can be
implemented with dynamic vs
static at the cost of dynamic indirection.
>
>
>
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 16:26 ` Jerin Jacob
@ 2020-05-15 16:52 ` Thomas Monjalon
2020-05-15 17:00 ` Jerin Jacob
2020-05-15 18:07 ` Nithin Dabilpuram
0 siblings, 2 replies; 36+ messages in thread
From: Thomas Monjalon @ 2020-05-15 16:52 UTC (permalink / raw)
To: Jerin Jacob
Cc: Nithin Dabilpuram, Olivier Matz, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
15/05/2020 18:26, Jerin Jacob:
> On Fri, May 15, 2020 at 8:40 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > 15/05/2020 15:44, Nithin Dabilpuram:
> > > On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> > > > 15/05/2020 12:08, Nithin Dabilpuram:
> > > > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > > > applications that will not use this feature.
> > > > > >
> > > > > > Would it be a problem to use a dynamic flag in this case?
> > > > > Since packet marking feature itself is already part of spec,
> > > > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > > > >
> > > > > It is not the case of a custom feature supported by a specific PMD.
> > > > > I believe when other PMD's implement packet marking, the same flags will
> > > > > suffice.
> > > >
> > > > A dynamic flag is not necessarily PMD-specific.
> > > > It is just avoiding consuming bits if the feature is not used by the application.
> > > > We must move more existing flags and fields to be dynamic.
> > > >
> > > > In general, all new flags and fields in mbuf should be dynamic.
> > > > And a work must be done to move existing stuff to free more space
> > > > for more dynamic features.
> > >
> > > My bad, I thought dynamic flags can only be used for PMD specific thing.
> > >
> > > There is however a cost of using dynamic flag which I think should be avoided
> > > for DPDK spec defined offloads, though it's fine for PMD specific things.
> > >
> > > Dynamic offload flags causes application and PMD to use non constant offset
> > > or shift which are looked up at init, instead of having a constant shift or
> > > offset. This indirection costs some cycles due to extra loads in fast path.
> >
> > Yes there is a cost. We described it quite clearly last year.
> > The default rule is now to add new flags and fields as dynamic.
> > In case the rule was not clear, I will send a patch to insert some
> > notes in the code and the doc.
>
> Yes. Please send a patch to document the rule. That makes life easy
> for everyone to make a boolean decision.
Yes, I will work on it.
> Here is the comment from mbuf: support dynamic fields and flags commit
> when accepted this patch.
>
> " The typical use case is a PMD that registers space for an offload
> feature, when the application requests to enable this feature. As
> the space in mbuf is limited, the space should only be reserved if it
> is going to be used (i.e when the application explicitly asks for it).
> "
OK, there is probably a documentation gap.
> If you are pushing this feature to dynamic mbuf filed then rte_tm
> subsystem needs to register dynamic field
> not the PMD as the feature is part of rte_tm spec.
Is there a function in rte_tm which initializes or configure the feature?
> > If you disagree with this new rule, you will have to give very good arguments.
>
> What would the definition of a good argument? as the same logic can be
> implemented with dynamic vs
> static at the cost of dynamic indirection.
I think the only exception to add a static flag or field is to demonstrate
how basic is the feature.
But I think all basic features are already integrated for years.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 16:52 ` Thomas Monjalon
@ 2020-05-15 17:00 ` Jerin Jacob
2020-05-15 18:07 ` Nithin Dabilpuram
1 sibling, 0 replies; 36+ messages in thread
From: Jerin Jacob @ 2020-05-15 17:00 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Nithin Dabilpuram, Olivier Matz, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Fri, May 15, 2020 at 10:22 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 15/05/2020 18:26, Jerin Jacob:
> > On Fri, May 15, 2020 at 8:40 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 15/05/2020 15:44, Nithin Dabilpuram:
> > > > On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> > > > > 15/05/2020 12:08, Nithin Dabilpuram:
> > > > > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > > > > applications that will not use this feature.
> > > > > > >
> > > > > > > Would it be a problem to use a dynamic flag in this case?
> > > > > > Since packet marking feature itself is already part of spec,
> > > > > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > > > > >
> > > > > > It is not the case of a custom feature supported by a specific PMD.
> > > > > > I believe when other PMD's implement packet marking, the same flags will
> > > > > > suffice.
> > > > >
> > > > > A dynamic flag is not necessarily PMD-specific.
> > > > > It is just avoiding consuming bits if the feature is not used by the application.
> > > > > We must move more existing flags and fields to be dynamic.
> > > > >
> > > > > In general, all new flags and fields in mbuf should be dynamic.
> > > > > And a work must be done to move existing stuff to free more space
> > > > > for more dynamic features.
> > > >
> > > > My bad, I thought dynamic flags can only be used for PMD specific thing.
> > > >
> > > > There is however a cost of using dynamic flag which I think should be avoided
> > > > for DPDK spec defined offloads, though it's fine for PMD specific things.
> > > >
> > > > Dynamic offload flags causes application and PMD to use non constant offset
> > > > or shift which are looked up at init, instead of having a constant shift or
> > > > offset. This indirection costs some cycles due to extra loads in fast path.
> > >
> > > Yes there is a cost. We described it quite clearly last year.
> > > The default rule is now to add new flags and fields as dynamic.
> > > In case the rule was not clear, I will send a patch to insert some
> > > notes in the code and the doc.
> >
> > Yes. Please send a patch to document the rule. That makes life easy
> > for everyone to make a boolean decision.
>
> Yes, I will work on it.
Thanks.
>
> > Here is the comment from mbuf: support dynamic fields and flags commit
> > when accepted this patch.
> >
> > " The typical use case is a PMD that registers space for an offload
> > feature, when the application requests to enable this feature. As
> > the space in mbuf is limited, the space should only be reserved if it
> > is going to be used (i.e when the application explicitly asks for it).
> > "
>
> OK, there is probably a documentation gap.
Obviously :-)
>
> > If you are pushing this feature to dynamic mbuf filed then rte_tm
> > subsystem needs to register dynamic field
> > not the PMD as the feature is part of rte_tm spec.
>
> Is there a function in rte_tm which initializes or configure the feature?
See rte_tm_mark_*
>
>
> > > If you disagree with this new rule, you will have to give very good arguments.
> >
> > What would the definition of a good argument? as the same logic can be
> > implemented with dynamic vs
> > static at the cost of dynamic indirection.
>
> I think the only exception to add a static flag or field is to demonstrate
> how basic is the feature.
> But I think all basic features are already integrated for years.
Yes. That's the path then let have a rule to not add any "new fields"
and "flags" to mbuf
and everything should be through dynamic.
>
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 16:52 ` Thomas Monjalon
2020-05-15 17:00 ` Jerin Jacob
@ 2020-05-15 18:07 ` Nithin Dabilpuram
1 sibling, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-15 18:07 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Jerin Jacob, Olivier Matz, Nithin Dabilpuram, Ferruh Yigit,
Andrew Rybchenko, Ori Kam, Cristian Dumitrescu, Anatoly Burakov,
John McNamara, Marko Kovacevic, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
On Fri, May 15, 2020 at 06:52:23PM +0200, Thomas Monjalon wrote:
> 15/05/2020 18:26, Jerin Jacob:
> > On Fri, May 15, 2020 at 8:40 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 15/05/2020 15:44, Nithin Dabilpuram:
> > > > On Fri, May 15, 2020 at 03:12:59PM +0200, Thomas Monjalon wrote:
> > > > > 15/05/2020 12:08, Nithin Dabilpuram:
> > > > > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > > > > applications that will not use this feature.
> > > > > > >
> > > > > > > Would it be a problem to use a dynamic flag in this case?
> > > > > > Since packet marking feature itself is already part of spec,
> > > > > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > > > > >
> > > > > > It is not the case of a custom feature supported by a specific PMD.
> > > > > > I believe when other PMD's implement packet marking, the same flags will
> > > > > > suffice.
> > > > >
> > > > > A dynamic flag is not necessarily PMD-specific.
> > > > > It is just avoiding consuming bits if the feature is not used by the application.
> > > > > We must move more existing flags and fields to be dynamic.
> > > > >
> > > > > In general, all new flags and fields in mbuf should be dynamic.
> > > > > And a work must be done to move existing stuff to free more space
> > > > > for more dynamic features.
> > > >
> > > > My bad, I thought dynamic flags can only be used for PMD specific thing.
> > > >
> > > > There is however a cost of using dynamic flag which I think should be avoided
> > > > for DPDK spec defined offloads, though it's fine for PMD specific things.
> > > >
> > > > Dynamic offload flags causes application and PMD to use non constant offset
> > > > or shift which are looked up at init, instead of having a constant shift or
> > > > offset. This indirection costs some cycles due to extra loads in fast path.
> > >
> > > Yes there is a cost. We described it quite clearly last year.
> > > The default rule is now to add new flags and fields as dynamic.
> > > In case the rule was not clear, I will send a patch to insert some
> > > notes in the code and the doc.
> >
> > Yes. Please send a patch to document the rule. That makes life easy
> > for everyone to make a boolean decision.
>
> Yes, I will work on it.
>
> > Here is the comment from mbuf: support dynamic fields and flags commit
> > when accepted this patch.
> >
> > " The typical use case is a PMD that registers space for an offload
> > feature, when the application requests to enable this feature. As
> > the space in mbuf is limited, the space should only be reserved if it
> > is going to be used (i.e when the application explicitly asks for it).
> > "
>
> OK, there is probably a documentation gap.
>
> > If you are pushing this feature to dynamic mbuf filed then rte_tm
> > subsystem needs to register dynamic field
> > not the PMD as the feature is part of rte_tm spec.
>
> Is there a function in rte_tm which initializes or configure the feature?
>
>
> > > If you disagree with this new rule, you will have to give very good arguments.
> >
> > What would the definition of a good argument? as the same logic can be
> > implemented with dynamic vs
> > static at the cost of dynamic indirection.
>
> I think the only exception to add a static flag or field is to demonstrate
> how basic is the feature.
> But I think all basic features are already integrated for years.
>
Vector implementations will always have huge cost due to dynamic offload
flags. Ofcourse, I don't have a vector implementation for packet marking but,
in future if there is such an offload flag with vector implementation
then hard enforcement will not work.
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-15 13:57 ` Nithin Dabilpuram
@ 2020-05-28 15:43 ` Nithin Dabilpuram
2020-05-30 15:12 ` Jerin Jacob
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-05-28 15:43 UTC (permalink / raw)
To: Ananyev, Konstantin, Cristian Dumitrescu
Cc: Olivier Matz, Jerin Jacob, Nithin Dabilpuram, Thomas Monjalon,
Yigit, Ferruh, Andrew Rybchenko, Ori Kam, Burakov, Anatoly,
Mcnamara, John, Kovacevic, Marko, dpdk-dev, Jerin Jacob,
Krzysztof Kanas
Hi Olivier,
On Fri, May 15, 2020 at 07:27:46PM +0530, Nithin Dabilpuram wrote:
> On Fri, May 15, 2020 at 10:30:30AM +0000, Ananyev, Konstantin wrote:
> >
> > > On Thu, May 14, 2020 at 10:29:31PM +0200, Olivier Matz wrote:
> > > > Hi Nithin,
> > > >
> > > > On Tue, May 05, 2020 at 11:49:20AM +0530, Nithin Dabilpuram wrote:
> > > > > On Mon, May 04, 2020 at 02:27:35PM +0200, Olivier Matz wrote:
> > > > > > On Mon, May 04, 2020 at 03:34:57PM +0530, Nithin Dabilpuram wrote:
> > > > > > > On Mon, May 04, 2020 at 11:16:40AM +0200, Olivier Matz wrote:
> > > > > > > > On Mon, May 04, 2020 at 01:57:06PM +0530, Nithin Dabilpuram wrote:
> > > > > > > > > Hi Olivier,
> > > > > > > > >
> > > > > > > > > On Mon, May 04, 2020 at 10:06:34AM +0200, Olivier Matz wrote:
> > > > > > > > > > External Email
> > > > > > > > > >
> > > > > > > > > > ----------------------------------------------------------------------
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > On Fri, May 01, 2020 at 04:48:21PM +0530, Jerin Jacob wrote:
> > > > > > > > > > > On Fri, Apr 17, 2020 at 12:53 PM Nithin Dabilpuram
> > > > > > > > > > > <nithind1988@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > >
> > > > > > > > > > > > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > > > > > > > > > > > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > > > > > > > > > > > packet marking.
> > > > > > > > > > > >
> > > > > > > > > > > > When packet marking feature in Traffic manager is enabled,
> > > > > > > > > > > > application has to the use the three new flags to indicate
> > > > > > > > > > > > to PMD on whether packet marking needs to be enabled on the
> > > > > > > > > > > > specific mbuf or not. By setting the three flags, it is
> > > > > > > > > > > > assumed by PMD that application has already verified the
> > > > > > > > > > > > applicability of marking on that specific packet and
> > > > > > > > > > > > PMD need not perform further checks as per RFC.
> > > > > > > > > > > >
> > > > > > > > > > > > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > >
> > > > > > > > > > > None of the ethdev TM driver implementations has supported packet
> > > > > > > > > > > marking support.
> > > > > > > > > > > rte_tm and rte_mbuf maintainers(Christian, Oliver), Could you review this patch?
> > > > > > > > > >
> > > > > > > > > > As you know, the number of mbuf flags is limited (only 18 bits are
> > > > > > > > > > remaining), so I think we should use them with care, i.e. for features
> > > > > > > > > > that are generic enough.
> > > > > > > > >
> > > > > > > > > I agree, but I believe this is one of the basic flags needed like other
> > > > > > > > > Tx checksum offload flags (like PKT_TX_IP_CKSUM, PKT_TX_IPV4, etc) which
> > > > > > > > > are needed to identify on which packets HW should/can apply packet marking.
> > > > > > > >
> > > > > > > > PKT_TX_IP_CKSUM tells the hardware to offload the checksum
> > > > > > > > calculation. This is pretty straightforward and there is no other
> > > > > > > > dependency than the offload feature advertised by the PMD.
> > > > > > > >
> > > > > > > > I'm sorry, I have not a lot of experience with rte_tm.h, so it's
> > > > > > > > difficult for me to have a global view of what is done for instance when
> > > > > > > > PKT_TX_MARK_VLAN_DEI is set, and what happens when it is not set.
> > > > > > > >
> > > > > > > > Can you confirm that my understanding below is correct? (or correct me
> > > > > > > > where I'm wrong)
> > > > > > > >
> > > > > > > > Before your patch:
> > > > > > > > - the application enables the port and traffic manager on it
> > > > > > > > - the application calls rte_tm_mark_vlan_dei() to select which traffic
> > > > > > > > class must be marked
> > > > > > > > - when a packet is transmitted, the traffic class is determined by the
> > > > > > > > hardware, and if the hardware recognizes a VLAN packet, the VLAN DEI
> > > > > > > > bit is set depending on traffic class
> > > > > > > >
> > > > > > > > The problem is for packets that cannot be recognized by the hardware,
> > > > > > > > correct?
> > > > > > >
> > > > > > > Yes. Octeontx2 HW always depends on application knowledge instead of walking
> > > > > > > through all the layers of packet data in Tx to identify what packet it is
> > > > > > > and where the l2, l3, l4 headers start for performance reasons.
> > > > > > >
> > > > > > > I believe there are other hardware too that have the same expectation
> > > > > > > and hence we have a need for PKT_TX_IPv4, PKT_TX_IPv6 kind of flags.
> > > > > > >
> > > > > > > Hence we want to make use of mbuf:tx_offload field and PKT_TX_* flags
> > > > > > > for identifying the packet and knowing what are its l2,l3,l4 offsets.
> > > > > >
> > > > > > The objective is to give an indication to the hardware that the packet has:
> > > > > > - an 802.1q header at offset X for PKT_TX_MARK_VLAN_DEI
> > > > > > - an IP/IPv6 header at offset X for PKT_TX_MARK_IP_DSCP
> > > > > > - an IP/IPv6 header at offset X and a TCP/SCTP header at offset Y for
> > > > > > PKT_TX_MARK_IP_ECN
> > > > > >
> > > > > > Just to be sure I'm getting the point, would it also work if with flags
> > > > > > like this:
> > > > > >
> > > > > > - an 802.1q header at offset X for PKT_TX_HAS_VLAN
> > > > > > - an IP/IPv6 header at offset X for PKT_TX_IPv4 or PKT_TX_IPv6
> > > > > > - a TCP/SCTP header at offset Y for PKT_TX_TCP/PKT_TX_SCTP (implies
> > > > > > PKT_TX_IPv4 or PKT_TX_IPv6)
> > > > > >
> > > > > > The underlying question is: do we need the flags to only describe the
> > > > > > content of the packet or do the flag also indicate that an action has to
> > > > > > be done?
> > > > >
> > > > > If we don't have a specific action based flag, then in future it might collide
> > > > > with other functionality and we will not be able to choose that specific
> > > > > offload. All the existing features are having specific flags, like TSO,
> > > > > CSUM.
> > > > >
> > > > > RFC wise, even when marking in enabled and packet is coloured, not all packets
> > > > > can be marked.
> > > > > For example when IP DSCP marking(RFC 2597) is enabled, marking is defined
> > > > > only with below 12 code points out of 64 code points (6 bits of DSCP).
> > > > >
> > > > > Class 1 Class 2 Class 3 Class 4
> > > > > +----------+----------+----------+----------+
> > > > > Low Drop Prec | 001010 | 010010 | 011010 | 100010 |
> > > > > Medium Drop Prec | 001100 | 010100 | 011100 | 100100 |
> > > > > High Drop Prec | 001110 | 010110 | 011110 | 100110 |
> > > > > +----------+----------+----------+----------+
> > > > >
> > > > > All other combinations of DSCP value can be used for some other purposes
> > > > > and hence packets with those values shouldn't be marked.
> > > > > Similar is the case with IP ECN marking for TCP/SCTP(RFC 3168).
> > > > >
> > > > > Having PMD or HW to check if the packet falls in the said class and then do
> > > > > marking will impact performance. Since application actually fills those values
> > > > > in packet, it will be more easy for them to say.
> > > > >
> > > > > >
> > > > > > > > So your patch is a way to force the hardware to recognize mark set the
> > > > > > > > VLAN DEI on packets that are not recognized as VLAN packets?
> > > > > > > >
> > > > > > > > How the is traffic class of the packet determined?
> > > > > > >
> > > > > > > Packet is coloured based on Single Rate[1] or Dual Rate[2] Shaping result
> > > > > > > and packet color determines traffic class. The exact behavior of
> > > > > > > packet color to traffic class mapping is mentioned in TM spec based on
> > > > > > > few other RFC's.
> > > > > > >
> > > > > > > [1] https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__tools.ietf.org_html_rfc2697&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> > > pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=05emGNkz3Qat3dtZIbEsmQDC5y9-tU9yItHX0x1aaJU&e=
> > > > > > > [2] https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__tools.ietf.org_html_rfc2698&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=
> > > pJDciSXpMy6TawycjvpYj_Jq5M5j_ywqhU8-keRI_ac&s=3VN2dIGSDt4vWM-FpPOOf-8SeVShl_t7QpXRU6Zw460&e=
> > > > > >
> > > > > > OK, so the traffic class does not depend on the packet type?
> > > > > Yes it doesn't. But where to update the traffic class is specific to packet
> > > > > type like DEI bit in VLAN or ECN field in IPv4/IPv6 or DSCP field in IPv4/IPv6.
> > > > > Also ECN marking is only valid for TCP/SCTP packets.
> > > > >
> > > > > >
> > > > > >
> > > > > > > > > > From what I understand, this feature is bound to octeontx2, so using a
> > > > > > > > > > mbuf dynamic flag would make more sense here. There are some examples in
> > > > > > > > > > dpdk repository, just grep for "dynflag".
> > > > > > > > >
> > > > > > > > > This is not octeontx2 specific flag but any "packet marking feature" enabled
> > > > > > > > > PMD would need these flags to identify on which packets marking needs to be
> > > > > > > > > done. This is the first PMD that supports packet marking feature and
> > > > > > > > > hence it was not exposed earlier.
> > > > > > > > >
> > > > > > > > > For example to mark VLAN DEI, PMD cannot always assume that there is preexisting
> > > > > > > > > VLAN header from Byte 12 as there is no gaurantee that ethernet header
> > > > > > > > > always starts at Byte 0 (Custom headers before ethernet hdr).
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Also, I think that the feature availability should be advertised through
> > > > > > > > > > an ethdev offload, so an application can know at initialization time
> > > > > > > > > > that these flags can be used.
> > > > > > > > >
> > > > > > > > > Feature availablity is already part of TM spec in rte_tm.h
> > > > > > > > > struct rte_tm_capabilities:mark_vlan_dei_supported
> > > > > > > > > struct rte_tm_capabilities:mark_ip_ecn_[sctp|tcp]_supported
> > > > > > > > > struct rte_tm_capabilities:mark_ip_dscp_supported
> > > > > > > >
> > > > > > > > Does this mean that any driver advertising this existing feature flag
> > > > > > > > has to support the new mbuf flags too? Shouldn't we have a specific
> > > > > > > > feature for it?
> > > > > > >
> > > > > > > Yes, I thought PMD's need to support both.
> > > > > > > I'm fine adding specific feature flag for the offload flags alone
> > > > > > > if you insist or if there are other PMD's which don't need the offload flags
> > > > > > > for packet marking. I was not able to find out about other PMD's as
> > > > > > > none of the existing PMD's support packet marking.
> > > > > >
> > > > > > Do you suggest that the behavior of the traffic manager marking should
> > > > > > be:
> > > > > >
> > > > > > a- the hardware tries to recognize tx packets, and mark them
> > > > > > accordingly. What packets are recognized depend on hardware.
> > > > > > b- if the mbuf has a specific flag, it helps the PMD and hardware to
> > > > > > recognize packets, so it can mark packets.
> > > > > >
> > > > > > For an application, a- is difficult to apprehend as it will be dependent
> > > > > > on hardware.
> > > > > >
> > > > > > Or do you suggest that packets should only be marked if there is a mbuf
> > > > > > flag? (only b-)
> > > > > Yes, I believe b- is the right thing.
> > > > >
> > > > > >
> > > > > > Do you confirm that there is no support at all for this feature today?
> > > > > > I mean, what was the usage of rte_tm_mark_vlan_dei() these last 3 years?
> > > > >
> > > > > Yes, it was not implemented/used. Because of such reasons, rte_tm.h is
> > > > > supposed to be experimental but was mistakenly marked stable.
> > > > > You can see related discussion in below threads about marking rte_tm.h
> > > > > experimental again in v20.11.
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> > > 2DApril_164970.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHN
> > > jN9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=-o2E-F9aHy3mrQw6xgO__RPXY9t8s3yjJn81X6Ius3k&e=
> > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2020-
> > > 2DMay_166221.html&d=DwIBAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=myqGwnIHNj
> > > N9IP7urxcVAB384qKoxlmm00p1gS7ttbw&s=gTKSzMmlhE75x4TP8IJB7NP5MVO-zxjmNRQ9bZ6MxwI&e=
> > > >
> > > > Thank you for the explanations. I also think b- is a better choice.
> > > >
> > > > I don't see any better approach than having a mbuf flag. However, I'm
> > > > still not fully convinced that a dynamic flag won't do the job. Taking
> > > > 3 additional flags (among 18 remaing) for this feature also means that
> > > > we have 3 flags less for dynamic flags for all applications, even for
> > > > applications that will not use this feature.
> >
> > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > Can it probably be squeezed somehow?
> > Let say we reserve one flag that this information is present or not, and
> > re-use one of rx-only fields for store additional information (packet_type, or so).
> > Or might be some other approach.
>
> We are fine with this approach where we define one bit in Tx offloads for pkt
> marking and and 3 bits reused from Rx offload flags area.
>
> For example:
>
> @@ -186,10 +186,16 @@ extern "C" {
>
> /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
>
> +/* Reused Rx offload bits for Tx offloads */
> +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> +
> #define PKT_FIRST_FREE (1ULL << 23)
> -#define PKT_LAST_FREE (1ULL << 40)
> +#define PKT_LAST_FREE (1ULL << 39)
>
> /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> +#define PKT_TX_MARK_EN (1ULL << 40)
>
> Is this fine ?
Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
and reuse unused Rx flag bits ?
>
>
> >
> > > >
> > > > Would it be a problem to use a dynamic flag in this case?
> > > Since packet marking feature itself is already part of spec,
> > > if we move the flags to PMD specific dynamic flag, then it creates a confusion.
> > >
> > > It is not the case of a custom feature supported by a specific PMD.
> > > I believe when other PMD's implement packet marking, the same flags will
> > > suffice.
> > > >
> > > > Thanks,
> > > > Olivier
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > > Nithin
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Olivier
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Please also see few comments below.
> > > > > > > >
> > > > > > > > > > > > ---
> > > > > > > > > > > > doc/guides/nics/features.rst | 14 ++++++++++++++
> > > > > > > > > > > > lib/librte_mbuf/rte_mbuf.c | 6 ++++++
> > > > > > > > > > > > lib/librte_mbuf/rte_mbuf_core.h | 36 ++++++++++++++++++++++++++++++++++--
> > > > > > > > > > > > 3 files changed, 54 insertions(+), 2 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > index edd21c4..bc978fb 100644
> > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > @@ -913,6 +913,20 @@ Supports to get Rx/Tx packet burst mode information.
> > > > > > > > > > > > * **[implements] eth_dev_ops**: ``rx_burst_mode_get``, ``tx_burst_mode_get``.
> > > > > > > > > > > > * **[related] API**: ``rte_eth_rx_burst_mode_get()``, ``rte_eth_tx_burst_mode_get()``.
> > > > > > > > > > > >
> > > > > > > > > > > > +.. _nic_features_traffic_manager_packet_marking_offload:
> > > > > > > > > > > > +
> > > > > > > > > > > > +Traffic Manager Packet marking offload
> > > > > > > > > > > > +--------------------------------------
> > > > > > > > > > > > +
> > > > > > > > > > > > +Supports enabling a packet marking offload specific mbuf.
> > > > > > > > > > > > +
> > > > > > > > > > > > +* **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_MARK_IP_DSCP``,
> > > > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_MARK_IP_ECN``, ``mbuf.ol_flags:PKT_TX_MARK_VLAN_DEI``,
> > > > > > > > > > > > + ``mbuf.ol_flags:PKT_TX_IPV4``, ``mbuf.ol_flags:PKT_TX_IPV6``.
> > > > > > > > > > > > +* **[uses] mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > +* **[related] API**: ``rte_tm_mark_ip_dscp()``, ``rte_tm_mark_ip_ecn()``,
> > > > > > > > > > > > + ``rte_tm_mark_vlan_dei()``.
> > > > > > > > > > > > +
> > > > > > > > > > > > .. _nic_features_other:
> > > > > > > > > > > >
> > > > > > > > > > > > Other dev ops not represented by a Feature
> > > > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > > index cd5794d..5c6896d 100644
> > > > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf.c
> > > > > > > > > > > > @@ -880,6 +880,9 @@ const char *rte_get_tx_ol_flag_name(uint64_t mask)
> > > > > > > > > > > > case PKT_TX_SEC_OFFLOAD: return "PKT_TX_SEC_OFFLOAD";
> > > > > > > > > > > > case PKT_TX_UDP_SEG: return "PKT_TX_UDP_SEG";
> > > > > > > > > > > > case PKT_TX_OUTER_UDP_CKSUM: return "PKT_TX_OUTER_UDP_CKSUM";
> > > > > > > > > > > > + case PKT_TX_MARK_VLAN_DEI: return "PKT_TX_MARK_VLAN_DEI";
> > > > > > > > > > > > + case PKT_TX_MARK_IP_DSCP: return "PKT_TX_MARK_IP_DSCP";
> > > > > > > > > > > > + case PKT_TX_MARK_IP_ECN: return "PKT_TX_MARK_IP_ECN";
> > > > > > > > > > > > default: return NULL;
> > > > > > > > > > > > }
> > > > > > > > > > > > }
> > > > > > > > > > > > @@ -916,6 +919,9 @@ rte_get_tx_ol_flag_list(uint64_t mask, char *buf, size_t buflen)
> > > > > > > > > > > > { PKT_TX_SEC_OFFLOAD, PKT_TX_SEC_OFFLOAD, NULL },
> > > > > > > > > > > > { PKT_TX_UDP_SEG, PKT_TX_UDP_SEG, NULL },
> > > > > > > > > > > > { PKT_TX_OUTER_UDP_CKSUM, PKT_TX_OUTER_UDP_CKSUM, NULL },
> > > > > > > > > > > > + { PKT_TX_MARK_VLAN_DEI, PKT_TX_MARK_VLAN_DEI, NULL },
> > > > > > > > > > > > + { PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_DSCP, NULL },
> > > > > > > > > > > > + { PKT_TX_MARK_IP_ECN, PKT_TX_MARK_IP_ECN, NULL },
> > > > > > > > > > > > };
> > > > > > > > > > > > const char *name;
> > > > > > > > > > > > unsigned int i;
> > > > > > > > > > > > diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > > index b9a59c8..d9f1290 100644
> > > > > > > > > > > > --- a/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > > +++ b/lib/librte_mbuf/rte_mbuf_core.h
> > > > > > > > > > > > @@ -187,11 +187,40 @@ extern "C" {
> > > > > > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > > > > > >
> > > > > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > > > > +#define PKT_LAST_FREE (1ULL << 37)
> > > > > > > > > > > >
> > > > > > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > > > > > >
> > > > > > > > > > > > /**
> > > > > > > > > > > > + * Packet marking offload flags. These flags indicated what kind
> > > > > > > > > > > > + * of packet marking needs to be applied on a given mbuf when
> > > > > > > > > > > > + * appropriate Traffic Manager configuration is in place.
> > > > > > > > > > > > + * When user set's these flags on a mbuf, below assumptions are made
> > > > > > > > > > > > + * 1) When PKT_TX_MARK_VLAN_DEI is set,
> > > > > > > > > > > > + * a) PMD assumes pkt to be a 802.1q packet.
> > > > > > > >
> > > > > > > > What does that imply?
> > > > > > >
> > > > > > > I meant by setting the flag, a packet has VLAN header adhering to IEEE 802.1Q spec.
> > > > > > >
> > > > > > > >
> > > > > > > > > > > > + * b) Application should also set mbuf.l2_len where 802.1Q header is
> > > > > > > > > > > > + * at (mbuf.l2_len - 6) offset.
> > > > > > > >
> > > > > > > > Why mbuf.l2_len - 6 ?
> > > > > > > L2 header when VLAN header is preset will be
> > > > > > > {custom header 'X' Bytes}:{Ethernet SRC+DST (12B)}:{VLAN Header (4B)}:{Ether Type (2B)}
> > > > > > > l2_len = X + 12 + 4 + 2
> > > > > > > So, VLAN header starts at (l2_len - 6) bytes.
> > > > > > >
> > > > > > > >
> > > > > > > > > > > > + * 2) When PKT_TX_MARK_IP_DSCP is set,
> > > > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6
> > > > > > > > > > > > + * to indicate whether if it is IPv4 packet or IPv6 packet
> > > > > > > > > > > > + * for DSCP marking. It should also set PKT_TX_IP_CKSUM if it is
> > > > > > > > > > > > + * IPv4 pkt.
> > > > > > > > > > > > + * b) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > > > + * 3) When PKT_TX_MARK_IP_ECN is set,
> > > > > > > > > > > > + * a) Application should also set either PKT_TX_IPV4 or PKT_TX_IPV6.
> > > > > > > > > > > > + * It should also set PKT_TX_IP_CKSUM if it is IPv4 pkt.
> > > > > > > > > > > > + * b) PMD will assume pkt L4 protocol is either TCP or SCTP and
> > > > > > > > > > > > + * ECN is set to 2'b01 or 2'b10 as per RFC 3168 and hence HW
> > > > > > > > > > > > + * can mark the packet for a configured color.
> > > > > > > > > > > > + * c) Application should also set mbuf.l2_len that indicates
> > > > > > > > > > > > + * start offset of L3 header.
> > > > > > > > > > > > + */
> > > > > > > > > > > > +#define PKT_TX_MARK_VLAN_DEI (1ULL << 38)
> > > > > > > > > > > > +#define PKT_TX_MARK_IP_DSCP (1ULL << 39)
> > > > > > > > > > > > +#define PKT_TX_MARK_IP_ECN (1ULL << 40)
> > > > > > > >
> > > > > > > > We should have one comment per define.
> > > > > > > Ack, will fix in V2.
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > > > > +
> > > > > > > > > > > > +/**
> > > > > > > > > > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > > > > > > > > > * outer UDP checksum in PMD. To use outer UDP checksum, the user needs to
> > > > > > > > > > > > * 1) Enable the following in mbuf,
> > > > > > > > > > > > @@ -384,7 +413,10 @@ extern "C" {
> > > > > > > > > > > > PKT_TX_MACSEC | \
> > > > > > > > > > > > PKT_TX_SEC_OFFLOAD | \
> > > > > > > > > > > > PKT_TX_UDP_SEG | \
> > > > > > > > > > > > - PKT_TX_OUTER_UDP_CKSUM)
> > > > > > > > > > > > + PKT_TX_OUTER_UDP_CKSUM | \
> > > > > > > > > > > > + PKT_TX_MARK_VLAN_DEI | \
> > > > > > > > > > > > + PKT_TX_MARK_IP_DSCP | \
> > > > > > > > > > > > + PKT_TX_MARK_IP_ECN)
> > > > > > > > > > > >
> > > > > > > > > > > > /**
> > > > > > > > > > > > * Mbuf having an external buffer attached. shinfo in mbuf must be filled.
> > > > > > > > > > > > --
> > > > > > > > > > > > 2.8.4
> > > > > > > > > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-28 15:43 ` Nithin Dabilpuram
@ 2020-05-30 15:12 ` Jerin Jacob
2020-06-02 10:53 ` Ananyev, Konstantin
0 siblings, 1 reply; 36+ messages in thread
From: Jerin Jacob @ 2020-05-30 15:12 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Ananyev, Konstantin, Cristian Dumitrescu, Olivier Matz,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
> > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > Can it probably be squeezed somehow?
> > > Let say we reserve one flag that this information is present or not, and
> > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > Or might be some other approach.
> >
> > We are fine with this approach where we define one bit in Tx offloads for pkt
> > marking and and 3 bits reused from Rx offload flags area.
> >
> > For example:
> >
> > @@ -186,10 +186,16 @@ extern "C" {
> >
> > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> >
> > +/* Reused Rx offload bits for Tx offloads */
> > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > +
> > #define PKT_FIRST_FREE (1ULL << 23)
> > -#define PKT_LAST_FREE (1ULL << 40)
> > +#define PKT_LAST_FREE (1ULL << 39)
> >
> > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > +#define PKT_TX_MARK_EN (1ULL << 40)
> >
> > Is this fine ?
>
> Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> and reuse unused Rx flag bits ?
+ Techboard
There is a related thread going on
http://mails.dpdk.org/archives/dev/2020-May/168810.html
If there is no consensus on email, then I would like to add this item
to the next TB meeting.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-05-30 15:12 ` Jerin Jacob
@ 2020-06-02 10:53 ` Ananyev, Konstantin
2020-06-02 14:25 ` Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Ananyev, Konstantin @ 2020-06-02 10:53 UTC (permalink / raw)
To: Jerin Jacob, Nithin Dabilpuram
Cc: Dumitrescu, Cristian, Olivier Matz, Nithin Dabilpuram,
Thomas Monjalon, Yigit, Ferruh, Andrew Rybchenko, Ori Kam,
Burakov, Anatoly, Mcnamara, John, Kovacevic, Marko, dpdk-dev,
Jerin Jacob, Krzysztof Kanas, techboard
Hi Jerin,
> > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > Can it probably be squeezed somehow?
> > > > Let say we reserve one flag that this information is present or not, and
> > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > Or might be some other approach.
> > >
> > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > marking and and 3 bits reused from Rx offload flags area.
> > >
> > > For example:
> > >
> > > @@ -186,10 +186,16 @@ extern "C" {
> > >
> > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > >
> > > +/* Reused Rx offload bits for Tx offloads */
> > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > +
> > > #define PKT_FIRST_FREE (1ULL << 23)
> > > -#define PKT_LAST_FREE (1ULL << 40)
> > > +#define PKT_LAST_FREE (1ULL << 39)
> > >
> > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > >
> > > Is this fine ?
> >
> > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > and reuse unused Rx flag bits ?
My thought was not about re-defining the flags (I think it is better to keep them intact),
but adding a union for one of rx-only fields (packet_type/rss/timestamp).
>
> + Techboard
>
> There is a related thread going on
> http://mails.dpdk.org/archives/dev/2020-May/168810.html
>
> If there is no consensus on email, then I would like to add this item
> to the next TB meeting.
Ok, I'll add that to tomorrow meeting agenda.
Konstantin
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-02 10:53 ` Ananyev, Konstantin
@ 2020-06-02 14:25 ` Nithin Dabilpuram
2020-06-03 8:28 ` Olivier Matz
2020-06-03 10:31 ` Ananyev, Konstantin
0 siblings, 2 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-06-02 14:25 UTC (permalink / raw)
To: Ananyev, Konstantin
Cc: Jerin Jacob, Dumitrescu, Cristian, Olivier Matz,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> Hi Jerin,
>
> > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > Can it probably be squeezed somehow?
> > > > > Let say we reserve one flag that this information is present or not, and
> > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > Or might be some other approach.
> > > >
> > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > marking and and 3 bits reused from Rx offload flags area.
> > > >
> > > > For example:
> > > >
> > > > @@ -186,10 +186,16 @@ extern "C" {
> > > >
> > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > >
> > > > +/* Reused Rx offload bits for Tx offloads */
> > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > +
> > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > >
> > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > >
> > > > Is this fine ?
> > >
> > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > and reuse unused Rx flag bits ?
>
> My thought was not about re-defining the flags (I think it is better to keep them intact),
> but adding a union for one of rx-only fields (packet_type/rss/timestamp).
Ok. Adding a union field at packet_type field is also fine like below.
@@ -187,9 +187,10 @@ extern "C" {
/* add new RX flags here, don't forget to update PKT_FIRST_FREE */
#define PKT_FIRST_FREE (1ULL << 23)
-#define PKT_LAST_FREE (1ULL << 40)
+#define PKT_LAST_FREE (1ULL << 39)
/* add new TX flags here, don't forget to update PKT_LAST_FREE */
+#define PKT_TX_MARK_EN (1ULL << 40)
/**
* Outer UDP checksum offload flag. This flag is used for enabling
@@ -461,6 +462,14 @@ enum {
#endif
};
+/* Tx packet marking flags in rte_mbuf::tx_mark.
+ * Valid only when PKT_TX_MARK_EN is set in
+ * rte_mbuf::ol_flags.
+ */
+#define TX_MARK_VLAN_DEI (1ULL << 0)
+#define TX_MARK_IP_DSCP (1ULL << 1)
+#define TX_MARK_IP_ECN (1ULL << 2)
+
/**
* The generic rte_mbuf, containing a packet mbuf.
*/
@@ -543,6 +552,10 @@ struct rte_mbuf {
};
uint32_t inner_l4_type:4; /**< Inner L4 type. */
};
+ struct {
+ uint32_t reserved:29;
+ uint32_t tx_mark:3;
+ };
};
Please correct me if this is not what you mean.
>
> >
> > + Techboard
> >
> > There is a related thread going on
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e=
> >
> > If there is no consensus on email, then I would like to add this item
> > to the next TB meeting.
>
> Ok, I'll add that to tomorrow meeting agenda.
> Konstantin
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-02 14:25 ` Nithin Dabilpuram
@ 2020-06-03 8:28 ` Olivier Matz
2020-06-03 10:44 ` Nithin Dabilpuram
2020-06-03 10:31 ` Ananyev, Konstantin
1 sibling, 1 reply; 36+ messages in thread
From: Olivier Matz @ 2020-06-03 8:28 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
Hi,
On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> > Hi Jerin,
> >
> > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > Can it probably be squeezed somehow?
> > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > Or might be some other approach.
> > > > >
> > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > marking and and 3 bits reused from Rx offload flags area.
> > > > >
> > > > > For example:
> > > > >
> > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > >
> > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > >
> > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > +
> > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > >
> > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > >
> > > > > Is this fine ?
> > > >
> > > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > > and reuse unused Rx flag bits ?
> >
> > My thought was not about re-defining the flags (I think it is better to keep them intact),
> > but adding a union for one of rx-only fields (packet_type/rss/timestamp).
>
> Ok. Adding a union field at packet_type field is also fine like below.
>
> @@ -187,9 +187,10 @@ extern "C" {
> /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
>
> #define PKT_FIRST_FREE (1ULL << 23)
> -#define PKT_LAST_FREE (1ULL << 40)
> +#define PKT_LAST_FREE (1ULL << 39)
>
> /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> +#define PKT_TX_MARK_EN (1ULL << 40)
>
> /**
> * Outer UDP checksum offload flag. This flag is used for enabling
> @@ -461,6 +462,14 @@ enum {
> #endif
> };
>
> +/* Tx packet marking flags in rte_mbuf::tx_mark.
> + * Valid only when PKT_TX_MARK_EN is set in
> + * rte_mbuf::ol_flags.
> + */
> +#define TX_MARK_VLAN_DEI (1ULL << 0)
> +#define TX_MARK_IP_DSCP (1ULL << 1)
> +#define TX_MARK_IP_ECN (1ULL << 2)
> +
> /**
> * The generic rte_mbuf, containing a packet mbuf.
> */
> @@ -543,6 +552,10 @@ struct rte_mbuf {
> };
> uint32_t inner_l4_type:4; /**< Inner L4 type. */
> };
> + struct {
> + uint32_t reserved:29;
> + uint32_t tx_mark:3;
> + };
> };
>
>
>
> Please correct me if this is not what you mean.
I'm not a big fan of reusing Rx fields or flags for Tx.
It's not obvious for an application than adding a tx_mark will overwrite
the packet_type. I understand that the risk is limited because packet_type
is Rx and the marks are Tx, but there is still one.
To summarize the different proposed approaches (please correct me if I'm wrong):
a- add 3 Tx mbuf flags
(-) consumes limited resource
b- add 3 dynamic flags
(-) slower
c- add 1 Tx flag and union with Rx field
(-) exclusive with Rx field
(-) still consumes one flag
My preference is still b-, for these reasons:
- There are many different DPDK use cases, and resources in mbuf is tight.
Recent contributions (rte_flow and ice driver) already made use of dynamic
fields/flags.
- When I implemented the dynamic fields/flags feature, I did a test which
showed that the cost of having a dynamic offset was few cycles (on my test
platform, it was~3 cycles for reading a field and ~2 cycles for writing a
field).
Regards,
Olivier
>
> >
> > >
> > > + Techboard
> > >
> > > There is a related thread going on
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e=
> > >
> > > If there is no consensus on email, then I would like to add this item
> > > to the next TB meeting.
> >
> > Ok, I'll add that to tomorrow meeting agenda.
> > Konstantin
> >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-02 14:25 ` Nithin Dabilpuram
2020-06-03 8:28 ` Olivier Matz
@ 2020-06-03 10:31 ` Ananyev, Konstantin
1 sibling, 0 replies; 36+ messages in thread
From: Ananyev, Konstantin @ 2020-06-03 10:31 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Jerin Jacob, Dumitrescu, Cristian, Olivier Matz,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
> > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > Can it probably be squeezed somehow?
> > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > Or might be some other approach.
> > > > >
> > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > marking and and 3 bits reused from Rx offload flags area.
> > > > >
> > > > > For example:
> > > > >
> > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > >
> > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > >
> > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > +
> > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > >
> > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > >
> > > > > Is this fine ?
> > > >
> > > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > > and reuse unused Rx flag bits ?
> >
> > My thought was not about re-defining the flags (I think it is better to keep them intact),
> > but adding a union for one of rx-only fields (packet_type/rss/timestamp).
>
> Ok. Adding a union field at packet_type field is also fine like below.
>
> @@ -187,9 +187,10 @@ extern "C" {
> /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
>
> #define PKT_FIRST_FREE (1ULL << 23)
> -#define PKT_LAST_FREE (1ULL << 40)
> +#define PKT_LAST_FREE (1ULL << 39)
>
> /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> +#define PKT_TX_MARK_EN (1ULL << 40)
>
> /**
> * Outer UDP checksum offload flag. This flag is used for enabling
> @@ -461,6 +462,14 @@ enum {
> #endif
> };
>
> +/* Tx packet marking flags in rte_mbuf::tx_mark.
> + * Valid only when PKT_TX_MARK_EN is set in
> + * rte_mbuf::ol_flags.
> + */
> +#define TX_MARK_VLAN_DEI (1ULL << 0)
> +#define TX_MARK_IP_DSCP (1ULL << 1)
> +#define TX_MARK_IP_ECN (1ULL << 2)
> +
> /**
> * The generic rte_mbuf, containing a packet mbuf.
> */
> @@ -543,6 +552,10 @@ struct rte_mbuf {
> };
> uint32_t inner_l4_type:4; /**< Inner L4 type. */
> };
> + struct {
> + uint32_t reserved:29;
> + uint32_t tx_mark:3;
> + };
> };
>
>
>
> Please correct me if this is not what you mean.
>
Yes, I thought about something like that.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 8:28 ` Olivier Matz
@ 2020-06-03 10:44 ` Nithin Dabilpuram
2020-06-03 11:38 ` Olivier Matz
0 siblings, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-06-03 10:44 UTC (permalink / raw)
To: Olivier Matz
Cc: Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> Hi,
>
> On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> > > Hi Jerin,
> > >
> > > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > > Can it probably be squeezed somehow?
> > > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > > Or might be some other approach.
> > > > > >
> > > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > > marking and and 3 bits reused from Rx offload flags area.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > > >
> > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > >
> > > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > > +
> > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > >
> > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > >
> > > > > > Is this fine ?
> > > > >
> > > > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > > > and reuse unused Rx flag bits ?
> > >
> > > My thought was not about re-defining the flags (I think it is better to keep them intact),
> > > but adding a union for one of rx-only fields (packet_type/rss/timestamp).
> >
> > Ok. Adding a union field at packet_type field is also fine like below.
> >
> > @@ -187,9 +187,10 @@ extern "C" {
> > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> >
> > #define PKT_FIRST_FREE (1ULL << 23)
> > -#define PKT_LAST_FREE (1ULL << 40)
> > +#define PKT_LAST_FREE (1ULL << 39)
> >
> > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > +#define PKT_TX_MARK_EN (1ULL << 40)
> >
> > /**
> > * Outer UDP checksum offload flag. This flag is used for enabling
> > @@ -461,6 +462,14 @@ enum {
> > #endif
> > };
> >
> > +/* Tx packet marking flags in rte_mbuf::tx_mark.
> > + * Valid only when PKT_TX_MARK_EN is set in
> > + * rte_mbuf::ol_flags.
> > + */
> > +#define TX_MARK_VLAN_DEI (1ULL << 0)
> > +#define TX_MARK_IP_DSCP (1ULL << 1)
> > +#define TX_MARK_IP_ECN (1ULL << 2)
> > +
> > /**
> > * The generic rte_mbuf, containing a packet mbuf.
> > */
> > @@ -543,6 +552,10 @@ struct rte_mbuf {
> > };
> > uint32_t inner_l4_type:4; /**< Inner L4 type. */
> > };
> > + struct {
> > + uint32_t reserved:29;
> > + uint32_t tx_mark:3;
> > + };
> > };
> >
> >
> >
> > Please correct me if this is not what you mean.
>
> I'm not a big fan of reusing Rx fields or flags for Tx.
> It's not obvious for an application than adding a tx_mark will overwrite
> the packet_type. I understand that the risk is limited because packet_type
> is Rx and the marks are Tx, but there is still one.
I'm also not a big fan but just wanted to take this approach so that,
it can both conserve space and also help fast path.
Reusing Rx area is however not a new thing as is already followed for
mbuf->txadapter field.
Apart from documentation issue, Is there any other issue or future
ramification with using Rx field's for Tx ?
If it is only about documentation, then we can add more documentation to make things clear.
>
> To summarize the different proposed approaches (please correct me if I'm wrong):
>
> a- add 3 Tx mbuf flags
> (-) consumes limited resource
>
> b- add 3 dynamic flags
> (-) slower
- Tx burst Vector implementation can't be done for this tx offload as
offset keeps changing.
>
> c- add 1 Tx flag and union with Rx field
> (-) exclusive with Rx field
> (-) still consumes one flag
>
> My preference is still b-, for these reasons:
>
> - There are many different DPDK use cases, and resources in mbuf is tight.
> Recent contributions (rte_flow and ice driver) already made use of dynamic
> fields/flags.
- Since RTE_FLOW metadata is 32-bit field, it is a clear candidate for
dynamic flags.
- ICE PMD's dynamic field is however a vendor specific field and only for
ICE PMD users.
In this case, it is just 1 bit out of 18 free bits available in ol_flags.
>
> - When I implemented the dynamic fields/flags feature, I did a test which
> showed that the cost of having a dynamic offset was few cycles (on my test
> platform, it was~3 cycles for reading a field and ~2 cycles for writing a
> field).
I think this cost is of the case where the address where the dyn_offset is
stored is already in cache as it needs to be read first.
>
> Regards,
> Olivier
>
> >
> > >
> > > >
> > > > + Techboard
> > > >
> > > > There is a related thread going on
> > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e=
> > > >
> > > > If there is no consensus on email, then I would like to add this item
> > > > to the next TB meeting.
> > >
> > > Ok, I'll add that to tomorrow meeting agenda.
> > > Konstantin
> > >
>
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 10:44 ` Nithin Dabilpuram
@ 2020-06-03 11:38 ` Olivier Matz
2020-06-03 12:52 ` Nithin Dabilpuram
2020-06-03 14:56 ` Thomas Monjalon
0 siblings, 2 replies; 36+ messages in thread
From: Olivier Matz @ 2020-06-03 11:38 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
Hi Nithin,
On Wed, Jun 03, 2020 at 04:14:14PM +0530, Nithin Dabilpuram wrote:
> On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> > Hi,
> >
> > On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> > > > Hi Jerin,
> > > >
> > > > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > > > Can it probably be squeezed somehow?
> > > > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > > > Or might be some other approach.
> > > > > > >
> > > > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > > > marking and and 3 bits reused from Rx offload flags area.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > > > >
> > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > >
> > > > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > > > +
> > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > > >
> > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > > >
> > > > > > > Is this fine ?
> > > > > >
> > > > > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > > > > and reuse unused Rx flag bits ?
> > > >
> > > > My thought was not about re-defining the flags (I think it is better to keep them intact),
> > > > but adding a union for one of rx-only fields (packet_type/rss/timestamp).
> > >
> > > Ok. Adding a union field at packet_type field is also fine like below.
> > >
> > > @@ -187,9 +187,10 @@ extern "C" {
> > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > >
> > > #define PKT_FIRST_FREE (1ULL << 23)
> > > -#define PKT_LAST_FREE (1ULL << 40)
> > > +#define PKT_LAST_FREE (1ULL << 39)
> > >
> > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > >
> > > /**
> > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > @@ -461,6 +462,14 @@ enum {
> > > #endif
> > > };
> > >
> > > +/* Tx packet marking flags in rte_mbuf::tx_mark.
> > > + * Valid only when PKT_TX_MARK_EN is set in
> > > + * rte_mbuf::ol_flags.
> > > + */
> > > +#define TX_MARK_VLAN_DEI (1ULL << 0)
> > > +#define TX_MARK_IP_DSCP (1ULL << 1)
> > > +#define TX_MARK_IP_ECN (1ULL << 2)
> > > +
> > > /**
> > > * The generic rte_mbuf, containing a packet mbuf.
> > > */
> > > @@ -543,6 +552,10 @@ struct rte_mbuf {
> > > };
> > > uint32_t inner_l4_type:4; /**< Inner L4 type. */
> > > };
> > > + struct {
> > > + uint32_t reserved:29;
> > > + uint32_t tx_mark:3;
> > > + };
> > > };
> > >
> > >
> > >
> > > Please correct me if this is not what you mean.
> >
> > I'm not a big fan of reusing Rx fields or flags for Tx.
> > It's not obvious for an application than adding a tx_mark will overwrite
> > the packet_type. I understand that the risk is limited because packet_type
> > is Rx and the marks are Tx, but there is still one.
>
> I'm also not a big fan but just wanted to take this approach so that,
> it can both conserve space and also help fast path.
> Reusing Rx area is however not a new thing as is already followed for
> mbuf->txadapter field.
Yes, and in my opinion this is something we should avoid when possible,
because it makes some features exclusive (ex: the big union with
sched/rss/adapter/usr/...).
> Apart from documentation issue, Is there any other issue or future
> ramification with using Rx field's for Tx ?
No, I don't see any other issue except the ones we already mentioned (doc, code clarity, ).
> If it is only about documentation, then we can add more documentation to make things clear.
> >
> > To summarize the different proposed approaches (please correct me if I'm wrong):
> >
> > a- add 3 Tx mbuf flags
> > (-) consumes limited resource
> >
> > b- add 3 dynamic flags
> > (-) slower
>
> - Tx burst Vector implementation can't be done for this tx offload as
> offset keeps changing.
A vector implementation can be done. But yes, it would be slower than
with a static flag.
> >
> > c- add 1 Tx flag and union with Rx field
> > (-) exclusive with Rx field
> > (-) still consumes one flag
> >
> > My preference is still b-, for these reasons:
> >
> > - There are many different DPDK use cases, and resources in mbuf is tight.
> > Recent contributions (rte_flow and ice driver) already made use of dynamic
> > fields/flags.
> - Since RTE_FLOW metadata is 32-bit field, it is a clear candidate for
> dynamic flags.
I'm not sure to get why it is a better candidate than packet marking.
You mean because it requires more room in mbuf?
> - ICE PMD's dynamic field is however a vendor specific field and only for
> ICE PMD users.
Yes, but ICE PMD users may be as important as packet marking users.
> In this case, it is just 1 bit out of 18 free bits available in ol_flags.
>
> >
> > - When I implemented the dynamic fields/flags feature, I did a test which
> > showed that the cost of having a dynamic offset was few cycles (on my test
> > platform, it was~3 cycles for reading a field and ~2 cycles for writing a
> > field).
>
> I think this cost is of the case where the address where the dyn_offset is
> stored is already in cache as it needs to be read first.
This fetch of the value (in case it is not in cache) can be done once per bulk,
so I'm not sure the impact would be high.
Regards,
Olivier
>
>
> >
> > Regards,
> > Olivier
> >
> > >
> > > >
> > > > >
> > > > > + Techboard
> > > > >
> > > > > There is a related thread going on
> > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e=
> > > > >
> > > > > If there is no consensus on email, then I would like to add this item
> > > > > to the next TB meeting.
> > > >
> > > > Ok, I'll add that to tomorrow meeting agenda.
> > > > Konstantin
> > > >
> >
> >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 11:38 ` Olivier Matz
@ 2020-06-03 12:52 ` Nithin Dabilpuram
2020-06-03 16:14 ` Slava Ovsiienko
2020-06-03 14:56 ` Thomas Monjalon
1 sibling, 1 reply; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-06-03 12:52 UTC (permalink / raw)
To: Olivier Matz
Cc: Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
On Wed, Jun 03, 2020 at 01:38:22PM +0200, Olivier Matz wrote:
> Hi Nithin,
>
> On Wed, Jun 03, 2020 at 04:14:14PM +0530, Nithin Dabilpuram wrote:
> > On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> > > Hi,
> > >
> > > On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > > > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> > > > > Hi Jerin,
> > > > >
> > > > > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > > > > Can it probably be squeezed somehow?
> > > > > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > > > > Or might be some other approach.
> > > > > > > >
> > > > > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > > > > marking and and 3 bits reused from Rx offload flags area.
> > > > > > > >
> > > > > > > > For example:
> > > > > > > >
> > > > > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > > > > >
> > > > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > > > > > >
> > > > > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > > > > +
> > > > > > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > > > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > > > >
> > > > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > > > >
> > > > > > > > Is this fine ?
> > > > > > >
> > > > > > > Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
> > > > > > > and reuse unused Rx flag bits ?
> > > > >
> > > > > My thought was not about re-defining the flags (I think it is better to keep them intact),
> > > > > but adding a union for one of rx-only fields (packet_type/rss/timestamp).
> > > >
> > > > Ok. Adding a union field at packet_type field is also fine like below.
> > > >
> > > > @@ -187,9 +187,10 @@ extern "C" {
> > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
> > > >
> > > > #define PKT_FIRST_FREE (1ULL << 23)
> > > > -#define PKT_LAST_FREE (1ULL << 40)
> > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > >
> > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE */
> > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > >
> > > > /**
> > > > * Outer UDP checksum offload flag. This flag is used for enabling
> > > > @@ -461,6 +462,14 @@ enum {
> > > > #endif
> > > > };
> > > >
> > > > +/* Tx packet marking flags in rte_mbuf::tx_mark.
> > > > + * Valid only when PKT_TX_MARK_EN is set in
> > > > + * rte_mbuf::ol_flags.
> > > > + */
> > > > +#define TX_MARK_VLAN_DEI (1ULL << 0)
> > > > +#define TX_MARK_IP_DSCP (1ULL << 1)
> > > > +#define TX_MARK_IP_ECN (1ULL << 2)
> > > > +
> > > > /**
> > > > * The generic rte_mbuf, containing a packet mbuf.
> > > > */
> > > > @@ -543,6 +552,10 @@ struct rte_mbuf {
> > > > };
> > > > uint32_t inner_l4_type:4; /**< Inner L4 type. */
> > > > };
> > > > + struct {
> > > > + uint32_t reserved:29;
> > > > + uint32_t tx_mark:3;
> > > > + };
> > > > };
> > > >
> > > >
> > > >
> > > > Please correct me if this is not what you mean.
> > >
> > > I'm not a big fan of reusing Rx fields or flags for Tx.
> > > It's not obvious for an application than adding a tx_mark will overwrite
> > > the packet_type. I understand that the risk is limited because packet_type
> > > is Rx and the marks are Tx, but there is still one.
> >
> > I'm also not a big fan but just wanted to take this approach so that,
> > it can both conserve space and also help fast path.
> > Reusing Rx area is however not a new thing as is already followed for
> > mbuf->txadapter field.
>
> Yes, and in my opinion this is something we should avoid when possible,
> because it makes some features exclusive (ex: the big union with
> sched/rss/adapter/usr/...).
>
> > Apart from documentation issue, Is there any other issue or future
> > ramification with using Rx field's for Tx ?
>
> No, I don't see any other issue except the ones we already mentioned (doc, code clarity, ).
>
> > If it is only about documentation, then we can add more documentation to make things clear.
> > >
> > > To summarize the different proposed approaches (please correct me if I'm wrong):
> > >
> > > a- add 3 Tx mbuf flags
> > > (-) consumes limited resource
> > >
> > > b- add 3 dynamic flags
> > > (-) slower
> >
> > - Tx burst Vector implementation can't be done for this tx offload as
> > offset keeps changing.
>
> A vector implementation can be done. But yes, it would be slower than
> with a static flag.
Very slow atleast in our HW as, we try to translate ol_flags to
HW descriptor flags in addition to extra operations to be done
like offset calculations etc.
So if we have fixed offsets, then it is easy to have static constant
128/256 bit words with offsets and use things like shuffle/table lookup
to reorganize multiple mbuf flags to descriptor fields in a single instruction.
>
> > >
> > > c- add 1 Tx flag and union with Rx field
> > > (-) exclusive with Rx field
> > > (-) still consumes one flag
> > >
> > > My preference is still b-, for these reasons:
> > >
> > > - There are many different DPDK use cases, and resources in mbuf is tight.
> > > Recent contributions (rte_flow and ice driver) already made use of dynamic
> > > fields/flags.
> > - Since RTE_FLOW metadata is 32-bit field, it is a clear candidate for
> > dynamic flags.
>
> I'm not sure to get why it is a better candidate than packet marking.
> You mean because it requires more room in mbuf?
Yes, I feel space consumption is one way to decide whether it should be
a dynfield or static field.
IMO, other parameter to judge could be whether the field definition/usage itself
is well know standard and is a part of RTE spec or its definition is vendor specific.
>
> > - ICE PMD's dynamic field is however a vendor specific field and only for
> > ICE PMD users.
>
> Yes, but ICE PMD users may be as important as packet marking users.
Agree, I only meant that the flag ICE PMD registered cannot be used for other PMD's
so by using dynamic field, we are avoiding wastage of a static field that is needed
only by one specific PMD irrespective of whether that PMD is probed or not.
>
> > In this case, it is just 1 bit out of 18 free bits available in ol_flags.
> >
> > >
> > > - When I implemented the dynamic fields/flags feature, I did a test which
> > > showed that the cost of having a dynamic offset was few cycles (on my test
> > > platform, it was~3 cycles for reading a field and ~2 cycles for writing a
> > > field).
> >
> > I think this cost is of the case where the address where the dyn_offset is
> > stored is already in cache as it needs to be read first.
>
> This fetch of the value (in case it is not in cache) can be done once per bulk,
> so I'm not sure the impact would be high.
Agreed, for bulk case offset loading should have less impact.
>
>
> Regards,
> Olivier
>
>
> >
> >
> > >
> > > Regards,
> > > Olivier
> > >
> > > >
> > > > >
> > > > > >
> > > > > > + Techboard
> > > > > >
> > > > > > There is a related thread going on
> > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e=
> > > > > >
> > > > > > If there is no consensus on email, then I would like to add this item
> > > > > > to the next TB meeting.
> > > > >
> > > > > Ok, I'll add that to tomorrow meeting agenda.
> > > > > Konstantin
> > > > >
> > >
> > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 11:38 ` Olivier Matz
2020-06-03 12:52 ` Nithin Dabilpuram
@ 2020-06-03 14:56 ` Thomas Monjalon
1 sibling, 0 replies; 36+ messages in thread
From: Thomas Monjalon @ 2020-06-03 14:56 UTC (permalink / raw)
To: Nithin Dabilpuram
Cc: dev, Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Yigit, Ferruh, Andrew Rybchenko, Ori Kam,
Burakov, Anatoly, Mcnamara, John, Kovacevic, Marko, dpdk-dev,
Jerin Jacob, Krzysztof Kanas, techboard, Olivier Matz
03/06/2020 13:38, Olivier Matz:
> On Wed, Jun 03, 2020 at 04:14:14PM +0530, Nithin Dabilpuram wrote:
> > On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> > > On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > > > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:
> > > > > > > > > I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
> > > > > > > > > Can it probably be squeezed somehow?
> > > > > > > > > Let say we reserve one flag that this information is present or not, and
> > > > > > > > > re-use one of rx-only fields for store additional information (packet_type, or so).
> > > > > > > > > Or might be some other approach.
> > > > > > > >
> > > > > > > > We are fine with this approach where we define one bit in Tx offloads for pkt
> > > > > > > > marking and and 3 bits reused from Rx offload flags area.
[...]
> > > I'm not a big fan of reusing Rx fields or flags for Tx.
> > > It's not obvious for an application than adding a tx_mark will overwrite
> > > the packet_type. I understand that the risk is limited because packet_type
> > > is Rx and the marks are Tx, but there is still one.
Mixing Rx and Tx info in the same field is a bad design pattern
which will create a lot of difficult bugs.
> > I'm also not a big fan but just wanted to take this approach so that,
> > it can both conserve space and also help fast path.
> > Reusing Rx area is however not a new thing as is already followed for
> > mbuf->txadapter field.
Yes there is a txadapter field union'ed with flow director and QoS.
This is a bad pattern that I highlighted in this presentation (slide 8):
https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/DynamicMbuf.pdf
> Yes, and in my opinion this is something we should avoid when possible,
> because it makes some features exclusive (ex: the big union with
> sched/rss/adapter/usr/...).
Yes, the "RSS union" must be cleaned-up, as some other mbuf parts.
> > Apart from documentation issue, Is there any other issue or future
> > ramification with using Rx field's for Tx ?
>
> No, I don't see any other issue except the ones we already mentioned
> (doc, code clarity, ).
"doc clarity" should be understood as the opposite of
"design leading inevitably to bugs".
> > If it is only about documentation, then we can add more documentation to make things clear.
More documentation won't make a bad design better, unfortunately.
> > > To summarize the different proposed approaches (please correct me if I'm wrong):
> > >
> > > a- add 3 Tx mbuf flags
> > > (-) consumes limited resource
> > >
> > > b- add 3 dynamic flags
> > > (-) slower
> >
> > - Tx burst Vector implementation can't be done for this tx offload as
> > offset keeps changing.
>
> A vector implementation can be done. But yes, it would be slower than
> with a static flag.
>
> > > c- add 1 Tx flag and union with Rx field
> > > (-) exclusive with Rx field
> > > (-) still consumes one flag
> > >
> > > My preference is still b-, for these reasons:
Me too, my preference is (b).
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 12:52 ` Nithin Dabilpuram
@ 2020-06-03 16:14 ` Slava Ovsiienko
2020-06-08 9:39 ` Nithin Dabilpuram
0 siblings, 1 reply; 36+ messages in thread
From: Slava Ovsiienko @ 2020-06-03 16:14 UTC (permalink / raw)
To: Nithin Dabilpuram, Olivier Matz
Cc: Ananyev, Konstantin, Jerin Jacob, Dumitrescu, Cristian,
Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Nithin Dabilpuram
> Sent: Wednesday, June 3, 2020 15:52
> To: Olivier Matz <olivier.matz@6wind.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Jerin Jacob
> <jerinjacobk@gmail.com>; Dumitrescu, Cristian
> <cristian.dumitrescu@intel.com>; Nithin Dabilpuram
> <nithind1988@gmail.com>; Thomas Monjalon <thomas@monjalon.net>;
> Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew Rybchenko
> <arybchenko@solarflare.com>; Ori Kam <orika@mellanox.com>; Burakov,
> Anatoly <anatoly.burakov@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; dpdk-dev <dev@dpdk.org>; Jerin Jacob
> <jerinj@marvell.com>; Krzysztof Kanas <kkanas@marvell.com>;
> techboard@dpdk.org
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for
> packet marking
>
> On Wed, Jun 03, 2020 at 01:38:22PM +0200, Olivier Matz wrote:
> > Hi Nithin,
> >
> > On Wed, Jun 03, 2020 at 04:14:14PM +0530, Nithin Dabilpuram wrote:
> > > On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> > > > Hi,
> > > >
> > > > On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > > > > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin
> wrote:
> > > > > > Hi Jerin,
> > > > > >
> > > > > > > > > > I also share Olivier's concern about consuming 3 bits in
> ol_flags for that feature.
> > > > > > > > > > Can it probably be squeezed somehow?
> > > > > > > > > > Let say we reserve one flag that this information is
> > > > > > > > > > present or not, and re-use one of rx-only fields for store
> additional information (packet_type, or so).
> > > > > > > > > > Or might be some other approach.
> > > > > > > > >
> > > > > > > > > We are fine with this approach where we define one bit
> > > > > > > > > in Tx offloads for pkt marking and and 3 bits reused from Rx
> offload flags area.
> > > > > > > > >
> > > > > > > > > For example:
> > > > > > > > >
> > > > > > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > > > > > >
> > > > > > > > > /* add new RX flags here, don't forget to update
> > > > > > > > > PKT_FIRST_FREE */
> > > > > > > > >
> > > > > > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > > > > > +
> > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23) -#define
> > > > > > > > > PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > > > > >
> > > > > > > > > /* add new TX flags here, don't forget to update
> > > > > > > > > PKT_LAST_FREE */
> > > > > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > > > > >
> > > > > > > > > Is this fine ?
> > > > > > > >
> > > > > > > > Any thoughts on this approach which uses only 1 bit in Tx
> > > > > > > > flags out of 18 and reuse unused Rx flag bits ?
> > > > > >
> > > > > > My thought was not about re-defining the flags (I think it is
> > > > > > better to keep them intact), but adding a union for one of rx-only
> fields (packet_type/rss/timestamp).
> > > > >
> > > > > Ok. Adding a union field at packet_type field is also fine like below.
> > > > >
> > > > > @@ -187,9 +187,10 @@ extern "C" {
> > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE
> > > > > */
> > > > >
> > > > > #define PKT_FIRST_FREE (1ULL << 23) -#define PKT_LAST_FREE
> > > > > (1ULL << 40)
> > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > >
> > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE
> > > > > */
> > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > >
> > > > > /**
> > > > > * Outer UDP checksum offload flag. This flag is used for
> > > > > enabling @@ -461,6 +462,14 @@ enum { #endif };
> > > > >
> > > > > +/* Tx packet marking flags in rte_mbuf::tx_mark.
> > > > > + * Valid only when PKT_TX_MARK_EN is set in
> > > > > + * rte_mbuf::ol_flags.
> > > > > + */
> > > > > +#define TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > +#define TX_MARK_IP_DSCP (1ULL << 1)
> > > > > +#define TX_MARK_IP_ECN (1ULL << 2)
> > > > > +
> > > > > /**
> > > > > * The generic rte_mbuf, containing a packet mbuf.
> > > > > */
> > > > > @@ -543,6 +552,10 @@ struct rte_mbuf {
> > > > > };
> > > > > uint32_t inner_l4_type:4; /**< Inner L4 type. */
> > > > > };
> > > > > + struct {
> > > > > + uint32_t reserved:29;
> > > > > + uint32_t tx_mark:3;
> > > > > + };
> > > > > };
> > > > >
> > > > >
> > > > >
> > > > > Please correct me if this is not what you mean.
> > > >
> > > > I'm not a big fan of reusing Rx fields or flags for Tx.
> > > > It's not obvious for an application than adding a tx_mark will
> > > > overwrite the packet_type. I understand that the risk is limited
> > > > because packet_type is Rx and the marks are Tx, but there is still one.
> > >
> > > I'm also not a big fan but just wanted to take this approach so
> > > that, it can both conserve space and also help fast path.
> > > Reusing Rx area is however not a new thing as is already followed
> > > for
> > > mbuf->txadapter field.
> >
> > Yes, and in my opinion this is something we should avoid when
> > possible, because it makes some features exclusive (ex: the big union
> > with sched/rss/adapter/usr/...).
> >
> > > Apart from documentation issue, Is there any other issue or future
> > > ramification with using Rx field's for Tx ?
> >
> > No, I don't see any other issue except the ones we already mentioned (doc,
> code clarity, ).
> >
> > > If it is only about documentation, then we can add more documentation
> to make things clear.
> > > >
> > > > To summarize the different proposed approaches (please correct me if
> I'm wrong):
> > > >
> > > > a- add 3 Tx mbuf flags
> > > > (-) consumes limited resource
> > > >
> > > > b- add 3 dynamic flags
> > > > (-) slower
> > >
> > > - Tx burst Vector implementation can't be done for this tx offload as
> > > offset keeps changing.
> >
> > A vector implementation can be done. But yes, it would be slower than
> > with a static flag.
>
> Very slow atleast in our HW as, we try to translate ol_flags to HW descriptor
> flags in addition to extra operations to be done like offset calculations etc.
The dynamic flag offset is not subject to be changed after registration.
So, flag offset can be converted once into appropriate mask and stored locally by PMD for further using in vector instructions.
The only difference - loaded variable mask instead of constant one, should not affect performance too much.
With the cmpeq/shifts the and results can be converted to any desired predefined mask (like static one).
>
> So if we have fixed offsets, then it is easy to have static constant
BTW, how this constant is loaded?
I suppose on x86 it would be rather mm_load?_si128(*constant array) - no difference
between dynamic and static masks at all. BTW, there is the hint - to make local copy of the
mask/offset in order to avoid cache-line concurrency in global variable storage.
> 128/256 bit words with offsets and use things like shuffle/table lookup to
> reorganize multiple mbuf flags to descriptor fields in a single instruction.
Offsets in the HW descriptor remain the fixed ones, so shuffle would still work OK.
Do you already have some vectorized implementation? It would be curious to have a look at.
I strongly share the concern about defining the static mbuf flags,
we should consider all ways to avoid doing this.
WBR, Slava
>
> >
> > > >
> > > > c- add 1 Tx flag and union with Rx field
> > > > (-) exclusive with Rx field
> > > > (-) still consumes one flag
> > > >
> > > > My preference is still b-, for these reasons:
> > > >
> > > > - There are many different DPDK use cases, and resources in mbuf is
> tight.
> > > > Recent contributions (rte_flow and ice driver) already made use of
> dynamic
> > > > fields/flags.
> > > - Since RTE_FLOW metadata is 32-bit field, it is a clear candidate
> > > for dynamic flags.
> >
> > I'm not sure to get why it is a better candidate than packet marking.
> > You mean because it requires more room in mbuf?
>
> Yes, I feel space consumption is one way to decide whether it should be a
> dynfield or static field.
>
> IMO, other parameter to judge could be whether the field definition/usage
> itself is well know standard and is a part of RTE spec or its definition is
> vendor specific.
>
> >
> > > - ICE PMD's dynamic field is however a vendor specific field and
> > > only for ICE PMD users.
> >
> > Yes, but ICE PMD users may be as important as packet marking users.
>
> Agree, I only meant that the flag ICE PMD registered cannot be used for
> other PMD's so by using dynamic field, we are avoiding wastage of a static
> field that is needed only by one specific PMD irrespective of whether that
> PMD is probed or not.
>
> >
> > > In this case, it is just 1 bit out of 18 free bits available in ol_flags.
> > >
> > > >
> > > > - When I implemented the dynamic fields/flags feature, I did a test
> which
> > > > showed that the cost of having a dynamic offset was few cycles (on
> my test
> > > > platform, it was~3 cycles for reading a field and ~2 cycles for writing a
> > > > field).
> > >
> > > I think this cost is of the case where the address where the
> > > dyn_offset is stored is already in cache as it needs to be read first.
> >
> > This fetch of the value (in case it is not in cache) can be done once
> > per bulk, so I'm not sure the impact would be high.
>
> Agreed, for bulk case offset loading should have less impact.
>
> >
> >
> > Regards,
> > Olivier
> >
> >
> > >
> > >
> > > >
> > > > Regards,
> > > > Olivier
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > + Techboard
> > > > > > >
> > > > > > > There is a related thread going on
> > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A
> > > > > > > %2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-
> 3A__ma
> > > > > > > ils.dpdk.org_archives_dev_2020-
> 2DMay_168810.html%26d%3DDwIGa
> > > > > > >
> Q%26c%3DnKjWec2b6R0mOyPaz7xtfQ%26r%3DFZ_tPCbgFOh18zwRPO9H0y
> D
> > > > > > >
> x8VW38vuapifdDfc8SFQ%26m%3DnyV4Rud03HW6DbWMpyvOCulQNkagmf
> o0w
> > > > > > >
> KtrwQ7zmmg%26s%3DVuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s
> > > > > > >
> %26e%3D&data=02%7C01%7Cviacheslavo%40mellanox.com%7C5e7a
> > > > > > >
> 9c93fd684e09267108d807bd0160%7Ca652971c7d2e4d9ba6a4d149256f4
> > > > > > >
> 61b%7C0%7C0%7C637267855650797843&sdata=r%2B0JIDapZocl6DD
> > > > > > > A8wbNd4LV0CX6zEdDoQJhBpMoELw%3D&reserved=0
> > > > > > >
> > > > > > > If there is no consensus on email, then I would like to add
> > > > > > > this item to the next TB meeting.
> > > > > >
> > > > > > Ok, I'll add that to tomorrow meeting agenda.
> > > > > > Konstantin
> > > > > >
> > > >
> > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-06-03 16:14 ` Slava Ovsiienko
@ 2020-06-08 9:39 ` Nithin Dabilpuram
0 siblings, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2020-06-08 9:39 UTC (permalink / raw)
To: Slava Ovsiienko
Cc: Olivier Matz, Ananyev, Konstantin, Jerin Jacob, Dumitrescu,
Cristian, Nithin Dabilpuram, Thomas Monjalon, Yigit, Ferruh,
Andrew Rybchenko, Ori Kam, Burakov, Anatoly, Mcnamara, John,
Kovacevic, Marko, dpdk-dev, Jerin Jacob, Krzysztof Kanas,
techboard
On Wed, Jun 03, 2020 at 04:14:13PM +0000, Slava Ovsiienko wrote:
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Nithin Dabilpuram
> > Sent: Wednesday, June 3, 2020 15:52
> > To: Olivier Matz <olivier.matz@6wind.com>
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Jerin Jacob
> > <jerinjacobk@gmail.com>; Dumitrescu, Cristian
> > <cristian.dumitrescu@intel.com>; Nithin Dabilpuram
> > <nithind1988@gmail.com>; Thomas Monjalon <thomas@monjalon.net>;
> > Yigit, Ferruh <ferruh.yigit@intel.com>; Andrew Rybchenko
> > <arybchenko@solarflare.com>; Ori Kam <orika@mellanox.com>; Burakov,
> > Anatoly <anatoly.burakov@intel.com>; Mcnamara, John
> > <john.mcnamara@intel.com>; Kovacevic, Marko
> > <marko.kovacevic@intel.com>; dpdk-dev <dev@dpdk.org>; Jerin Jacob
> > <jerinj@marvell.com>; Krzysztof Kanas <kkanas@marvell.com>;
> > techboard@dpdk.org
> > Subject: Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for
> > packet marking
> >
> > On Wed, Jun 03, 2020 at 01:38:22PM +0200, Olivier Matz wrote:
> > > Hi Nithin,
> > >
> > > On Wed, Jun 03, 2020 at 04:14:14PM +0530, Nithin Dabilpuram wrote:
> > > > On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:
> > > > > Hi,
> > > > >
> > > > > On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:
> > > > > > On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin
> > wrote:
> > > > > > > Hi Jerin,
> > > > > > >
> > > > > > > > > > > I also share Olivier's concern about consuming 3 bits in
> > ol_flags for that feature.
> > > > > > > > > > > Can it probably be squeezed somehow?
> > > > > > > > > > > Let say we reserve one flag that this information is
> > > > > > > > > > > present or not, and re-use one of rx-only fields for store
> > additional information (packet_type, or so).
> > > > > > > > > > > Or might be some other approach.
> > > > > > > > > >
> > > > > > > > > > We are fine with this approach where we define one bit
> > > > > > > > > > in Tx offloads for pkt marking and and 3 bits reused from Rx
> > offload flags area.
> > > > > > > > > >
> > > > > > > > > > For example:
> > > > > > > > > >
> > > > > > > > > > @@ -186,10 +186,16 @@ extern "C" {
> > > > > > > > > >
> > > > > > > > > > /* add new RX flags here, don't forget to update
> > > > > > > > > > PKT_FIRST_FREE */
> > > > > > > > > >
> > > > > > > > > > +/* Reused Rx offload bits for Tx offloads */
> > > > > > > > > > +#define PKT_X_TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > > > > > +#define PKT_X_TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > > > > > +#define PKT_X_TX_MARK_IP_ECN (1ULL << 2)
> > > > > > > > > > +
> > > > > > > > > > #define PKT_FIRST_FREE (1ULL << 23) -#define
> > > > > > > > > > PKT_LAST_FREE (1ULL << 40)
> > > > > > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > > > > > >
> > > > > > > > > > /* add new TX flags here, don't forget to update
> > > > > > > > > > PKT_LAST_FREE */
> > > > > > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > > > > > >
> > > > > > > > > > Is this fine ?
> > > > > > > > >
> > > > > > > > > Any thoughts on this approach which uses only 1 bit in Tx
> > > > > > > > > flags out of 18 and reuse unused Rx flag bits ?
> > > > > > >
> > > > > > > My thought was not about re-defining the flags (I think it is
> > > > > > > better to keep them intact), but adding a union for one of rx-only
> > fields (packet_type/rss/timestamp).
> > > > > >
> > > > > > Ok. Adding a union field at packet_type field is also fine like below.
> > > > > >
> > > > > > @@ -187,9 +187,10 @@ extern "C" {
> > > > > > /* add new RX flags here, don't forget to update PKT_FIRST_FREE
> > > > > > */
> > > > > >
> > > > > > #define PKT_FIRST_FREE (1ULL << 23) -#define PKT_LAST_FREE
> > > > > > (1ULL << 40)
> > > > > > +#define PKT_LAST_FREE (1ULL << 39)
> > > > > >
> > > > > > /* add new TX flags here, don't forget to update PKT_LAST_FREE
> > > > > > */
> > > > > > +#define PKT_TX_MARK_EN (1ULL << 40)
> > > > > >
> > > > > > /**
> > > > > > * Outer UDP checksum offload flag. This flag is used for
> > > > > > enabling @@ -461,6 +462,14 @@ enum { #endif };
> > > > > >
> > > > > > +/* Tx packet marking flags in rte_mbuf::tx_mark.
> > > > > > + * Valid only when PKT_TX_MARK_EN is set in
> > > > > > + * rte_mbuf::ol_flags.
> > > > > > + */
> > > > > > +#define TX_MARK_VLAN_DEI (1ULL << 0)
> > > > > > +#define TX_MARK_IP_DSCP (1ULL << 1)
> > > > > > +#define TX_MARK_IP_ECN (1ULL << 2)
> > > > > > +
> > > > > > /**
> > > > > > * The generic rte_mbuf, containing a packet mbuf.
> > > > > > */
> > > > > > @@ -543,6 +552,10 @@ struct rte_mbuf {
> > > > > > };
> > > > > > uint32_t inner_l4_type:4; /**< Inner L4 type. */
> > > > > > };
> > > > > > + struct {
> > > > > > + uint32_t reserved:29;
> > > > > > + uint32_t tx_mark:3;
> > > > > > + };
> > > > > > };
> > > > > >
> > > > > >
> > > > > >
> > > > > > Please correct me if this is not what you mean.
> > > > >
> > > > > I'm not a big fan of reusing Rx fields or flags for Tx.
> > > > > It's not obvious for an application than adding a tx_mark will
> > > > > overwrite the packet_type. I understand that the risk is limited
> > > > > because packet_type is Rx and the marks are Tx, but there is still one.
> > > >
> > > > I'm also not a big fan but just wanted to take this approach so
> > > > that, it can both conserve space and also help fast path.
> > > > Reusing Rx area is however not a new thing as is already followed
> > > > for
> > > > mbuf->txadapter field.
> > >
> > > Yes, and in my opinion this is something we should avoid when
> > > possible, because it makes some features exclusive (ex: the big union
> > > with sched/rss/adapter/usr/...).
> > >
> > > > Apart from documentation issue, Is there any other issue or future
> > > > ramification with using Rx field's for Tx ?
> > >
> > > No, I don't see any other issue except the ones we already mentioned (doc,
> > code clarity, ).
> > >
> > > > If it is only about documentation, then we can add more documentation
> > to make things clear.
> > > > >
> > > > > To summarize the different proposed approaches (please correct me if
> > I'm wrong):
> > > > >
> > > > > a- add 3 Tx mbuf flags
> > > > > (-) consumes limited resource
> > > > >
> > > > > b- add 3 dynamic flags
> > > > > (-) slower
> > > >
> > > > - Tx burst Vector implementation can't be done for this tx offload as
> > > > offset keeps changing.
> > >
> > > A vector implementation can be done. But yes, it would be slower than
> > > with a static flag.
> >
> > Very slow atleast in our HW as, we try to translate ol_flags to HW descriptor
> > flags in addition to extra operations to be done like offset calculations etc.
>
> The dynamic flag offset is not subject to be changed after registration.
> So, flag offset can be converted once into appropriate mask and stored locally by PMD for further using in vector instructions.
> The only difference - loaded variable mask instead of constant one, should not affect performance too much.
> With the cmpeq/shifts the and results can be converted to any desired predefined mask (like static one).
>
> >
> > So if we have fixed offsets, then it is easy to have static constant
>
> BTW, how this constant is loaded?
> I suppose on x86 it would be rather mm_load?_si128(*constant array) - no difference
> between dynamic and static masks at all. BTW, there is the hint - to make local copy of the
> mask/offset in order to avoid cache-line concurrency in global variable storage.
>
> > 128/256 bit words with offsets and use things like shuffle/table lookup to
> > reorganize multiple mbuf flags to descriptor fields in a single instruction.
>
> Offsets in the HW descriptor remain the fixed ones, so shuffle would still work OK.
> Do you already have some vectorized implementation? It would be curious to have a look at.
Agree that the constants need to be loaded from memory, but there is also a
logic created onto transforming that data to descriptor which is not always straight forward.
For example, see otx2_tx.c,
// mbuf has tx offload field L2_LEN of 7 bits, L3_LEN of 9 bits which are not on byte boundary.
// Below are the transformations done to get them to byte boundary.
/* Get tx_offload for ol2, ol3, l2, l3 lengths from mbuf*/
/*
* Operation result:
* E(8):OL2_LEN(7):OL3_LEN(9):E(24):L3_LEN(9):L2_LEN(7)
* E(8):OL2_LEN(7):OL3_LEN(9):E(24):L3_LEN(9):L2_LEN(7)
*/
asm volatile ("LD1 {%[a].D}[0],[%[in]]\n\t" :
[a]"+w"(senddesc01_w1) :
[in]"r"(mbuf0 + 2) : "memory");
/*
* Operation Result:
* E(47):L3_LEN(9):L2_LEN(7+z)
* E(47):L3_LEN(9):L2_LEN(7+z)
*/
senddesc01_w1 = vshlq_n_u64(senddesc01_w1, 1);
senddesc23_w1 = vshlq_n_u64(senddesc23_w1, 1);
/*
* Result:
* E(48):L3_LEN(8):L2_LEN(z+7)
* E(48):L3_LEN(8):L2_LEN(z+7)
*/
const int8x16_t tshft3 = {
-1, 0, 8, 8, 8, 8, 8, 8,
-1, 0, 8, 8, 8, 8, 8, 8,
};
senddesc01_w1 = vshlq_u8(senddesc01_w1, tshft3);
senddesc23_w1 = vshlq_u8(senddesc23_w1, tshft3);
We cannot get all the above logic done runtime for every position of mbuf->l2_len if
the field position keeps changing for every application run.
>
> I strongly share the concern about defining the static mbuf flags,
> we should consider all ways to avoid doing this.
>
> WBR, Slava
> >
> > >
> > > > >
> > > > > c- add 1 Tx flag and union with Rx field
> > > > > (-) exclusive with Rx field
> > > > > (-) still consumes one flag
> > > > >
> > > > > My preference is still b-, for these reasons:
> > > > >
> > > > > - There are many different DPDK use cases, and resources in mbuf is
> > tight.
> > > > > Recent contributions (rte_flow and ice driver) already made use of
> > dynamic
> > > > > fields/flags.
> > > > - Since RTE_FLOW metadata is 32-bit field, it is a clear candidate
> > > > for dynamic flags.
> > >
> > > I'm not sure to get why it is a better candidate than packet marking.
> > > You mean because it requires more room in mbuf?
> >
> > Yes, I feel space consumption is one way to decide whether it should be a
> > dynfield or static field.
> >
> > IMO, other parameter to judge could be whether the field definition/usage
> > itself is well know standard and is a part of RTE spec or its definition is
> > vendor specific.
> >
> > >
> > > > - ICE PMD's dynamic field is however a vendor specific field and
> > > > only for ICE PMD users.
> > >
> > > Yes, but ICE PMD users may be as important as packet marking users.
> >
> > Agree, I only meant that the flag ICE PMD registered cannot be used for
> > other PMD's so by using dynamic field, we are avoiding wastage of a static
> > field that is needed only by one specific PMD irrespective of whether that
> > PMD is probed or not.
> >
> > >
> > > > In this case, it is just 1 bit out of 18 free bits available in ol_flags.
> > > >
> > > > >
> > > > > - When I implemented the dynamic fields/flags feature, I did a test
> > which
> > > > > showed that the cost of having a dynamic offset was few cycles (on
> > my test
> > > > > platform, it was~3 cycles for reading a field and ~2 cycles for writing a
> > > > > field).
> > > >
> > > > I think this cost is of the case where the address where the
> > > > dyn_offset is stored is already in cache as it needs to be read first.
> > >
> > > This fetch of the value (in case it is not in cache) can be done once
> > > per bulk, so I'm not sure the impact would be high.
> >
> > Agreed, for bulk case offset loading should have less impact.
> >
> > >
> > >
> > > Regards,
> > > Olivier
> > >
> > >
> > > >
> > > >
> > > > >
> > > > > Regards,
> > > > > Olivier
> > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > + Techboard
> > > > > > > >
> > > > > > > > There is a related thread going on
> > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__eur03.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A&d=DwIFAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=5IzeBXssneRLE7EeAyOUnytITHfcmhQu-eRLr18e5U0&s=aCn4On5jrJAfrGkQm_ftPodlBQo3QiozQM76MU9S8j8&e=
> > > > > > > > %2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-
> > 3A__ma
> > > > > > > > ils.dpdk.org_archives_dev_2020-
> > 2DMay_168810.html%26d%3DDwIGa
> > > > > > > >
> > Q%26c%3DnKjWec2b6R0mOyPaz7xtfQ%26r%3DFZ_tPCbgFOh18zwRPO9H0y
> > D
> > > > > > > >
> > x8VW38vuapifdDfc8SFQ%26m%3DnyV4Rud03HW6DbWMpyvOCulQNkagmf
> > o0w
> > > > > > > >
> > KtrwQ7zmmg%26s%3DVuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s
> > > > > > > >
> > %26e%3D&data=02%7C01%7Cviacheslavo%40mellanox.com%7C5e7a
> > > > > > > >
> > 9c93fd684e09267108d807bd0160%7Ca652971c7d2e4d9ba6a4d149256f4
> > > > > > > >
> > 61b%7C0%7C0%7C637267855650797843&sdata=r%2B0JIDapZocl6DD
> > > > > > > > A8wbNd4LV0CX6zEdDoQJhBpMoELw%3D&reserved=0
> > > > > > > >
> > > > > > > > If there is no consensus on email, then I would like to add
> > > > > > > > this item to the next TB meeting.
> > > > > > >
> > > > > > > Ok, I'll add that to tomorrow meeting agenda.
> > > > > > > Konstantin
> > > > > > >
> > > > >
> > > > >
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking
2020-04-17 7:22 [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Nithin Dabilpuram
` (2 preceding siblings ...)
2020-05-01 11:18 ` [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Jerin Jacob
@ 2023-07-31 12:54 ` Thomas Monjalon
2023-08-14 8:11 ` Nithin Dabilpuram
3 siblings, 1 reply; 36+ messages in thread
From: Thomas Monjalon @ 2023-07-31 12:54 UTC (permalink / raw)
To: jerinj, Nithin Dabilpuram
Cc: ferruh.yigit, arybchenko, orika, cristian.dumitrescu,
anatoly.burakov, John McNamara, Marko Kovacevic, Olivier Matz,
dev, kkanas, Nithin Dabilpuram
Please can you remind what was decided for this feature?
I think I should remove it from the roadmap.
17/04/2020 09:22, Nithin Dabilpuram:
> From: Nithin Dabilpuram <ndabilpuram@marvell.com>
>
> Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> packet marking.
>
> When packet marking feature in Traffic manager is enabled,
> application has to the use the three new flags to indicate
> to PMD on whether packet marking needs to be enabled on the
> specific mbuf or not. By setting the three flags, it is
> assumed by PMD that application has already verified the
> applicability of marking on that specific packet and
> PMD need not perform further checks as per RFC.
>
> Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking
2023-07-31 12:54 ` [dpdk-dev] " Thomas Monjalon
@ 2023-08-14 8:11 ` Nithin Dabilpuram
0 siblings, 0 replies; 36+ messages in thread
From: Nithin Dabilpuram @ 2023-08-14 8:11 UTC (permalink / raw)
To: Thomas Monjalon
Cc: jerinj, Nithin Dabilpuram, ferruh.yigit, arybchenko, orika,
cristian.dumitrescu, anatoly.burakov, John McNamara,
Marko Kovacevic, Olivier Matz, dev, kkanas
At last it was suggested to use dynamic ol_flags in mbuf for this
feature. We decided to postpone this since it effects our fast path
performance.
You can remove it from the roadmap.
Thanks
Nithin
On Mon, Jul 31, 2023 at 6:24 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Please can you remind what was decided for this feature?
> I think I should remove it from the roadmap.
>
>
> 17/04/2020 09:22, Nithin Dabilpuram:
> > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> >
> > Introduce PKT_TX_MARK_IP_DSCP, PKT_TX_MARK_IP_ECN
> > and PKT_TX_MARK_VLAN_DEI Tx offload flags to support
> > packet marking.
> >
> > When packet marking feature in Traffic manager is enabled,
> > application has to the use the three new flags to indicate
> > to PMD on whether packet marking needs to be enabled on the
> > specific mbuf or not. By setting the three flags, it is
> > assumed by PMD that application has already verified the
> > applicability of marking on that specific packet and
> > PMD need not perform further checks as per RFC.
> >
> > Signed-off-by: Krzysztof Kanas <kkanas@marvell.com>
> > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>
>
>
>
>
^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2023-08-14 8:12 UTC | newest]
Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-17 7:22 [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 2/3] net/octeontx2: add tm packet marking cb Nithin Dabilpuram
2020-04-17 7:22 ` [dpdk-dev] [PATCH 3/3] net/octeontx2: add Tx packet marking offload support Nithin Dabilpuram
2020-05-01 11:18 ` [dpdk-dev] [PATCH 1/3] mbuf: add Tx offloads for packet marking Jerin Jacob
2020-05-04 8:06 ` Olivier Matz
2020-05-04 8:27 ` [dpdk-dev] [EXT] " Nithin Dabilpuram
2020-05-04 9:16 ` Olivier Matz
2020-05-04 10:04 ` Nithin Dabilpuram
2020-05-04 12:27 ` Olivier Matz
2020-05-05 6:19 ` Nithin Dabilpuram
2020-05-13 12:28 ` Nithin Dabilpuram
2020-05-14 20:29 ` Olivier Matz
2020-05-15 10:08 ` Nithin Dabilpuram
2020-05-15 10:30 ` Ananyev, Konstantin
2020-05-15 13:57 ` Nithin Dabilpuram
2020-05-28 15:43 ` Nithin Dabilpuram
2020-05-30 15:12 ` Jerin Jacob
2020-06-02 10:53 ` Ananyev, Konstantin
2020-06-02 14:25 ` Nithin Dabilpuram
2020-06-03 8:28 ` Olivier Matz
2020-06-03 10:44 ` Nithin Dabilpuram
2020-06-03 11:38 ` Olivier Matz
2020-06-03 12:52 ` Nithin Dabilpuram
2020-06-03 16:14 ` Slava Ovsiienko
2020-06-08 9:39 ` Nithin Dabilpuram
2020-06-03 14:56 ` Thomas Monjalon
2020-06-03 10:31 ` Ananyev, Konstantin
2020-05-15 13:12 ` Thomas Monjalon
2020-05-15 13:44 ` Nithin Dabilpuram
2020-05-15 15:10 ` Thomas Monjalon
2020-05-15 16:26 ` Jerin Jacob
2020-05-15 16:52 ` Thomas Monjalon
2020-05-15 17:00 ` Jerin Jacob
2020-05-15 18:07 ` Nithin Dabilpuram
2023-07-31 12:54 ` [dpdk-dev] " Thomas Monjalon
2023-08-14 8:11 ` Nithin Dabilpuram
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).