[dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Akhil Goyal]

From: Nithin Dabilpuram <ndabilpuram@marvell.com>

For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
set, rte_security_set_pkt_metadata() needs to be called for pkts
to associate a Security session with a mbuf before submitting
to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
set some opaque metadata in mbuf for PMD's use.
This patch updates documentation that rte_security_set_pkt_metadata()
should be called only with mbuf containing Layer 3 and above data.
This behaviour is consistent with existing PMD's such as ixgbe.

On Tx, not all net PMD's/HW can parse packet and identify
L2 header and L3 header locations on Tx. This is inline with other
Tx offloads requirements such as L3 checksum, L4 checksum offload,
etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
HW to be able to generate checksum. Since Inline IPSec is also
such a Tx offload, some PMD's at least need mbuf.l2_len to be
valid to find L3 header and perform Outbound IPSec processing.
Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Reviewed-by: Akhil Goyal <gakhil@marvell.com>
---
 doc/guides/nics/features.rst           | 2 ++
 doc/guides/prog_guide/rte_security.rst | 6 +++++-
 lib/mbuf/rte_mbuf_core.h               | 2 ++
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index 403c2b03a..414baf14f 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
index f72bc8a78..7b68c698d 100644
--- a/doc/guides/prog_guide/rte_security.rst
+++ b/doc/guides/prog_guide/rte_security.rst
@@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
 
 For Inline Crypto and Inline protocol offload, device specific defined metadata is
 updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
-``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
+``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
+should be called on mbuf only with Layer 3 and above data present and
+``mbuf.data_off`` should be pointing to Layer 3 Header. Once called,
+Layer 3 and above data cannot be modified or moved around unless
+``rte_security_set_pkt_metadata()`` is called again.
 
 For inline protocol offloaded ingress traffic, the application can register a
 pointer, ``userdata`` , in the security session. When the packet is received,
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f58..9d8e3ddc8 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.25.1

[dpdk-dev] [PATCH 2/2] examples/ipsec-secgw: modify event mode inline path

[Akhil Goyal]

From: Nithin Dabilpuram <ndabilpuram@marvell.com>

Align event mode path for Tx inline IPsec processing to adhere to
security spec. Call rte_security_set_pkt_metadata() only with
mbuf containing L3 header and above. Also update mbuf.l2_len
with L2 header size.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Reviewed-by: Akhil Goyal <gakhil@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 50 +++++++++++++++++++++--------
 2 files changed, 38 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d3498..7ad94cb82 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22df5..401fd6186 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -43,6 +48,8 @@ update_mac_addrs(struct rte_mbuf *pkt, uint16_t portid)
 {
 	struct rte_ether_hdr *ethhdr;
 
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	ethhdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *);
 	memcpy(&ethhdr->s_addr, &ethaddr_tbl[portid].src, RTE_ETHER_ADDR_LEN);
 	memcpy(&ethhdr->d_addr, &ethaddr_tbl[portid].dst, RTE_ETHER_ADDR_LEN);
@@ -60,7 +67,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +103,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +365,11 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	/* Remove L2 header before metadata set */
+	rte_pktmbuf_adj(pkt, RTE_ETHER_HDR_LEN);
+
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -366,6 +377,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	/* Get the port to which this pkt need to be submitted */
 	port_id = sa->portid;
 
+	/* Add L2 header for processing */
+	rte_pktmbuf_prepend(pkt, RTE_ETHER_HDR_LEN);
+
 send_pkt:
 	/* Update mac addresses */
 	update_mac_addrs(pkt, port_id);
@@ -398,7 +412,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +426,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +438,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +476,25 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
+			/* Remove L2 header before metadata set */
+			rte_pktmbuf_adj(pkt, RTE_ETHER_HDR_LEN);
+
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Add L2 header for processing */
+			rte_pktmbuf_prepend(pkt, RTE_ETHER_HDR_LEN);
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.25.1

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Akhil Goyal]

Hi Konstantin/ Olivier,

Could you please review this patch? This has also an update in documentation of mbuf.

Regards, 
Akhil

> From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> 
> For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> set, rte_security_set_pkt_metadata() needs to be called for pkts
> to associate a Security session with a mbuf before submitting
> to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> set some opaque metadata in mbuf for PMD's use.
> This patch updates documentation that rte_security_set_pkt_metadata()
> should be called only with mbuf containing Layer 3 and above data.
> This behaviour is consistent with existing PMD's such as ixgbe.
> 
> On Tx, not all net PMD's/HW can parse packet and identify
> L2 header and L3 header locations on Tx. This is inline with other
> Tx offloads requirements such as L3 checksum, L4 checksum offload,
> etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> HW to be able to generate checksum. Since Inline IPSec is also
> such a Tx offload, some PMD's at least need mbuf.l2_len to be
> valid to find L3 header and perform Outbound IPSec processing.
> Hence, this patch updates documentation to enforce setting
> mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> for Inline IPSec Crypto / Protocol offload processing to
> work on Tx.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Reviewed-by: Akhil Goyal <gakhil@marvell.com>

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> 
> From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> 
> For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> set, rte_security_set_pkt_metadata() needs to be called for pkts
> to associate a Security session with a mbuf before submitting
> to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> set some opaque metadata in mbuf for PMD's use.
> This patch updates documentation that rte_security_set_pkt_metadata()
> should be called only with mbuf containing Layer 3 and above data.
> This behaviour is consistent with existing PMD's such as ixgbe.
> 
> On Tx, not all net PMD's/HW can parse packet and identify
> L2 header and L3 header locations on Tx. This is inline with other
> Tx offloads requirements such as L3 checksum, L4 checksum offload,
> etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> HW to be able to generate checksum. Since Inline IPSec is also
> such a Tx offload, some PMD's at least need mbuf.l2_len to be
> valid to find L3 header and perform Outbound IPSec processing.
> Hence, this patch updates documentation to enforce setting
> mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> for Inline IPSec Crypto / Protocol offload processing to
> work on Tx.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  doc/guides/nics/features.rst           | 2 ++
>  doc/guides/prog_guide/rte_security.rst | 6 +++++-
>  lib/mbuf/rte_mbuf_core.h               | 2 ++
>  3 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index 403c2b03a..414baf14f 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> 
>  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
>  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> +* **[uses]       mbuf**: ``mbuf.l2_len``.
>  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
>    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
>  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> 
>  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
>  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> +* **[uses]       mbuf**: ``mbuf.l2_len``.
>  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
>    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
>    ``capabilities_get``.
> diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> index f72bc8a78..7b68c698d 100644
> --- a/doc/guides/prog_guide/rte_security.rst
> +++ b/doc/guides/prog_guide/rte_security.rst
> @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> 
>  For Inline Crypto and Inline protocol offload, device specific defined metadata is
>  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> +should be called on mbuf only with Layer 3 and above data present and
> +``mbuf.data_off`` should be pointing to Layer 3 Header.

Hmm... not sure why mbuf.data_off should point to L3 hdr.
Who will add L2 hdr to the packet in that case?
Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?

> Once called,
> +Layer 3 and above data cannot be modified or moved around unless
> +``rte_security_set_pkt_metadata()`` is called again.
> 
>  For inline protocol offloaded ingress traffic, the application can register a
>  pointer, ``userdata`` , in the security session. When the packet is received,
> diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> index bb38d7f58..9d8e3ddc8 100644
> --- a/lib/mbuf/rte_mbuf_core.h
> +++ b/lib/mbuf/rte_mbuf_core.h
> @@ -228,6 +228,8 @@ extern "C" {
> 
>  /**
>   * Request security offload processing on the TX packet.
> + * To use Tx security offload, the user needs to fill l2_len in mbuf
> + * indicating L2 header size and where L3 header starts.
>   */
>  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> 
> --
> 2.25.1

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Tue, Jul 06, 2021 at 10:56:10AM +0000, Ananyev, Konstantin wrote:
> 
> > 
> > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > 
> > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > to associate a Security session with a mbuf before submitting
> > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > set some opaque metadata in mbuf for PMD's use.
> > This patch updates documentation that rte_security_set_pkt_metadata()
> > should be called only with mbuf containing Layer 3 and above data.
> > This behaviour is consistent with existing PMD's such as ixgbe.
> > 
> > On Tx, not all net PMD's/HW can parse packet and identify
> > L2 header and L3 header locations on Tx. This is inline with other
> > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > HW to be able to generate checksum. Since Inline IPSec is also
> > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > valid to find L3 header and perform Outbound IPSec processing.
> > Hence, this patch updates documentation to enforce setting
> > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > for Inline IPSec Crypto / Protocol offload processing to
> > work on Tx.
> > 
> > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > ---
> >  doc/guides/nics/features.rst           | 2 ++
> >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> >  3 files changed, 9 insertions(+), 1 deletion(-)
> > 
> > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > index 403c2b03a..414baf14f 100644
> > --- a/doc/guides/nics/features.rst
> > +++ b/doc/guides/nics/features.rst
> > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > 
> >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > 
> >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> >    ``capabilities_get``.
> > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > index f72bc8a78..7b68c698d 100644
> > --- a/doc/guides/prog_guide/rte_security.rst
> > +++ b/doc/guides/prog_guide/rte_security.rst
> > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > 
> >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > +should be called on mbuf only with Layer 3 and above data present and
> > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> 
> Hmm... not sure why mbuf.data_off should point to L3 hdr.
> Who will add L2 hdr to the packet in that case?
> Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?

That is the semantics I was trying to define. I think below are the sequence of
operations to be done for ipsec processing,

1. receive_pkt()
2. strip_l2_hdr()
3. Do policy lookup ()
4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
particular SA. Now pkt only has L3 and above data.
5. Do route_lookup()
6. add_l2hdr() which might be different from stripped l2hdr.
7. Send packet out.

The above sequence is what I believe the current poll mode worker thread in
ipsec-secgw is following. While in event mode, step 2 and step 6 are missing.

This patch is trying to enforce semantics as above so that
rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
called.

I also think above sequence is what Linux kernel stack or other stacks follow.
Does it makes sense ?

> 
> > Once called,
> > +Layer 3 and above data cannot be modified or moved around unless
> > +``rte_security_set_pkt_metadata()`` is called again.
> > 
> >  For inline protocol offloaded ingress traffic, the application can register a
> >  pointer, ``userdata`` , in the security session. When the packet is received,
> > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > index bb38d7f58..9d8e3ddc8 100644
> > --- a/lib/mbuf/rte_mbuf_core.h
> > +++ b/lib/mbuf/rte_mbuf_core.h
> > @@ -228,6 +228,8 @@ extern "C" {
> > 
> >  /**
> >   * Request security offload processing on the TX packet.
> > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > + * indicating L2 header size and where L3 header starts.
> >   */
> >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > 
> > --
> > 2.25.1
> 

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> On Tue, Jul 06, 2021 at 10:56:10AM +0000, Ananyev, Konstantin wrote:
> >
> > >
> > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > >
> > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > to associate a Security session with a mbuf before submitting
> > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > set some opaque metadata in mbuf for PMD's use.
> > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > should be called only with mbuf containing Layer 3 and above data.
> > > This behaviour is consistent with existing PMD's such as ixgbe.
> > >
> > > On Tx, not all net PMD's/HW can parse packet and identify
> > > L2 header and L3 header locations on Tx. This is inline with other
> > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > HW to be able to generate checksum. Since Inline IPSec is also
> > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > valid to find L3 header and perform Outbound IPSec processing.
> > > Hence, this patch updates documentation to enforce setting
> > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > for Inline IPSec Crypto / Protocol offload processing to
> > > work on Tx.
> > >
> > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > ---
> > >  doc/guides/nics/features.rst           | 2 ++
> > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > index 403c2b03a..414baf14f 100644
> > > --- a/doc/guides/nics/features.rst
> > > +++ b/doc/guides/nics/features.rst
> > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > >
> > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > >
> > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > >    ``capabilities_get``.
> > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > index f72bc8a78..7b68c698d 100644
> > > --- a/doc/guides/prog_guide/rte_security.rst
> > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > >
> > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > +should be called on mbuf only with Layer 3 and above data present and
> > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> >
> > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > Who will add L2 hdr to the packet in that case?
> > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> 
> That is the semantics I was trying to define. I think below are the sequence of
> operations to be done for ipsec processing,
> 
> 1. receive_pkt()
> 2. strip_l2_hdr()
> 3. Do policy lookup ()
> 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> particular SA. Now pkt only has L3 and above data.
> 5. Do route_lookup()
> 6. add_l2hdr() which might be different from stripped l2hdr.
> 7. Send packet out.
> 
> The above sequence is what I believe the current poll mode worker thread in
> ipsec-secgw is following.

That's just a sample app, it doesn't mean it has to be the only possible way.

> While in event mode, step 2 and step 6 are missing.

I think this L2 hdr manipulation is totally optional.
If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().   

> 
> This patch is trying to enforce semantics as above so that
> rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> called.
> 
> I also think above sequence is what Linux kernel stack or other stacks follow.
> Does it makes sense ?
> 
> >
> > > Once called,
> > > +Layer 3 and above data cannot be modified or moved around unless
> > > +``rte_security_set_pkt_metadata()`` is called again.
> > >
> > >  For inline protocol offloaded ingress traffic, the application can register a
> > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > index bb38d7f58..9d8e3ddc8 100644
> > > --- a/lib/mbuf/rte_mbuf_core.h
> > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > @@ -228,6 +228,8 @@ extern "C" {
> > >
> > >  /**
> > >   * Request security offload processing on the TX packet.
> > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > + * indicating L2 header size and where L3 header starts.
> > >   */
> > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > >
> > > --
> > > 2.25.1
> >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Tue, Jul 06, 2021 at 12:42:34PM +0000, Ananyev, Konstantin wrote:
> 
> > On Tue, Jul 06, 2021 at 10:56:10AM +0000, Ananyev, Konstantin wrote:
> > >
> > > >
> > > > From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > >
> > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > to associate a Security session with a mbuf before submitting
> > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > set some opaque metadata in mbuf for PMD's use.
> > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > should be called only with mbuf containing Layer 3 and above data.
> > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > >
> > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > Hence, this patch updates documentation to enforce setting
> > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > work on Tx.
> > > >
> > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > ---
> > > >  doc/guides/nics/features.rst           | 2 ++
> > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > index 403c2b03a..414baf14f 100644
> > > > --- a/doc/guides/nics/features.rst
> > > > +++ b/doc/guides/nics/features.rst
> > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > >
> > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > >
> > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > >    ``capabilities_get``.
> > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > index f72bc8a78..7b68c698d 100644
> > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > >
> > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > >
> > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > Who will add L2 hdr to the packet in that case?
> > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > 
> > That is the semantics I was trying to define. I think below are the sequence of
> > operations to be done for ipsec processing,
> > 
> > 1. receive_pkt()
> > 2. strip_l2_hdr()
> > 3. Do policy lookup ()
> > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > particular SA. Now pkt only has L3 and above data.
> > 5. Do route_lookup()
> > 6. add_l2hdr() which might be different from stripped l2hdr.
> > 7. Send packet out.
> > 
> > The above sequence is what I believe the current poll mode worker thread in
> > ipsec-secgw is following.
> 
> That's just a sample app, it doesn't mean it has to be the only possible way.
> 
> > While in event mode, step 2 and step 6 are missing.
> 
> I think this L2 hdr manipulation is totally optional.
> If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think 
having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.

> then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().  

This is also fine with us.
> 
> > 
> > This patch is trying to enforce semantics as above so that
> > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > called.
> > 
> > I also think above sequence is what Linux kernel stack or other stacks follow.
> > Does it makes sense ?
> > 
> > >
> > > > Once called,
> > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > >
> > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > index bb38d7f58..9d8e3ddc8 100644
> > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > @@ -228,6 +228,8 @@ extern "C" {
> > > >
> > > >  /**
> > > >   * Request security offload processing on the TX packet.
> > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > + * indicating L2 header size and where L3 header starts.
> > > >   */
> > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > >
> > > > --
> > > > 2.25.1
> > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > to associate a Security session with a mbuf before submitting
> > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > >
> > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > Hence, this patch updates documentation to enforce setting
> > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > work on Tx.
> > > > >
> > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > ---
> > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > index 403c2b03a..414baf14f 100644
> > > > > --- a/doc/guides/nics/features.rst
> > > > > +++ b/doc/guides/nics/features.rst
> > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > >
> > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > >
> > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > >    ``capabilities_get``.
> > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > index f72bc8a78..7b68c698d 100644
> > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > >
> > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > >
> > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > Who will add L2 hdr to the packet in that case?
> > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > >
> > > That is the semantics I was trying to define. I think below are the sequence of
> > > operations to be done for ipsec processing,
> > >
> > > 1. receive_pkt()
> > > 2. strip_l2_hdr()
> > > 3. Do policy lookup ()
> > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > particular SA. Now pkt only has L3 and above data.
> > > 5. Do route_lookup()
> > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > 7. Send packet out.
> > >
> > > The above sequence is what I believe the current poll mode worker thread in
> > > ipsec-secgw is following.
> >
> > That's just a sample app, it doesn't mean it has to be the only possible way.
> >
> > > While in event mode, step 2 and step 6 are missing.
> >
> > I think this L2 hdr manipulation is totally optional.
> > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> 
> > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> 
> This is also fine with us.

Ok, so to make sure we are on the same page, you propose:
1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
    at [mbuf.l2_len, mbuf.pkt_len) can't be modified?

Is that correct understanding?
If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?  

> >
> > >
> > > This patch is trying to enforce semantics as above so that
> > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > called.
> > >
> > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > Does it makes sense ?
> > >
> > > >
> > > > > Once called,
> > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > >
> > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > >
> > > > >  /**
> > > > >   * Request security offload processing on the TX packet.
> > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > + * indicating L2 header size and where L3 header starts.
> > > > >   */
> > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > >
> > > > > --
> > > > > 2.25.1
> > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Tue, Jul 06, 2021 at 02:07:17PM +0000, Ananyev, Konstantin wrote:
> 
> 
> > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > to associate a Security session with a mbuf before submitting
> > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > >
> > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > work on Tx.
> > > > > >
> > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > ---
> > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > index 403c2b03a..414baf14f 100644
> > > > > > --- a/doc/guides/nics/features.rst
> > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > >
> > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > >
> > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > >    ``capabilities_get``.
> > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > >
> > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > >
> > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > Who will add L2 hdr to the packet in that case?
> > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > >
> > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > operations to be done for ipsec processing,
> > > >
> > > > 1. receive_pkt()
> > > > 2. strip_l2_hdr()
> > > > 3. Do policy lookup ()
> > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > particular SA. Now pkt only has L3 and above data.
> > > > 5. Do route_lookup()
> > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > 7. Send packet out.
> > > >
> > > > The above sequence is what I believe the current poll mode worker thread in
> > > > ipsec-secgw is following.
> > >
> > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > >
> > > > While in event mode, step 2 and step 6 are missing.
> > >
> > > I think this L2 hdr manipulation is totally optional.
> > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > 
> > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > 
> > This is also fine with us.
> 
> Ok, so to make sure we are on the same page, you propose:
> 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
>     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
Yes.

> 
> Is that correct understanding?
> If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?  

Since our PMD doesn't have a prepare function, I missed that but, since
rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
are callbacks from same PMD, do you see any issue ? 

The restriction is from user side, data is not supposed to be modified unless
rte_security_set_pkt_metadata() is called again.

If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
security, my only argument was that since there is already a hit in
rte_security_set_pkt_metadata() to PMD specific callback and
struct rte_security_session is passed as an argument to it, it is more benefitial to
do security related pre-processing there.

Also rte_eth_tx_prepare() if implemented will be called for both security and
non-security pkts.


> 
> > >
> > > >
> > > > This patch is trying to enforce semantics as above so that
> > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > called.
> > > >
> > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > Does it makes sense ?
> > > >
> > > > >
> > > > > > Once called,
> > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > >
> > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > >
> > > > > >  /**
> > > > > >   * Request security offload processing on the TX packet.
> > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > >   */
> > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > >
> > > > > > --
> > > > > > 2.25.1
> > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > >
> > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > work on Tx.
> > > > > > >
> > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > ---
> > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > >
> > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > >
> > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > >
> > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > >    ``capabilities_get``.
> > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > >
> > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > >
> > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > >
> > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > operations to be done for ipsec processing,
> > > > >
> > > > > 1. receive_pkt()
> > > > > 2. strip_l2_hdr()
> > > > > 3. Do policy lookup ()
> > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > particular SA. Now pkt only has L3 and above data.
> > > > > 5. Do route_lookup()
> > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > 7. Send packet out.
> > > > >
> > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > ipsec-secgw is following.
> > > >
> > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > >
> > > > > While in event mode, step 2 and step 6 are missing.
> > > >
> > > > I think this L2 hdr manipulation is totally optional.
> > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > >
> > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > >
> > > This is also fine with us.
> >
> > Ok, so to make sure we are on the same page, you propose:
> > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> Yes.
> 
> >
> > Is that correct understanding?
> > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> 
> Since our PMD doesn't have a prepare function, I missed that but, since
> rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> are callbacks from same PMD, do you see any issue ?
> 
> The restriction is from user side, data is not supposed to be modified unless
> rte_security_set_pkt_metadata() is called again.

Yep, I do have a concern here.
Right now it is perfectly valid to do something like that:
rte_security_set_pkt_metadata(..., mb, ...);
/* can modify contents of the packet */
rte_eth_tx_prepare(..., &mb, 1);
rte_eth_tx_burst(..., &mb, 1);

With the new restrictions you are proposing it wouldn't be allowed any more.

> 
> If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> security,

Yes, that was my thought. 

> my only argument was that since there is already a hit in
> rte_security_set_pkt_metadata() to PMD specific callback and
> struct rte_security_session is passed as an argument to it, it is more benefitial to
> do security related pre-processing there.

Yes, it would be extra callback call that way.
Though tx_prepare() accepts burst of packets, so the overhead
of function call will be spread around the whole burst, and I presume
shouldn't be too high.

> Also rte_eth_tx_prepare() if implemented will be called for both security and
> non-security pkts.

Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
modifications are required for the packet. 

> 
> >
> > > >
> > > > >
> > > > > This patch is trying to enforce semantics as above so that
> > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > called.
> > > > >
> > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > Does it makes sense ?
> > > > >
> > > > > >
> > > > > > > Once called,
> > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > >
> > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > >
> > > > > > >  /**
> > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > >   */
> > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > >
> > > > > > > --
> > > > > > > 2.25.1
> > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Wed, Jul 07, 2021 at 09:59:10AM +0000, Ananyev, Konstantin wrote:
> 
> > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > >
> > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > work on Tx.
> > > > > > > >
> > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > ---
> > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > >
> > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > >
> > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > >
> > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > >    ``capabilities_get``.
> > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > >
> > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > >
> > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > >
> > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > operations to be done for ipsec processing,
> > > > > >
> > > > > > 1. receive_pkt()
> > > > > > 2. strip_l2_hdr()
> > > > > > 3. Do policy lookup ()
> > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > 5. Do route_lookup()
> > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > 7. Send packet out.
> > > > > >
> > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > ipsec-secgw is following.
> > > > >
> > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > >
> > > > > > While in event mode, step 2 and step 6 are missing.
> > > > >
> > > > > I think this L2 hdr manipulation is totally optional.
> > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > >
> > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > >
> > > > This is also fine with us.
> > >
> > > Ok, so to make sure we are on the same page, you propose:
> > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > Yes.
> > 
> > >
> > > Is that correct understanding?
> > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > 
> > Since our PMD doesn't have a prepare function, I missed that but, since
> > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > are callbacks from same PMD, do you see any issue ?
> > 
> > The restriction is from user side, data is not supposed to be modified unless
> > rte_security_set_pkt_metadata() is called again.
> 
> Yep, I do have a concern here.
> Right now it is perfectly valid to do something like that:
> rte_security_set_pkt_metadata(..., mb, ...);
> /* can modify contents of the packet */
> rte_eth_tx_prepare(..., &mb, 1);
> rte_eth_tx_burst(..., &mb, 1);
> 
> With the new restrictions you are proposing it wouldn't be allowed any more.
You can still modify L2 header and IPSEC is only concerned about L3 and above.

I think insisting that rte_security_set_pkt_metadata() be called after all L3
and above header modifications is no a problem. I guess existing ixgbe/txgbe
PMD which are the ones only implementing the call back are already expecting the
same ?

> 
> > 
> > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > security,
> 
> Yes, that was my thought. 
> 
> > my only argument was that since there is already a hit in
> > rte_security_set_pkt_metadata() to PMD specific callback and
> > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > do security related pre-processing there.
> 
> Yes, it would be extra callback call that way.
> Though tx_prepare() accepts burst of packets, so the overhead
> of function call will be spread around the whole burst, and I presume
> shouldn't be too high.
> 
> > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > non-security pkts.
> 
> Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> modifications are required for the packet. 

But the major issues I see are

1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
   In our case, we need to know the security session details to do things.
2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
   macro RTE_ETHDEV_TX_PREPARE_NOOP. 
3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
   struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.

So I think instead of enforcing yet another callback tx_prepare() for inline security
processing, it can be done via security specific set_pkt_metadata(). I'm fine to
introduce a burst call for the same(I was thinking to propose it in future) to
compensate for the overhead.

If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
rte_mbuf had space for struct rte_security_session pointer, 
then then I guess it would have been better to do the way you proposed.

> 
> > 
> > >
> > > > >
> > > > > >
> > > > > > This patch is trying to enforce semantics as above so that
> > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > called.
> > > > > >
> > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > Does it makes sense ?
> > > > > >
> > > > > > >
> > > > > > > > Once called,
> > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > >
> > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > >
> > > > > > > >  /**
> > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > >   */
> > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.25.1
> > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > >
> > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > work on Tx.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > ---
> > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > >
> > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > >
> > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > >
> > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > >    ``capabilities_get``.
> > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > >
> > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > >
> > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > >
> > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > operations to be done for ipsec processing,
> > > > > > >
> > > > > > > 1. receive_pkt()
> > > > > > > 2. strip_l2_hdr()
> > > > > > > 3. Do policy lookup ()
> > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > 5. Do route_lookup()
> > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > 7. Send packet out.
> > > > > > >
> > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > ipsec-secgw is following.
> > > > > >
> > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > >
> > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > >
> > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > >
> > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > > >
> > > > > This is also fine with us.
> > > >
> > > > Ok, so to make sure we are on the same page, you propose:
> > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > Yes.
> > >
> > > >
> > > > Is that correct understanding?
> > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > >
> > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > are callbacks from same PMD, do you see any issue ?
> > >
> > > The restriction is from user side, data is not supposed to be modified unless
> > > rte_security_set_pkt_metadata() is called again.
> >
> > Yep, I do have a concern here.
> > Right now it is perfectly valid to do something like that:
> > rte_security_set_pkt_metadata(..., mb, ...);
> > /* can modify contents of the packet */
> > rte_eth_tx_prepare(..., &mb, 1);
> > rte_eth_tx_burst(..., &mb, 1);
> >
> > With the new restrictions you are proposing it wouldn't be allowed any more.
> You can still modify L2 header and IPSEC is only concerned about L3 and above.
> 
> I think insisting that rte_security_set_pkt_metadata() be called after all L3
> and above header modifications is no a problem. I guess existing ixgbe/txgbe
> PMD which are the ones only implementing the call back are already expecting the
> same ?

AFAIK, no there are no such requirements for ixgbe or txgbe.
All that ixgbe callback does - store session related data inside mbuf.
It's only expectation to have ESP trailer at the proper place (after ICV):

union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
                                rte_security_dynfield(m);
  mdata->enc = 1;
  mdata->sa_idx = ic_session->sa_index;
  mdata->pad_len = ixgbe_crypto_compute_pad_len(m);

Then this data will be used by tx_burst() function.

> 
> >
> > >
> > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > security,
> >
> > Yes, that was my thought.
> >
> > > my only argument was that since there is already a hit in
> > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > do security related pre-processing there.
> >
> > Yes, it would be extra callback call that way.
> > Though tx_prepare() accepts burst of packets, so the overhead
> > of function call will be spread around the whole burst, and I presume
> > shouldn't be too high.
> >
> > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > non-security pkts.
> >
> > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > modifications are required for the packet.
> 
> But the major issues I see are
> 
> 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
>    In our case, we need to know the security session details to do things.

I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?

> 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
>    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
>    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.

Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
specially for that - to store secuiryt related data inside the mbuf.
Yes your PMD has to request it at initialization time, but I suppose it is not a big deal. 

> So I think instead of enforcing yet another callback tx_prepare() for inline security
> processing, it can be done via security specific set_pkt_metadata(). 

But what you proposing introduces new limitations and might existing functionality.
BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
itself is not an option?

> I'm fine to
> introduce a burst call for the same(I was thinking to propose it in future) to
> compensate for the overhead.
> 
> If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> rte_mbuf had space for struct rte_security_session pointer,

But it does, see above.
In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field. 
If your PMD requires more data to be associated with mbuf
- you can request it via mbuf_dynfield and store there whatever is needed.

> then then I guess it would have been better to do the way you proposed.
> 
> >
> > >
> > > >
> > > > > >
> > > > > > >
> > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > called.
> > > > > > >
> > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > Does it makes sense ?
> > > > > > >
> > > > > > > >
> > > > > > > > > Once called,
> > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > >
> > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > >
> > > > > > > > >  /**
> > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > >   */
> > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Sat, Jul 10, 2021 at 12:57:19PM +0000, Ananyev, Konstantin wrote:
> 
> > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > >
> > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > work on Tx.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > ---
> > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > >
> > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > >
> > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > >
> > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > >
> > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > >
> > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > operations to be done for ipsec processing,
> > > > > > > >
> > > > > > > > 1. receive_pkt()
> > > > > > > > 2. strip_l2_hdr()
> > > > > > > > 3. Do policy lookup ()
> > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > 5. Do route_lookup()
> > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > 7. Send packet out.
> > > > > > > >
> > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > ipsec-secgw is following.
> > > > > > >
> > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > >
> > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > >
> > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > >
> > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > > > >
> > > > > > This is also fine with us.
> > > > >
> > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > Yes.
> > > >
> > > > >
> > > > > Is that correct understanding?
> > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > >
> > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > are callbacks from same PMD, do you see any issue ?
> > > >
> > > > The restriction is from user side, data is not supposed to be modified unless
> > > > rte_security_set_pkt_metadata() is called again.
> > >
> > > Yep, I do have a concern here.
> > > Right now it is perfectly valid to do something like that:
> > > rte_security_set_pkt_metadata(..., mb, ...);
> > > /* can modify contents of the packet */
> > > rte_eth_tx_prepare(..., &mb, 1);
> > > rte_eth_tx_burst(..., &mb, 1);
> > >
> > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > 
> > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > PMD which are the ones only implementing the call back are already expecting the
> > same ?
> 
> AFAIK, no there are no such requirements for ixgbe or txgbe.
> All that ixgbe callback does - store session related data inside mbuf.
> It's only expectation to have ESP trailer at the proper place (after ICV):

This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
have ESP trailer updated or when mbuf->pkt_len = 0

> 
> union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
>                                 rte_security_dynfield(m);
>   mdata->enc = 1;
>   mdata->sa_idx = ic_session->sa_index;
>   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> 
> Then this data will be used by tx_burst() function.
So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(), 
mbuf data / packet len cannot be modified right as if modified, then tx_burst()
will be using incorrect pad len ? 

This patch is also trying to add similar restriction on when
rte_security_set_pkt_metadata() should be called and what cannot be done after
calling rte_security_set_pkt_metadata().


> 
> > 
> > >
> > > >
> > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > security,
> > >
> > > Yes, that was my thought.
> > >
> > > > my only argument was that since there is already a hit in
> > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > do security related pre-processing there.
> > >
> > > Yes, it would be extra callback call that way.
> > > Though tx_prepare() accepts burst of packets, so the overhead
> > > of function call will be spread around the whole burst, and I presume
> > > shouldn't be too high.
> > >
> > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > non-security pkts.
> > >
> > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > modifications are required for the packet.
> > 
> > But the major issues I see are
> > 
> > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> >    In our case, we need to know the security session details to do things.
> 
> I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?

We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata() 
just for storing session pointer in rte_security_dynfield consumes unnecessary
cycles per pkt.


> 
> > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> 
> Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> specially for that - to store secuiryt related data inside the mbuf.
> Yes your PMD has to request it at initialization time, but I suppose it is not a big deal. 
> 
> > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > processing, it can be done via security specific set_pkt_metadata(). 
> 
> But what you proposing introduces new limitations and might existing functionality.
> BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> itself is not an option?

We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
rte_security_set_pkt_metadata().

Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
set, then, user needs to update struct rte_security_session's sess_private_data in a in 
rte_security_dynfield like below ?

<snip>

static inline void                                                                
inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,                      
        struct rte_mbuf *mb[], uint16_t num)                                      
{                                                                                 
        uint32_t i, ol_flags;                                                     
                                                                                  
        ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;      
        for (i = 0; i != num; i++) {                                              
                                                                                  
                mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;                            

                if (ol_flags != 0)                                                
                        rte_security_set_pkt_metadata(ss->security.ctx,           
                                ss->security.ses, mb[i], NULL);                   
		else
                	*rte_security_dynfield(mb[i]) =                           
                                (uint64_t)ss->security.ses->sess_private_data;    


If the above can be done, then in our PMD, we will not have a callback for
set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
in capabilities. 


> 
> > I'm fine to
> > introduce a burst call for the same(I was thinking to propose it in future) to
> > compensate for the overhead.
> > 
> > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > rte_mbuf had space for struct rte_security_session pointer,
> 
> But it does, see above.
> In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field. 
> If your PMD requires more data to be associated with mbuf
> - you can request it via mbuf_dynfield and store there whatever is needed.
> 
> > then then I guess it would have been better to do the way you proposed.
> > 
> > >
> > > >
> > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > called.
> > > > > > > >
> > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > Does it makes sense ?
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > Once called,
> > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > >
> > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > >
> > > > > > > > > >  /**
> > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > >   */
> > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > 2.25.1
> > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

Adding more rte_security and PMD maintainers into the loop.

> > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > >
> > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > work on Tx.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > >
> > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > >
> > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > >
> > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > >
> > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > >
> > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > >
> > > > > > > > > 1. receive_pkt()
> > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > 5. Do route_lookup()
> > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > 7. Send packet out.
> > > > > > > > >
> > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > ipsec-secgw is following.
> > > > > > > >
> > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > >
> > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > >
> > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > >
> > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > > > > >
> > > > > > > This is also fine with us.
> > > > > >
> > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > Yes.
> > > > >
> > > > > >
> > > > > > Is that correct understanding?
> > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > >
> > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > are callbacks from same PMD, do you see any issue ?
> > > > >
> > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > rte_security_set_pkt_metadata() is called again.
> > > >
> > > > Yep, I do have a concern here.
> > > > Right now it is perfectly valid to do something like that:
> > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > /* can modify contents of the packet */
> > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > rte_eth_tx_burst(..., &mb, 1);
> > > >
> > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > >
> > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > PMD which are the ones only implementing the call back are already expecting the
> > > same ?
> >
> > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > All that ixgbe callback does - store session related data inside mbuf.
> > It's only expectation to have ESP trailer at the proper place (after ICV):
> 
> This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> have ESP trailer updated or when mbuf->pkt_len = 0
> 
> >
> > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> >                                 rte_security_dynfield(m);
> >   mdata->enc = 1;
> >   mdata->sa_idx = ic_session->sa_index;
> >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> >
> > Then this data will be used by tx_burst() function.
> So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> will be using incorrect pad len ?

No, pkt_len can be modified.
Though ESP trailer pad_len can't.

> 
> This patch is also trying to add similar restriction on when
> rte_security_set_pkt_metadata() should be called and what cannot be done after
> calling rte_security_set_pkt_metadata().

No, I don't think it is really the same.
Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
that ESP packet is already formed and trailer contains valid data.
In fact, I think this pad_len calculation can be moved to actual TX function.

> 
> >
> > >
> > > >
> > > > >
> > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > security,
> > > >
> > > > Yes, that was my thought.
> > > >
> > > > > my only argument was that since there is already a hit in
> > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > do security related pre-processing there.
> > > >
> > > > Yes, it would be extra callback call that way.
> > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > of function call will be spread around the whole burst, and I presume
> > > > shouldn't be too high.
> > > >
> > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > non-security pkts.
> > > >
> > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > modifications are required for the packet.
> > >
> > > But the major issues I see are
> > >
> > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > >    In our case, we need to know the security session details to do things.
> >
> > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> 
> We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> just for storing session pointer in rte_security_dynfield consumes unnecessary
> cycles per pkt.

In fact there are two function calls: one for rte_security_set_pkt_metadata(),
second for  instance->ops->set_pkt_metadata() callback.
Which off-course way too expensive for such simple operation.
Actually same thought for rte_security_get_userdata().
Both of these functions belong to data-path and ideally have to be as fast as possible.
Probably 21.11 is a right timeframe for that.
 
> >
> > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> >
> > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > specially for that - to store secuiryt related data inside the mbuf.
> > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> >
> > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > processing, it can be done via security specific set_pkt_metadata().
> >
> > But what you proposing introduces new limitations and might existing functionality.
> > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > itself is not an option?
> 
> We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> rte_security_set_pkt_metadata().
> 
> Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> set, then, user needs to update struct rte_security_session's sess_private_data in a in
> rte_security_dynfield like below ?
> 
> <snip>
> 
> static inline void
> inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
>         struct rte_mbuf *mb[], uint16_t num)
> {
>         uint32_t i, ol_flags;
> 
>         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
>         for (i = 0; i != num; i++) {
> 
>                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> 
>                 if (ol_flags != 0)
>                         rte_security_set_pkt_metadata(ss->security.ctx,
>                                 ss->security.ses, mb[i], NULL);
> 		else
>                 	*rte_security_dynfield(mb[i]) =
>                                 (uint64_t)ss->security.ses->sess_private_data;
> 
> 
> If the above can be done, then in our PMD, we will not have a callback for
> set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> in capabilities.

That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
So all existing apps that use this API will have to be changed.
We'd better avoid such changes unless there is really good reason for that.
So, I'd suggest to tweak your idea a bit:

1) change rte_security_set_pkt_metadata():
if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
otherwise just: rte_security_dynfield(m) = sess->session_private_data;
(fast-path)

2) consider to make rte_security_set_pkt_metadata() inline function. 
We probably can have some special flag inside struct rte_security_ctx,
or even store inside ctx a pointer to set_pkt_metadata() itself.

As a brief code snippet:

struct rte_security_ctx {
        void *device;
        /**< Crypto/ethernet device attached */
        const struct rte_security_ops *ops;
        /**< Pointer to security ops for the device */
        uint16_t sess_cnt;
        /**< Number of sessions attached to this context */
+     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);   
}; 

static inline int
rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
                              struct rte_security_session *sess,
                              struct rte_mbuf *m, void *params)
{
     /* fast-path */
      if (instance->set_pkt_mdata == NULL) {
             *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
             return 0; 
       /* slow path */ 
       } else
           return instance->set_pkt_mdata(instance->device, sess, m, params);
}

That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require 
some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
(ctx_create()), but hopefully will benefit everyone.

> 
> >
> > > I'm fine to
> > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > compensate for the overhead.
> > >
> > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > rte_mbuf had space for struct rte_security_session pointer,
> >
> > But it does, see above.
> > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > If your PMD requires more data to be associated with mbuf
> > - you can request it via mbuf_dynfield and store there whatever is needed.
> >
> > > then then I guess it would have been better to do the way you proposed.
> > >
> > > >
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > called.
> > > > > > > > >
> > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > Does it makes sense ?
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Once called,
> > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > >
> > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > >
> > > > > > > > > > >  /**
> > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > >   */
> > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > 2.25.1
> > > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> 
> Adding more rte_security and PMD maintainers into the loop.
> 
> > > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > > work on Tx.
> > > > > > > > > > > >
> > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > > ---
> > > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > > >
> > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > > >
> > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > > >
> > > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > > >
> > > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > > >
> > > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > > >
> > > > > > > > > > 1. receive_pkt()
> > > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > > 5. Do route_lookup()
> > > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > > 7. Send packet out.
> > > > > > > > > >
> > > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > > ipsec-secgw is following.
> > > > > > > > >
> > > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > > >
> > > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > > >
> > > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > > >
> > > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > > > > > >
> > > > > > > > This is also fine with us.
> > > > > > >
> > > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > > Yes.
> > > > > >
> > > > > > >
> > > > > > > Is that correct understanding?
> > > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > > >
> > > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > > are callbacks from same PMD, do you see any issue ?
> > > > > >
> > > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > > rte_security_set_pkt_metadata() is called again.
> > > > >
> > > > > Yep, I do have a concern here.
> > > > > Right now it is perfectly valid to do something like that:
> > > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > > /* can modify contents of the packet */
> > > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > > rte_eth_tx_burst(..., &mb, 1);
> > > > >
> > > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > > >
> > > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > > PMD which are the ones only implementing the call back are already expecting the
> > > > same ?
> > >
> > > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > > All that ixgbe callback does - store session related data inside mbuf.
> > > It's only expectation to have ESP trailer at the proper place (after ICV):
> >
> > This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> > have ESP trailer updated or when mbuf->pkt_len = 0
> >
> > >
> > > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> > >                                 rte_security_dynfield(m);
> > >   mdata->enc = 1;
> > >   mdata->sa_idx = ic_session->sa_index;
> > >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> > >
> > > Then this data will be used by tx_burst() function.
> > So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> > mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> > will be using incorrect pad len ?
> 
> No, pkt_len can be modified.
> Though ESP trailer pad_len can't.
> 
> >
> > This patch is also trying to add similar restriction on when
> > rte_security_set_pkt_metadata() should be called and what cannot be done after
> > calling rte_security_set_pkt_metadata().
> 
> No, I don't think it is really the same.
> Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
> that ESP packet is already formed and trailer contains valid data.
> In fact, I think this pad_len calculation can be moved to actual TX function.
> 
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > > security,
> > > > >
> > > > > Yes, that was my thought.
> > > > >
> > > > > > my only argument was that since there is already a hit in
> > > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > > do security related pre-processing there.
> > > > >
> > > > > Yes, it would be extra callback call that way.
> > > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > > of function call will be spread around the whole burst, and I presume
> > > > > shouldn't be too high.
> > > > >
> > > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > > non-security pkts.
> > > > >
> > > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > > modifications are required for the packet.
> > > >
> > > > But the major issues I see are
> > > >
> > > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > > >    In our case, we need to know the security session details to do things.
> > >
> > > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> >
> > We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> > just for storing session pointer in rte_security_dynfield consumes unnecessary
> > cycles per pkt.
> 
> In fact there are two function calls: one for rte_security_set_pkt_metadata(),
> second for  instance->ops->set_pkt_metadata() callback.
> Which off-course way too expensive for such simple operation.
> Actually same thought for rte_security_get_userdata().
> Both of these functions belong to data-path and ideally have to be as fast as possible.
> Probably 21.11 is a right timeframe for that.
> 
> > >
> > > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> > >
> > > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > > specially for that - to store secuiryt related data inside the mbuf.
> > > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> > >
> > > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > > processing, it can be done via security specific set_pkt_metadata().
> > >
> > > But what you proposing introduces new limitations and might existing functionality.
> > > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > > itself is not an option?
> >
> > We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> > rte_security_set_pkt_metadata().
> >
> > Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> > set, then, user needs to update struct rte_security_session's sess_private_data in a in
> > rte_security_dynfield like below ?
> >
> > <snip>
> >
> > static inline void
> > inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
> >         struct rte_mbuf *mb[], uint16_t num)
> > {
> >         uint32_t i, ol_flags;
> >
> >         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
> >         for (i = 0; i != num; i++) {
> >
> >                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> >
> >                 if (ol_flags != 0)
> >                         rte_security_set_pkt_metadata(ss->security.ctx,
> >                                 ss->security.ses, mb[i], NULL);
> > 		else
> >                 	*rte_security_dynfield(mb[i]) =
> >                                 (uint64_t)ss->security.ses->sess_private_data;
> >
> >
> > If the above can be done, then in our PMD, we will not have a callback for
> > set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> > in capabilities.
> 
> That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
> So all existing apps that use this API will have to be changed.
> We'd better avoid such changes unless there is really good reason for that.
> So, I'd suggest to tweak your idea a bit:
> 
> 1) change rte_security_set_pkt_metadata():
> if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
> otherwise just: rte_security_dynfield(m) = sess->session_private_data;
> (fast-path)
> 
> 2) consider to make rte_security_set_pkt_metadata() inline function.
> We probably can have some special flag inside struct rte_security_ctx,
> or even store inside ctx a pointer to set_pkt_metadata() itself.

After another thoughts some new flags might be better.
Then later, if we'll realize that set_pkt_metadata() and get_useradata()
are not really used by PMDs, it might be easier to deprecate these callbacks.

> 
> As a brief code snippet:
> 
> struct rte_security_ctx {
>         void *device;
>         /**< Crypto/ethernet device attached */
>         const struct rte_security_ops *ops;
>         /**< Pointer to security ops for the device */
>         uint16_t sess_cnt;
>         /**< Number of sessions attached to this context */
> +     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);
> };
> 
> static inline int
> rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>                               struct rte_security_session *sess,
>                               struct rte_mbuf *m, void *params)
> {
>      /* fast-path */
>       if (instance->set_pkt_mdata == NULL) {
>              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
>              return 0;
>        /* slow path */
>        } else
>            return instance->set_pkt_mdata(instance->device, sess, m, params);
> }
> 
> That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require
> some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
> (ctx_create()), but hopefully will benefit everyone.
> 
> >
> > >
> > > > I'm fine to
> > > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > > compensate for the overhead.
> > > >
> > > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > > rte_mbuf had space for struct rte_security_session pointer,
> > >
> > > But it does, see above.
> > > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > > If your PMD requires more data to be associated with mbuf
> > > - you can request it via mbuf_dynfield and store there whatever is needed.
> > >
> > > > then then I guess it would have been better to do the way you proposed.
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > > called.
> > > > > > > > > >
> > > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > > Does it makes sense ?
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > Once called,
> > > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > > >
> > > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > > >
> > > > > > > > > > > >  /**
> > > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > > >   */
> > > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > 2.25.1
> > > > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Tue, Jul 13, 2021 at 02:08:18PM +0000, Ananyev, Konstantin wrote:
> 
> > 
> > Adding more rte_security and PMD maintainers into the loop.
> > 
> > > > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > > > work on Tx.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > > > >
> > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > > > >
> > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > > > >
> > > > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > > > >
> > > > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > > > >
> > > > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > > > >
> > > > > > > > > > > 1. receive_pkt()
> > > > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > > > 5. Do route_lookup()
> > > > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > > > 7. Send packet out.
> > > > > > > > > > >
> > > > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > > > ipsec-secgw is following.
> > > > > > > > > >
> > > > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > > > >
> > > > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > > > >
> > > > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > > > >
> > > > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling rte_security_set_pkt_metadata().
> > > > > > > > >
> > > > > > > > > This is also fine with us.
> > > > > > > >
> > > > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > > > Yes.
> > > > > > >
> > > > > > > >
> > > > > > > > Is that correct understanding?
> > > > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > > > >
> > > > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > > > are callbacks from same PMD, do you see any issue ?
> > > > > > >
> > > > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > > > rte_security_set_pkt_metadata() is called again.
> > > > > >
> > > > > > Yep, I do have a concern here.
> > > > > > Right now it is perfectly valid to do something like that:
> > > > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > > > /* can modify contents of the packet */
> > > > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > > > rte_eth_tx_burst(..., &mb, 1);
> > > > > >
> > > > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > > > >
> > > > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > > > PMD which are the ones only implementing the call back are already expecting the
> > > > > same ?
> > > >
> > > > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > > > All that ixgbe callback does - store session related data inside mbuf.
> > > > It's only expectation to have ESP trailer at the proper place (after ICV):
> > >
> > > This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> > > have ESP trailer updated or when mbuf->pkt_len = 0
> > >
> > > >
> > > > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> > > >                                 rte_security_dynfield(m);
> > > >   mdata->enc = 1;
> > > >   mdata->sa_idx = ic_session->sa_index;
> > > >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> > > >
> > > > Then this data will be used by tx_burst() function.
> > > So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> > > mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> > > will be using incorrect pad len ?
> > 
> > No, pkt_len can be modified.
> > Though ESP trailer pad_len can't.
> > 
> > >
> > > This patch is also trying to add similar restriction on when
> > > rte_security_set_pkt_metadata() should be called and what cannot be done after
> > > calling rte_security_set_pkt_metadata().
> > 
> > No, I don't think it is really the same.
> > Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
> > that ESP packet is already formed and trailer contains valid data.
> > In fact, I think this pad_len calculation can be moved to actual TX function.
> > 
> > >
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > > > security,
> > > > > >
> > > > > > Yes, that was my thought.
> > > > > >
> > > > > > > my only argument was that since there is already a hit in
> > > > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > > > do security related pre-processing there.
> > > > > >
> > > > > > Yes, it would be extra callback call that way.
> > > > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > > > of function call will be spread around the whole burst, and I presume
> > > > > > shouldn't be too high.
> > > > > >
> > > > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > > > non-security pkts.
> > > > > >
> > > > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > > > modifications are required for the packet.
> > > > >
> > > > > But the major issues I see are
> > > > >
> > > > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > > > >    In our case, we need to know the security session details to do things.
> > > >
> > > > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> > >
> > > We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> > > just for storing session pointer in rte_security_dynfield consumes unnecessary
> > > cycles per pkt.
> > 
> > In fact there are two function calls: one for rte_security_set_pkt_metadata(),
> > second for  instance->ops->set_pkt_metadata() callback.
> > Which off-course way too expensive for such simple operation.
> > Actually same thought for rte_security_get_userdata().
> > Both of these functions belong to data-path and ideally have to be as fast as possible.
> > Probably 21.11 is a right timeframe for that.
> > 
> > > >
> > > > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > > > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > > > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> > > >
> > > > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > > > specially for that - to store secuiryt related data inside the mbuf.
> > > > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> > > >
> > > > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > > > processing, it can be done via security specific set_pkt_metadata().
> > > >
> > > > But what you proposing introduces new limitations and might existing functionality.
> > > > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > > > itself is not an option?
> > >
> > > We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> > > rte_security_set_pkt_metadata().
> > >
> > > Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> > > set, then, user needs to update struct rte_security_session's sess_private_data in a in
> > > rte_security_dynfield like below ?
> > >
> > > <snip>
> > >
> > > static inline void
> > > inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
> > >         struct rte_mbuf *mb[], uint16_t num)
> > > {
> > >         uint32_t i, ol_flags;
> > >
> > >         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
> > >         for (i = 0; i != num; i++) {
> > >
> > >                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> > >
> > >                 if (ol_flags != 0)
> > >                         rte_security_set_pkt_metadata(ss->security.ctx,
> > >                                 ss->security.ses, mb[i], NULL);
> > > 		else
> > >                 	*rte_security_dynfield(mb[i]) =
> > >                                 (uint64_t)ss->security.ses->sess_private_data;
> > >
> > >
> > > If the above can be done, then in our PMD, we will not have a callback for
> > > set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> > > in capabilities.
> > 
> > That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
> > So all existing apps that use this API will have to be changed.
> > We'd better avoid such changes unless there is really good reason for that.
> > So, I'd suggest to tweak your idea a bit:
> > 
> > 1) change rte_security_set_pkt_metadata():
> > if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
> > otherwise just: rte_security_dynfield(m) = sess->session_private_data;
> > (fast-path)
> > 
> > 2) consider to make rte_security_set_pkt_metadata() inline function.
> > We probably can have some special flag inside struct rte_security_ctx,
> > or even store inside ctx a pointer to set_pkt_metadata() itself.
> 
> After another thoughts some new flags might be better.
> Then later, if we'll realize that set_pkt_metadata() and get_useradata()
> are not really used by PMDs, it might be easier to deprecate these callbacks.

Thanks, I agree with your thoughts. I'll submit a V2 with above change, new flags and 
set_pkt_metadata() and get_userdata() function pointers moved to rte_security_ctx for
review so that it can be targeted for 21.11. 

Even with flags moving set_pkt_metadata() and get_userdata() function pointers is still needed
as we need to make rte_security_set_pkt_metadata() API inline while struct rte_security_ops is not
exposed to user. I think this is fine as it is inline with how fast path function pointers
of rte_ethdev and rte_cryptodev are currently placed.

> 
> > 
> > As a brief code snippet:
> > 
> > struct rte_security_ctx {
> >         void *device;
> >         /**< Crypto/ethernet device attached */
> >         const struct rte_security_ops *ops;
> >         /**< Pointer to security ops for the device */
> >         uint16_t sess_cnt;
> >         /**< Number of sessions attached to this context */
> > +     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);
> > };
> > 
> > static inline int
> > rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> >                               struct rte_security_session *sess,
> >                               struct rte_mbuf *m, void *params)
> > {
> >      /* fast-path */
> >       if (instance->set_pkt_mdata == NULL) {
> >              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
> >              return 0;
> >        /* slow path */
> >        } else
> >            return instance->set_pkt_mdata(instance->device, sess, m, params);
> > }
> > 
> > That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require
> > some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
> > (ctx_create()), but hopefully will benefit everyone.
> > 
> > >
> > > >
> > > > > I'm fine to
> > > > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > > > compensate for the overhead.
> > > > >
> > > > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > > > rte_mbuf had space for struct rte_security_session pointer,
> > > >
> > > > But it does, see above.
> > > > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > > > If your PMD requires more data to be associated with mbuf
> > > > - you can request it via mbuf_dynfield and store there whatever is needed.
> > > >
> > > > > then then I guess it would have been better to do the way you proposed.
> > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > > > called.
> > > > > > > > > > >
> > > > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > > > Does it makes sense ?
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > Once called,
> > > > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > > > >
> > > > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > > > >
> > > > > > > > > > > > >  /**
> > > > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > > > >   */
> > > > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > 2.25.1
> > > > > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> > >
> > > Adding more rte_security and PMD maintainers into the loop.
> > >
> > > > > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > > > > work on Tx.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > > > > >
> > > > > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > > > > >
> > > > > > > > > > > > 1. receive_pkt()
> > > > > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > > > > 5. Do route_lookup()
> > > > > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > > > > 7. Send packet out.
> > > > > > > > > > > >
> > > > > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > > > > ipsec-secgw is following.
> > > > > > > > > > >
> > > > > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > > > > >
> > > > > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > > > > >
> > > > > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > > > > >
> > > > > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling
> rte_security_set_pkt_metadata().
> > > > > > > > > >
> > > > > > > > > > This is also fine with us.
> > > > > > > > >
> > > > > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Is that correct understanding?
> > > > > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > > > > >
> > > > > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > > > > are callbacks from same PMD, do you see any issue ?
> > > > > > > >
> > > > > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > > > > rte_security_set_pkt_metadata() is called again.
> > > > > > >
> > > > > > > Yep, I do have a concern here.
> > > > > > > Right now it is perfectly valid to do something like that:
> > > > > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > > > > /* can modify contents of the packet */
> > > > > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > > > > rte_eth_tx_burst(..., &mb, 1);
> > > > > > >
> > > > > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > > > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > > > > >
> > > > > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > > > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > > > > PMD which are the ones only implementing the call back are already expecting the
> > > > > > same ?
> > > > >
> > > > > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > > > > All that ixgbe callback does - store session related data inside mbuf.
> > > > > It's only expectation to have ESP trailer at the proper place (after ICV):
> > > >
> > > > This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> > > > have ESP trailer updated or when mbuf->pkt_len = 0
> > > >
> > > > >
> > > > > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> > > > >                                 rte_security_dynfield(m);
> > > > >   mdata->enc = 1;
> > > > >   mdata->sa_idx = ic_session->sa_index;
> > > > >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> > > > >
> > > > > Then this data will be used by tx_burst() function.
> > > > So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> > > > mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> > > > will be using incorrect pad len ?
> > >
> > > No, pkt_len can be modified.
> > > Though ESP trailer pad_len can't.
> > >
> > > >
> > > > This patch is also trying to add similar restriction on when
> > > > rte_security_set_pkt_metadata() should be called and what cannot be done after
> > > > calling rte_security_set_pkt_metadata().
> > >
> > > No, I don't think it is really the same.
> > > Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
> > > that ESP packet is already formed and trailer contains valid data.
> > > In fact, I think this pad_len calculation can be moved to actual TX function.
> > >
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > > > > security,
> > > > > > >
> > > > > > > Yes, that was my thought.
> > > > > > >
> > > > > > > > my only argument was that since there is already a hit in
> > > > > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > > > > do security related pre-processing there.
> > > > > > >
> > > > > > > Yes, it would be extra callback call that way.
> > > > > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > > > > of function call will be spread around the whole burst, and I presume
> > > > > > > shouldn't be too high.
> > > > > > >
> > > > > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > > > > non-security pkts.
> > > > > > >
> > > > > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > > > > modifications are required for the packet.
> > > > > >
> > > > > > But the major issues I see are
> > > > > >
> > > > > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > > > > >    In our case, we need to know the security session details to do things.
> > > > >
> > > > > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> > > >
> > > > We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> > > > just for storing session pointer in rte_security_dynfield consumes unnecessary
> > > > cycles per pkt.
> > >
> > > In fact there are two function calls: one for rte_security_set_pkt_metadata(),
> > > second for  instance->ops->set_pkt_metadata() callback.
> > > Which off-course way too expensive for such simple operation.
> > > Actually same thought for rte_security_get_userdata().
> > > Both of these functions belong to data-path and ideally have to be as fast as possible.
> > > Probably 21.11 is a right timeframe for that.
> > >
> > > > >
> > > > > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > > > > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > > > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > > > > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> > > > >
> > > > > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > > > > specially for that - to store secuiryt related data inside the mbuf.
> > > > > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> > > > >
> > > > > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > > > > processing, it can be done via security specific set_pkt_metadata().
> > > > >
> > > > > But what you proposing introduces new limitations and might existing functionality.
> > > > > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > > > > itself is not an option?
> > > >
> > > > We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> > > > rte_security_set_pkt_metadata().
> > > >
> > > > Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> > > > set, then, user needs to update struct rte_security_session's sess_private_data in a in
> > > > rte_security_dynfield like below ?
> > > >
> > > > <snip>
> > > >
> > > > static inline void
> > > > inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
> > > >         struct rte_mbuf *mb[], uint16_t num)
> > > > {
> > > >         uint32_t i, ol_flags;
> > > >
> > > >         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
> > > >         for (i = 0; i != num; i++) {
> > > >
> > > >                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> > > >
> > > >                 if (ol_flags != 0)
> > > >                         rte_security_set_pkt_metadata(ss->security.ctx,
> > > >                                 ss->security.ses, mb[i], NULL);
> > > > 		else
> > > >                 	*rte_security_dynfield(mb[i]) =
> > > >                                 (uint64_t)ss->security.ses->sess_private_data;
> > > >
> > > >
> > > > If the above can be done, then in our PMD, we will not have a callback for
> > > > set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> > > > in capabilities.
> > >
> > > That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
> > > So all existing apps that use this API will have to be changed.
> > > We'd better avoid such changes unless there is really good reason for that.
> > > So, I'd suggest to tweak your idea a bit:
> > >
> > > 1) change rte_security_set_pkt_metadata():
> > > if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
> > > otherwise just: rte_security_dynfield(m) = sess->session_private_data;
> > > (fast-path)
> > >
> > > 2) consider to make rte_security_set_pkt_metadata() inline function.
> > > We probably can have some special flag inside struct rte_security_ctx,
> > > or even store inside ctx a pointer to set_pkt_metadata() itself.
> >
> > After another thoughts some new flags might be better.
> > Then later, if we'll realize that set_pkt_metadata() and get_useradata()
> > are not really used by PMDs, it might be easier to deprecate these callbacks.
> 
> Thanks, I agree with your thoughts. I'll submit a V2 with above change, new flags and
> set_pkt_metadata() and get_userdata() function pointers moved to rte_security_ctx for
> review so that it can be targeted for 21.11.
> 
> Even with flags moving set_pkt_metadata() and get_userdata() function pointers is still needed
> as we need to make rte_security_set_pkt_metadata() API inline while struct rte_security_ops is not
> exposed to user. I think this is fine as it is inline with how fast path function pointers
> of rte_ethdev and rte_cryptodev are currently placed.

My thought was we can get away with just flags only.
Something like that:
rte_security.h:

...

enum {
	RTE_SEC_CTX_F_FAST_SET_MDATA = 0x1,
              RTE_SEC_CTX_F_FAST_GET_UDATA = 0x2,
}; 

struct rte_security_ctx {
        void *device;
        /**< Crypto/ethernet device attached */
        const struct rte_security_ops *ops;
        /**< Pointer to security ops for the device */
        uint16_t sess_cnt;
        /**< Number of sessions attached to this context */
       uint32_t flags;
};

extern int
__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
                               struct rte_security_session *sess,
                               struct rte_mbuf *m, void *params); 

static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
                               struct rte_security_session *sess,
                               struct rte_mbuf *m, void *params)
{
      /* fast-path */
       if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
              return 0;
        /* slow path */
        } else
            return __rte_security_set_pkt_metadata (instance->device, sess, m, params);
}

rte_security.c: 

...
/* existing one, just renamed */
int
__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
                              struct rte_security_session *sess,
                              struct rte_mbuf *m, void *params)
{
#ifdef RTE_DEBUG
        RTE_PTR_OR_ERR_RET(sess, -EINVAL);
        RTE_PTR_OR_ERR_RET(instance, -EINVAL);
        RTE_PTR_OR_ERR_RET(instance->ops, -EINVAL);
#endif
        RTE_FUNC_PTR_OR_ERR_RET(*instance->ops->set_pkt_metadata, -ENOTSUP);
        return instance->ops->set_pkt_metadata(instance->device,
                                               sess, m, params);
}


I think both ways are possible (flags vs actual func pointers) and both have
some pluses and minuses.
I suppose the main choice here what do we think should be the future of
set_pkt_metadata() and rte_security_get_userdata(). 
If we think that they will be useful for some future PMDs and we want to keep them,
then probably storing actual func pointers inside ctx is a better approach.
If not, then flags seems like a better one, as in that case we can eventually
deprecate and remove these callbacks.
From what I see right now, custom callbacks seems excessive,
and rte_security_dynfield is enough.
But might be there are some future plans that would require them?   
 
> 
> >
> > >
> > > As a brief code snippet:
> > >
> > > struct rte_security_ctx {
> > >         void *device;
> > >         /**< Crypto/ethernet device attached */
> > >         const struct rte_security_ops *ops;
> > >         /**< Pointer to security ops for the device */
> > >         uint16_t sess_cnt;
> > >         /**< Number of sessions attached to this context */
> > > +     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);
> > > };
> > >
> > > static inline int
> > > rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> > >                               struct rte_security_session *sess,
> > >                               struct rte_mbuf *m, void *params)
> > > {
> > >      /* fast-path */
> > >       if (instance->set_pkt_mdata == NULL) {
> > >              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
> > >              return 0;
> > >        /* slow path */
> > >        } else
> > >            return instance->set_pkt_mdata(instance->device, sess, m, params);
> > > }
> > >
> > > That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require
> > > some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
> > > (ctx_create()), but hopefully will benefit everyone.
> > >
> > > >
> > > > >
> > > > > > I'm fine to
> > > > > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > > > > compensate for the overhead.
> > > > > >
> > > > > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > > > > rte_mbuf had space for struct rte_security_session pointer,
> > > > >
> > > > > But it does, see above.
> > > > > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > > > > If your PMD requires more data to be associated with mbuf
> > > > > - you can request it via mbuf_dynfield and store there whatever is needed.
> > > > >
> > > > > > then then I guess it would have been better to do the way you proposed.
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > > > > called.
> > > > > > > > > > > >
> > > > > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > > > > Does it makes sense ?
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Once called,
> > > > > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  /**
> > > > > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > > > > >   */
> > > > > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > 2.25.1
> > > > > > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

On Wed, Jul 14, 2021 at 11:09:08AM +0000, Ananyev, Konstantin wrote:
> > > >
> > > > Adding more rte_security and PMD maintainers into the loop.
> > > >
> > > > > > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > > > > > work on Tx.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > > > > > >
> > > > > > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. receive_pkt()
> > > > > > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > > > > > 5. Do route_lookup()
> > > > > > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > > > > > 7. Send packet out.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > > > > > ipsec-secgw is following.
> > > > > > > > > > > >
> > > > > > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > > > > > >
> > > > > > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > > > > > >
> > > > > > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > > > > > >
> > > > > > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling
> > rte_security_set_pkt_metadata().
> > > > > > > > > > >
> > > > > > > > > > > This is also fine with us.
> > > > > > > > > >
> > > > > > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > > > > > Yes.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Is that correct understanding?
> > > > > > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > > > > > >
> > > > > > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > > > > > are callbacks from same PMD, do you see any issue ?
> > > > > > > > >
> > > > > > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > > > > > rte_security_set_pkt_metadata() is called again.
> > > > > > > >
> > > > > > > > Yep, I do have a concern here.
> > > > > > > > Right now it is perfectly valid to do something like that:
> > > > > > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > > > > > /* can modify contents of the packet */
> > > > > > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > > > > > rte_eth_tx_burst(..., &mb, 1);
> > > > > > > >
> > > > > > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > > > > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > > > > > >
> > > > > > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > > > > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > > > > > PMD which are the ones only implementing the call back are already expecting the
> > > > > > > same ?
> > > > > >
> > > > > > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > > > > > All that ixgbe callback does - store session related data inside mbuf.
> > > > > > It's only expectation to have ESP trailer at the proper place (after ICV):
> > > > >
> > > > > This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> > > > > have ESP trailer updated or when mbuf->pkt_len = 0
> > > > >
> > > > > >
> > > > > > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> > > > > >                                 rte_security_dynfield(m);
> > > > > >   mdata->enc = 1;
> > > > > >   mdata->sa_idx = ic_session->sa_index;
> > > > > >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> > > > > >
> > > > > > Then this data will be used by tx_burst() function.
> > > > > So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> > > > > mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> > > > > will be using incorrect pad len ?
> > > >
> > > > No, pkt_len can be modified.
> > > > Though ESP trailer pad_len can't.
> > > >
> > > > >
> > > > > This patch is also trying to add similar restriction on when
> > > > > rte_security_set_pkt_metadata() should be called and what cannot be done after
> > > > > calling rte_security_set_pkt_metadata().
> > > >
> > > > No, I don't think it is really the same.
> > > > Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
> > > > that ESP packet is already formed and trailer contains valid data.
> > > > In fact, I think this pad_len calculation can be moved to actual TX function.
> > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > > > > > security,
> > > > > > > >
> > > > > > > > Yes, that was my thought.
> > > > > > > >
> > > > > > > > > my only argument was that since there is already a hit in
> > > > > > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > > > > > do security related pre-processing there.
> > > > > > > >
> > > > > > > > Yes, it would be extra callback call that way.
> > > > > > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > > > > > of function call will be spread around the whole burst, and I presume
> > > > > > > > shouldn't be too high.
> > > > > > > >
> > > > > > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > > > > > non-security pkts.
> > > > > > > >
> > > > > > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > > > > > modifications are required for the packet.
> > > > > > >
> > > > > > > But the major issues I see are
> > > > > > >
> > > > > > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > > > > > >    In our case, we need to know the security session details to do things.
> > > > > >
> > > > > > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> > > > >
> > > > > We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> > > > > just for storing session pointer in rte_security_dynfield consumes unnecessary
> > > > > cycles per pkt.
> > > >
> > > > In fact there are two function calls: one for rte_security_set_pkt_metadata(),
> > > > second for  instance->ops->set_pkt_metadata() callback.
> > > > Which off-course way too expensive for such simple operation.
> > > > Actually same thought for rte_security_get_userdata().
> > > > Both of these functions belong to data-path and ideally have to be as fast as possible.
> > > > Probably 21.11 is a right timeframe for that.
> > > >
> > > > > >
> > > > > > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > > > > > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > > > > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > > > > > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> > > > > >
> > > > > > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > > > > > specially for that - to store secuiryt related data inside the mbuf.
> > > > > > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> > > > > >
> > > > > > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > > > > > processing, it can be done via security specific set_pkt_metadata().
> > > > > >
> > > > > > But what you proposing introduces new limitations and might existing functionality.
> > > > > > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > > > > > itself is not an option?
> > > > >
> > > > > We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> > > > > rte_security_set_pkt_metadata().
> > > > >
> > > > > Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> > > > > set, then, user needs to update struct rte_security_session's sess_private_data in a in
> > > > > rte_security_dynfield like below ?
> > > > >
> > > > > <snip>
> > > > >
> > > > > static inline void
> > > > > inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
> > > > >         struct rte_mbuf *mb[], uint16_t num)
> > > > > {
> > > > >         uint32_t i, ol_flags;
> > > > >
> > > > >         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
> > > > >         for (i = 0; i != num; i++) {
> > > > >
> > > > >                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> > > > >
> > > > >                 if (ol_flags != 0)
> > > > >                         rte_security_set_pkt_metadata(ss->security.ctx,
> > > > >                                 ss->security.ses, mb[i], NULL);
> > > > > 		else
> > > > >                 	*rte_security_dynfield(mb[i]) =
> > > > >                                 (uint64_t)ss->security.ses->sess_private_data;
> > > > >
> > > > >
> > > > > If the above can be done, then in our PMD, we will not have a callback for
> > > > > set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> > > > > in capabilities.
> > > >
> > > > That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
> > > > So all existing apps that use this API will have to be changed.
> > > > We'd better avoid such changes unless there is really good reason for that.
> > > > So, I'd suggest to tweak your idea a bit:
> > > >
> > > > 1) change rte_security_set_pkt_metadata():
> > > > if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
> > > > otherwise just: rte_security_dynfield(m) = sess->session_private_data;
> > > > (fast-path)
> > > >
> > > > 2) consider to make rte_security_set_pkt_metadata() inline function.
> > > > We probably can have some special flag inside struct rte_security_ctx,
> > > > or even store inside ctx a pointer to set_pkt_metadata() itself.
> > >
> > > After another thoughts some new flags might be better.
> > > Then later, if we'll realize that set_pkt_metadata() and get_useradata()
> > > are not really used by PMDs, it might be easier to deprecate these callbacks.
> > 
> > Thanks, I agree with your thoughts. I'll submit a V2 with above change, new flags and
> > set_pkt_metadata() and get_userdata() function pointers moved to rte_security_ctx for
> > review so that it can be targeted for 21.11.
> > 
> > Even with flags moving set_pkt_metadata() and get_userdata() function pointers is still needed
> > as we need to make rte_security_set_pkt_metadata() API inline while struct rte_security_ops is not
> > exposed to user. I think this is fine as it is inline with how fast path function pointers
> > of rte_ethdev and rte_cryptodev are currently placed.
> 
> My thought was we can get away with just flags only.
> Something like that:
> rte_security.h:
> 
> ...
> 
> enum {
> 	RTE_SEC_CTX_F_FAST_SET_MDATA = 0x1,
>               RTE_SEC_CTX_F_FAST_GET_UDATA = 0x2,
> }; 
> 
> struct rte_security_ctx {
>         void *device;
>         /**< Crypto/ethernet device attached */
>         const struct rte_security_ops *ops;
>         /**< Pointer to security ops for the device */
>         uint16_t sess_cnt;
>         /**< Number of sessions attached to this context */
>        uint32_t flags;
> };
> 
> extern int
> __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>                                struct rte_security_session *sess,
>                                struct rte_mbuf *m, void *params); 
> 
> static inline int
>  rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>                                struct rte_security_session *sess,
>                                struct rte_mbuf *m, void *params)
> {
>       /* fast-path */
>        if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
>               *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
>               return 0;
>         /* slow path */
>         } else
>             return __rte_security_set_pkt_metadata (instance->device, sess, m, params);
> }
> 
> rte_security.c: 
> 
> ...
> /* existing one, just renamed */
> int
> __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>                               struct rte_security_session *sess,
>                               struct rte_mbuf *m, void *params)
> {
> #ifdef RTE_DEBUG
>         RTE_PTR_OR_ERR_RET(sess, -EINVAL);
>         RTE_PTR_OR_ERR_RET(instance, -EINVAL);
>         RTE_PTR_OR_ERR_RET(instance->ops, -EINVAL);
> #endif
>         RTE_FUNC_PTR_OR_ERR_RET(*instance->ops->set_pkt_metadata, -ENOTSUP);
>         return instance->ops->set_pkt_metadata(instance->device,
>                                                sess, m, params);
> }
> 
> 
> I think both ways are possible (flags vs actual func pointers) and both have
> some pluses and minuses.
> I suppose the main choice here what do we think should be the future of
> set_pkt_metadata() and rte_security_get_userdata(). 
> If we think that they will be useful for some future PMDs and we want to keep them,
> then probably storing actual func pointers inside ctx is a better approach.
> If not, then flags seems like a better one, as in that case we can eventually
> deprecate and remove these callbacks.
> From what I see right now, custom callbacks seems excessive,
> and rte_security_dynfield is enough.
> But might be there are some future plans that would require them?   

Above method is also fine. Moving fn pointers to rte_security_ctx can be
done later if other PMD's need it.

Atleast our HW PMD's doesn't plan to use set_pkt_metada()/get_user_data() 
fn pointers in future if above is implemented.

>  
> > 
> > >
> > > >
> > > > As a brief code snippet:
> > > >
> > > > struct rte_security_ctx {
> > > >         void *device;
> > > >         /**< Crypto/ethernet device attached */
> > > >         const struct rte_security_ops *ops;
> > > >         /**< Pointer to security ops for the device */
> > > >         uint16_t sess_cnt;
> > > >         /**< Number of sessions attached to this context */
> > > > +     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);
> > > > };
> > > >
> > > > static inline int
> > > > rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> > > >                               struct rte_security_session *sess,
> > > >                               struct rte_mbuf *m, void *params)
> > > > {
> > > >      /* fast-path */
> > > >       if (instance->set_pkt_mdata == NULL) {
> > > >              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
> > > >              return 0;
> > > >        /* slow path */
> > > >        } else
> > > >            return instance->set_pkt_mdata(instance->device, sess, m, params);
> > > > }
> > > >
> > > > That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require
> > > > some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
> > > > (ctx_create()), but hopefully will benefit everyone.
> > > >
> > > > >
> > > > > >
> > > > > > > I'm fine to
> > > > > > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > > > > > compensate for the overhead.
> > > > > > >
> > > > > > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > > > > > rte_mbuf had space for struct rte_security_session pointer,
> > > > > >
> > > > > > But it does, see above.
> > > > > > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > > > > > If your PMD requires more data to be associated with mbuf
> > > > > > - you can request it via mbuf_dynfield and store there whatever is needed.
> > > > > >
> > > > > > > then then I guess it would have been better to do the way you proposed.
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > > > > > called.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > > > > > Does it makes sense ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Once called,
> > > > > > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  /**
> > > > > > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > > > > > >   */
> > > > > > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > 2.25.1
> > > > > > > > > > > > > >

Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> -----Original Message-----
> From: Nithin Dabilpuram <nithind1988@gmail.com>
> Sent: Wednesday, July 14, 2021 2:30 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Cc: Akhil Goyal <gakhil@marvell.com>; dev@dpdk.org; hemant.agrawal@nxp.com; thomas@monjalon.net; g.singh@nxp.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Zhang, Roy Fan <roy.fan.zhang@intel.com>; olivier.matz@6wind.com; jerinj@marvell.com; Doherty, Declan
> <declan.doherty@intel.com>; Nicolau, Radu <radu.nicolau@intel.com>; jiawenwu@trustnetic.com; jianwang@trustnetic.com
> Subject: Re: [dpdk-dev] [PATCH 1/2] security: enforce semantics for Tx inline processing
> 
> On Wed, Jul 14, 2021 at 11:09:08AM +0000, Ananyev, Konstantin wrote:
> > > > >
> > > > > Adding more rte_security and PMD maintainers into the loop.
> > > > >
> > > > > > > > > > > > > > > > For Tx inline processing, when RTE_SECURITY_TX_OLOAD_NEED_MDATA is
> > > > > > > > > > > > > > > > set, rte_security_set_pkt_metadata() needs to be called for pkts
> > > > > > > > > > > > > > > > to associate a Security session with a mbuf before submitting
> > > > > > > > > > > > > > > > to Ethdev Tx. This is apart from setting PKT_TX_SEC_OFFLOAD in
> > > > > > > > > > > > > > > > mbuf.ol_flags. rte_security_set_pkt_metadata() is also used to
> > > > > > > > > > > > > > > > set some opaque metadata in mbuf for PMD's use.
> > > > > > > > > > > > > > > > This patch updates documentation that rte_security_set_pkt_metadata()
> > > > > > > > > > > > > > > > should be called only with mbuf containing Layer 3 and above data.
> > > > > > > > > > > > > > > > This behaviour is consistent with existing PMD's such as ixgbe.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tx, not all net PMD's/HW can parse packet and identify
> > > > > > > > > > > > > > > > L2 header and L3 header locations on Tx. This is inline with other
> > > > > > > > > > > > > > > > Tx offloads requirements such as L3 checksum, L4 checksum offload,
> > > > > > > > > > > > > > > > etc, where mbuf.l2_len, mbuf.l3_len etc, needs to be set for
> > > > > > > > > > > > > > > > HW to be able to generate checksum. Since Inline IPSec is also
> > > > > > > > > > > > > > > > such a Tx offload, some PMD's at least need mbuf.l2_len to be
> > > > > > > > > > > > > > > > valid to find L3 header and perform Outbound IPSec processing.
> > > > > > > > > > > > > > > > Hence, this patch updates documentation to enforce setting
> > > > > > > > > > > > > > > > mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> > > > > > > > > > > > > > > > for Inline IPSec Crypto / Protocol offload processing to
> > > > > > > > > > > > > > > > work on Tx.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > > > > > > > > > > > > > > > Reviewed-by: Akhil Goyal <gakhil@marvell.com>
> > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > >  doc/guides/nics/features.rst           | 2 ++
> > > > > > > > > > > > > > > >  doc/guides/prog_guide/rte_security.rst | 6 +++++-
> > > > > > > > > > > > > > > >  lib/mbuf/rte_mbuf_core.h               | 2 ++
> > > > > > > > > > > > > > > >  3 files changed, 9 insertions(+), 1 deletion(-)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > > index 403c2b03a..414baf14f 100644
> > > > > > > > > > > > > > > > --- a/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > > +++ b/doc/guides/nics/features.rst
> > > > > > > > > > > > > > > > @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
> > > > > > > > > > > > > > > >  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > > > @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
> > > > > > > > > > > > > > > >  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> > > > > > > > > > > > > > > > +* **[uses]       mbuf**: ``mbuf.l2_len``.
> > > > > > > > > > > > > > > >  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
> > > > > > > > > > > > > > > >    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
> > > > > > > > > > > > > > > >    ``capabilities_get``.
> > > > > > > > > > > > > > > > diff --git a/doc/guides/prog_guide/rte_security.rst b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > > index f72bc8a78..7b68c698d 100644
> > > > > > > > > > > > > > > > --- a/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > > +++ b/doc/guides/prog_guide/rte_security.rst
> > > > > > > > > > > > > > > > @@ -560,7 +560,11 @@ created by the application is attached to the security session by the API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  For Inline Crypto and Inline protocol offload, device specific defined metadata is
> > > > > > > > > > > > > > > >  updated in the mbuf using ``rte_security_set_pkt_metadata()`` if
> > > > > > > > > > > > > > > > -``DEV_TX_OFFLOAD_SEC_NEED_MDATA`` is set.
> > > > > > > > > > > > > > > > +``RTE_SECURITY_TX_OLOAD_NEED_MDATA`` is set. ``rte_security_set_pkt_metadata()``
> > > > > > > > > > > > > > > > +should be called on mbuf only with Layer 3 and above data present and
> > > > > > > > > > > > > > > > +``mbuf.data_off`` should be pointing to Layer 3 Header.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hmm... not sure why mbuf.data_off should point to L3 hdr.
> > > > > > > > > > > > > > > Who will add L2 hdr to the packet in that case?
> > > > > > > > > > > > > > > Or did you mean ``mbuf.data_off + mbuf.l2_len`` here?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That is the semantics I was trying to define. I think below are the sequence of
> > > > > > > > > > > > > > operations to be done for ipsec processing,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. receive_pkt()
> > > > > > > > > > > > > > 2. strip_l2_hdr()
> > > > > > > > > > > > > > 3. Do policy lookup ()
> > > > > > > > > > > > > > 4. Call rte_security_set_pkt_metadata() if pkt needs to be encrypted with a
> > > > > > > > > > > > > > particular SA. Now pkt only has L3 and above data.
> > > > > > > > > > > > > > 5. Do route_lookup()
> > > > > > > > > > > > > > 6. add_l2hdr() which might be different from stripped l2hdr.
> > > > > > > > > > > > > > 7. Send packet out.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The above sequence is what I believe the current poll mode worker thread in
> > > > > > > > > > > > > > ipsec-secgw is following.
> > > > > > > > > > > > >
> > > > > > > > > > > > > That's just a sample app, it doesn't mean it has to be the only possible way.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > While in event mode, step 2 and step 6 are missing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think this L2 hdr manipulation is totally optional.
> > > > > > > > > > > > > If your rte_security_set_pkt_metadata() implementation really needs to know L3 hdr offset (not sure why?),
> > > > > > > > > > > > Since rte_security_set_pkt_metadata() is PMD specific function ptr call, we are currently doing some pre-processing
> > > > > > > > > > > > here before submitting packet to inline IPSec via rte_eth_tx_burst(). This saves us cycles later in rte_eth_tx_burst().
> > > > > > > > > > > > If we cannot know for sure, the pkt content at the time of rte_security_set_pkt_metadata() call, then I think
> > > > > > > > > > > > having a PMD specific callback is not much of use except for saving SA priv data to rte_mbuf.
> > > > > > > > > > > >
> > > > > > > > > > > > > then I suppose we can add a requirement that l2_len has to be set properly before calling
> > > rte_security_set_pkt_metadata().
> > > > > > > > > > > >
> > > > > > > > > > > > This is also fine with us.
> > > > > > > > > > >
> > > > > > > > > > > Ok, so to make sure we are on the same page, you propose:
> > > > > > > > > > > 1. before calling rte_security_set_pkt_metadata() mbuf.l2_len should be properly set.
> > > > > > > > > > > 2. after rte_security_set_pkt_metadata() and before rte_eth_tx_burst() packet contents
> > > > > > > > > > >     at [mbuf.l2_len, mbuf.pkt_len) can't be modified?
> > > > > > > > > > Yes.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Is that correct understanding?
> > > > > > > > > > > If yes, I wonder how 2) will correlate with rte_eth_tx_prepare() concept?
> > > > > > > > > >
> > > > > > > > > > Since our PMD doesn't have a prepare function, I missed that but, since
> > > > > > > > > > rte_security_set_pkt_metadata() is only used for Inline Crypto/Protocol via
> > > > > > > > > > a rte_eth_dev, and both rte_security_set_pkt_metadata() and rte_eth_tx_prepare()
> > > > > > > > > > are callbacks from same PMD, do you see any issue ?
> > > > > > > > > >
> > > > > > > > > > The restriction is from user side, data is not supposed to be modified unless
> > > > > > > > > > rte_security_set_pkt_metadata() is called again.
> > > > > > > > >
> > > > > > > > > Yep, I do have a concern here.
> > > > > > > > > Right now it is perfectly valid to do something like that:
> > > > > > > > > rte_security_set_pkt_metadata(..., mb, ...);
> > > > > > > > > /* can modify contents of the packet */
> > > > > > > > > rte_eth_tx_prepare(..., &mb, 1);
> > > > > > > > > rte_eth_tx_burst(..., &mb, 1);
> > > > > > > > >
> > > > > > > > > With the new restrictions you are proposing it wouldn't be allowed any more.
> > > > > > > > You can still modify L2 header and IPSEC is only concerned about L3 and above.
> > > > > > > >
> > > > > > > > I think insisting that rte_security_set_pkt_metadata() be called after all L3
> > > > > > > > and above header modifications is no a problem. I guess existing ixgbe/txgbe
> > > > > > > > PMD which are the ones only implementing the call back are already expecting the
> > > > > > > > same ?
> > > > > > >
> > > > > > > AFAIK, no there are no such requirements for ixgbe or txgbe.
> > > > > > > All that ixgbe callback does - store session related data inside mbuf.
> > > > > > > It's only expectation to have ESP trailer at the proper place (after ICV):
> > > > > >
> > > > > > This implies rte_security_set_pkt_metadata() cannot be called when mbuf does't
> > > > > > have ESP trailer updated or when mbuf->pkt_len = 0
> > > > > >
> > > > > > >
> > > > > > > union ixgbe_crypto_tx_desc_md *mdata = (union ixgbe_crypto_tx_desc_md *)
> > > > > > >                                 rte_security_dynfield(m);
> > > > > > >   mdata->enc = 1;
> > > > > > >   mdata->sa_idx = ic_session->sa_index;
> > > > > > >   mdata->pad_len = ixgbe_crypto_compute_pad_len(m);
> > > > > > >
> > > > > > > Then this data will be used by tx_burst() function.
> > > > > > So it implies that after above rte_security_set_pkt_metadata() call, and before tx_burst(),
> > > > > > mbuf data / packet len cannot be modified right as if modified, then tx_burst()
> > > > > > will be using incorrect pad len ?
> > > > >
> > > > > No, pkt_len can be modified.
> > > > > Though ESP trailer pad_len can't.
> > > > >
> > > > > >
> > > > > > This patch is also trying to add similar restriction on when
> > > > > > rte_security_set_pkt_metadata() should be called and what cannot be done after
> > > > > > calling rte_security_set_pkt_metadata().
> > > > >
> > > > > No, I don't think it is really the same.
> > > > > Also, IMO, inside ixgbe set_pkt_metadata() implementaion we probably shouldn't silently imply
> > > > > that ESP packet is already formed and trailer contains valid data.
> > > > > In fact, I think this pad_len calculation can be moved to actual TX function.
> > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If your question is can't we do the preprocessing in rte_eth_tx_prepare() for
> > > > > > > > > > security,
> > > > > > > > >
> > > > > > > > > Yes, that was my thought.
> > > > > > > > >
> > > > > > > > > > my only argument was that since there is already a hit in
> > > > > > > > > > rte_security_set_pkt_metadata() to PMD specific callback and
> > > > > > > > > > struct rte_security_session is passed as an argument to it, it is more benefitial to
> > > > > > > > > > do security related pre-processing there.
> > > > > > > > >
> > > > > > > > > Yes, it would be extra callback call that way.
> > > > > > > > > Though tx_prepare() accepts burst of packets, so the overhead
> > > > > > > > > of function call will be spread around the whole burst, and I presume
> > > > > > > > > shouldn't be too high.
> > > > > > > > >
> > > > > > > > > > Also rte_eth_tx_prepare() if implemented will be called for both security and
> > > > > > > > > > non-security pkts.
> > > > > > > > >
> > > > > > > > > Yes, but tx_prepare() can distinguish (by ol_flags and/or other field contents) which
> > > > > > > > > modifications are required for the packet.
> > > > > > > >
> > > > > > > > But the major issues I see are
> > > > > > > >
> > > > > > > > 1. tx_prepare() doesn't take rte_security_session as argument though ol_flags has security flag.
> > > > > > > >    In our case, we need to know the security session details to do things.
> > > > > > >
> > > > > > > I suppose you can store pointer to session (or so) inside mbuf in rte_security_dynfield, no?
> > > > > >
> > > > > > We can do. But having to call PMD specific function call via rte_security_set_pkt_metadata()
> > > > > > just for storing session pointer in rte_security_dynfield consumes unnecessary
> > > > > > cycles per pkt.
> > > > >
> > > > > In fact there are two function calls: one for rte_security_set_pkt_metadata(),
> > > > > second for  instance->ops->set_pkt_metadata() callback.
> > > > > Which off-course way too expensive for such simple operation.
> > > > > Actually same thought for rte_security_get_userdata().
> > > > > Both of these functions belong to data-path and ideally have to be as fast as possible.
> > > > > Probably 21.11 is a right timeframe for that.
> > > > >
> > > > > > >
> > > > > > > > 2. AFAIU tx_prepare() is not mandatory as per spec and even by default disabled under compile time
> > > > > > > >    macro RTE_ETHDEV_TX_PREPARE_NOOP.
> > > > > > > > 3. Even if we do tx_prepare(), rte_security_set_pkt_mdata() is mandatory to associate
> > > > > > > >    struct rte_security_session to a pkt as unlike ol_flags, there is no direct space to do the same.
> > > > > > >
> > > > > > > Didn't get you here, obviously we do have rte_security_dynfield inside mbuf,
> > > > > > > specially for that - to store secuiryt related data inside the mbuf.
> > > > > > > Yes your PMD has to request it at initialization time, but I suppose it is not a big deal.
> > > > > > >
> > > > > > > > So I think instead of enforcing yet another callback tx_prepare() for inline security
> > > > > > > > processing, it can be done via security specific set_pkt_metadata().
> > > > > > >
> > > > > > > But what you proposing introduces new limitations and might existing functionality.
> > > > > > > BTW, if you don't like to use tx_prepare() - why doing these calculations inside tx_burst()
> > > > > > > itself is not an option?
> > > > > >
> > > > > > We can do things in tx_burst() but if we are doing it there, then we want to avoid having callback for
> > > > > > rte_security_set_pkt_metadata().
> > > > > >
> > > > > > Are you fine if we can update the spec that "When DEV_TX_OFFLOAD_SEC_NEED_MDATA is not
> > > > > > set, then, user needs to update struct rte_security_session's sess_private_data in a in
> > > > > > rte_security_dynfield like below ?
> > > > > >
> > > > > > <snip>
> > > > > >
> > > > > > static inline void
> > > > > > inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,
> > > > > >         struct rte_mbuf *mb[], uint16_t num)
> > > > > > {
> > > > > >         uint32_t i, ol_flags;
> > > > > >
> > > > > >         ol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;
> > > > > >         for (i = 0; i != num; i++) {
> > > > > >
> > > > > >                 mb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;
> > > > > >
> > > > > >                 if (ol_flags != 0)
> > > > > >                         rte_security_set_pkt_metadata(ss->security.ctx,
> > > > > >                                 ss->security.ses, mb[i], NULL);
> > > > > > 		else
> > > > > >                 	*rte_security_dynfield(mb[i]) =
> > > > > >                                 (uint64_t)ss->security.ses->sess_private_data;
> > > > > >
> > > > > >
> > > > > > If the above can be done, then in our PMD, we will not have a callback for
> > > > > > set_pkt_metadata() and DEV_TX_OFFLOAD_SEC_NEED_MDATA will also be not set
> > > > > > in capabilities.
> > > > >
> > > > > That's an interesting idea, but what you propose is the change in current rte_security API behaviour.
> > > > > So all existing apps that use this API will have to be changed.
> > > > > We'd better avoid such changes unless there is really good reason for that.
> > > > > So, I'd suggest to tweak your idea a bit:
> > > > >
> > > > > 1) change rte_security_set_pkt_metadata():
> > > > > if ops->set_pkt_metadata != NULL, then call it (existing behaviour)
> > > > > otherwise just: rte_security_dynfield(m) = sess->session_private_data;
> > > > > (fast-path)
> > > > >
> > > > > 2) consider to make rte_security_set_pkt_metadata() inline function.
> > > > > We probably can have some special flag inside struct rte_security_ctx,
> > > > > or even store inside ctx a pointer to set_pkt_metadata() itself.
> > > >
> > > > After another thoughts some new flags might be better.
> > > > Then later, if we'll realize that set_pkt_metadata() and get_useradata()
> > > > are not really used by PMDs, it might be easier to deprecate these callbacks.
> > >
> > > Thanks, I agree with your thoughts. I'll submit a V2 with above change, new flags and
> > > set_pkt_metadata() and get_userdata() function pointers moved to rte_security_ctx for
> > > review so that it can be targeted for 21.11.
> > >
> > > Even with flags moving set_pkt_metadata() and get_userdata() function pointers is still needed
> > > as we need to make rte_security_set_pkt_metadata() API inline while struct rte_security_ops is not
> > > exposed to user. I think this is fine as it is inline with how fast path function pointers
> > > of rte_ethdev and rte_cryptodev are currently placed.
> >
> > My thought was we can get away with just flags only.
> > Something like that:
> > rte_security.h:
> >
> > ...
> >
> > enum {
> > 	RTE_SEC_CTX_F_FAST_SET_MDATA = 0x1,
> >               RTE_SEC_CTX_F_FAST_GET_UDATA = 0x2,
> > };
> >
> > struct rte_security_ctx {
> >         void *device;
> >         /**< Crypto/ethernet device attached */
> >         const struct rte_security_ops *ops;
> >         /**< Pointer to security ops for the device */
> >         uint16_t sess_cnt;
> >         /**< Number of sessions attached to this context */
> >        uint32_t flags;
> > };
> >
> > extern int
> > __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> >                                struct rte_security_session *sess,
> >                                struct rte_mbuf *m, void *params);
> >
> > static inline int
> >  rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> >                                struct rte_security_session *sess,
> >                                struct rte_mbuf *m, void *params)
> > {
> >       /* fast-path */
> >        if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
> >               *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
> >               return 0;
> >         /* slow path */
> >         } else
> >             return __rte_security_set_pkt_metadata (instance->device, sess, m, params);
> > }
> >
> > rte_security.c:
> >
> > ...
> > /* existing one, just renamed */
> > int
> > __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> >                               struct rte_security_session *sess,
> >                               struct rte_mbuf *m, void *params)
> > {
> > #ifdef RTE_DEBUG
> >         RTE_PTR_OR_ERR_RET(sess, -EINVAL);
> >         RTE_PTR_OR_ERR_RET(instance, -EINVAL);
> >         RTE_PTR_OR_ERR_RET(instance->ops, -EINVAL);
> > #endif
> >         RTE_FUNC_PTR_OR_ERR_RET(*instance->ops->set_pkt_metadata, -ENOTSUP);
> >         return instance->ops->set_pkt_metadata(instance->device,
> >                                                sess, m, params);
> > }
> >
> >
> > I think both ways are possible (flags vs actual func pointers) and both have
> > some pluses and minuses.
> > I suppose the main choice here what do we think should be the future of
> > set_pkt_metadata() and rte_security_get_userdata().
> > If we think that they will be useful for some future PMDs and we want to keep them,
> > then probably storing actual func pointers inside ctx is a better approach.
> > If not, then flags seems like a better one, as in that case we can eventually
> > deprecate and remove these callbacks.
> > From what I see right now, custom callbacks seems excessive,
> > and rte_security_dynfield is enough.
> > But might be there are some future plans that would require them?
> 
> Above method is also fine. Moving fn pointers to rte_security_ctx can be
> done later if other PMD's need it.

Yes, agree.

> 
> Atleast our HW PMD's doesn't plan to use set_pkt_metada()/get_user_data()
> fn pointers in future if above is implemented.
> 
> >
> > >
> > > >
> > > > >
> > > > > As a brief code snippet:
> > > > >
> > > > > struct rte_security_ctx {
> > > > >         void *device;
> > > > >         /**< Crypto/ethernet device attached */
> > > > >         const struct rte_security_ops *ops;
> > > > >         /**< Pointer to security ops for the device */
> > > > >         uint16_t sess_cnt;
> > > > >         /**< Number of sessions attached to this context */
> > > > > +     int (*set_pkt_mdata)(void *, struct rte_security_session *, struct rte_mbuf *,  void *);
> > > > > };
> > > > >
> > > > > static inline int
> > > > > rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> > > > >                               struct rte_security_session *sess,
> > > > >                               struct rte_mbuf *m, void *params)
> > > > > {
> > > > >      /* fast-path */
> > > > >       if (instance->set_pkt_mdata == NULL) {
> > > > >              *rte_security_dynfield(m) = (rte_security_dynfield_t)(session->sess_priv_data);
> > > > >              return 0;
> > > > >        /* slow path */
> > > > >        } else
> > > > >            return instance->set_pkt_mdata(instance->device, sess, m, params);
> > > > > }
> > > > >
> > > > > That probably would be an ABI breakage (new fileld in rte_security_ctx) and would require
> > > > > some trivial changes for all existing PMDs that use RTE_SECURITY_TX_OFLOAD_NEED_MDATA
> > > > > (ctx_create()), but hopefully will benefit everyone.
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > I'm fine to
> > > > > > > > introduce a burst call for the same(I was thinking to propose it in future) to
> > > > > > > > compensate for the overhead.
> > > > > > > >
> > > > > > > > If rte_security_set_pkt_metadata() was not a PMD specific function ptr call and
> > > > > > > > rte_mbuf had space for struct rte_security_session pointer,
> > > > > > >
> > > > > > > But it does, see above.
> > > > > > > In fact it even more flexible - because it is driver specific, you are not limited to one 64-bit field.
> > > > > > > If your PMD requires more data to be associated with mbuf
> > > > > > > - you can request it via mbuf_dynfield and store there whatever is needed.
> > > > > > >
> > > > > > > > then then I guess it would have been better to do the way you proposed.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This patch is trying to enforce semantics as above so that
> > > > > > > > > > > > > > rte_security_set_pkt_metadata() can predict what comes in the pkt when he is
> > > > > > > > > > > > > > called.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also think above sequence is what Linux kernel stack or other stacks follow.
> > > > > > > > > > > > > > Does it makes sense ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Once called,
> > > > > > > > > > > > > > > > +Layer 3 and above data cannot be modified or moved around unless
> > > > > > > > > > > > > > > > +``rte_security_set_pkt_metadata()`` is called again.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  For inline protocol offloaded ingress traffic, the application can register a
> > > > > > > > > > > > > > > >  pointer, ``userdata`` , in the security session. When the packet is received,
> > > > > > > > > > > > > > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > > index bb38d7f58..9d8e3ddc8 100644
> > > > > > > > > > > > > > > > --- a/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > > +++ b/lib/mbuf/rte_mbuf_core.h
> > > > > > > > > > > > > > > > @@ -228,6 +228,8 @@ extern "C" {
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  /**
> > > > > > > > > > > > > > > >   * Request security offload processing on the TX packet.
> > > > > > > > > > > > > > > > + * To use Tx security offload, the user needs to fill l2_len in mbuf
> > > > > > > > > > > > > > > > + * indicating L2 header size and where L3 header starts.
> > > > > > > > > > > > > > > >   */
> > > > > > > > > > > > > > > >  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > 2.25.1
> > > > > > > > > > > > > > >

[dpdk-dev] [PATCH v2 0/3] security: Improve inline fast path routines

[Nithin Dabilpuram]

Improvements to Inline inbound and outbound processing fast path routines
rte_security_set_pkt_metadata() and rte_security_get_userdata() to make
them inline functions and also provide mechanism for drivers to support
fast userdata and metadata access instead of driver specific per-pkt
function callbacks.

This series updates requirements of mbuf fields to be updated for outbound
inline processing.

Nithin Dabilpuram (3):
  security: enforce semantics for Tx inline processing
  security: add option for faster udata or mdata access
  examples/ipsec-secgw: update L2 length for Tx

v2:
- Remove restrictions on rte_security_set_pkt_metadata() w.r.t pkt content
- Add inline functions for rte_security_set_pkt_metadata() and 
  rte_security_get_userdata() and also faster mdata, udata access via
  patch 2/3

 doc/guides/nics/features.rst        |  2 ++
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 42 ++++++++++++++++++++-----------
 lib/mbuf/rte_mbuf_core.h            |  2 ++
 lib/security/rte_security.c         |  8 +++---
 lib/security/rte_security.h         | 49 ++++++++++++++++++++++++++++++++++---
 lib/security/version.map            |  2 ++
 7 files changed, 85 insertions(+), 22 deletions(-)

-- 
2.8.4

[dpdk-dev] [PATCH v2 1/3] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

Not all net PMD's/HW can parse packet and identify L2 header and
L3 header locations on Tx. This is inline with other Tx offloads
requirements such as L3 checksum, L4 checksum offload, etc,
where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
able to generate checksum. Since Inline IPSec is also such a Tx
offload, some PMD's at least need mbuf.l2_len to be valid to
find L3 header and perform Outbound IPSec processing.

Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 doc/guides/nics/features.rst | 2 ++
 lib/mbuf/rte_mbuf_core.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a96e12d..4fce8cd 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f..9d8e3dd 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.8.4

[dpdk-dev] [PATCH v2 2/3] security: add option for faster udata or mdata access

[Nithin Dabilpuram]

Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
methods to set pkt metadata on Inline outbound and get userdata
after Inline inbound processing is always driver specific callbacks.

For drivers that do not have much to do in the callbacks but just
to update metadata in rte_security dynamic field and get userdata
from rte_security dynamic field, having to just to PMD specific
callback is costly per packet operation. This patch provides
a mechanism to do the same in inline function and avoid function
pointer jump if a driver supports the same.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 lib/security/rte_security.c |  8 ++++----
 lib/security/rte_security.h | 49 +++++++++++++++++++++++++++++++++++++++++----
 lib/security/version.map    |  2 ++
 3 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
index e8116d5..fe81ed3 100644
--- a/lib/security/rte_security.c
+++ b/lib/security/rte_security.c
@@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
 }
 
 int
-rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
-			      struct rte_security_session *sess,
-			      struct rte_mbuf *m, void *params)
+__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+				struct rte_security_session *sess,
+				struct rte_mbuf *m, void *params)
 {
 #ifdef RTE_DEBUG
 	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
@@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 }
 
 void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
 {
 	void *userdata = NULL;
 
diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
index 88d31de..da1108b 100644
--- a/lib/security/rte_security.h
+++ b/lib/security/rte_security.h
@@ -71,8 +71,18 @@ struct rte_security_ctx {
 	/**< Pointer to security ops for the device */
 	uint16_t sess_cnt;
 	/**< Number of sessions attached to this context */
+	uint32_t flags;
+	/**< Flags for security context */
 };
 
+#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
+/**< Driver uses fast metadata update without using driver specific callback */
+
+#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
+/**< Driver provides udata using fast method without using driver specific
+ * callback.
+ */
+
 /**
  * IPSEC tunnel parameters
  *
@@ -493,6 +503,12 @@ static inline bool rte_security_dynfield_is_registered(void)
 	return rte_security_dynfield_offset >= 0;
 }
 
+/** Function to call PMD specific function pointer set_pkt_metadata() */
+__rte_experimental
+extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+					   struct rte_security_session *sess,
+					   struct rte_mbuf *m, void *params);
+
 /**
  *  Updates the buffer with device-specific defined metadata
  *
@@ -506,10 +522,27 @@ static inline bool rte_security_dynfield_is_registered(void)
  *  - On success, zero.
  *  - On failure, a negative value.
  */
-int
+static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 			      struct rte_security_session *sess,
-			      struct rte_mbuf *mb, void *params);
+			      struct rte_mbuf *mb, void *params)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
+		*rte_security_dynfield(mb) =
+			(rte_security_dynfield_t)(sess->sess_private_data);
+		return 0;
+	}
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_set_pkt_metadata(instance->device, sess, mb,
+					       params);
+}
+
+/** Function to call PMD specific function pointer get_userdata() */
+__rte_experimental
+extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
+					 uint64_t md);
 
 /**
  * Get userdata associated with the security session. Device specific metadata
@@ -529,8 +562,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
  *  - On failure, NULL
  */
 __rte_experimental
-void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
+static inline void *
+rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
+		return (void *)md;
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_get_userdata(instance, md);
+}
 
 /**
  * Attach a session to a symmetric crypto operation
diff --git a/lib/security/version.map b/lib/security/version.map
index 2277555..e1c8148 100644
--- a/lib/security/version.map
+++ b/lib/security/version.map
@@ -20,4 +20,6 @@ EXPERIMENTAL {
 	rte_security_get_userdata;
 	rte_security_session_stats_get;
 	rte_security_session_update;
+	__rte_security_set_pkt_metadata;
+	__rte_security_get_userdata;
 };
-- 
2.8.4

[dpdk-dev] [PATCH v2 3/3] examples/ipsec-secgw: update L2 length for Tx

[Nithin Dabilpuram]

Update mbuf.l2_len with L2 header size for outbound
inline processing.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 42 ++++++++++++++++++++++++-------------
 2 files changed, 30 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d34..7ad94cb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22d..9c359cb 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -43,6 +48,9 @@ update_mac_addrs(struct rte_mbuf *pkt, uint16_t portid)
 {
 	struct rte_ether_hdr *ethhdr;
 
+	/* Provide L2 len for Outbound processing */
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	ethhdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *);
 	memcpy(&ethhdr->s_addr, &ethaddr_tbl[portid].src, RTE_ETHER_ADDR_LEN);
 	memcpy(&ethhdr->d_addr, &ethaddr_tbl[portid].dst, RTE_ETHER_ADDR_LEN);
@@ -60,7 +68,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +104,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +366,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
+
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -398,7 +408,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +422,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +434,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +472,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Provide L2 len for Outbound processing */
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.8.4

[dpdk-dev] [PATCH v3 0/3] security: Improve inline fast path routines

[Nithin Dabilpuram]

Improvements to Inline inbound and outbound processing fast path routines
rte_security_set_pkt_metadata() and rte_security_get_userdata() to make
them inline functions and also provide mechanism for drivers to support
fast userdata and metadata access instead of driver specific per-pkt
function callbacks.

This series updates requirements of mbuf fields to be updated for outbound
inline processing.

Nithin Dabilpuram (3):
  security: enforce semantics for Tx inline processing
  security: add option for faster udata or mdata access
  examples/ipsec-secgw: update event mode inline path

v3:
- Rebased and fixed compilation issue with rte_security_get_userdata() on
  32-bit platform
- Updated l2_len on patch 3/3 only for outbound.

v2:
- Remove restrictions on rte_security_set_pkt_metadata() w.r.t pkt content
- Add inline functions for rte_security_set_pkt_metadata() and 
  rte_security_get_userdata() and also faster mdata, udata access via
  patch 2/3

 doc/guides/nics/features.rst        |  2 ++
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++-----------
 lib/mbuf/rte_mbuf_core.h            |  2 ++
 lib/security/rte_security.c         |  8 +++---
 lib/security/rte_security.h         | 49 ++++++++++++++++++++++++++++++++++---
 lib/security/version.map            |  2 ++
 7 files changed, 84 insertions(+), 22 deletions(-)

-- 
2.8.4

[dpdk-dev] [PATCH v3 1/3] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

Not all net PMD's/HW can parse packet and identify L2 header and
L3 header locations on Tx. This is inline with other Tx offloads
requirements such as L3 checksum, L4 checksum offload, etc,
where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
able to generate checksum. Since Inline IPSec is also such a Tx
offload, some PMD's at least need mbuf.l2_len to be valid to
find L3 header and perform Outbound IPSec processing.

Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 doc/guides/nics/features.rst | 2 ++
 lib/mbuf/rte_mbuf_core.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a96e12d..4fce8cd 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f..9d8e3dd 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.8.4

[dpdk-dev] [PATCH v3 2/3] security: add option for faster udata or mdata access

[Nithin Dabilpuram]

Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
methods to set pkt metadata on Inline outbound and get userdata
after Inline inbound processing is always driver specific callbacks.

For drivers that do not have much to do in the callbacks but just
to update metadata in rte_security dynamic field and get userdata
from rte_security dynamic field, having to just to PMD specific
callback is costly per packet operation. This patch provides
a mechanism to do the same in inline function and avoid function
pointer jump if a driver supports the same.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 lib/security/rte_security.c |  8 ++++----
 lib/security/rte_security.h | 49 +++++++++++++++++++++++++++++++++++++++++----
 lib/security/version.map    |  2 ++
 3 files changed, 51 insertions(+), 8 deletions(-)

diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
index e8116d5..fe81ed3 100644
--- a/lib/security/rte_security.c
+++ b/lib/security/rte_security.c
@@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
 }
 
 int
-rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
-			      struct rte_security_session *sess,
-			      struct rte_mbuf *m, void *params)
+__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+				struct rte_security_session *sess,
+				struct rte_mbuf *m, void *params)
 {
 #ifdef RTE_DEBUG
 	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
@@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 }
 
 void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
 {
 	void *userdata = NULL;
 
diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
index 88d31de..06267f6 100644
--- a/lib/security/rte_security.h
+++ b/lib/security/rte_security.h
@@ -71,8 +71,18 @@ struct rte_security_ctx {
 	/**< Pointer to security ops for the device */
 	uint16_t sess_cnt;
 	/**< Number of sessions attached to this context */
+	uint32_t flags;
+	/**< Flags for security context */
 };
 
+#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
+/**< Driver uses fast metadata update without using driver specific callback */
+
+#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
+/**< Driver provides udata using fast method without using driver specific
+ * callback.
+ */
+
 /**
  * IPSEC tunnel parameters
  *
@@ -493,6 +503,12 @@ static inline bool rte_security_dynfield_is_registered(void)
 	return rte_security_dynfield_offset >= 0;
 }
 
+/** Function to call PMD specific function pointer set_pkt_metadata() */
+__rte_experimental
+extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+					   struct rte_security_session *sess,
+					   struct rte_mbuf *m, void *params);
+
 /**
  *  Updates the buffer with device-specific defined metadata
  *
@@ -506,10 +522,27 @@ static inline bool rte_security_dynfield_is_registered(void)
  *  - On success, zero.
  *  - On failure, a negative value.
  */
-int
+static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 			      struct rte_security_session *sess,
-			      struct rte_mbuf *mb, void *params);
+			      struct rte_mbuf *mb, void *params)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
+		*rte_security_dynfield(mb) =
+			(rte_security_dynfield_t)(sess->sess_private_data);
+		return 0;
+	}
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_set_pkt_metadata(instance->device, sess, mb,
+					       params);
+}
+
+/** Function to call PMD specific function pointer get_userdata() */
+__rte_experimental
+extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
+					 uint64_t md);
 
 /**
  * Get userdata associated with the security session. Device specific metadata
@@ -529,8 +562,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
  *  - On failure, NULL
  */
 __rte_experimental
-void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
+static inline void *
+rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
+		return (void *)(uintptr_t)md;
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_get_userdata(instance, md);
+}
 
 /**
  * Attach a session to a symmetric crypto operation
diff --git a/lib/security/version.map b/lib/security/version.map
index 2277555..e1c8148 100644
--- a/lib/security/version.map
+++ b/lib/security/version.map
@@ -20,4 +20,6 @@ EXPERIMENTAL {
 	rte_security_get_userdata;
 	rte_security_session_stats_get;
 	rte_security_session_update;
+	__rte_security_set_pkt_metadata;
+	__rte_security_get_userdata;
 };
-- 
2.8.4

[dpdk-dev] [PATCH v3 3/3] examples/ipsec-secgw: update event mode inline path

[Nithin Dabilpuram]

Update mbuf.l2_len with L2 header size for outbound
inline processing.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++++++-------------
 2 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d34..7ad94cb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22d..c545497 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -60,7 +65,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +101,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +363,8 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -367,6 +373,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	port_id = sa->portid;
 
 send_pkt:
+	/* Provide L2 len for Outbound processing */
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	/* Update mac addresses */
 	update_mac_addrs(pkt, port_id);
 
@@ -398,7 +407,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +421,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +433,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +471,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Provide L2 len for Outbound processing */
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.8.4

[dpdk-dev] [PATCH v4 0/4] security: Improve inline fast path routines

[Nithin Dabilpuram]

Improvements to Inline inbound and outbound processing fast path routines
rte_security_set_pkt_metadata() and rte_security_get_userdata() to make
them inline functions and also provide mechanism for drivers to support
fast userdata and metadata access instead of driver specific per-pkt
function callbacks.

This series updates requirements of mbuf fields to be updated for outbound
inline processing.

Nithin Dabilpuram (4):
  security: enforce semantics for Tx inline processing
  security: add option for faster udata or mdata access
  examples/ipsec-secgw: update event mode inline path
  doc: remove deprecation notice for security fast path change

v4:
- Removed entry from deprecation notice.
- Fixed issue with rte_security_set_pkt_metadata() to pass instance instead
  of device ptr to non-inline C function.

v3:
- Rebased and fixed compilation issue with rte_security_get_userdata() on
  32-bit platform
- Updated l2_len on patch 3/3 only for outbound.

v2:
- Remove restrictions on rte_security_set_pkt_metadata() w.r.t pkt content
- Add inline functions for rte_security_set_pkt_metadata() and 
  rte_security_get_userdata() and also faster mdata, udata access via
  patch 2/3

 doc/guides/nics/features.rst         |  2 ++
 doc/guides/rel_notes/deprecation.rst |  4 ---
 examples/ipsec-secgw/ipsec-secgw.c   |  2 ++
 examples/ipsec-secgw/ipsec_worker.c  | 41 +++++++++++++++++++-----------
 lib/mbuf/rte_mbuf_core.h             |  2 ++
 lib/security/rte_security.c          |  8 +++---
 lib/security/rte_security.h          | 48 +++++++++++++++++++++++++++++++++---
 lib/security/version.map             |  2 ++
 8 files changed, 83 insertions(+), 26 deletions(-)

-- 
2.8.4

[dpdk-dev] [PATCH v4 1/4] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

Not all net PMD's/HW can parse packet and identify L2 header and
L3 header locations on Tx. This is inline with other Tx offloads
requirements such as L3 checksum, L4 checksum offload, etc,
where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
able to generate checksum. Since Inline IPSec is also such a Tx
offload, some PMD's at least need mbuf.l2_len to be valid to
find L3 header and perform Outbound IPSec processing.

Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 doc/guides/nics/features.rst | 2 ++
 lib/mbuf/rte_mbuf_core.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a96e12d..4fce8cd 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f..9d8e3dd 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.8.4

[dpdk-dev] [PATCH v4 2/4] security: add option for faster udata or mdata access

[Nithin Dabilpuram]

Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
methods to set pkt metadata on Inline outbound and get userdata
after Inline inbound processing is always driver specific callbacks.

For drivers that do not have much to do in the callbacks but just
to update metadata in rte_security dynamic field and get userdata
from rte_security dynamic field, having to just to PMD specific
callback is costly per packet operation. This patch provides
a mechanism to do the same in inline function and avoid function
pointer jump if a driver supports the same.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 lib/security/rte_security.c |  8 ++++----
 lib/security/rte_security.h | 48 +++++++++++++++++++++++++++++++++++++++++----
 lib/security/version.map    |  2 ++
 3 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
index e8116d5..fe81ed3 100644
--- a/lib/security/rte_security.c
+++ b/lib/security/rte_security.c
@@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
 }
 
 int
-rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
-			      struct rte_security_session *sess,
-			      struct rte_mbuf *m, void *params)
+__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+				struct rte_security_session *sess,
+				struct rte_mbuf *m, void *params)
 {
 #ifdef RTE_DEBUG
 	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
@@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 }
 
 void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
 {
 	void *userdata = NULL;
 
diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
index 88d31de..1f73dee 100644
--- a/lib/security/rte_security.h
+++ b/lib/security/rte_security.h
@@ -71,8 +71,18 @@ struct rte_security_ctx {
 	/**< Pointer to security ops for the device */
 	uint16_t sess_cnt;
 	/**< Number of sessions attached to this context */
+	uint32_t flags;
+	/**< Flags for security context */
 };
 
+#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
+/**< Driver uses fast metadata update without using driver specific callback */
+
+#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
+/**< Driver provides udata using fast method without using driver specific
+ * callback.
+ */
+
 /**
  * IPSEC tunnel parameters
  *
@@ -493,6 +503,12 @@ static inline bool rte_security_dynfield_is_registered(void)
 	return rte_security_dynfield_offset >= 0;
 }
 
+/** Function to call PMD specific function pointer set_pkt_metadata() */
+__rte_experimental
+extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+					   struct rte_security_session *sess,
+					   struct rte_mbuf *m, void *params);
+
 /**
  *  Updates the buffer with device-specific defined metadata
  *
@@ -506,10 +522,26 @@ static inline bool rte_security_dynfield_is_registered(void)
  *  - On success, zero.
  *  - On failure, a negative value.
  */
-int
+static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 			      struct rte_security_session *sess,
-			      struct rte_mbuf *mb, void *params);
+			      struct rte_mbuf *mb, void *params)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
+		*rte_security_dynfield(mb) =
+			(rte_security_dynfield_t)(sess->sess_private_data);
+		return 0;
+	}
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_set_pkt_metadata(instance, sess, mb, params);
+}
+
+/** Function to call PMD specific function pointer get_userdata() */
+__rte_experimental
+extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
+					 uint64_t md);
 
 /**
  * Get userdata associated with the security session. Device specific metadata
@@ -529,8 +561,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
  *  - On failure, NULL
  */
 __rte_experimental
-void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
+static inline void *
+rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
+		return (void *)(uintptr_t)md;
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_get_userdata(instance, md);
+}
 
 /**
  * Attach a session to a symmetric crypto operation
diff --git a/lib/security/version.map b/lib/security/version.map
index 2277555..e1c8148 100644
--- a/lib/security/version.map
+++ b/lib/security/version.map
@@ -20,4 +20,6 @@ EXPERIMENTAL {
 	rte_security_get_userdata;
 	rte_security_session_stats_get;
 	rte_security_session_update;
+	__rte_security_set_pkt_metadata;
+	__rte_security_get_userdata;
 };
-- 
2.8.4

[dpdk-dev] [PATCH v4 3/4] examples/ipsec-secgw: update event mode inline path

[Nithin Dabilpuram]

Update mbuf.l2_len with L2 header size for outbound
inline processing.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++++++-------------
 2 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d34..7ad94cb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22d..c545497 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -60,7 +65,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +101,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +363,8 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -367,6 +373,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	port_id = sa->portid;
 
 send_pkt:
+	/* Provide L2 len for Outbound processing */
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	/* Update mac addresses */
 	update_mac_addrs(pkt, port_id);
 
@@ -398,7 +407,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +421,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +433,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +471,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Provide L2 len for Outbound processing */
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.8.4

[dpdk-dev] [PATCH v4 4/4] doc: remove deprecation notice for security fast path change

[Nithin Dabilpuram]

Remove deprecation notice for security fast path change.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
 doc/guides/rel_notes/deprecation.rst | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 76a4abf..2e38a36 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -279,10 +279,6 @@ Deprecation Notices
   content. On Linux and FreeBSD, supported prior to DPDK 20.11,
   original structure will be kept until DPDK 21.11.
 
-* security: The functions ``rte_security_set_pkt_metadata`` and
-  ``rte_security_get_userdata`` will be made inline functions and additional
-  flags will be added in structure ``rte_security_ctx`` in DPDK 21.11.
-
 * cryptodev: The structure ``rte_crypto_op`` would be updated to reduce
   reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
   information from the crypto/security operation. This field will be used to
-- 
2.8.4

Re: [dpdk-dev] [PATCH v4 4/4] doc: remove deprecation notice for security fast path change

[Akhil Goyal]

> Remove deprecation notice for security fast path change.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst | 4 ----
>  1 file changed, 4 deletions(-)
> 
> diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> index 76a4abf..2e38a36 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -279,10 +279,6 @@ Deprecation Notices
>    content. On Linux and FreeBSD, supported prior to DPDK 20.11,
>    original structure will be kept until DPDK 21.11.
> 
> -* security: The functions ``rte_security_set_pkt_metadata`` and
> -  ``rte_security_get_userdata`` will be made inline functions and additional
> -  flags will be added in structure ``rte_security_ctx`` in DPDK 21.11.
> -
Corresponding update in release notes is also needed when a deprecation notice is removed.
You can update in the ABI/API changes section of the release notes.
Also squash this patch in the 2/4 patch. Doc changes should be part of the patch.

Regards

Re: [dpdk-dev] [PATCH v4 1/4] security: enforce semantics for Tx inline processing

[Akhil Goyal]

> Not all net PMD's/HW can parse packet and identify L2 header and
> L3 header locations on Tx. This is inline with other Tx offloads
> requirements such as L3 checksum, L4 checksum offload, etc,
> where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
> able to generate checksum. Since Inline IPSec is also such a Tx
> offload, some PMD's at least need mbuf.l2_len to be valid to
> find L3 header and perform Outbound IPSec processing.
> 
> Hence, this patch updates documentation to enforce setting
> mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> for Inline IPSec Crypto / Protocol offload processing to
> work on Tx.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

Re: [dpdk-dev] [PATCH v4 2/4] security: add option for faster udata or mdata access

[Akhil Goyal]

> -----Original Message-----
> From: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Sent: Thursday, August 12, 2021 6:03 PM
> To: dev@dpdk.org
> Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Akhil Goyal
> <gakhil@marvell.com>; hemant.agrawal@nxp.com; thomas@monjalon.net;
> g.singh@nxp.com; ferruh.yigit@intel.com; roy.fan.zhang@intel.com;
> olivier.matz@6wind.com; declan.doherty@intel.com;
> radu.nicolau@intel.com; jiawenwu@trustnetic.com;
> konstantin.ananyev@intel.com; Nithin Kumar Dabilpuram
> <ndabilpuram@marvell.com>
> Subject: [PATCH v4 2/4] security: add option for faster udata or mdata access
> 
> Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
> methods to set pkt metadata on Inline outbound and get userdata
> after Inline inbound processing is always driver specific callbacks.
> 
> For drivers that do not have much to do in the callbacks but just
> to update metadata in rte_security dynamic field and get userdata
> from rte_security dynamic field, having to just to PMD specific
> callback is costly per packet operation. This patch provides
> a mechanism to do the same in inline function and avoid function
> pointer jump if a driver supports the same.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> ---
>  lib/security/rte_security.c |  8 ++++----
>  lib/security/rte_security.h | 48
> +++++++++++++++++++++++++++++++++++++++++----
>  lib/security/version.map    |  2 ++
>  3 files changed, 50 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
> index e8116d5..fe81ed3 100644
> --- a/lib/security/rte_security.c
> +++ b/lib/security/rte_security.c
> @@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx
> *instance,
>  }
> 
>  int
> -rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> -			      struct rte_security_session *sess,
> -			      struct rte_mbuf *m, void *params)
> +__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> +				struct rte_security_session *sess,
> +				struct rte_mbuf *m, void *params)
>  {
>  #ifdef RTE_DEBUG
>  	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
> @@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct
> rte_security_ctx *instance,
>  }
> 
>  void *
> -rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
> +__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
>  {
>  	void *userdata = NULL;
> 
> diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
> index 88d31de..1f73dee 100644
> --- a/lib/security/rte_security.h
> +++ b/lib/security/rte_security.h
> @@ -71,8 +71,18 @@ struct rte_security_ctx {
>  	/**< Pointer to security ops for the device */
>  	uint16_t sess_cnt;
>  	/**< Number of sessions attached to this context */
> +	uint32_t flags;
> +	/**< Flags for security context */
>  };
> 
> +#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
> +/**< Driver uses fast metadata update without using driver specific callback
> */
> +
> +#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
> +/**< Driver provides udata using fast method without using driver specific
> + * callback.
> + */
> +
>  /**
>   * IPSEC tunnel parameters
>   *
> @@ -493,6 +503,12 @@ static inline bool
> rte_security_dynfield_is_registered(void)
>  	return rte_security_dynfield_offset >= 0;
>  }
> 
> +/** Function to call PMD specific function pointer set_pkt_metadata() */
> +__rte_experimental
> +extern int __rte_security_set_pkt_metadata(struct rte_security_ctx
> *instance,
> +					   struct rte_security_session *sess,
> +					   struct rte_mbuf *m, void *params);
> +
>  /**
>   *  Updates the buffer with device-specific defined metadata
>   *
> @@ -506,10 +522,26 @@ static inline bool
> rte_security_dynfield_is_registered(void)
>   *  - On success, zero.
>   *  - On failure, a negative value.
>   */
> -int
> +static inline int
>  rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>  			      struct rte_security_session *sess,
> -			      struct rte_mbuf *mb, void *params);
> +			      struct rte_mbuf *mb, void *params)
> +{
> +	/* Fast Path */
> +	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
> +		*rte_security_dynfield(mb) =
> +			(rte_security_dynfield_t)(sess->sess_private_data);
> +		return 0;
> +	}
> +
> +	/* Jump to PMD specific function pointer */
> +	return __rte_security_set_pkt_metadata(instance, sess, mb,
> params);
> +}
> +
> +/** Function to call PMD specific function pointer get_userdata() */
> +__rte_experimental
> +extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
> +					 uint64_t md);
> 
>  /**
>   * Get userdata associated with the security session. Device specific
> metadata
> @@ -529,8 +561,16 @@ rte_security_set_pkt_metadata(struct
> rte_security_ctx *instance,
>   *  - On failure, NULL
>   */
>  __rte_experimental
> -void *
> -rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
> +static inline void *
> +rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
> +{
> +	/* Fast Path */
> +	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
> +		return (void *)(uintptr_t)md;
> +
> +	/* Jump to PMD specific function pointer */
> +	return __rte_security_get_userdata(instance, md);
> +}
> 
>  /**
>   * Attach a session to a symmetric crypto operation
> diff --git a/lib/security/version.map b/lib/security/version.map
> index 2277555..e1c8148 100644
> --- a/lib/security/version.map
> +++ b/lib/security/version.map
> @@ -20,4 +20,6 @@ EXPERIMENTAL {
>  	rte_security_get_userdata;
>  	rte_security_session_stats_get;
>  	rte_security_session_update;
> +	__rte_security_set_pkt_metadata;
> +	__rte_security_get_userdata;
>  };
> --
> 2.8.4

Re: [dpdk-dev] [PATCH v4 2/4] security: add option for faster udata or mdata access

[Akhil Goyal]

> Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
> methods to set pkt metadata on Inline outbound and get userdata
> after Inline inbound processing is always driver specific callbacks.
> 
> For drivers that do not have much to do in the callbacks but just
> to update metadata in rte_security dynamic field and get userdata
> from rte_security dynamic field, having to just to PMD specific
> callback is costly per packet operation. This patch provides
> a mechanism to do the same in inline function and avoid function
> pointer jump if a driver supports the same.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> ---
Acked-by: Akhil Goyal <gakhil@marvell.com>

Re: [dpdk-dev] [PATCH v4 3/4] examples/ipsec-secgw: update event mode inline path

[Akhil Goyal]

> Update mbuf.l2_len with L2 header size for outbound
> inline processing.
> 
> This patch also fixes a bug in arg parsing.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> ---
Acked-by: Akhil Goyal <gakhil@marvell.com>

[dpdk-dev] [PATCH v5 0/3] security: Improve inline fast path routines

[Nithin Dabilpuram]

Improvements to Inline inbound and outbound processing fast path routines
rte_security_set_pkt_metadata() and rte_security_get_userdata() to make
them inline functions and also provide mechanism for drivers to support
fast userdata and metadata access instead of driver specific per-pkt
function callbacks.

This series updates requirements of mbuf fields to be updated for outbound
inline processing.

Nithin Dabilpuram (3):
  security: enforce semantics for Tx inline processing
  security: add option for faster udata or mdata access
  examples/ipsec-secgw: update event mode inline path

v5:
- Squash 4/4 patch to 2/4 and update release notes

v4:
- Removed entry from deprecation notice.
- Fixed issue with rte_security_set_pkt_metadata() to pass instance instead
  of device ptr to non-inline C function.

v3:
- Rebased and fixed compilation issue with rte_security_get_userdata() on
  32-bit platform
- Updated l2_len on patch 3/3 only for outbound.

v2:
- Remove restrictions on rte_security_set_pkt_metadata() w.r.t pkt content
- Add inline functions for rte_security_set_pkt_metadata() and 
  rte_security_get_userdata() and also faster mdata, udata access via
  patch 2/3

 doc/guides/nics/features.rst           |  2 ++
 doc/guides/rel_notes/deprecation.rst   |  4 ---
 doc/guides/rel_notes/release_21_08.rst |  6 +++++
 examples/ipsec-secgw/ipsec-secgw.c     |  2 ++
 examples/ipsec-secgw/ipsec_worker.c    | 41 +++++++++++++++++++----------
 lib/mbuf/rte_mbuf_core.h               |  2 ++
 lib/security/rte_security.c            |  8 +++---
 lib/security/rte_security.h            | 48 +++++++++++++++++++++++++++++++---
 lib/security/version.map               |  2 ++
 9 files changed, 89 insertions(+), 26 deletions(-)

-- 
2.8.4

[dpdk-dev] [PATCH v5 1/3] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

Not all net PMD's/HW can parse packet and identify L2 header and
L3 header locations on Tx. This is inline with other Tx offloads
requirements such as L3 checksum, L4 checksum offload, etc,
where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
able to generate checksum. Since Inline IPSec is also such a Tx
offload, some PMD's at least need mbuf.l2_len to be valid to
find L3 header and perform Outbound IPSec processing.

Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 doc/guides/nics/features.rst | 2 ++
 lib/mbuf/rte_mbuf_core.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a96e12d..4fce8cd 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f..9d8e3dd 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.8.4

[dpdk-dev] [PATCH v5 2/3] security: add option for faster udata or mdata access

[Nithin Dabilpuram]

Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
methods to set pkt metadata on Inline outbound and get userdata
after Inline inbound processing is always driver specific callbacks.

For drivers that do not have much to do in the callbacks but just
to update metadata in rte_security dynamic field and get userdata
from rte_security dynamic field, having to just to PMD specific
callback is costly per packet operation. This patch provides
a mechanism to do the same in inline function and avoid function
pointer jump if a driver supports the same.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 doc/guides/rel_notes/deprecation.rst   |  4 ---
 doc/guides/rel_notes/release_21_08.rst |  6 +++++
 lib/security/rte_security.c            |  8 +++---
 lib/security/rte_security.h            | 48 +++++++++++++++++++++++++++++++---
 lib/security/version.map               |  2 ++
 5 files changed, 56 insertions(+), 12 deletions(-)

diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 59445a6..70ef45e 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -276,10 +276,6 @@ Deprecation Notices
   content. On Linux and FreeBSD, supported prior to DPDK 20.11,
   original structure will be kept until DPDK 21.11.
 
-* security: The functions ``rte_security_set_pkt_metadata`` and
-  ``rte_security_get_userdata`` will be made inline functions and additional
-  flags will be added in structure ``rte_security_ctx`` in DPDK 21.11.
-
 * cryptodev: The structure ``rte_crypto_op`` would be updated to reduce
   reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
   information from the crypto/security operation. This field will be used to
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index b4cbf2d..59ff15a 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -223,6 +223,12 @@ ABI Changes
 
 * No ABI change that would break compatibility with 20.11.
 
+* security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
+  routines used by Inline outbound and Inline inbound security processing are
+  made inline and enhanced to do simple 64-bit set/get for PMD's that donot
+  have much processing in PMD specific callbacks but just 64-bit set/get.
+  This avoids a per-pkt function pointer jump overhead for such PMD's.
+
 
 Known Issues
 ------------
diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
index e8116d5..fe81ed3 100644
--- a/lib/security/rte_security.c
+++ b/lib/security/rte_security.c
@@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
 }
 
 int
-rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
-			      struct rte_security_session *sess,
-			      struct rte_mbuf *m, void *params)
+__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+				struct rte_security_session *sess,
+				struct rte_mbuf *m, void *params)
 {
 #ifdef RTE_DEBUG
 	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
@@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 }
 
 void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
 {
 	void *userdata = NULL;
 
diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
index 2e136d7..3124134 100644
--- a/lib/security/rte_security.h
+++ b/lib/security/rte_security.h
@@ -71,8 +71,18 @@ struct rte_security_ctx {
 	/**< Pointer to security ops for the device */
 	uint16_t sess_cnt;
 	/**< Number of sessions attached to this context */
+	uint32_t flags;
+	/**< Flags for security context */
 };
 
+#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
+/**< Driver uses fast metadata update without using driver specific callback */
+
+#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
+/**< Driver provides udata using fast method without using driver specific
+ * callback.
+ */
+
 /**
  * IPSEC tunnel parameters
  *
@@ -494,6 +504,12 @@ static inline bool rte_security_dynfield_is_registered(void)
 	return rte_security_dynfield_offset >= 0;
 }
 
+/** Function to call PMD specific function pointer set_pkt_metadata() */
+__rte_experimental
+extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+					   struct rte_security_session *sess,
+					   struct rte_mbuf *m, void *params);
+
 /**
  *  Updates the buffer with device-specific defined metadata
  *
@@ -507,10 +523,26 @@ static inline bool rte_security_dynfield_is_registered(void)
  *  - On success, zero.
  *  - On failure, a negative value.
  */
-int
+static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 			      struct rte_security_session *sess,
-			      struct rte_mbuf *mb, void *params);
+			      struct rte_mbuf *mb, void *params)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
+		*rte_security_dynfield(mb) =
+			(rte_security_dynfield_t)(sess->sess_private_data);
+		return 0;
+	}
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_set_pkt_metadata(instance, sess, mb, params);
+}
+
+/** Function to call PMD specific function pointer get_userdata() */
+__rte_experimental
+extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
+					 uint64_t md);
 
 /**
  * Get userdata associated with the security session. Device specific metadata
@@ -530,8 +562,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
  *  - On failure, NULL
  */
 __rte_experimental
-void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
+static inline void *
+rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
+		return (void *)(uintptr_t)md;
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_get_userdata(instance, md);
+}
 
 /**
  * Attach a session to a symmetric crypto operation
diff --git a/lib/security/version.map b/lib/security/version.map
index c44c7f5..45ace9c 100644
--- a/lib/security/version.map
+++ b/lib/security/version.map
@@ -20,4 +20,6 @@ EXPERIMENTAL {
 	rte_security_get_userdata;
 	rte_security_session_stats_get;
 	rte_security_session_update;
+	__rte_security_set_pkt_metadata;
+	__rte_security_get_userdata;
 };
-- 
2.8.4

[dpdk-dev] [PATCH v5 3/3] examples/ipsec-secgw: update event mode inline path

[Nithin Dabilpuram]

Update mbuf.l2_len with L2 header size for outbound
inline processing.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++++++-------------
 2 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d34..7ad94cb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22d..c545497 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -60,7 +65,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +101,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +363,8 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -367,6 +373,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	port_id = sa->portid;
 
 send_pkt:
+	/* Provide L2 len for Outbound processing */
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	/* Update mac addresses */
 	update_mac_addrs(pkt, port_id);
 
@@ -398,7 +407,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +421,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +433,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +471,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Provide L2 len for Outbound processing */
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.8.4

Re: [dpdk-dev] [PATCH v5 1/3] security: enforce semantics for Tx inline processing

[Ananyev, Konstantin]

> 
> Not all net PMD's/HW can parse packet and identify L2 header and
> L3 header locations on Tx. This is inline with other Tx offloads
> requirements such as L3 checksum, L4 checksum offload, etc,
> where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
> able to generate checksum. Since Inline IPSec is also such a Tx
> offload, some PMD's at least need mbuf.l2_len to be valid to
> find L3 header and perform Outbound IPSec processing.
> 
> Hence, this patch updates documentation to enforce setting
> mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> for Inline IPSec Crypto / Protocol offload processing to
> work on Tx.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  doc/guides/nics/features.rst | 2 ++
>  lib/mbuf/rte_mbuf_core.h     | 2 ++
>  2 files changed, 4 insertions(+)
> 
> diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
> index a96e12d..4fce8cd 100644
> --- a/doc/guides/nics/features.rst
> +++ b/doc/guides/nics/features.rst
> @@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
> 
>  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
>  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> +* **[uses]       mbuf**: ``mbuf.l2_len``.
>  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
>    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
>  * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
> @@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
> 
>  * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
>  * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
> +* **[uses]       mbuf**: ``mbuf.l2_len``.
>  * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
>    ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
>    ``capabilities_get``.
> diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
> index bb38d7f..9d8e3dd 100644
> --- a/lib/mbuf/rte_mbuf_core.h
> +++ b/lib/mbuf/rte_mbuf_core.h
> @@ -228,6 +228,8 @@ extern "C" {
> 
>  /**
>   * Request security offload processing on the TX packet.
> + * To use Tx security offload, the user needs to fill l2_len in mbuf
> + * indicating L2 header size and where L3 header starts.
>   */
>  #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
> 
> --

Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

> 2.8.4

Re: [dpdk-dev] [PATCH v5 2/3] security: add option for faster udata or mdata access

[Ananyev, Konstantin]

> 
> Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
> methods to set pkt metadata on Inline outbound and get userdata
> after Inline inbound processing is always driver specific callbacks.
> 
> For drivers that do not have much to do in the callbacks but just
> to update metadata in rte_security dynamic field and get userdata
> from rte_security dynamic field, having to just to PMD specific
> callback is costly per packet operation. This patch provides
> a mechanism to do the same in inline function and avoid function
> pointer jump if a driver supports the same.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  doc/guides/rel_notes/deprecation.rst   |  4 ---
>  doc/guides/rel_notes/release_21_08.rst |  6 +++++
>  lib/security/rte_security.c            |  8 +++---
>  lib/security/rte_security.h            | 48 +++++++++++++++++++++++++++++++---
>  lib/security/version.map               |  2 ++
>  5 files changed, 56 insertions(+), 12 deletions(-)
> 
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 59445a6..70ef45e 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -276,10 +276,6 @@ Deprecation Notices
>    content. On Linux and FreeBSD, supported prior to DPDK 20.11,
>    original structure will be kept until DPDK 21.11.
> 
> -* security: The functions ``rte_security_set_pkt_metadata`` and
> -  ``rte_security_get_userdata`` will be made inline functions and additional
> -  flags will be added in structure ``rte_security_ctx`` in DPDK 21.11.
> -
>  * cryptodev: The structure ``rte_crypto_op`` would be updated to reduce
>    reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
>    information from the crypto/security operation. This field will be used to
> diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
> index b4cbf2d..59ff15a 100644
> --- a/doc/guides/rel_notes/release_21_08.rst
> +++ b/doc/guides/rel_notes/release_21_08.rst
> @@ -223,6 +223,12 @@ ABI Changes
> 
>  * No ABI change that would break compatibility with 20.11.
> 
> +* security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
> +  routines used by Inline outbound and Inline inbound security processing are
> +  made inline and enhanced to do simple 64-bit set/get for PMD's that donot
> +  have much processing in PMD specific callbacks but just 64-bit set/get.
> +  This avoids a per-pkt function pointer jump overhead for such PMD's.
> +
> 
>  Known Issues
>  ------------
> diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
> index e8116d5..fe81ed3 100644
> --- a/lib/security/rte_security.c
> +++ b/lib/security/rte_security.c
> @@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
>  }
> 
>  int
> -rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> -			      struct rte_security_session *sess,
> -			      struct rte_mbuf *m, void *params)
> +__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> +				struct rte_security_session *sess,
> +				struct rte_mbuf *m, void *params)
>  {
>  #ifdef RTE_DEBUG
>  	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
> @@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>  }
> 
>  void *
> -rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
> +__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
>  {
>  	void *userdata = NULL;
> 
> diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
> index 2e136d7..3124134 100644
> --- a/lib/security/rte_security.h
> +++ b/lib/security/rte_security.h
> @@ -71,8 +71,18 @@ struct rte_security_ctx {
>  	/**< Pointer to security ops for the device */
>  	uint16_t sess_cnt;
>  	/**< Number of sessions attached to this context */
> +	uint32_t flags;
> +	/**< Flags for security context */
>  };
> 
> +#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
> +/**< Driver uses fast metadata update without using driver specific callback */

Probably worth to mention somewhere that it is driver responsibility to call 
rte_security_dynfield_register() to expose that flag.

> +
> +#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
> +/**< Driver provides udata using fast method without using driver specific
> + * callback.
> + */
> +
>  /**
>   * IPSEC tunnel parameters
>   *
> @@ -494,6 +504,12 @@ static inline bool rte_security_dynfield_is_registered(void)
>  	return rte_security_dynfield_offset >= 0;
>  }
> 
> +/** Function to call PMD specific function pointer set_pkt_metadata() */
> +__rte_experimental
> +extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
> +					   struct rte_security_session *sess,
> +					   struct rte_mbuf *m, void *params);
> +
>  /**
>   *  Updates the buffer with device-specific defined metadata
>   *
> @@ -507,10 +523,26 @@ static inline bool rte_security_dynfield_is_registered(void)
>   *  - On success, zero.
>   *  - On failure, a negative value.
>   */
> -int
> +static inline int
>  rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>  			      struct rte_security_session *sess,
> -			      struct rte_mbuf *mb, void *params);
> +			      struct rte_mbuf *mb, void *params)
> +{
> +	/* Fast Path */
> +	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
> +		*rte_security_dynfield(mb) =
> +			(rte_security_dynfield_t)(sess->sess_private_data);
> +		return 0;
> +	}
> +
> +	/* Jump to PMD specific function pointer */
> +	return __rte_security_set_pkt_metadata(instance, sess, mb, params);
> +}
> +
> +/** Function to call PMD specific function pointer get_userdata() */
> +__rte_experimental
> +extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
> +					 uint64_t md);
> 
>  /**
>   * Get userdata associated with the security session. Device specific metadata
> @@ -530,8 +562,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
>   *  - On failure, NULL
>   */
>  __rte_experimental
> -void *
> -rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
> +static inline void *
> +rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
> +{
> +	/* Fast Path */
> +	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
> +		return (void *)(uintptr_t)md;
> +
> +	/* Jump to PMD specific function pointer */
> +	return __rte_security_get_userdata(instance, md);
> +}
> 
>  /**
>   * Attach a session to a symmetric crypto operation
> diff --git a/lib/security/version.map b/lib/security/version.map
> index c44c7f5..45ace9c 100644
> --- a/lib/security/version.map
> +++ b/lib/security/version.map
> @@ -20,4 +20,6 @@ EXPERIMENTAL {
>  	rte_security_get_userdata;
>  	rte_security_session_stats_get;
>  	rte_security_session_update;
> +	__rte_security_set_pkt_metadata;
> +	__rte_security_get_userdata;
>  };
> --
> 2.8.4

Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

Re: [dpdk-dev] [PATCH v5 3/3] examples/ipsec-secgw: update event mode inline path

[Ananyev, Konstantin]

> Update mbuf.l2_len with L2 header size for outbound
> inline processing.
> 
> This patch also fixes a bug in arg parsing.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Akhil Goyal <gakhil@marvell.com>
> ---
>  examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
>  examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++++++-------------
>  2 files changed, 29 insertions(+), 14 deletions(-)
> 
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> index f252d34..7ad94cb 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
>  	char *end = NULL;
>  	unsigned long pm;
> 
> +	errno = 0;
> +
>  	/* parse hexadecimal string */
>  	pm = strtoul(portmask, &end, 16);
>  	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
> diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
> index 647e22d..c545497 100644
> --- a/examples/ipsec-secgw/ipsec_worker.c
> +++ b/examples/ipsec-secgw/ipsec_worker.c
> @@ -12,6 +12,11 @@
>  #include "ipsec-secgw.h"
>  #include "ipsec_worker.h"
> 
> +struct port_drv_mode_data {
> +	struct rte_security_session *sess;
> +	struct rte_security_ctx *ctx;
> +};
> +
>  static inline enum pkt_type
>  process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
>  {
> @@ -60,7 +65,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
> 
>  static inline void
>  prepare_out_sessions_tbl(struct sa_ctx *sa_out,
> -		struct rte_security_session **sess_tbl, uint16_t size)
> +			 struct port_drv_mode_data *data,
> +			 uint16_t size)
>  {
>  	struct rte_ipsec_session *pri_sess;
>  	struct ipsec_sa *sa;
> @@ -95,9 +101,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
>  		}
> 
>  		/* Use only first inline session found for a given port */
> -		if (sess_tbl[sa->portid])
> +		if (data[sa->portid].sess)
>  			continue;
> -		sess_tbl[sa->portid] = pri_sess->security.ses;
> +		data[sa->portid].sess = pri_sess->security.ses;
> +		data[sa->portid].ctx = pri_sess->security.ctx;
>  	}
>  }
> 
> @@ -356,9 +363,8 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
>  		goto drop_pkt_and_exit;
>  	}
> 
> -	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
> -		*(struct rte_security_session **)rte_security_dynfield(pkt) =
> -				sess->security.ses;
> +	rte_security_set_pkt_metadata(sess->security.ctx,
> +				      sess->security.ses, pkt, NULL);
> 
>  	/* Mark the packet for Tx security offload */
>  	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
> @@ -367,6 +373,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
>  	port_id = sa->portid;
> 
>  send_pkt:
> +	/* Provide L2 len for Outbound processing */
> +	pkt->l2_len = RTE_ETHER_HDR_LEN;
> +
>  	/* Update mac addresses */
>  	update_mac_addrs(pkt, port_id);
> 
> @@ -398,7 +407,7 @@ static void
>  ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
>  		uint8_t nb_links)
>  {
> -	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
> +	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
>  	unsigned int nb_rx = 0;
>  	struct rte_mbuf *pkt;
>  	struct rte_event ev;
> @@ -412,6 +421,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
>  		return;
>  	}
> 
> +	memset(&data, 0, sizeof(struct port_drv_mode_data));
> +
>  	/* Get core ID */
>  	lcore_id = rte_lcore_id();
> 
> @@ -422,8 +433,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
>  	 * Prepare security sessions table. In outbound driver mode
>  	 * we always use first session configured for a given port
>  	 */
> -	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
> -			RTE_MAX_ETHPORTS);
> +	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
> +				 RTE_MAX_ETHPORTS);
> 
>  	RTE_LOG(INFO, IPSEC,
>  		"Launching event mode worker (non-burst - Tx internal port - "
> @@ -460,19 +471,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
> 
>  		if (!is_unprotected_port(port_id)) {
> 
> -			if (unlikely(!sess_tbl[port_id])) {
> +			if (unlikely(!data[port_id].sess)) {
>  				rte_pktmbuf_free(pkt);
>  				continue;
>  			}
> 
>  			/* Save security session */
> -			if (rte_security_dynfield_is_registered())
> -				*(struct rte_security_session **)
> -					rte_security_dynfield(pkt) =
> -						sess_tbl[port_id];
> +			rte_security_set_pkt_metadata(data[port_id].ctx,
> +						      data[port_id].sess, pkt,
> +						      NULL);
> 
>  			/* Mark the packet for Tx security offload */
>  			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
> +
> +			/* Provide L2 len for Outbound processing */
> +			pkt->l2_len = RTE_ETHER_HDR_LEN;
>  		}
> 
>  		/*
> --

Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

> 2.8.4

[dpdk-dev] [PATCH v6 0/3] security: Improve inline fast path routines

[Nithin Dabilpuram]

Improvements to Inline inbound and outbound processing fast path routines
rte_security_set_pkt_metadata() and rte_security_get_userdata() to make
them inline functions and also provide mechanism for drivers to support
fast userdata and metadata access instead of driver specific per-pkt
function callbacks.

This series updates requirements of mbuf fields to be updated for outbound
inline processing.

Nithin Dabilpuram (3):
  security: enforce semantics for Tx inline processing
  security: add option for faster udata or mdata access
  examples/ipsec-secgw: update event mode inline path

v6:
- Addressed comment from Konstantin.

v5:
- Squash 4/4 patch to 2/4 and update release notes

v4:
- Removed entry from deprecation notice.
- Fixed issue with rte_security_set_pkt_metadata() to pass instance instead
  of device ptr to non-inline C function.

v3:
- Rebased and fixed compilation issue with rte_security_get_userdata() on
  32-bit platform
- Updated l2_len on patch 3/3 only for outbound.

v2:
- Remove restrictions on rte_security_set_pkt_metadata() w.r.t pkt content
- Add inline functions for rte_security_set_pkt_metadata() and 
  rte_security_get_userdata() and also faster mdata, udata access via
  patch 2/3

 doc/guides/nics/features.rst           |  2 ++
 doc/guides/rel_notes/deprecation.rst   |  4 ---
 doc/guides/rel_notes/release_21_08.rst |  6 +++++
 examples/ipsec-secgw/ipsec-secgw.c     |  2 ++
 examples/ipsec-secgw/ipsec_worker.c    | 41 ++++++++++++++++++----------
 lib/mbuf/rte_mbuf_core.h               |  2 ++
 lib/security/rte_security.c            |  8 +++---
 lib/security/rte_security.h            | 49 +++++++++++++++++++++++++++++++---
 lib/security/version.map               |  2 ++
 9 files changed, 90 insertions(+), 26 deletions(-)

-- 
2.8.4

[dpdk-dev] [PATCH v6 1/3] security: enforce semantics for Tx inline processing

[Nithin Dabilpuram]

Not all net PMD's/HW can parse packet and identify L2 header and
L3 header locations on Tx. This is inline with other Tx offloads
requirements such as L3 checksum, L4 checksum offload, etc,
where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
able to generate checksum. Since Inline IPSec is also such a Tx
offload, some PMD's at least need mbuf.l2_len to be valid to
find L3 header and perform Outbound IPSec processing.

Hence, this patch updates documentation to enforce setting
mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
for Inline IPSec Crypto / Protocol offload processing to
work on Tx.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 doc/guides/nics/features.rst | 2 ++
 lib/mbuf/rte_mbuf_core.h     | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst
index a96e12d..4fce8cd 100644
--- a/doc/guides/nics/features.rst
+++ b/doc/guides/nics/features.rst
@@ -430,6 +430,7 @@ of protocol operations. See Security library and PMD documentation for more deta
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``capabilities_get``.
 * **[provides] rte_eth_dev_info**: ``rx_offload_capa,rx_queue_offload_capa:DEV_RX_OFFLOAD_SECURITY``,
@@ -451,6 +452,7 @@ protocol operations. See security library and PMD documentation for more details
 
 * **[uses]       rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_SECURITY``,
 * **[uses]       rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_SECURITY``.
+* **[uses]       mbuf**: ``mbuf.l2_len``.
 * **[implements] rte_security_ops**: ``session_create``, ``session_update``,
   ``session_stats_get``, ``session_destroy``, ``set_pkt_metadata``, ``get_userdata``,
   ``capabilities_get``.
diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h
index bb38d7f..9d8e3dd 100644
--- a/lib/mbuf/rte_mbuf_core.h
+++ b/lib/mbuf/rte_mbuf_core.h
@@ -228,6 +228,8 @@ extern "C" {
 
 /**
  * Request security offload processing on the TX packet.
+ * To use Tx security offload, the user needs to fill l2_len in mbuf
+ * indicating L2 header size and where L3 header starts.
  */
 #define PKT_TX_SEC_OFFLOAD	(1ULL << 43)
 
-- 
2.8.4

[dpdk-dev] [PATCH v6 2/3] security: add option for faster udata or mdata access

[Nithin Dabilpuram]

Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
methods to set pkt metadata on Inline outbound and get userdata
after Inline inbound processing is always driver specific callbacks.

For drivers that do not have much to do in the callbacks but just
to update metadata in rte_security dynamic field and get userdata
from rte_security dynamic field, having to just to PMD specific
callback is costly per packet operation. This patch provides
a mechanism to do the same in inline function and avoid function
pointer jump if a driver supports the same.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 doc/guides/rel_notes/deprecation.rst   |  4 ---
 doc/guides/rel_notes/release_21_08.rst |  6 +++++
 lib/security/rte_security.c            |  8 +++---
 lib/security/rte_security.h            | 49 +++++++++++++++++++++++++++++++---
 lib/security/version.map               |  2 ++
 5 files changed, 57 insertions(+), 12 deletions(-)

diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 59445a6..70ef45e 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -276,10 +276,6 @@ Deprecation Notices
   content. On Linux and FreeBSD, supported prior to DPDK 20.11,
   original structure will be kept until DPDK 21.11.
 
-* security: The functions ``rte_security_set_pkt_metadata`` and
-  ``rte_security_get_userdata`` will be made inline functions and additional
-  flags will be added in structure ``rte_security_ctx`` in DPDK 21.11.
-
 * cryptodev: The structure ``rte_crypto_op`` would be updated to reduce
   reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
   information from the crypto/security operation. This field will be used to
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index b4cbf2d..dd92461 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -223,6 +223,12 @@ ABI Changes
 
 * No ABI change that would break compatibility with 20.11.
 
+* security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
+  routines used by Inline outbound and Inline inbound security processing are
+  made inline and enhanced to do simple 64-bit set/get for PMD's that do not
+  have much processing in PMD specific callbacks but just 64-bit set/get.
+  This avoids a per pkt function pointer jump overhead for such PMD's.
+
 
 Known Issues
 ------------
diff --git a/lib/security/rte_security.c b/lib/security/rte_security.c
index e8116d5..fe81ed3 100644
--- a/lib/security/rte_security.c
+++ b/lib/security/rte_security.c
@@ -122,9 +122,9 @@ rte_security_session_destroy(struct rte_security_ctx *instance,
 }
 
 int
-rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
-			      struct rte_security_session *sess,
-			      struct rte_mbuf *m, void *params)
+__rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+				struct rte_security_session *sess,
+				struct rte_mbuf *m, void *params)
 {
 #ifdef RTE_DEBUG
 	RTE_PTR_OR_ERR_RET(sess, -EINVAL);
@@ -137,7 +137,7 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 }
 
 void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+__rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
 {
 	void *userdata = NULL;
 
diff --git a/lib/security/rte_security.h b/lib/security/rte_security.h
index 2e136d7..2446ab0 100644
--- a/lib/security/rte_security.h
+++ b/lib/security/rte_security.h
@@ -71,8 +71,19 @@ struct rte_security_ctx {
 	/**< Pointer to security ops for the device */
 	uint16_t sess_cnt;
 	/**< Number of sessions attached to this context */
+	uint32_t flags;
+	/**< Flags for security context */
 };
 
+#define RTE_SEC_CTX_F_FAST_SET_MDATA 0x00000001
+/**< Driver uses fast metadata update without using driver specific callback */
+
+#define RTE_SEC_CTX_F_FAST_GET_UDATA 0x00000002
+/**< Driver provides udata using fast method without using driver specific
+ * callback. For fast mdata and udata, mbuf dynamic field would be registered
+ * by driver via rte_security_dynfield_register().
+ */
+
 /**
  * IPSEC tunnel parameters
  *
@@ -494,6 +505,12 @@ static inline bool rte_security_dynfield_is_registered(void)
 	return rte_security_dynfield_offset >= 0;
 }
 
+/** Function to call PMD specific function pointer set_pkt_metadata() */
+__rte_experimental
+extern int __rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
+					   struct rte_security_session *sess,
+					   struct rte_mbuf *m, void *params);
+
 /**
  *  Updates the buffer with device-specific defined metadata
  *
@@ -507,10 +524,26 @@ static inline bool rte_security_dynfield_is_registered(void)
  *  - On success, zero.
  *  - On failure, a negative value.
  */
-int
+static inline int
 rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
 			      struct rte_security_session *sess,
-			      struct rte_mbuf *mb, void *params);
+			      struct rte_mbuf *mb, void *params)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_SET_MDATA) {
+		*rte_security_dynfield(mb) =
+			(rte_security_dynfield_t)(sess->sess_private_data);
+		return 0;
+	}
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_set_pkt_metadata(instance, sess, mb, params);
+}
+
+/** Function to call PMD specific function pointer get_userdata() */
+__rte_experimental
+extern void *__rte_security_get_userdata(struct rte_security_ctx *instance,
+					 uint64_t md);
 
 /**
  * Get userdata associated with the security session. Device specific metadata
@@ -530,8 +563,16 @@ rte_security_set_pkt_metadata(struct rte_security_ctx *instance,
  *  - On failure, NULL
  */
 __rte_experimental
-void *
-rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md);
+static inline void *
+rte_security_get_userdata(struct rte_security_ctx *instance, uint64_t md)
+{
+	/* Fast Path */
+	if (instance->flags & RTE_SEC_CTX_F_FAST_GET_UDATA)
+		return (void *)(uintptr_t)md;
+
+	/* Jump to PMD specific function pointer */
+	return __rte_security_get_userdata(instance, md);
+}
 
 /**
  * Attach a session to a symmetric crypto operation
diff --git a/lib/security/version.map b/lib/security/version.map
index c44c7f5..45ace9c 100644
--- a/lib/security/version.map
+++ b/lib/security/version.map
@@ -20,4 +20,6 @@ EXPERIMENTAL {
 	rte_security_get_userdata;
 	rte_security_session_stats_get;
 	rte_security_session_update;
+	__rte_security_set_pkt_metadata;
+	__rte_security_get_userdata;
 };
-- 
2.8.4

[dpdk-dev] [PATCH v6 3/3] examples/ipsec-secgw: update event mode inline path

[Nithin Dabilpuram]

Update mbuf.l2_len with L2 header size for outbound
inline processing.

This patch also fixes a bug in arg parsing.

Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  2 ++
 examples/ipsec-secgw/ipsec_worker.c | 41 ++++++++++++++++++++++++-------------
 2 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f252d34..7ad94cb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1495,6 +1495,8 @@ parse_portmask(const char *portmask)
 	char *end = NULL;
 	unsigned long pm;
 
+	errno = 0;
+
 	/* parse hexadecimal string */
 	pm = strtoul(portmask, &end, 16);
 	if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 647e22d..c545497 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -12,6 +12,11 @@
 #include "ipsec-secgw.h"
 #include "ipsec_worker.h"
 
+struct port_drv_mode_data {
+	struct rte_security_session *sess;
+	struct rte_security_ctx *ctx;
+};
+
 static inline enum pkt_type
 process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
 {
@@ -60,7 +65,8 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int port_id)
 
 static inline void
 prepare_out_sessions_tbl(struct sa_ctx *sa_out,
-		struct rte_security_session **sess_tbl, uint16_t size)
+			 struct port_drv_mode_data *data,
+			 uint16_t size)
 {
 	struct rte_ipsec_session *pri_sess;
 	struct ipsec_sa *sa;
@@ -95,9 +101,10 @@ prepare_out_sessions_tbl(struct sa_ctx *sa_out,
 		}
 
 		/* Use only first inline session found for a given port */
-		if (sess_tbl[sa->portid])
+		if (data[sa->portid].sess)
 			continue;
-		sess_tbl[sa->portid] = pri_sess->security.ses;
+		data[sa->portid].sess = pri_sess->security.ses;
+		data[sa->portid].ctx = pri_sess->security.ctx;
 	}
 }
 
@@ -356,9 +363,8 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 		goto drop_pkt_and_exit;
 	}
 
-	if (sess->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
-		*(struct rte_security_session **)rte_security_dynfield(pkt) =
-				sess->security.ses;
+	rte_security_set_pkt_metadata(sess->security.ctx,
+				      sess->security.ses, pkt, NULL);
 
 	/* Mark the packet for Tx security offload */
 	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
@@ -367,6 +373,9 @@ process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	port_id = sa->portid;
 
 send_pkt:
+	/* Provide L2 len for Outbound processing */
+	pkt->l2_len = RTE_ETHER_HDR_LEN;
+
 	/* Update mac addresses */
 	update_mac_addrs(pkt, port_id);
 
@@ -398,7 +407,7 @@ static void
 ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		uint8_t nb_links)
 {
-	struct rte_security_session *sess_tbl[RTE_MAX_ETHPORTS] = { NULL };
+	struct port_drv_mode_data data[RTE_MAX_ETHPORTS];
 	unsigned int nb_rx = 0;
 	struct rte_mbuf *pkt;
 	struct rte_event ev;
@@ -412,6 +421,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 		return;
 	}
 
+	memset(&data, 0, sizeof(struct port_drv_mode_data));
+
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
@@ -422,8 +433,8 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 	 * Prepare security sessions table. In outbound driver mode
 	 * we always use first session configured for a given port
 	 */
-	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, sess_tbl,
-			RTE_MAX_ETHPORTS);
+	prepare_out_sessions_tbl(socket_ctx[socket_id].sa_out, data,
+				 RTE_MAX_ETHPORTS);
 
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
@@ -460,19 +471,21 @@ ipsec_wrkr_non_burst_int_port_drv_mode(struct eh_event_link_info *links,
 
 		if (!is_unprotected_port(port_id)) {
 
-			if (unlikely(!sess_tbl[port_id])) {
+			if (unlikely(!data[port_id].sess)) {
 				rte_pktmbuf_free(pkt);
 				continue;
 			}
 
 			/* Save security session */
-			if (rte_security_dynfield_is_registered())
-				*(struct rte_security_session **)
-					rte_security_dynfield(pkt) =
-						sess_tbl[port_id];
+			rte_security_set_pkt_metadata(data[port_id].ctx,
+						      data[port_id].sess, pkt,
+						      NULL);
 
 			/* Mark the packet for Tx security offload */
 			pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+			/* Provide L2 len for Outbound processing */
+			pkt->l2_len = RTE_ETHER_HDR_LEN;
 		}
 
 		/*
-- 
2.8.4

Re: [dpdk-dev] [PATCH v6 1/3] security: enforce semantics for Tx inline processing

[Akhil Goyal]

> 
> Not all net PMD's/HW can parse packet and identify L2 header and
> L3 header locations on Tx. This is inline with other Tx offloads
> requirements such as L3 checksum, L4 checksum offload, etc,
> where mbuf.l2_len, mbuf.l3_len etc, needs to be set for HW to be
> able to generate checksum. Since Inline IPSec is also such a Tx
> offload, some PMD's at least need mbuf.l2_len to be valid to
> find L3 header and perform Outbound IPSec processing.
> 
> Hence, this patch updates documentation to enforce setting
> mbuf.l2_len while setting PKT_TX_SEC_OFFLOAD in mbuf.ol_flags
> for Inline IPSec Crypto / Protocol offload processing to
> work on Tx.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Akhil Goyal <gakhil@marvell.com>

Title updated as mbuf: enforce semantics for Tx inline IPSec processing

Series Applied to dpdk-next-crypto

Re: [dpdk-dev] [PATCH v6 2/3] security: add option for faster udata or mdata access

[Thomas Monjalon]

15/09/2021 18:30, Nithin Dabilpuram:
> Currently rte_security_set_pkt_metadata() and rte_security_get_userdata()
> methods to set pkt metadata on Inline outbound and get userdata
> after Inline inbound processing is always driver specific callbacks.
> 
> For drivers that do not have much to do in the callbacks but just
> to update metadata in rte_security dynamic field and get userdata
> from rte_security dynamic field, having to just to PMD specific
> callback is costly per packet operation. This patch provides
> a mechanism to do the same in inline function and avoid function
> pointer jump if a driver supports the same.
> 
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Akhil Goyal <gakhil@marvell.com>
[...]
> --- a/doc/guides/rel_notes/release_21_08.rst
> +++ b/doc/guides/rel_notes/release_21_08.rst
> @@ -223,6 +223,12 @@ ABI Changes
>  
>  * No ABI change that would break compatibility with 20.11.
>  
> +* security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
> +  routines used by Inline outbound and Inline inbound security processing are
> +  made inline and enhanced to do simple 64-bit set/get for PMD's that do not
> +  have much processing in PMD specific callbacks but just 64-bit set/get.
> +  This avoids a per pkt function pointer jump overhead for such PMD's.

Please pay attention it is not the right release notes.

Re: [dpdk-dev] [EXT] Re: [PATCH v6 2/3] security: add option for faster udata or mdata access

[Akhil Goyal]

> > --- a/doc/guides/rel_notes/release_21_08.rst
> > +++ b/doc/guides/rel_notes/release_21_08.rst
> > @@ -223,6 +223,12 @@ ABI Changes
> >
> >  * No ABI change that would break compatibility with 20.11.
> >
> > +* security: ``rte_security_set_pkt_metadata`` and
> ``rte_security_get_userdata``
> > +  routines used by Inline outbound and Inline inbound security processing
> are
> > +  made inline and enhanced to do simple 64-bit set/get for PMD's that do
> not
> > +  have much processing in PMD specific callbacks but just 64-bit set/get.
> > +  This avoids a per pkt function pointer jump overhead for such PMD's.
> 
> Please pay attention it is not the right release notes.
> 
Fixed... My bad.