RE: [PATCH v4] net/ice: refactor proto_ext to remove global variable

["Ling, Jin" <jin.ling@intel.com>]

Tested-by: Jin Ling <jin.ling@intel.com>

-----Original Message-----
From: Kevin Liu <kevinx.liu@intel.com> 
Sent: 2022年8月26日 18:15
To: dev@dpdk.org
Cc: Yang, Qiming <qiming.yang@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; Yang, SteveX <stevex.yang@intel.com>; Liu, KevinX <kevinx.liu@intel.com>
Subject: [PATCH v4] net/ice: refactor proto_ext to remove global variable

The ice has the feature to extract protocol fields into flex descriptor by programming per queue. However, the dynamic field for proto_ext are allocated by PMD, it is the responsibility of application to reserved the field, before start DPDK.

Application with parse the offset and proto_ext name to PMD with devargs.
Remove related private API in 'rte_pmd_ice.h' and 'rte_pmd_ice.h' file.

Signed-off-by: Kevin Liu <kevinx.liu@intel.com>

---
v2: Delete doc content related to 'rte_pmd_ice.h'.
---
v3: Delete doc content related to 'rte_pmd_ice.h'.
---
v4: refine code and change the check mode of dynamic field.
---
 doc/api/doxy-api-index.md         |   1 -
 doc/api/doxy-api.conf.in          |   1 -
 doc/guides/nics/ice.rst           |  33 ++--
 drivers/net/ice/ice_ddp_package.c |   1 -
 drivers/net/ice/ice_ethdev.c      | 113 ++++++++++----
 drivers/net/ice/ice_ethdev.h      |   7 +
 drivers/net/ice/ice_rxtx.c        |  45 ++----
 drivers/net/ice/ice_rxtx.h        |   1 +
 drivers/net/ice/ice_testpmd.c     |   2 +-
 drivers/net/ice/meson.build       |   2 -
 drivers/net/ice/rte_pmd_ice.h     | 247 ------------------------------
 drivers/net/ice/version.map       |   7 -
 12 files changed, 122 insertions(+), 338 deletions(-)  delete mode 100644 drivers/net/ice/rte_pmd_ice.h

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index 186a258be4..e578800cea 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -44,7 +44,6 @@ The public API headers are grouped by topics:
   [KNI](@ref rte_kni.h),
   [ixgbe](@ref rte_pmd_ixgbe.h),
   [i40e](@ref rte_pmd_i40e.h),
-  [ice](@ref rte_pmd_ice.h),
   [iavf](@ref rte_pmd_iavf.h),
   [ioat](@ref rte_ioat_rawdev.h),
   [bnxt](@ref rte_pmd_bnxt.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index 608494a7c0..6fab7436d7 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -18,7 +18,6 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/drivers/net/dpaa2 \
                           @TOPDIR@/drivers/net/i40e \
                           @TOPDIR@/drivers/net/iavf \
-                          @TOPDIR@/drivers/net/ice \
                           @TOPDIR@/drivers/net/ixgbe \
                           @TOPDIR@/drivers/net/mlx5 \
                           @TOPDIR@/drivers/net/softnic \ diff --git a/doc/guides/nics/ice.rst b/doc/guides/nics/ice.rst index 6b903b9bbc..432c6fd7ed 100644
--- a/doc/guides/nics/ice.rst
+++ b/doc/guides/nics/ice.rst
@@ -110,29 +110,43 @@ Runtime Config Options
 
   The argument format is::
 
-      -a 18:00.0,proto_xtr=<queues:protocol>[<queues:protocol>...]
-      -a 18:00.0,proto_xtr=<protocol>
+      18:00.0,proto_xtr=<queues:protocol>[<queues:protocol>...],field_offs=<offset>, \
+      field_name=<name>
+      
+ 18:00.0,proto_xtr=<protocol>,field_offs=<offset>,field_name=<name>
 
   Queues are grouped by ``(`` and ``)`` within the group. The ``-`` character
   is used as a range separator and ``,`` is used as a single number separator.
   The grouping ``()`` can be omitted for single element group. If no queues are
   specified, PMD will use this protocol extraction type for all queues.
+  ``field_offs`` is the offset of mbuf dynamic field for protocol extraction data.
+  ``field_name`` is the name of mbuf dynamic field for protocol extraction data.
+  ``field_offs`` and ``field_name`` will be checked whether it is 
+ valid. If it is invalid  the value needs to be reconfigured.
 
   Protocol is : ``vlan, ipv4, ipv6, ipv6_flow, tcp, ip_offset``.
 
   .. code-block:: console
 
-    dpdk-testpmd -a 18:00.0,proto_xtr='[(1,2-3,8-9):tcp,10-13:vlan]'
+    dpdk-testpmd -c 0xff -- -i
+    port stop 0
+    port detach 0
+    port attach 
+ 18:00.0,proto_xtr='[(1,2-3,8-9):tcp,10-13:vlan]',field_offs=92,field_n
+ ame=pmd_dyn
 
   This setting means queues 1, 2-3, 8-9 are TCP extraction, queues 10-13 are
-  VLAN extraction, other queues run with no protocol extraction.
+  VLAN extraction, other queues run with no protocol extraction. The 
+ offset of mbuf  dynamic field is 92 for all queues with protocol extraction.
 
   .. code-block:: console
 
-    dpdk-testpmd -a 18:00.0,proto_xtr=vlan,proto_xtr='[(1,2-3,8-9):tcp,10-23:ipv6]'
+    dpdk-testpmd -c 0xff -- -i
+    port stop 0
+    port detach 0
+    port attach 18:00.0,proto_xtr=vlan,proto_xtr='[(1,2-3,8-9):tcp,10-23:ipv6]', \
+    field_offs=92,field_name=pmd_dyn
 
   This setting means queues 1, 2-3, 8-9 are TCP extraction, queues 10-23 are
-  IPv6 extraction, other queues use the default VLAN extraction.
+  IPv6 extraction, other queues use the default VLAN extraction. The 
+ offset of mbuf  dynamic field is 92 for all queues with protocol extraction.
 
   The extraction metadata is copied into the registered dynamic mbuf field, and
   the related dynamic mbuf flags is set.
@@ -211,13 +225,6 @@ Runtime Config Options
 
   IPHDR2 - Outer/Single IPv6 Header offset.
 
-  Use ``rte_net_ice_dynf_proto_xtr_metadata_get`` to access the protocol
-  extraction metadata, and use ``RTE_PKT_RX_DYNF_PROTO_XTR_*`` to get the
-  metadata type of ``struct rte_mbuf::ol_flags``.
-
-  The ``rte_net_ice_dump_proto_xtr_metadata`` routine shows how to
-  access the protocol extraction result in ``struct rte_mbuf``.
-
 - ``Hardware debug mask log support`` (default ``0``)
 
   User can enable the related hardware debug mask such as ICE_DBG_NVM::
diff --git a/drivers/net/ice/ice_ddp_package.c b/drivers/net/ice/ice_ddp_package.c
index c7b5dc7ee7..86c0d6d499 100644
--- a/drivers/net/ice/ice_ddp_package.c
+++ b/drivers/net/ice/ice_ddp_package.c
@@ -7,7 +7,6 @@
 #include <rte_tailq.h>
 
 #include "ice_ethdev.h"
-#include "rte_pmd_ice.h"
 
 #define ICE_BUFF_SEG_HEADER_FLAG   0x1
 #define ICE_PKG_HDR_HEADR_PART1    1
diff --git a/drivers/net/ice/ice_ethdev.c b/drivers/net/ice/ice_ethdev.c index b2300790ae..ed21f5c569 100644
--- a/drivers/net/ice/ice_ethdev.c
+++ b/drivers/net/ice/ice_ethdev.c
@@ -20,7 +20,6 @@
 #include "base/ice_common.h"
 #include "base/ice_ptp_hw.h"
 
-#include "rte_pmd_ice.h"
 #include "ice_ethdev.h"
 #include "ice_rxtx.h"
 #include "ice_generic_flow.h"
@@ -29,6 +28,8 @@
 #define ICE_SAFE_MODE_SUPPORT_ARG "safe-mode-support"
 #define ICE_PIPELINE_MODE_SUPPORT_ARG  "pipeline-mode-support"
 #define ICE_PROTO_XTR_ARG         "proto_xtr"
+#define ICE_FIELD_OFFS_ARG		  "field_offs"
+#define ICE_FIELD_NAME_ARG		  "field_name"
 #define ICE_HW_DEBUG_MASK_ARG     "hw_debug_mask"
 #define ICE_ONE_PPS_OUT_ARG       "pps_out"
 #define ICE_RX_LOW_LATENCY_ARG    "rx_low_latency"
@@ -42,6 +43,8 @@ static const char * const ice_valid_args[] = {
 	ICE_SAFE_MODE_SUPPORT_ARG,
 	ICE_PIPELINE_MODE_SUPPORT_ARG,
 	ICE_PROTO_XTR_ARG,
+	ICE_FIELD_OFFS_ARG,
+	ICE_FIELD_NAME_ARG,
 	ICE_HW_DEBUG_MASK_ARG,
 	ICE_ONE_PPS_OUT_ARG,
 	ICE_RX_LOW_LATENCY_ARG,
@@ -50,16 +53,8 @@ static const char * const ice_valid_args[] = {
 
 #define PPS_OUT_DELAY_NS  1
 
-static const struct rte_mbuf_dynfield ice_proto_xtr_metadata_param = {
-	.name = "intel_pmd_dynfield_proto_xtr_metadata",
-	.size = sizeof(uint32_t),
-	.align = __alignof__(uint32_t),
-	.flags = 0,
-};
-
 struct proto_xtr_ol_flag {
 	const struct rte_mbuf_dynflag param;
-	uint64_t *ol_flag;
 	bool required;
 };
 
@@ -67,23 +62,17 @@ static bool ice_proto_xtr_hw_support[PROTO_XTR_MAX];
 
 static struct proto_xtr_ol_flag ice_proto_xtr_ol_flag_params[] = {
 	[PROTO_XTR_VLAN] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_vlan" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_vlan_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_vlan" }},
 	[PROTO_XTR_IPV4] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv4" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_ipv4_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv4" }},
 	[PROTO_XTR_IPV6] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv6" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_ipv6_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv6" }},
 	[PROTO_XTR_IPV6_FLOW] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv6_flow" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_ipv6_flow" }},
 	[PROTO_XTR_TCP] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_tcp" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_tcp_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_tcp" }},
 	[PROTO_XTR_IP_OFFSET] = {
-		.param = { .name = "intel_pmd_dynflag_proto_xtr_ip_offset" },
-		.ol_flag = &rte_net_ice_dynflag_proto_xtr_ip_offset_mask },
+		.param = { .name = "intel_pmd_dynflag_proto_xtr_ip_offset" }}
 };
 
 #define ICE_OS_DEFAULT_PKG_NAME		"ICE OS Default Package"
@@ -612,6 +601,45 @@ handle_proto_xtr_arg(__rte_unused const char *key, const char *value,
 	return 0;
 }
 
+static int
+handle_field_offs_arg(__rte_unused const char *key, const char *value,
+				void *offs_args)
+{
+	uint8_t *offset = offs_args;
+
+	if (value == NULL || offs_args == NULL)
+		return -EINVAL;
+
+	if (!isdigit(*value))
+		return -1;
+
+	*offset = atoi(value);
+
+	return 0;
+}
+
+static int
+handle_field_name_arg(__rte_unused const char *key, const char *value,
+				void *name_args)
+{
+	char *name = name_args;
+	int ret;
+
+	if (name == NULL || name_args == NULL)
+		return -EINVAL;
+	if (isdigit(*value))
+		return -1;
+
+	ret = strlcpy(name, value, RTE_MBUF_DYN_NAMESIZE);
+	if (ret < 0 || ret >= RTE_MBUF_DYN_NAMESIZE) {
+		PMD_DRV_LOG(ERR,
+			    "The protocol extraction field name too long : '%s'",
+			    name);
+		return -1;
+	}
+	return 0;
+}
+
 static void
 ice_check_proto_xtr_support(struct ice_hw *hw)  { @@ -1400,7 +1428,7 @@ ice_init_proto_xtr(struct rte_eth_dev *dev)
 	struct ice_hw *hw = ICE_PF_TO_HW(pf);
 	const struct proto_xtr_ol_flag *ol_flag;
 	bool proto_xtr_enable = false;
-	int offset;
+	int offset, field_offs;
 	uint16_t i;
 
 	pf->proto_xtr = rte_zmalloc(NULL, pf->lan_nb_qps, 0); @@ -1422,27 +1450,35 @@ ice_init_proto_xtr(struct rte_eth_dev *dev)
 		}
 	}
 
-	if (likely(!proto_xtr_enable))
+	if (likely(!proto_xtr_enable)) {
+		ad->devargs.xtr_field_offs = -1;
 		return;
+	}
 
 	ice_check_proto_xtr_support(hw);
 
-	offset = rte_mbuf_dynfield_register(&ice_proto_xtr_metadata_param);
-	if (unlikely(offset == -1)) {
-		PMD_DRV_LOG(ERR,
-			    "Protocol extraction metadata is disabled in mbuf with error %d",
-			    -rte_errno);
+	/*check mbuf dynfield*/
+	field_offs = rte_mbuf_dynfield_lookup(ad->devargs.xtr_field_name, NULL);
+	if (ad->devargs.xtr_field_offs == field_offs) {
+		PMD_DRV_LOG(DEBUG,
+			"Protocol extraction metadata offset in mbuf is : %d",
+			ad->devargs.xtr_field_offs);
+	} else {
+		PMD_DRV_LOG(ERR, "Invalid field offset or name, no match dynfield, [%d],[%s]",
+			ad->devargs.xtr_field_offs, ad->devargs.xtr_field_name);
+		ad->devargs.xtr_field_offs = -1;
 		return;
 	}
 
 	PMD_DRV_LOG(DEBUG,
 		    "Protocol extraction metadata offset in mbuf is : %d",
-		    offset);
-	rte_net_ice_dynfield_proto_xtr_metadata_offs = offset;
+		    ad->devargs.xtr_field_offs);
 
 	for (i = 0; i < RTE_DIM(ice_proto_xtr_ol_flag_params); i++) {
 		ol_flag = &ice_proto_xtr_ol_flag_params[i];
 
+		ad->devargs.xtr_flag_offs[i] = 0xff;
+
 		if (!ol_flag->required)
 			continue;
 
@@ -1450,7 +1486,7 @@ ice_init_proto_xtr(struct rte_eth_dev *dev)
 			PMD_DRV_LOG(ERR,
 				    "Protocol extraction type %u is not supported in hardware",
 				    i);
-			rte_net_ice_dynfield_proto_xtr_metadata_offs = -1;
+			ad->devargs.xtr_field_offs = -1;
 			break;
 		}
 
@@ -1460,14 +1496,15 @@ ice_init_proto_xtr(struct rte_eth_dev *dev)
 				    "Protocol extraction offload '%s' failed to register with error %d",
 				    ol_flag->param.name, -rte_errno);
 
-			rte_net_ice_dynfield_proto_xtr_metadata_offs = -1;
+			ad->devargs.xtr_field_offs = -1;
 			break;
 		}
 
 		PMD_DRV_LOG(DEBUG,
 			    "Protocol extraction offload '%s' offset in mbuf is : %d",
 			    ol_flag->param.name, offset);
-		*ol_flag->ol_flag = 1ULL << offset;
+
+		ad->devargs.xtr_flag_offs[i] = offset;
 	}
 }
 
@@ -2010,6 +2047,16 @@ static int ice_parse_devargs(struct rte_eth_dev *dev)
 	if (ret)
 		goto bail;
 
+	ret = rte_kvargs_process(kvlist, ICE_FIELD_OFFS_ARG,
+				 &handle_field_offs_arg, &ad->devargs.xtr_field_offs);
+	if (ret)
+		goto bail;
+
+	ret = rte_kvargs_process(kvlist, ICE_FIELD_NAME_ARG,
+				 &handle_field_name_arg, &ad->devargs.xtr_field_name);
+	if (ret)
+		goto bail;
+
 	ret = rte_kvargs_process(kvlist, ICE_SAFE_MODE_SUPPORT_ARG,
 				 &parse_bool, &ad->devargs.safe_mode_support);
 	if (ret)
diff --git a/drivers/net/ice/ice_ethdev.h b/drivers/net/ice/ice_ethdev.h index ec23dae665..62d9654841 100644
--- a/drivers/net/ice/ice_ethdev.h
+++ b/drivers/net/ice/ice_ethdev.h
@@ -565,6 +565,10 @@ struct ice_devargs {
 	uint8_t proto_xtr[ICE_MAX_QUEUE_NUM];
 	uint8_t pin_idx;
 	uint8_t pps_out_ena;
+	int xtr_field_offs;
+	uint8_t xtr_flag_offs[PROTO_XTR_MAX];
+	/* Name of the field. */
+	char xtr_field_name[RTE_MBUF_DYN_NAMESIZE];
 };
 
 /**
@@ -725,4 +729,7 @@ ice_align_floor(int n)
 	((phy_type) & ICE_PHY_TYPE_HIGH_100G_AUI2_AOC_ACC) || \
 	((phy_type) & ICE_PHY_TYPE_HIGH_100G_AUI2))
 
+__rte_experimental
+int rte_pmd_ice_dump_package(uint16_t port, uint8_t **buff, uint32_t 
+*size);
+
 #endif /* _ICE_ETHDEV_H_ */
diff --git a/drivers/net/ice/ice_rxtx.c b/drivers/net/ice/ice_rxtx.c index bfb3a16ae2..5af7c0c8f6 100644
--- a/drivers/net/ice/ice_rxtx.c
+++ b/drivers/net/ice/ice_rxtx.c
@@ -6,7 +6,6 @@
 #include <rte_net.h>
 #include <rte_vect.h>
 
-#include "rte_pmd_ice.h"
 #include "ice_rxtx.h"
 #include "ice_rxtx_vec_common.h"
 
@@ -15,16 +14,11 @@
 		RTE_MBUF_F_TX_TCP_SEG |		 \
 		RTE_MBUF_F_TX_OUTER_IP_CKSUM)
 
-/* Offset of mbuf dynamic field for protocol extraction data */ -int rte_net_ice_dynfield_proto_xtr_metadata_offs = -1;
-
-/* Mask of mbuf dynamic flags for protocol extraction type */ -uint64_t rte_net_ice_dynflag_proto_xtr_vlan_mask;
-uint64_t rte_net_ice_dynflag_proto_xtr_ipv4_mask;
-uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_mask;
-uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
-uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;
-uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
+/**
+ * The mbuf dynamic field pointer for protocol extraction metadata.
+ */
+#define ICE_DYNF_PROTO_XTR_METADATA(m, n) \
+	RTE_MBUF_DYNFIELD((m), (n), uint32_t *)
 
 static int
 ice_monitor_callback(const uint64_t value, @@ -160,7 +154,7 @@ ice_rxd_to_pkt_fields_by_comms_aux_v1(struct ice_rx_queue *rxq,
 		if (metadata) {
 			mb->ol_flags |= rxq->xtr_ol_flag;
 
-			*RTE_NET_ICE_DYNF_PROTO_XTR_METADATA(mb) = metadata;
+			*ICE_DYNF_PROTO_XTR_METADATA(mb, rxq->xtr_field_offs) = metadata;
 		}
 	}
 #else
@@ -200,7 +194,7 @@ ice_rxd_to_pkt_fields_by_comms_aux_v2(struct ice_rx_queue *rxq,
 		if (metadata) {
 			mb->ol_flags |= rxq->xtr_ol_flag;
 
-			*RTE_NET_ICE_DYNF_PROTO_XTR_METADATA(mb) = metadata;
+			*ICE_DYNF_PROTO_XTR_METADATA(mb, rxq->xtr_field_offs) = metadata;
 		}
 	}
 #else
@@ -226,29 +220,12 @@ ice_select_rxd_to_pkt_fields_handler(struct ice_rx_queue *rxq, uint32_t rxdid)
 
 	switch (rxdid) {
 	case ICE_RXDID_COMMS_AUX_VLAN:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_vlan_mask;
-		break;
-
 	case ICE_RXDID_COMMS_AUX_IPV4:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_ipv4_mask;
-		break;
-
 	case ICE_RXDID_COMMS_AUX_IPV6:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_ipv6_mask;
-		break;
-
 	case ICE_RXDID_COMMS_AUX_IPV6_FLOW:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
-		break;
-
 	case ICE_RXDID_COMMS_AUX_TCP:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_tcp_mask;
-		break;
-
 	case ICE_RXDID_COMMS_AUX_IP_OFFSET:
-		rxq->xtr_ol_flag = rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
 		break;
-
 	case ICE_RXDID_COMMS_GENERIC:
 		/* fallthrough */
 	case ICE_RXDID_COMMS_OVS:
@@ -260,7 +237,7 @@ ice_select_rxd_to_pkt_fields_handler(struct ice_rx_queue *rxq, uint32_t rxdid)
 		break;
 	}
 
-	if (!rte_net_ice_dynf_proto_xtr_metadata_avail())
+	if (rxq->xtr_field_offs == -1)
 		rxq->xtr_ol_flag = 0;
 }
 
@@ -346,7 +323,7 @@ ice_program_hw_rx_queue(struct ice_rx_queue *rxq)
 		return -EINVAL;
 	}
 
-	ice_select_rxd_to_pkt_fields_handler(rxq, rxdid);
+	rxq->rxdid = rxdid;
 
 	/* Enable Flexible Descriptors in the queue context which
 	 * allows this driver to select a specific receive descriptor format @@ -1121,6 +1098,10 @@ ice_rx_queue_setup(struct rte_eth_dev *dev,
 	rxq->rx_deferred_start = rx_conf->rx_deferred_start;
 	rxq->proto_xtr = pf->proto_xtr != NULL ?
 			 pf->proto_xtr[queue_idx] : PROTO_XTR_NONE;
+	if (rxq->proto_xtr != PROTO_XTR_NONE &&
+			ad->devargs.xtr_flag_offs[rxq->proto_xtr] != 0xff)
+		rxq->xtr_ol_flag = 1ULL << ad->devargs.xtr_flag_offs[rxq->proto_xtr];
+	rxq->xtr_field_offs = ad->devargs.xtr_field_offs;
 
 	/* Allocate the maximum number of RX ring hardware descriptor. */
 	len = ICE_MAX_RING_DESC;
diff --git a/drivers/net/ice/ice_rxtx.h b/drivers/net/ice/ice_rxtx.h index f5337d5284..6c08c175dc 100644
--- a/drivers/net/ice/ice_rxtx.h
+++ b/drivers/net/ice/ice_rxtx.h
@@ -88,6 +88,7 @@ struct ice_rx_queue {
 	bool q_set; /* indicate if rx queue has been configured */
 	bool rx_deferred_start; /* don't start this queue in dev start */
 	uint8_t proto_xtr; /* Protocol extraction from flexible descriptor */
+	int xtr_field_offs; /*Protocol extraction matedata offset*/
 	uint64_t xtr_ol_flag; /* Protocol extraction offload flag */
 	uint32_t rxdid; /* Receive Flex Descriptor profile ID */
 	ice_rx_release_mbufs_t rx_rel_mbufs;
diff --git a/drivers/net/ice/ice_testpmd.c b/drivers/net/ice/ice_testpmd.c index 2de9b36503..10ae9d5bdd 100644
--- a/drivers/net/ice/ice_testpmd.c
+++ b/drivers/net/ice/ice_testpmd.c
@@ -2,12 +2,12 @@
  * Copyright(c) 2022 Intel Corporation.
  */
 
-#include <rte_pmd_ice.h>
 
 #include <cmdline_parse_num.h>
 #include <cmdline_parse_string.h>
 
 #include "testpmd.h"
+#include "ice_ethdev.h"
 
 /* Fixed size for ICE ddp runtime configure */
 #define ICE_BUFF_SIZE	0x000c9000
diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build index 1045919687..528e77613e 100644
--- a/drivers/net/ice/meson.build
+++ b/drivers/net/ice/meson.build
@@ -78,5 +78,3 @@ sources += files(
         'ice_dcf_parent.c',
         'ice_dcf_sched.c',
 )
-
-headers = files('rte_pmd_ice.h')
diff --git a/drivers/net/ice/rte_pmd_ice.h b/drivers/net/ice/rte_pmd_ice.h deleted file mode 100644 index 53c81ccf4e..0000000000
--- a/drivers/net/ice/rte_pmd_ice.h
+++ /dev/null
@@ -1,247 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2019 Intel Corporation
- */
-
-#ifndef _RTE_PMD_ICE_H_
-#define _RTE_PMD_ICE_H_
-
-/**
- * @file rte_pmd_ice.h
- *
- * ice PMD specific functions.
- *
- * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
- *
- */
-
-#include <stdio.h>
-#include <rte_mbuf.h>
-#include <rte_mbuf_dyn.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-/**
- * The supported network protocol extraction metadata format.
- */
-union rte_net_ice_proto_xtr_metadata {
-	uint32_t metadata;
-
-	struct {
-		uint16_t data0;
-		uint16_t data1;
-	} raw;
-
-	struct {
-		uint16_t stag_vid:12,
-			 stag_dei:1,
-			 stag_pcp:3;
-		uint16_t ctag_vid:12,
-			 ctag_dei:1,
-			 ctag_pcp:3;
-	} vlan;
-
-	struct {
-		uint16_t protocol:8,
-			 ttl:8;
-		uint16_t tos:8,
-			 ihl:4,
-			 version:4;
-	} ipv4;
-
-	struct {
-		uint16_t hoplimit:8,
-			 nexthdr:8;
-		uint16_t flowhi4:4,
-			 tc:8,
-			 version:4;
-	} ipv6;
-
-	struct {
-		uint16_t flowlo16;
-		uint16_t flowhi4:4,
-			 tc:8,
-			 version:4;
-	} ipv6_flow;
-
-	struct {
-		uint16_t fin:1,
-			 syn:1,
-			 rst:1,
-			 psh:1,
-			 ack:1,
-			 urg:1,
-			 ece:1,
-			 cwr:1,
-			 res1:4,
-			 doff:4;
-		uint16_t rsvd;
-	} tcp;
-
-	uint32_t ip_ofs;
-};
-
-/* Offset of mbuf dynamic field for protocol extraction data */ -extern int rte_net_ice_dynfield_proto_xtr_metadata_offs;
-
-/* Mask of mbuf dynamic flags for protocol extraction type */ -extern uint64_t rte_net_ice_dynflag_proto_xtr_vlan_mask;
-extern uint64_t rte_net_ice_dynflag_proto_xtr_ipv4_mask;
-extern uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_mask;
-extern uint64_t rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
-extern uint64_t rte_net_ice_dynflag_proto_xtr_tcp_mask;
-extern uint64_t rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
-
-/**
- * The mbuf dynamic field pointer for protocol extraction metadata.
- */
-#define RTE_NET_ICE_DYNF_PROTO_XTR_METADATA(m) \
-	RTE_MBUF_DYNFIELD((m), \
-			  rte_net_ice_dynfield_proto_xtr_metadata_offs, \
-			  uint32_t *)
-
-/**
- * The mbuf dynamic flag for VLAN protocol extraction metadata, it is valid
- * when dev_args 'proto_xtr' has 'vlan' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_VLAN \
-	(rte_net_ice_dynflag_proto_xtr_vlan_mask)
-
-/**
- * The mbuf dynamic flag for IPv4 protocol extraction metadata, it is valid
- * when dev_args 'proto_xtr' has 'ipv4' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_IPV4 \
-	(rte_net_ice_dynflag_proto_xtr_ipv4_mask)
-
-/**
- * The mbuf dynamic flag for IPv6 protocol extraction metadata, it is valid
- * when dev_args 'proto_xtr' has 'ipv6' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_IPV6 \
-	(rte_net_ice_dynflag_proto_xtr_ipv6_mask)
-
-/**
- * The mbuf dynamic flag for IPv6 with flow protocol extraction metadata, it is
- * valid when dev_args 'proto_xtr' has 'ipv6_flow' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_IPV6_FLOW \
-	(rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask)
-
-/**
- * The mbuf dynamic flag for TCP protocol extraction metadata, it is valid
- * when dev_args 'proto_xtr' has 'tcp' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_TCP \
-	(rte_net_ice_dynflag_proto_xtr_tcp_mask)
-
-/**
- * The mbuf dynamic flag for IP_OFFSET extraction metadata, it is valid
- * when dev_args 'proto_xtr' has 'ip_offset' specified.
- */
-#define RTE_PKT_RX_DYNF_PROTO_XTR_IP_OFFSET \
-	(rte_net_ice_dynflag_proto_xtr_ip_offset_mask)
-
-/**
- * Check if mbuf dynamic field for protocol extraction metadata is registered.
- *
- * @return
- *   True if registered, false otherwise.
- */
-__rte_experimental
-static __rte_always_inline int
-rte_net_ice_dynf_proto_xtr_metadata_avail(void)
-{
-	return rte_net_ice_dynfield_proto_xtr_metadata_offs != -1;
-}
-
-/**
- * Get the mbuf dynamic field for protocol extraction metadata.
- *
- * @param m
- *    The pointer to the mbuf.
- * @return
- *   The saved protocol extraction metadata.
- */
-__rte_experimental
-static __rte_always_inline uint32_t
-rte_net_ice_dynf_proto_xtr_metadata_get(struct rte_mbuf *m) -{
-	return *RTE_NET_ICE_DYNF_PROTO_XTR_METADATA(m);
-}
-
-/**
- * Dump the mbuf dynamic field for protocol extraction metadata.
- *
- * @param m
- *    The pointer to the mbuf.
- */
-__rte_experimental
-static inline void
-rte_net_ice_dump_proto_xtr_metadata(struct rte_mbuf *m) -{
-	union rte_net_ice_proto_xtr_metadata data;
-
-	if (!rte_net_ice_dynf_proto_xtr_metadata_avail())
-		return;
-
-	data.metadata = rte_net_ice_dynf_proto_xtr_metadata_get(m);
-
-	if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_VLAN)
-		printf(" - Protocol Extraction:[0x%04x:0x%04x],vlan,stag=%u:%u:%u,ctag=%u:%u:%u",
-		       data.raw.data0, data.raw.data1,
-		       data.vlan.stag_pcp,
-		       data.vlan.stag_dei,
-		       data.vlan.stag_vid,
-		       data.vlan.ctag_pcp,
-		       data.vlan.ctag_dei,
-		       data.vlan.ctag_vid);
-	else if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_IPV4)
-		printf(" - Protocol Extraction:[0x%04x:0x%04x],ipv4,ver=%u,hdrlen=%u,tos=%u,ttl=%u,proto=%u",
-		       data.raw.data0, data.raw.data1,
-		       data.ipv4.version,
-		       data.ipv4.ihl,
-		       data.ipv4.tos,
-		       data.ipv4.ttl,
-		       data.ipv4.protocol);
-	else if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_IPV6)
-		printf(" - Protocol Extraction:[0x%04x:0x%04x],ipv6,ver=%u,tc=%u,flow_hi4=0x%x,nexthdr=%u,hoplimit=%u",
-		       data.raw.data0, data.raw.data1,
-		       data.ipv6.version,
-		       data.ipv6.tc,
-		       data.ipv6.flowhi4,
-		       data.ipv6.nexthdr,
-		       data.ipv6.hoplimit);
-	else if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_IPV6_FLOW)
-		printf(" - Protocol Extraction:[0x%04x:0x%04x],ipv6_flow,ver=%u,tc=%u,flow=0x%x%04x",
-		       data.raw.data0, data.raw.data1,
-		       data.ipv6_flow.version,
-		       data.ipv6_flow.tc,
-		       data.ipv6_flow.flowhi4,
-		       data.ipv6_flow.flowlo16);
-	else if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_TCP)
-		printf(" - Protocol Extraction:[0x%04x:0x%04x],tcp,doff=%u,flags=%s%s%s%s%s%s%s%s",
-		       data.raw.data0, data.raw.data1,
-		       data.tcp.doff,
-		       data.tcp.cwr ? "C" : "",
-		       data.tcp.ece ? "E" : "",
-		       data.tcp.urg ? "U" : "",
-		       data.tcp.ack ? "A" : "",
-		       data.tcp.psh ? "P" : "",
-		       data.tcp.rst ? "R" : "",
-		       data.tcp.syn ? "S" : "",
-		       data.tcp.fin ? "F" : "");
-	else if (m->ol_flags & RTE_PKT_RX_DYNF_PROTO_XTR_IP_OFFSET)
-		printf(" - Protocol Offset:ip_offset=%u",
-		       data.ip_ofs);
-}
-
-__rte_experimental
-int rte_pmd_ice_dump_package(uint16_t port, uint8_t **buff, uint32_t *size);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _RTE_PMD_ICE_H_ */
diff --git a/drivers/net/ice/version.map b/drivers/net/ice/version.map index 60a3f17393..620152e080 100644
--- a/drivers/net/ice/version.map
+++ b/drivers/net/ice/version.map
@@ -6,12 +6,5 @@ EXPERIMENTAL {
 	global:
 
 	# added in 19.11
-	rte_net_ice_dynfield_proto_xtr_metadata_offs;
-	rte_net_ice_dynflag_proto_xtr_vlan_mask;
-	rte_net_ice_dynflag_proto_xtr_ipv4_mask;
-	rte_net_ice_dynflag_proto_xtr_ipv6_mask;
-	rte_net_ice_dynflag_proto_xtr_ipv6_flow_mask;
-	rte_net_ice_dynflag_proto_xtr_tcp_mask;
-	rte_net_ice_dynflag_proto_xtr_ip_offset_mask;
 	rte_pmd_ice_dump_package;
 };
--
2.34.1