RE: [RFC] lib/ethdev: introduce table driven APIs

["Zhang, Qi Z" <qi.z.zhang@intel.com>]

Hi Ori:

	Thank you for your review!
	Comment inline.
	Please let me know if anything I missed.

Thanks
Qi

> -----Original Message-----
> From: Ori Kam <orika@nvidia.com>
> Sent: Thursday, June 15, 2023 2:31 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; NBU-Contact-Thomas Monjalon
> (EXTERNAL) <thomas@monjalon.net>; david.marchand@redhat.com;
> Richardson, Bruce <bruce.richardson@intel.com>; jerinj@marvell.com;
> ferruh.yigit@amd.com
> Cc: Mcnamara, John <john.mcnamara@intel.com>; Zhang, Helin
> <helin.zhang@intel.com>; techboard@dpdk.org; dev@dpdk.org
> Subject: RE: [RFC] lib/ethdev: introduce table driven APIs
> 
> Hi Qi,
> 
> 
> 1. it may be useful to get some general calling flow what comes from the
> application, what comes from the compiler.
> Simple example will be good.

An example of decap VXLAN TCP flow is explained in problem statement (http://mails.dpdk.org/archives/dev/2023-May/267719.html)
covering the following information.

1. the p4 source code, the definition of the table and actions
2. the table / action hints generated by the compiler, details to each fields.
3. How the Control Plane Application utilizes the P4 Runtime API to program the rule with the respective table and action IDs

The DPDK PMD is responsible for loading the hints generated by the compiler. 
This enables the PMD to accept requests from the P4 Runtime and reject any incompatible request.

> 
> 2. I gave some comments about names but those are in low priority, first, we
> need to understand what is the basic flow.
> 
> 3. in your suggested API there is no use of RTE_FLOW and I assume that the
> PMD implementation will just offload the rules to the HW.
> if so this will result in duplicate APIs and maintenance issues.
> I think we should leverage the fact that P4 has a complier which can gives us
> all the input we need.

The purpose of the RFC  is to introduce a front-end API that addresses the previously mentioned problem statement.

In order to facilitate P4 runtime, the implementation of a front-end table-driven API shows promise. 
This API simplifies the user's task of enabling a P4 runtime backend, eliminating the need to handle the translation of table and action IDs into the rte_flow protocols.

Some customers may still prefer the protocol-based rte_flow approach, especially if they do not make use of P4 runtime.
So, I believe that this proposal serves as an extension to the rte_flow API, providing support for new use case, but not a duplicate implementation.

> 
> 4. From reading the code, I have the assumption that the complier programs
> the HW and the PMD reads this info am I correct?

Yes, typically, the compiler compiles the P4 source code into a binary format that is then programmed into the hardware. 
In addition, the compiler also produces a hint file, often in JSON format, which serves as a guide for the PMD.

> 
> 5. as I see it there should be some control stage where everything is
> configured and then application just offload the rules, there is no reason to
> query data about keys and tables.

I assume you are question about the learning API which I think it offers below potential benefits. 

1. It aids in diagnostics, such as using testpmd to examine tables, debug entries, or expose pipeline definition through telemetric ...
2. It assists applications in negotiating with hardware when maintaining different pipeline deployments, ensuring compatibility.

> 
> 6. I didn't fully review the API below but have some comments inline.
> 
> 
> Thanks,
> Ori
> 
> > -----Original Message-----
> > From: Qi Zhang <qi.z.zhang@intel.com>
> > Sent: Monday, June 12, 2023 2:16 PM
> >
> > The patch addresses the problem statement [1] by introducing a set of
> > "Table-Driven" APIs in rte_flow.
> >
> > This approach is inspired by p4land/tdi [2] and is particularly
> > beneficial for P4 programmable network controller drivers.
> > It provides the following advantages:
> >
> > * Easy integration of DPDK as a P4 runtime [3] backend.
> > * Reduced effort for PMDs to enable flow offloading when the packet
> >   processing unit is abstracted as a pipeline of match/action tables
> >   in low-level drivers.
> >
> > The new APIs can be categoried into 5 types
> >
> > 1. Learning APIs
> >
> >    Retrieve information about attributes supported by each table
> >    and action specification in the current pipeline.
> >
> >    rte_flow_table_list_get
> >    rte_flow_table_info_get
> >    rte_flow_table_info_get_by_name
> >    rte_flow_table_key_field_info_get
> >    rte_flow_table_key_field_info_get_by_name
> >    rte_flow_action_spec_list_group
> >    rte_flow_action_spec_info_get
> >    rte_flow_action_spec_info_get_by_name
> >    rte_flow_action_spec_field_info_get_by_name
> >
> 
> What is the idea to query the tables? Isn't the idea that application will
> create them?

This is explained above.

> 
> > 2. Key / Action Object Management API:
> >
> >    Create, destroy, and clone key and action objects.
> >
> >    rte_flow_table_key_create
> 
> This creates a new table?

No, it creates a key object (or instance) for a table, 
You can assume memory will be allocated to store the key values with below key_field APIs.

> >    rte_flow_table_key_destroy
> >    rte_flow_table_key_clone
> >    rte_flow_table_key_field_set
> >    rte_flow_table_key_field_set_by_name
> >    rte_flow_table_key_field_set_with_mask
> >    rte_flow_table_key_field_set_with_mask_by_name
> >    rte_flow_table_key_field_set_with_range
> >    rte_flow_table_key_field_set_with_range_by_name
> >    rte_flow_table_key_field_set_with_prefix_
> >    rte_flow_table_key_field_set_with_prefix_by_name
> >
> 
> Why do we need so many functions? We can create a table with the keys?
> I assume that keys are the matching values right?

A key object is composed by a set of fields, each fields has different match type, it could be exact match, wildcard (mask is needed) , range (min, max) or lpm (a prefix bit width is required)


> 
> >    rte_flow_table_action_create
> 
> This creates a single action in selected table?


Yes, this created an action instance, which can be paired with a key object and be added into a table as a rule entry.

The action instance is a little bit difference with rte_flow_action..

One action instance is composed at of action fields, typically each action field can be mapped to rte_flow_aciton,  (to queue, drop ...) from hardware's pointer of view.

P4 use below syntax define table / actions

table decap_vxlan_tcp_table {
      .....
	actions = {
        @tableonly decap_vxlan_fwd;
        @tableonly decap_vxlan_dnat_fwd;
        @tableonly decap_vxlan_snat_fwd;
        @defaultonly set_exception;
    }  
}

Table decap_vxlan_tcp_table  accept 4 kind of action, but only one action can be selected for each rule.

action decap_vxlan_fwd(PortId_t port_id) {
    meta.mod_action = (bit<11>)VXLAN_DECAP_OUTER_IPV4;
    send_to_port(port_id);
}

Here the port_id is the only field of the action decap_vxlan_fwd, this is exposed by the compiler.


> 
> >    rte_flow_table_action_destroy
> >    rte_flow_table_action_clone
> >    rte_flow_table_action_field_get
> >    rte_flow_table_action_field_set
> >    rte_flow_table_action_field_set_by_name
> >
> 
> Again I don't understand why so many functions.
> 
> > 3. Table Entry Update Synchronized APIs:
> >
> >    Enable synchronized table updates, ensuring the updates are
> >    run-to-completion.
> >
> >    rte_flow_table_entry_add
> >    rte_flow_table_entry_query
> >    rte_flow_table_entry_del
> >    rte_flow_table_entry_count_query
> >    rte_flow_table_default_action_set
> >    rte_flow_table_default_action_cancel
> >
> > 4. Table Entry Update Asynchronized APIs
> >
> >    Provide asynchronous table update mode using a
> >    prepare/commit/pull pattern.
> >
> >    rte_flow_table_entry_add_prepare
> >    rte_flow_table_entry_del_prepare
> >    rte_flow_table_update_status_commit
> >    rte_flow_table_update_status_pull
> >
> In general I think async function should have the word async

Of cause, we can add.
> 
> > 5. DPDK to PNA Interpretation APIs
> >
> >    Facilitate APIs that map the DPDK context to the P4 Portable
> >    NIC Architecture (PNA) context, enabling interoperability between
> >    DPDK and PNA applications
> >
> 
> This API is for registering queues in DPDK which can be used by non DPDK
> application?

PNA (P4 Portable NIC Architecture) is an ongoing specification. https://p4.org/p4-spec/docs/PNA.html
It defines the hardware abstraction with some data structure (e.g. port, queue..)

That's why I think it may worth to have some help APIs that allows user to convert from DPDK context to PNA.
Currently we can take this as an open, I will not insist to have them, just what to get more inputs.

> 
> >    rte_flow_pna_port_get
> >    rte_flow_pna_rx_queue_get
> >    rte_flow_pna_tx_queue_get
> >
> > Follow the example in Problem Statement [1], to create a rule for
> > table decap_vxlan_tcp_table with action decap_vxlan_fwd, we can use
> > the following code.
> >
> > Code Snippet:
> >
> > /* Get the table info */
> > struct rte_flow_table_info tbl_info;
> > rte_flow_table_info_get_by_name(port_id, "decap_vxlan_tcp_table",
> > &tbl_info);
> >
> > /* Create the key */
> > struct rte_flow_table_key *key;
> > rte_flow_table_key_create(port_id, tbl_info->id, &key);
> >
> > /* Set the key fields */
> > rte_flow_table_key_field_set_by_name(port_id, key, "wire_port",
> > &wire_port, 2);
> 
> This function should be called per each entery right?

Yes, but again, we can always aggregate all key fields  into an array with one API calls.

> The values should arrive from gRPC right?

For P4runtime case, yes, its from gPRC client, but gRPC will aggregate.

> 
> > rte_flow_table_key_field_set_by_name(port_id, key, "tun_ip_src",
> > &tun_ip_src, 4); rte_flow_table_key_field_set_by_name(port_id, key,
> > "tun_ip_dst", &tun_ip_dst, 4);
> > rte_flow_table_key_field_set_by_name(port_id, key, "vni", &vni, 3);
> > rte_flow_table_key_field_set_by_name(port_id, key, "ipv4_src",
> > &ipv4_src, 4); rte_flow_table_key_field_set_by_name(port_id, key,
> > "ipv4_dst", &ipv4_dst, 4);
> > rte_flow_table_key_field_set_by_name(port_id, key, "src_port",
> > &src_port, 2); rte_flow_table_key_field_set_by_name(port_id, key,
> > "dst_port", &dst_port, 2);
> >
> > /* Get the action spec info */
> > struct rte_flow_action_spec_info as_info;
> > rte_flow_action_spec_info_get_by_name(port_id, "decap_vxlan_fwd",
> > &as_info);
> >
> Where are we getting the action from is this the result of the complier?

Yes, an actions spec defines all the fields name / size / byte order/ bitwidth of an action type.

> 
> > /* Create the action */
> > struct rte_flow_table_action *action;
> > rte_flow_table_action_create(port_id, as_info->id, &action);
> >
> > /* Set the action fields */
> > rte_flow_table_action_field_set_by_name(port_id, action, "mod_id",
> > &mod_id, 3); rte_flow_table_action_field_set_by_name(port_id, action,
> > "port_id", &target_port_id, 2);
> >
> > /* Add the entry */
> > rte_flow_table_entry_add(port_id, tbl_info->id, key, action);
> >
> > /* destroy key and action */
> > rte_flow_table_action_destroy(port_id, action);
> > rte_flow_table_key_destroy(port_id, key);
> >
> > ...
> >
> > Below code demonstrates how to use the prepare/commit/pull for high
> > performance table entry updates.
> >
> > Code Snipped:
> >
> > struct rte_flow_table_key *keys[BATCH_SIZE]; struct
> > rte_flow_table_action *actions[BATCH_SIZE]; struct
> > rte_flow_table_update_status stats[BATCH_SIZE];
> >
> > /* Create Keys and Actions */
> > for (i = 0; i < BATCH_SIZE; i++) {
> >     rte_flow_table_key_create(port_id, table_id, &keys[i]);
> 
> This is the configuration stage right?

Yes, basically this is trying to reserve a set of key objects in memory.

> 
> >     /* set key field */
> >     rte_flow_table_key_field_set(...)
> >
> >     rte_flow_table_action_create(port_id, table_id, spec_id, &actions[i]);
> >     /* set action field */
> >     rte_flow_table_action_field_set(...)
> > }
> >
> > /* program loop */
> > While (condition = true) {
> >
> >     /* Prepare entry adding */
> >     for (i = 0; i < BATCH_SIZE; i++) {
> >         struct rte_flow_table_key *key = keys[i];
> >         struct rte_flow_table_action *action = actions[i];
> >
> >         rte_flow_table_entry_add_prepare(port_id, TABLE_ID, key,
> > action);
> 
> For each call you add complete rule or just one matching and one action?

The action here is an instance of action 
An action instance is composed of a set of action fields which I have explained previous.

> Why not give a list of actions and keys to avoid extra function calls?

Of cause it is worthwhile to support what you suggested here.


> 
> >     }
> >
> >     /* Commit to hardware */
> >     rte_flow_table_update_commit(port_id);
> >
> >     /* pull status */
> >     int count = 0;
> >     while (count < BATCH_SIZE) {
> >         count += rte_flow_table_update_status_pull(port_id, stats,
> > BATCH_SIZE, NULL);
> >     }
> >
> >     /* reused Key and Action object */
> >     for (i = 0; i< BATCH_SIZE; i++) {
> >             struct rte_flow_table_key *key = stats[i].key;
> >             struct rte_flow_table_action *action = stats[i].action;
> >
> >             rte_flow_table_key_field_set(...);
> >             rte_flow_table_action_field_set(...)
> >     }
> > }
> >
> > ...
> >
> > NOTE: For simplicity, error check and the rte_flow_error parameter for
> > each API has been omitted:
> >
> > [1]. http://mails.dpdk.org/archives/dev/2023-May/267719.html
> > [2]. https://github.com/p4lang/tdi/
> > [3]. https://p4.org/p4-spec/p4runtime/main/P4Runtime-Spec.html
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >  lib/ethdev/rte_flow_table.h | 1261
> > +++++++++++++++++++++++++++++++++++
> >  1 file changed, 1261 insertions(+)
> >  create mode 100644 lib/ethdev/rte_flow_table.h
> >
> > diff --git a/lib/ethdev/rte_flow_table.h b/lib/ethdev/rte_flow_table.h
> > new file mode 100644 index 0000000000..31edf57a0f
> > --- /dev/null
> > +++ b/lib/ethdev/rte_flow_table.h
> > @@ -0,0 +1,1261 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright 2023 Intel Corporation.
> > + */
> > +
> > +#ifndef RTE_FLOW_TABLE_H_
> > +#define RTE_FLOW_TABLE_H_
> > +
> > +#include <stdint.h>
> > +#include <stdbool.h>
> > +
> > +/**
> > + * Max number of key field in a table.
> > + */
> > +#define RTE_FLOW_TABLE_KEY_FIELD_NUM_MAX	256
> > +/**
> > + * Max number of action spec in a table.
> > + */
> > +#define RTE_FLOW_TABLE_ACTION_SPEC_NUM_MAX	64
> > +/**
> > + * Max number of field in an action spec.
> > + */
> > +#define RTE_FLOW_ACTION_SPEC_FIELD_NUM_MAX	16
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Table key match type.
> > + *
> > + * To specify the key match type of a table.
> > + */
> > +enum rte_flow_table_key_match_type {
> > +	RTE_FLOW_TABLE_KEY_MATCH_TYPE_EXACT, /**< Exact match. */
> > +	RTE_FLOW_TABLE_KEY_MATCH_TYPE_WILDCARD, /**< Wildcard
> > match. */
> > +	RTE_FLOW_TABLE_KEY_MATCH_TYPE_RANGE, /**< Range match. */
> > +	RTE_FLOW_TABLE_KEY_MATCH_TYPE_LPM, /**< longest prefix
> > match. */
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Byte order.
> > + *
> > + * To specify the byte order of table key / action field value in bytes.
> > + */
> > +enum rte_flow_byte_order {
> > +	RTE_FLOW_BYTE_ORDER_HOST, /**< follow host byte order. */
> > +	RTE_FLOW_BYTE_ORDER_NETWORK, /**< follow network byte
> > order. */
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Flow rule table info.
> > + *
> > + * A structure stores the properties of a flow rule table.
> > + * Typically, a flow rule table represents to a P4 table which
> > +describe a
> > + * match/action unit in packet process pipeline.
> > + */
> > +struct rte_flow_table_info {
> > +	uint32_t id; /**< Identifier of a table within the ethdev. */
> > +	const char *name; /**< Name of the table. */
> > +	const char *annotation; /**< Human readable message about this
> > table. */
> > +	uint16_t key_field_num; /**< Number of key field. */
> > +	uint32_t key_fields[RTE_FLOW_TABLE_KEY_FIELD_NUM_MAX]; /**<
> > Key field id array. */
> > +	uint16_t action_spec_num; /**< Number of action spec. */
> > +	uint32_t
> > action_specs[RTE_FLOW_TABLE_ACTION_SPEC_NUM_MAX]; /**< Action
> spec id
> > array */
> > +};
> > +
> 
> Why do we need this info?
> 
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Table key field info.
> > + *
> > + * A structure stores the properties of a table key field.
> > + */
> > +struct rte_flow_table_key_field_info {
> > +	uint32_t table_id; /**< Identifier of a table within the ethdev. */
> > +	uint32_t field_id; /**< Identifier of the key field within the table. */
> > +	const char *name;  /**< Name of the key field. */
> > +	const char *annotation; /**< Human readable message about this
> > key field. */
> > +	enum rte_flow_table_key_match_type match_type; /**< Key match
> > type. */
> > +	uint16_t bit_width; /**< Bit width of the field value. */
> > +	uint16_t byte_width; /**< Number of bytes to store the field value.
> > */
> > +	/**
> > +	 * Byte order of the byte array that store the key value.
> > +	 */
> > +	enum rte_flow_byte_order byte_order; };
> > +
> 
> Like above why is it needed?
> 
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Action spec info.
> > + *
> > + * A structure stores the properties of a action specification.
> > + * Typically, a action specification represents a P4 Action.
> > + */
> > +struct rte_flow_action_spec_info {
> > +	uint32_t id; /**< Identifier of a action spec within the ethdev. */
> > +	const char *name; /**< Name of the action spec. */
> > +	const char *annotation; /**< Human readable message about this
> > action spec */
> > +	uint16_t field_num; /**< Number of fields */
> > +	uint32_t fields[RTE_FLOW_ACTION_SPEC_FIELD_NUM_MAX]; /**<
> > Field id array */
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Action spec field info.
> > + *
> > + * A structure stores the properties of a action spec field.
> > + */
> > +struct rte_flow_action_spec_field_info {
> > +	uint32_t spec_id; /**< Identifier of a action spec within the ethdev.
> > */
> > +	uint32_t field_id; /**< Identifier of the field within the action spec.
> > */
> > +	const char *name; /**< Name of the field. */
> > +	const char *annotation; /**< Human readable message about this
> > action spec. */
> > +	uint16_t bit_width; /**< Bit width of the field value */
> > +	uint16_t byte_width; /**< Number of bytes to store the field value.
> > */
> > +	/**
> > +	 * Byte order of the byte array that stores the key value.
> > +	 */
> > +	enum rte_flow_byte_order byte_order; };
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Table Key object.
> > + *
> > + * A structure represent a table key object, should be created /
> > +destroyed
> > by
> > + * rte_flow_table_key_create and rte_flow_table_key_destroy.
> > + */
> > +struct rte_flow_table_key {
> > +	uint32_t table_id; /**< Indicate which table the key instance
> > +belongs
> > to. */
> > +	int ref_cnt; /**< Reference count, in async ops it prevents the
> > +object
> > be destoried .*/
> > +	uint8_t data[]; /**< PMD specific data. */ };
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Action object.
> > + *
> > + * A structure represent a table action object, should be created /
> > destroyed by
> > + * rte_flow_table_action_create and rte_flow_table_action_destroy.
> > + */
> > +struct rte_flow_table_action {
> > +	uint32_t table_id; /**< Indicate which table the action instance
> > belongs to. */
> > +	uint32_t spec_id; /**< Indicate which action spec the action follow.
> > */
> > +	int ref_cnt; /**< Reference count, in async ops it prevents the
> > +object
> > be destoried .*/
> > +	uint8_t data[]; /**< PMD specific data. */ };
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * ID list.
> > + *
> > + * An id list with variant size, should be created by
> > + * rte_flow_table_list_popup or rte_flow_action_spec_list_popup.
> > + *
> > + * Application need to free the list by rte_free.
> > + */
> > +struct rte_flow_id_list {
> > +	uint32_t num; /**< Number of the id list */
> > +	uint32_t ids[]; /**< ID array */
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Popup table id list.
> > + *
> > + * A variant size list that store all table identifiers will be created.
> > + * Application need to free the list by rte_free.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[out] list
> > + *    A variant size id list, store all table identifiers of current ethernet
> > + *    device.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_list_popup(uint16_t port_id,
> > +			  struct rte_flow_id_list **list,
> > +			  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get table info by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[out] info
> > + *    Pointer to store the table info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_info_get(uint16_t port_id,
> > +			uint32_t table_id,
> > +			struct rte_flow_table_info *info,
> > +			struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > +* @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get table info by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] name
> > + *    Table name.
> > + * @param[out] info
> > + *    Pointer to store the table info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_info_get_by_name(uint16_t port_id,
> > +				const char *name,
> > +				struct rte_flow_table_info *info,
> > +				struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get table key info by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] field_id
> > + *    Key field identifier.
> > + * @param[info] info
> > + *    Pointer to store the table key field info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_info_get(uint16_t port_id,
> > +				  uint32_t table_id,
> > +				  uint32_t field_id,
> > +				  struct rte_flow_table_key_field_info *info,
> > +				  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Popup action spec id list.
> > + *
> > + * A variant size list that store all action spec identifiers will be created.
> > + * Application need to free the list by rte_free.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_action_spec_list_popup(uint16_t port_id,
> > +				struct rte_flow_id_list **list,
> > +				struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get action spec info by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] spec_id
> > + *    Action spec identifier.
> > + * @info[out] info
> > + *    Pointer to store the action spec info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_action_spec_info_get(uint16_t port_id,
> > +			      uint32_t spec_id,
> > +			      struct rte_flow_action_spec_info *info,
> > +			      struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get action spec info by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] name
> > + *    Action spec name.
> > + * @info[out] info
> > + *    Pointer to store the action spec info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_action_spec_info_get_by_name(uint16_t port_id,
> > +				      const char *name,
> > +				      struct rte_flow_action_spec_info *info,
> > +				      struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get action spec field info by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] spec_id
> > + *    Action spec identifier.
> > + * @param[in] field_id
> > + *    Field identifier.
> > + * @param[out] info
> > + *    Pointer to store the action spec field info.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_action_spec_field_info_get(uint16_t port_id,
> > +				    uint32_t spec_id,
> > +				    uint32_t field_id,
> > +				    struct rte_flow_action_spec_field_info
> > *info,
> > +				    struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Create a table key object.
> > + *
> > + * Application need to call rte_flow_table_key_destroy to free the
> > +key
> > object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[out] key
> > + *    Table key object created by PMD.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_create(uint16_t port_id,
> > +			  uint32_t table_id,
> > +			  struct rte_flow_table_key **key,
> > +			  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Destroy a table key object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to destroy.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_destroy(uint16_t port_id,
> > +			   struct rte_flow_table_key *key,
> > +			   struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Create an table action object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] spec_id
> > + *    Action spec identifier.
> > + * @param[out] action
> > + *    Action key created by PMD.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_create(uint16_t port_id,
> > +			     uint32_t table_id,
> > +			     uint32_t spec_id,
> > +			     struct rte_flow_table_action **action,
> > +			     struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Destroy an table action object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] action
> > + *    Action object to destroy.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_destroy(uint16_t port_id,
> > +			      struct rte_flow_table_action *action,
> > +			      struct rte_flow_error *error);
> > +
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set table key field value by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] field_id
> > + *    key field identifier.
> > + * @param[in] value
> > + *    Byte array to store the value
> > + * @param[in] size
> > + *    Size of the byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set(uint16_t port_id,
> > +			     struct rte_flow_table_key *key,
> > +			     uint32_t field_id,
> > +			     const uint8_t *value,
> > +			     uint16_t size,
> > +			     struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set table key field value by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] name
> > + *    key field name.
> > + * @param[in] value
> > + *    Byte array to store the value to match.
> > + * @param[in] size
> > + *    Size of the byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_by_name(uint16_t port_id,
> > +				     struct rte_flow_table_key *key,
> > +				     const char *name,
> > +				     const uint8_t *value,
> > +				     uint16_t size,
> > +				     struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set wildcard match key field by identifier.
> > + *
> > + * For wildcard match, only a bit set in mask should be matched.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] field_id
> > + *    Key field identifier.
> > + * @param[in] value
> > + *    Byte array stores the value to match.
> > + * @param[in] mask
> > + *    Byte array stores the bit mask.
> > + * @param[in] size
> > + *    Size of value and mask byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_mask(uint16_t port_id,
> > +				       struct rte_flow_table_key *key,
> > +				       uint32_t field_id,
> > +				       const uint8_t *value,
> > +				       const uint8_t *mask,
> > +				       uint16_t size,
> > +				       struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set wildcard match key field by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] name
> > + *    Key field name.
> > + * @param[in] value
> > + *    Byte array stores the value to match.
> > + * @param[in] mask
> > + *    Byte array stores the bit mask.
> > + * @param[in] size
> > + *    Size of value and mask byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_mask_by_name(uint16_t port_id,
> > +					       struct rte_flow_table_key *key,
> > +					       const char *name,
> > +					       const uint8_t *value,
> > +					       const uint8_t *mask,
> > +					       uint16_t size,
> > +					       struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set range match key field by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] field_id
> > + *    Key field identifier.
> > + * @param[in] min
> > + *    Byte array stores the min value of the range to match
> > + * @param[in] max
> > + *    Byte array stores the max value of the range to match
> > + * @param[in] size
> > + *    Size of the min and max byte array
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_range(uint16_t port_id,
> > +				        struct rte_flow_table_key *key,
> > +					uint32_t field_id,
> > +					const uint8_t *min,
> > +					const uint8_t *max,
> > +					uint16_t size,
> > +					struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set range match key field by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] name
> > + *    Key field name.
> > + * @param[in] min
> > + *    Byte array stores the min value of the range to match
> > + * @param[in] max
> > + *    Byte array stores the max value of the range to match
> > + * @param[in] size
> > + *    Size of the min and max byte array
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_range_by_name(uint16_t port_id,
> > +						struct rte_flow_table_key
> > *key,
> > +						const char *name,
> > +						const uint8_t *min,
> > +						const uint8_t *max,
> > +						uint16_t size,
> > +						struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set lpm match key field by identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] field_id
> > + *    Key field identifier.
> > + * @param[in] value
> > + *    Byte array stores the value to match.
> > + * @param[in] size
> > + *    Size of value byte array.
> > + * @param[in] prefix
> > + *    Bits of the prefix to match, must <= (8 * size)
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_prefix(uint16_t port_id,
> > +					 struct rte_flow_table_key *key,
> > +					 uint32_t field_id,
> > +					 const uint8_t *value,
> > +					 uint16_t size,
> > +					 uint16_t prefix,
> > +					 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set lpm match key field by name.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to update.
> > + * @param[in] name
> > + *    Key field name.
> > + * @param[in] value
> > + *    Byte array stores the value to match.
> > + * @param[in] size
> > + *    Size of value byte array.
> > + * @param[in] prefix
> > + *    Bits of the prefix to match, must <= (8 * size)
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_field_set_with_prefix_by_name(uint16_t port_id,
> > +						 struct rte_flow_table_key
> > *key,
> > +						 const char* name,
> > +						 const uint8_t *value,
> > +						 uint16_t size,
> > +						 uint16_t prefix,
> > +						 struct rte_flow_error
> > *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set action field value.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] action
> > + *    Action object to update.
> > + * @param[in] field_id
> > + *    Field identifier.
> > + * @param[in] value
> > + *    Byte array stores the value of the field.
> > + * @param[in] size
> > + *    Size of the byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_field_set(uint16_t port_id,
> > +				struct rte_flow_table_action *action,
> > +				uint32_t field_id,
> > +				const uint8_t *value,
> > +				uint16_t size,
> > +				struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * get action field value, application may use
> > +rte_flow_table_entry_query
> > + * to query by key and use this API to figure out each action field.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] action
> > + *    Action object to query.
> > + * @param[in] field_id
> > + *    Field identifier.
> > + * @param[out] value
> > + *    Byte array stores the value of the field.
> > + * @param[in | out] size
> > + *    Input as size of the byte array, return the size of the value.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_field_get(uint16_t port_id,
> > +				const struct rte_flow_table_action *action,
> > +				uint32_t field_id,
> > +				uint8_t *value,
> > +				uint16_t *size,
> > +				struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] action
> > + *    Action object to update.
> > + * @param[in] name
> > + *    Field name.
> > + * @param[in] value
> > + *    Byte array stores the value of the field.
> > + * @param[in] size
> > + *    Size of the byte array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_field_set_by_name(uint16_t port_id,
> > +					struct rte_flow_action *action,
> > +					const char *name,
> > +					const uint8_t *value,
> > +					uint16_t size,
> > +					struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Set table default action.
> > + *
> > + * The default action will take effect when a packet hit no rules.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier
> > + * @param[in] action
> > + *    Default action object.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_default_action_set(uint16_t port_id,
> > +				  uint32_t table_id,
> > +				  const struct rte_flow_table_action *action,
> > +				  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Cancel table default action
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_default_action_cancel(uint32_t port_id,
> > +				     uint32_t table_id,
> > +				     struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Add matching rule as a table entry, the rule take effect
> > +immediately
> > + * after the API call.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object.
> > + * @param[in] action
> > + *    Action object.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_add(uint16_t port_id,
> > +			 uint32_t table_id,
> > +			 const struct rte_flow_table_key *key,
> > +			 const struct rte_flow_table_action *action,
> > +			 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Query action of a table entry.
> > + *
> > + * If success, a new rte_flow_table_action object will be created.
> > + * Use rte_flow_table_action_destroy to free the resource.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object.
> > + * @param[out] action
> > + *    Action object returned.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_query(uint16_t port_id,
> > +			   uint32_t table_id,
> > +			   const struct rte_flow_table_key *key,
> > +			   struct rte_flow_table_action **action,
> > +			   struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Delete a table entry, this take effect immeidatly after the API call.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object to match.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_del(uint16_t port_id,
> > +			 uint32_t table_id,
> > +			 const struct rte_flow_table_key *key,
> > +			 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Query rule hit counters.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object to match.
> > + * @param[out] count
> > + *    Pointer stores the hit counters.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_count_query(uint16_t port_id,
> > +				 uint32_t table_id,
> > +				 const struct rte_flow_table_key *key,
> > +				 struct rte_flow_query_count *count,
> > +				 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Clone a table key object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] key
> > + *    Table key object to clone.
> > + * @param[out] new_key
> > + *    New table key object be created by PMD.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_key_clone(uint16_t port_id,
> > +			 const struct rte_flow_table_key *key,
> > +			 struct rte_flow_table_key **new_key,
> > +			 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Clone a action object.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] action
> > + *    Action object to clone.
> > + * @param[out] new_action
> > + *    New action object be created by PMD.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_action_clone(uint16_t port_id,
> > +			    const struct rte_flow_action *action,
> > +			    struct rte_flow_action **new_action,
> > +			    struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Prepare table entry adding.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object.
> > + * @param[in] action
> > + *    Action object.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_add_prepare(uint16_t port_id,
> > +				 uint32_t table_id,
> > +				 struct rte_flow_table_key *key,
> > +				 struct rte_flow_table_action *action,
> > +				 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Prepare table entry deletion.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] table_id
> > + *    Table identifier.
> > + * @param[in] key
> > + *    Table key object to match.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_entry_del_prepare(uint16_t port_id,
> > +				 uint32_t table_id,
> > +				 struct rte_flow_table_key *key,
> > +				 struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Commit all prepared adding and deletion requests.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_update_commit(uint16_t port_id,
> > +			     struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Table entry operation type.
> > + */
> > +
> > +enum rte_flow_table_update_op {
> > +	RTE_FLOW_TABLE_ENTRY_OP_ADD, /* Add an entry */
> > +	RTE_FLOW_TABLE_ENTRY_OP_DEL, /* Delete an entry */
> > +	RTE_FLOW_TABLE_ENTRY_OP_QRY, /* Query an entry */ };
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Table entry update status.
> > + */
> > +struct rte_flow_table_update_status {
> > +	struct rte_flow_table_key *key; /**< Table key object of the entry */
> > +	struct rte_flow_table_action *action; /**< Action object of the
> > +entry
> > */
> > +	enum rte_flow_table_update_op op; /**< Operation type */
> > +	enum rte_flow_error_type err; /**< Error type */ };
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Pull table entry update status.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[out] stats
> > + *    An array stores the status of all finished entry adding / delete
> > + *    requests.
> > + * @param[in] size
> > + *    Size of the input array.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    >=0 on success, indiates the number of status be pulled.
> > + *    A negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_table_update_status_pull(uint16_t port_id,
> > +				  struct rte_flow_table_update_status *stats,
> > +				  int size,
> > +				  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get PNA port identifier.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] ethdev_port_id
> > + *    Ethdev port identifier maps to the required PNA port.
> > + * @param[out] pna_port_id
> > + *    Pointer stores the PNA port identifier.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_pna_port_get(uint16_t port_id,
> > +		      uint16_t ethdev_port_id,
> > +		      uint32_t *hw_port_id,
> > +		      struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get a PNA queue identifer from a ethdev Rx queue.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] ethdev_port_id
> > + *    Ethdev port identifier the Rx queue belongs to.
> > + * @param[in] ethdev_queue_id
> > + *    Ethdev Rx queue index that maps to the required PNA queue
> identifier.
> > + * @param[out] pna_queue_id
> > + *    Pointer stores the PNA queue identifier.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_pna_rx_queue_get(uint16_t port_id,
> > +			  uint16_t ethdev_port_id,
> > +			  uint16_t ethdev_queue_id,
> > +			  uint32_t *hw_queue_id,
> > +			  struct rte_flow_error *error);
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Get a PNA queue identifer from a ethdev Tx queue.
> > + *
> > + * @param[in] port_id
> > + *    Port identifier of the Ethernet device.
> > + * @param[in] ethdev_port_id
> > + *    Ethdev port identifier the Tx queue belongs to.
> > + * @param[in] ethdev_queue_id
> > + *    Ethdev Tx queue index that maps to the required PNA queue
> identifier.
> > + * @param[out] pna_queue_id
> > + *    Pointer stores the PNA queue identifier.
> > + * @param[out] error
> > + *    Perform verbose error reporting if not NULL. PMDs initialize this
> > + *    structure in case of error only.
> > + *
> > + * @return
> > + *    0 on success, a negative errno value otherwise and rte_errno is set.
> > + */
> > +__rte_experimental int
> > +rte_flow_pna_tx_queue_get(uint16_t port_id,
> > +			  uint16_t ethdev_port_id,
> > +			  uint16_t ethdev_queue_id,
> > +			  uint32_t *hw_queue_id,
> > +			  struct rte_flow_error *error);
> > +#endif
> > --
> > 2.31.1