Re: [dpdk-dev] [PATCH v4 2/5] app/test-flow-perf: add insertion rate calculation

[Andrew Rybchenko <arybchenko@solarflare.com>]

On 4/30/20 1:33 PM, Wisam Jaddo wrote:
> Add insertion rate calculation feature into flow
> performance application.
> 
> The application now provide the ability to test
> insertion rate of specific rte_flow rule, by
> stressing it to the NIC, and calculate the
> insertion rate.
> 
> The application offers some options in the command
> line, to configure which rule to apply.
> 
> After that the application will start producing
> rules with same pattern but increasing the outer IP
> source address by 1 each time, thus it will give
> different flow each time, and all other items will
> have open masks.
> 
> The current design have single core insertion rate.
> In the future we may have a multi core insertion
> rate measurement support in the app.
> 
> Signed-off-by: Wisam Jaddo <wisamm@mellanox.com>
> ---
>  app/test-flow-perf/Makefile          |   3 +
>  app/test-flow-perf/actions_gen.c     |  86 ++++++
>  app/test-flow-perf/actions_gen.h     |  48 ++++
>  app/test-flow-perf/flow_gen.c        | 176 ++++++++++++
>  app/test-flow-perf/flow_gen.h        |  61 ++++
>  app/test-flow-perf/items_gen.c       | 265 +++++++++++++++++
>  app/test-flow-perf/items_gen.h       |  68 +++++
>  app/test-flow-perf/main.c            | 416 +++++++++++++++++++++++++--
>  app/test-flow-perf/meson.build       |   8 +
>  app/test-flow-perf/user_parameters.h |  15 +
>  doc/guides/tools/flow-perf.rst       | 186 +++++++++++-
>  11 files changed, 1307 insertions(+), 25 deletions(-)
>  create mode 100644 app/test-flow-perf/actions_gen.c
>  create mode 100644 app/test-flow-perf/actions_gen.h
>  create mode 100644 app/test-flow-perf/flow_gen.c
>  create mode 100644 app/test-flow-perf/flow_gen.h
>  create mode 100644 app/test-flow-perf/items_gen.c
>  create mode 100644 app/test-flow-perf/items_gen.h
> 
> diff --git a/app/test-flow-perf/Makefile b/app/test-flow-perf/Makefile
> index 45b1fb1464..968c7c60dd 100644
> --- a/app/test-flow-perf/Makefile
> +++ b/app/test-flow-perf/Makefile
> @@ -19,6 +19,9 @@ CFLAGS += -Wno-unused-function
>  #
>  # all source are stored in SRCS-y
>  #
> +SRCS-y += actions_gen.c
> +SRCS-y += flow_gen.c
> +SRCS-y += items_gen.c
>  SRCS-y += main.c
>  
>  include $(RTE_SDK)/mk/rte.app.mk
> diff --git a/app/test-flow-perf/actions_gen.c b/app/test-flow-perf/actions_gen.c
> new file mode 100644
> index 0000000000..564ed820e4
> --- /dev/null
> +++ b/app/test-flow-perf/actions_gen.c
> @@ -0,0 +1,86 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * The file contains the implementations of actions generators.
> + * Each generator is responsible for preparing it's action instance
> + * and initializing it with needed data.
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + **/
> +
> +#include <sys/types.h>
> +#include <rte_malloc.h>
> +#include <rte_flow.h>
> +#include <rte_ethdev.h>
> +
> +#include "actions_gen.h"
> +#include "user_parameters.h"
> +
> +void
> +gen_mark(void)
> +{
> +	mark_action.id = MARK_ID;
> +}
> +
> +void
> +gen_queue(uint16_t queue)
> +{
> +	queue_action.index = queue;
> +}
> +
> +void
> +gen_jump(uint16_t next_table)
> +{
> +	jump_action.group = next_table;
> +}

It looks like global structures are used above.
It is hardly future-proof taking above words
about multi-core insertion into account.

> +void
> +gen_rss(uint16_t *queues, uint16_t queues_number)
> +{
> +	uint16_t queue;
> +	struct action_rss_data *rss_data;

It is better to have empty line here, since right now
it looks very misleading taking into account empty
line before NULL check below.

> +	rss_data = rte_malloc("rss_data",
> +		sizeof(struct action_rss_data), 0);
> +
> +	if (rss_data == NULL)
> +		rte_exit(EXIT_FAILURE, "No Memory available!");
> +
> +	*rss_data = (struct action_rss_data){
> +		.conf = (struct rte_flow_action_rss){
> +			.func = RTE_ETH_HASH_FUNCTION_DEFAULT,
> +			.level = 0,
> +			.types = ETH_RSS_IP,

Why does it differ from port RSS types defaults?

> +			.key_len = 0,
> +			.queue_num = queues_number,
> +			.key = 0,
> +			.queue = rss_data->queue,
> +		},
> +		.key = { 0 },
> +		.queue = { 0 },
> +	};
> +
> +	for (queue = 0; queue < queues_number; queue++)
> +		rss_data->queue[queue] = queues[queue];
> +
> +	rss_action = &rss_data->conf;
> +}
> +
> +void
> +gen_set_meta(void)
> +{
> +	meta_action.data = RTE_BE32(META_DATA);
> +	meta_action.mask = RTE_BE32(0xffffffff);
> +}
> +
> +void
> +gen_set_tag(void)
> +{
> +	tag_action.data = RTE_BE32(META_DATA);
> +	tag_action.mask = RTE_BE32(0xffffffff);
> +	tag_action.index = TAG_INDEX;
> +}
> +
> +void
> +gen_port_id(void)
> +{
> +	port_id.id = PORT_ID_DST;
> +}

Global structures again.

> diff --git a/app/test-flow-perf/actions_gen.h b/app/test-flow-perf/actions_gen.h
> new file mode 100644
> index 0000000000..556d48b871
> --- /dev/null
> +++ b/app/test-flow-perf/actions_gen.h
> @@ -0,0 +1,48 @@
> +/** SPDX-License-Identifier: BSD-3-Clause
> + *
> + * This file contains the functions definitions to
> + * generate each supported action.
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + **/
> +
> +#ifndef _ACTION_GEN_
> +#define _ACTION_GEN_
> +
> +struct rte_flow_action_mark mark_action;
> +struct rte_flow_action_queue queue_action;
> +struct rte_flow_action_jump jump_action;
> +struct rte_flow_action_rss *rss_action;
> +struct rte_flow_action_set_meta meta_action;
> +struct rte_flow_action_set_tag tag_action;
> +struct rte_flow_action_port_id port_id;
> +
> +/* Storage for struct rte_flow_action_rss including external data. */
> +struct action_rss_data {
> +	struct rte_flow_action_rss conf;
> +	uint8_t key[64];
> +	uint16_t queue[128];
> +} action_rss_data;
> +
> +void
> +gen_mark(void);
> +
> +void
> +gen_queue(uint16_t queue);
> +
> +void
> +gen_jump(uint16_t next_table);
> +
> +void
> +gen_rss(uint16_t *queues, uint16_t queues_number);
> +
> +void
> +gen_set_meta(void);
> +
> +void
> +gen_set_tag(void);
> +
> +void
> +gen_port_id(void);
> +
> +#endif
> diff --git a/app/test-flow-perf/flow_gen.c b/app/test-flow-perf/flow_gen.c
> new file mode 100644
> index 0000000000..2d42deace9
> --- /dev/null
> +++ b/app/test-flow-perf/flow_gen.c
> @@ -0,0 +1,176 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * The file contains the implementations of the method to
> + * fill items, actions & attributes in their corresponding
> + * arrays, and then generate rte_flow rule.
> + *
> + * After the generation. The rule goes to validation then
> + * creation state and then return the results.
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#include <stdint.h>
> +
> +#include "flow_gen.h"
> +#include "items_gen.h"
> +#include "actions_gen.h"
> +#include "user_parameters.h"
> +
> +
> +static void
> +fill_attributes(struct rte_flow_attr *attr,
> +	uint8_t flow_attrs, uint16_t group)
> +{
> +	if (flow_attrs & INGRESS)
> +		attr->ingress = 1;
> +	if (flow_attrs & EGRESS)
> +		attr->egress = 1;
> +	if (flow_attrs & TRANSFER)
> +		attr->transfer = 1;
> +	attr->group = group;
> +}
> +
> +static void
> +fill_items(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint16_t flow_items, uint32_t outer_ip_src)
> +{
> +	uint8_t items_counter = 0;
> +
> +	if (flow_items & META_ITEM)
> +		add_meta_data(items, items_counter++);
> +	if (flow_items & TAG_ITEM)
> +		add_meta_tag(items, items_counter++);
> +	if (flow_items & ETH_ITEM)
> +		add_ether(items, items_counter++);
> +	if (flow_items & VLAN_ITEM)
> +		add_vlan(items, items_counter++);
> +	if (flow_items & IPV4_ITEM)
> +		add_ipv4(items, items_counter++, outer_ip_src);
> +	if (flow_items & IPV6_ITEM)
> +		add_ipv6(items, items_counter++, outer_ip_src);
> +	if (flow_items & TCP_ITEM)
> +		add_tcp(items, items_counter++);
> +	if (flow_items & UDP_ITEM)
> +		add_udp(items, items_counter++);
> +	if (flow_items & VXLAN_ITEM)
> +		add_vxlan(items, items_counter++);
> +	if (flow_items & VXLAN_GPE_ITEM)
> +		add_vxlan_gpe(items, items_counter++);
> +	if (flow_items & GRE_ITEM)
> +		add_gre(items, items_counter++);
> +	if (flow_items & GENEVE_ITEM)
> +		add_geneve(items, items_counter++);
> +	if (flow_items & GTP_ITEM)
> +		add_gtp(items, items_counter++);
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_END;
> +}
> +
> +static void
> +fill_actions(struct rte_flow_action actions[MAX_ACTIONS_NUM],
> +	uint16_t flow_actions, uint32_t counter, uint16_t next_table)
> +{
> +	uint8_t actions_counter = 0;
> +	uint16_t queues[RXQs];
> +	uint16_t hairpin_queues[HAIRPIN_QUEUES];
> +	uint16_t i;
> +	struct rte_flow_action_count count_action;
> +	uint8_t temp = counter & 0xff;
> +
> +	/* None-fate actions */
> +	if (flow_actions & MARK_ACTION) {
> +		if (!counter)
> +			gen_mark();
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_MARK;
> +		actions[actions_counter++].conf = &mark_action;
> +	}
> +	if (flow_actions & COUNT_ACTION) {
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_COUNT;
> +		actions[actions_counter++].conf = &count_action;
> +	}
> +	if (flow_actions & META_ACTION) {
> +		if (!counter)
> +			gen_set_meta();
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_SET_META;
> +		actions[actions_counter++].conf = &meta_action;
> +	}
> +	if (flow_actions & TAG_ACTION) {
> +		if (!counter)
> +			gen_set_tag();
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_SET_TAG;
> +		actions[actions_counter++].conf = &tag_action;
> +	}
> +
> +	/* Fate actions */
> +	if (flow_actions & QUEUE_ACTION) {
> +		gen_queue(counter % RXQs);
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_QUEUE;
> +		actions[actions_counter++].conf = &queue_action;
> +	}
> +	if (flow_actions & RSS_ACTION) {
> +		for (i = 0; i < RXQs; i++)
> +			queues[i] = (temp >> (i << 1)) & 0x3;
> +		gen_rss(queues, RXQs);
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_RSS;
> +		actions[actions_counter++].conf = rss_action;
> +	}
> +	if (flow_actions & JUMP_ACTION) {
> +		if (!counter)
> +			gen_jump(next_table);
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_JUMP;
> +		actions[actions_counter++].conf = &jump_action;
> +	}
> +	if (flow_actions & PORT_ID_ACTION) {
> +		if (!counter)
> +			gen_port_id();
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_PORT_ID;
> +		actions[actions_counter++].conf = &port_id;
> +	}
> +	if (flow_actions & DROP_ACTION)
> +		actions[actions_counter++].type = RTE_FLOW_ACTION_TYPE_DROP;
> +	if (flow_actions & HAIRPIN_QUEUE_ACTION) {
> +		gen_queue((counter % HAIRPIN_QUEUES) + RXQs);
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_QUEUE;
> +		actions[actions_counter++].conf = &queue_action;
> +	}
> +	if (flow_actions & HAIRPIN_RSS_ACTION) {
> +		for (i = 0; i < HAIRPIN_QUEUES; i++)
> +			hairpin_queues[i] = ((temp >> (i << 1)) & 0x3) + RXQs;
> +		gen_rss(hairpin_queues, RXQs);
> +		actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_RSS;
> +		actions[actions_counter++].conf = rss_action;
> +	}
> +
> +	actions[actions_counter].type = RTE_FLOW_ACTION_TYPE_END;
> +}
> +
> +struct rte_flow *
> +generate_flow(uint16_t port_id,
> +	uint16_t group,
> +	uint8_t flow_attrs,
> +	uint16_t flow_items,
> +	uint16_t flow_actions,
> +	uint16_t next_table,
> +	uint32_t outer_ip_src,
> +	struct rte_flow_error *error)
> +{
> +	struct rte_flow_attr attr;
> +	struct rte_flow_item items[MAX_ITEMS_NUM];
> +	struct rte_flow_action actions[MAX_ACTIONS_NUM];
> +	struct rte_flow *flow = NULL;
> +
> +	memset(items, 0, sizeof(items));
> +	memset(actions, 0, sizeof(actions));
> +	memset(&attr, 0, sizeof(struct rte_flow_attr));
> +
> +	fill_attributes(&attr, flow_attrs, group);
> +
> +	fill_actions(actions, flow_actions,
> +			outer_ip_src, next_table);
> +
> +	fill_items(items, flow_items, outer_ip_src);
> +
> +	flow = rte_flow_create(port_id, &attr, items, actions, error);
> +	return flow;
> +}
> diff --git a/app/test-flow-perf/flow_gen.h b/app/test-flow-perf/flow_gen.h
> new file mode 100644
> index 0000000000..99cb9e3791
> --- /dev/null
> +++ b/app/test-flow-perf/flow_gen.h
> @@ -0,0 +1,61 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * This file contains the items, actions and attributes
> + * definition. And the methods to prepare and fill items,
> + * actions and attributes to generate rte_flow rule.
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#ifndef _FLOW_GEN_
> +#define _FLOW_GEN_
> +
> +#include <stdint.h>
> +#include <rte_flow.h>
> +
> +#include "user_parameters.h"
> +
> +/* Items */
> +#define ETH_ITEM       0x0001
> +#define IPV4_ITEM      0x0002
> +#define IPV6_ITEM      0x0004
> +#define VLAN_ITEM      0x0008
> +#define TCP_ITEM       0x0010
> +#define UDP_ITEM       0x0020
> +#define VXLAN_ITEM     0x0040
> +#define VXLAN_GPE_ITEM 0x0080
> +#define GRE_ITEM       0x0100
> +#define GENEVE_ITEM    0x0200
> +#define GTP_ITEM       0x0400
> +#define META_ITEM      0x0800
> +#define TAG_ITEM       0x1000

May be it is better to use (1UL << RTE_FLOW_ITEM_TYPE_ETH) as
flags in bitmask? E.g. define macro:
#define FLOW_ITEM_MASK(_x) \
    (UINT64_C(1) << ETH_FLOW_ITEM_ ## _x)
and use as
    FLOW_ITEM_MASK(ETH)

> +
> +/* Actions */
> +#define QUEUE_ACTION   0x0001
> +#define MARK_ACTION    0x0002
> +#define JUMP_ACTION    0x0004
> +#define RSS_ACTION     0x0008
> +#define COUNT_ACTION   0x0010
> +#define META_ACTION    0x0020
> +#define TAG_ACTION     0x0040
> +#define DROP_ACTION    0x0080
> +#define PORT_ID_ACTION 0x0100
> +#define HAIRPIN_QUEUE_ACTION 0x0200
> +#define HAIRPIN_RSS_ACTION   0x0400

same as above

> +
> +/* Attributes */
> +#define INGRESS  0x0001
> +#define EGRESS   0x0002
> +#define TRANSFER 0x0004
> +
> +struct rte_flow *
> +generate_flow(uint16_t port_id,
> +	uint16_t group,
> +	uint8_t flow_attrs,
> +	uint16_t flow_items,
> +	uint16_t flow_actions,
> +	uint16_t next_table,
> +	uint32_t outer_ip_src,
> +	struct rte_flow_error *error);
> +
> +#endif
> diff --git a/app/test-flow-perf/items_gen.c b/app/test-flow-perf/items_gen.c
> new file mode 100644
> index 0000000000..fb9733d4e7
> --- /dev/null
> +++ b/app/test-flow-perf/items_gen.c
> @@ -0,0 +1,265 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * This file contain the implementations of the items
> + * related methods. Each Item have a method to prepare
> + * the item and add it into items array in given index.
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#include <stdint.h>
> +#include <rte_flow.h>
> +
> +#include "items_gen.h"
> +#include "user_parameters.h"
> +
> +static struct rte_flow_item_eth eth_spec;
> +static struct rte_flow_item_eth eth_mask;
> +static struct rte_flow_item_vlan vlan_spec;
> +static struct rte_flow_item_vlan vlan_mask;
> +static struct rte_flow_item_ipv4 ipv4_spec;
> +static struct rte_flow_item_ipv4 ipv4_mask;
> +static struct rte_flow_item_ipv6 ipv6_spec;
> +static struct rte_flow_item_ipv6 ipv6_mask;
> +static struct rte_flow_item_udp udp_spec;
> +static struct rte_flow_item_udp udp_mask;
> +static struct rte_flow_item_tcp tcp_spec;
> +static struct rte_flow_item_tcp tcp_mask;
> +static struct rte_flow_item_vxlan vxlan_spec;
> +static struct rte_flow_item_vxlan vxlan_mask;
> +static struct rte_flow_item_vxlan_gpe vxlan_gpe_spec;
> +static struct rte_flow_item_vxlan_gpe vxlan_gpe_mask;
> +static struct rte_flow_item_gre gre_spec;
> +static struct rte_flow_item_gre gre_mask;
> +static struct rte_flow_item_geneve geneve_spec;
> +static struct rte_flow_item_geneve geneve_mask;
> +static struct rte_flow_item_gtp gtp_spec;
> +static struct rte_flow_item_gtp gtp_mask;
> +static struct rte_flow_item_meta meta_spec;
> +static struct rte_flow_item_meta meta_mask;
> +static struct rte_flow_item_tag tag_spec;
> +static struct rte_flow_item_tag tag_mask;

Any global variable usually very bad if you have
a plan to use many CPU cores using threads.

> +
> +
> +void
> +add_ether(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	memset(&eth_spec, 0, sizeof(struct rte_flow_item_eth));
> +	memset(&eth_mask, 0, sizeof(struct rte_flow_item_eth));
> +	eth_spec.type = 0;
> +	eth_mask.type = 0;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_ETH;
> +	items[items_counter].spec = &eth_spec;
> +	items[items_counter].mask = &eth_mask;
> +}
> +
> +void
> +add_vlan(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint16_t vlan_value = VLAN_VALUE;
> +	memset(&vlan_spec, 0, sizeof(struct rte_flow_item_vlan));
> +	memset(&vlan_mask, 0, sizeof(struct rte_flow_item_vlan));
> +
> +	vlan_spec.tci = RTE_BE16(vlan_value);
> +	vlan_mask.tci = RTE_BE16(0xffff);
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_VLAN;
> +	items[items_counter].spec = &vlan_spec;
> +	items[items_counter].mask = &vlan_mask;
> +}
> +
> +void
> +add_ipv4(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter, uint32_t src_ipv4)
> +{
> +	memset(&ipv4_spec, 0, sizeof(struct rte_flow_item_ipv4));
> +	memset(&ipv4_mask, 0, sizeof(struct rte_flow_item_ipv4));
> +
> +	ipv4_spec.hdr.src_addr = src_ipv4;
> +	ipv4_mask.hdr.src_addr = 0xffffffff;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_IPV4;
> +	items[items_counter].spec = &ipv4_spec;
> +	items[items_counter].mask = &ipv4_mask;
> +}
> +
> +
> +void
> +add_ipv6(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter, int src_ipv6)
> +{
> +	memset(&ipv6_spec, 0, sizeof(struct rte_flow_item_ipv6));
> +	memset(&ipv6_mask, 0, sizeof(struct rte_flow_item_ipv6));
> +
> +	/** Set ipv6 src **/
> +	memset(&ipv6_spec.hdr.src_addr, src_ipv6,
> +					sizeof(ipv6_spec.hdr.src_addr) / 2);
> +
> +	/** Full mask **/
> +	memset(&ipv6_mask.hdr.src_addr, 1,
> +					sizeof(ipv6_spec.hdr.src_addr));
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_IPV6;
> +	items[items_counter].spec = &ipv6_spec;
> +	items[items_counter].mask = &ipv6_mask;
> +}
> +
> +void
> +add_tcp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	memset(&tcp_spec, 0, sizeof(struct rte_flow_item_tcp));
> +	memset(&tcp_mask, 0, sizeof(struct rte_flow_item_tcp));
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_TCP;
> +	items[items_counter].spec = &tcp_spec;
> +	items[items_counter].mask = &tcp_mask;
> +}
> +
> +void
> +add_udp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	memset(&udp_spec, 0, sizeof(struct rte_flow_item_udp));
> +	memset(&udp_mask, 0, sizeof(struct rte_flow_item_udp));
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_UDP;
> +	items[items_counter].spec = &udp_spec;
> +	items[items_counter].mask = &udp_mask;
> +}
> +
> +void
> +add_vxlan(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t vni_value = VNI_VALUE;
> +	uint8_t i;
> +	memset(&vxlan_spec, 0, sizeof(struct rte_flow_item_vxlan));
> +	memset(&vxlan_mask, 0, sizeof(struct rte_flow_item_vxlan));
> +
> +	/* Set standard vxlan vni */
> +	for (i = 0; i < 3; i++) {
> +		vxlan_spec.vni[2 - i] = vni_value >> (i * 8);
> +		vxlan_mask.vni[2 - i] = 0xff;
> +	}
> +
> +	/* Standard vxlan flags **/
> +	vxlan_spec.flags = 0x8;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_VXLAN;
> +	items[items_counter].spec = &vxlan_spec;
> +	items[items_counter].mask = &vxlan_mask;
> +}
> +
> +void
> +add_vxlan_gpe(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t vni_value = VNI_VALUE;
> +	uint8_t i;
> +	memset(&vxlan_gpe_spec, 0, sizeof(struct rte_flow_item_vxlan_gpe));
> +	memset(&vxlan_gpe_mask, 0, sizeof(struct rte_flow_item_vxlan_gpe));
> +
> +	/* Set vxlan-gpe vni */
> +	for (i = 0; i < 3; i++) {
> +		vxlan_gpe_spec.vni[2 - i] = vni_value >> (i * 8);
> +		vxlan_gpe_mask.vni[2 - i] = 0xff;
> +	}
> +
> +	/* vxlan-gpe flags */
> +	vxlan_gpe_spec.flags = 0x0c;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_VXLAN_GPE;
> +	items[items_counter].spec = &vxlan_gpe_spec;
> +	items[items_counter].mask = &vxlan_gpe_mask;
> +}
> +
> +void
> +add_gre(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint16_t proto = GRE_PROTO;
> +	memset(&gre_spec, 0, sizeof(struct rte_flow_item_gre));
> +	memset(&gre_mask, 0, sizeof(struct rte_flow_item_gre));
> +
> +	gre_spec.protocol = RTE_BE16(proto);
> +	gre_mask.protocol = 0xffff;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_GRE;
> +	items[items_counter].spec = &gre_spec;
> +	items[items_counter].mask = &gre_mask;
> +}
> +
> +void
> +add_geneve(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t vni_value = VNI_VALUE;
> +	uint8_t i;
> +	memset(&geneve_spec, 0, sizeof(struct rte_flow_item_geneve));
> +	memset(&geneve_mask, 0, sizeof(struct rte_flow_item_geneve));
> +
> +	for (i = 0; i < 3; i++) {
> +		geneve_spec.vni[2 - i] = vni_value >> (i * 8);
> +		geneve_mask.vni[2 - i] = 0xff;
> +	}
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_GENEVE;
> +	items[items_counter].spec = &geneve_spec;
> +	items[items_counter].mask = &geneve_mask;
> +}
> +
> +void
> +add_gtp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t teid_value = TEID_VALUE;
> +	memset(&gtp_spec, 0, sizeof(struct rte_flow_item_gtp));
> +	memset(&gtp_mask, 0, sizeof(struct rte_flow_item_gtp));
> +
> +	gtp_spec.teid = RTE_BE32(teid_value);
> +	gtp_mask.teid = RTE_BE32(0xffffffff);
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_GTP;
> +	items[items_counter].spec = &gtp_spec;
> +	items[items_counter].mask = &gtp_mask;
> +}
> +
> +void
> +add_meta_data(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t data = META_DATA;
> +	memset(&meta_spec, 0, sizeof(struct rte_flow_item_meta));
> +	memset(&meta_mask, 0, sizeof(struct rte_flow_item_meta));
> +
> +	meta_spec.data = RTE_BE32(data);
> +	meta_mask.data = RTE_BE32(0xffffffff);
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_META;
> +	items[items_counter].spec = &meta_spec;
> +	items[items_counter].mask = &meta_mask;
> +}
> +
> +
> +void
> +add_meta_tag(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter)
> +{
> +	uint32_t data = META_DATA;
> +	uint8_t index = TAG_INDEX;
> +	memset(&tag_spec, 0, sizeof(struct rte_flow_item_tag));
> +	memset(&tag_mask, 0, sizeof(struct rte_flow_item_tag));
> +
> +	tag_spec.data = RTE_BE32(data);
> +	tag_mask.data = RTE_BE32(0xffffffff);
> +	tag_spec.index = index;
> +	tag_mask.index = 0xff;
> +
> +	items[items_counter].type = RTE_FLOW_ITEM_TYPE_TAG;
> +	items[items_counter].spec = &tag_spec;
> +	items[items_counter].mask = &tag_mask;
> +}
> diff --git a/app/test-flow-perf/items_gen.h b/app/test-flow-perf/items_gen.h
> new file mode 100644
> index 0000000000..0b01385951
> --- /dev/null
> +++ b/app/test-flow-perf/items_gen.h
> @@ -0,0 +1,68 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + *
> + * This file contains the items related methods
> + *
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#ifndef _ITEMS_GEN_
> +#define _ITEMS_GEN_
> +
> +#include <stdint.h>
> +#include <rte_flow.h>
> +
> +#include "user_parameters.h"
> +
> +void
> +add_ether(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_vlan(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_ipv4(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter, uint32_t src_ipv4);
> +
> +void
> +add_ipv6(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter, int src_ipv6);
> +
> +void
> +add_udp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_tcp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_vxlan(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_vxlan_gpe(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_gre(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_geneve(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_gtp(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_meta_data(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +void
> +add_meta_tag(struct rte_flow_item items[MAX_ITEMS_NUM],
> +	uint8_t items_counter);
> +
> +#endif
> diff --git a/app/test-flow-perf/main.c b/app/test-flow-perf/main.c
> index 156b9ef553..115af4f302 100644
> --- a/app/test-flow-perf/main.c
> +++ b/app/test-flow-perf/main.c
> @@ -49,29 +49,119 @@
>  #include <rte_cycles.h>
>  #include <rte_memory.h>
>  
> +#include "flow_gen.h"
>  #include "user_parameters.h"
>  
> -static uint32_t nb_lcores;
> +#define MAX_ITERATIONS 100
> +
> +struct rte_flow *flow;
> +static uint8_t flow_group;
> +
> +static uint16_t flow_items;
> +static uint16_t flow_actions;
> +static uint8_t flow_attrs;
> +static volatile bool force_quit;
> +static volatile bool dump_iterations;
>  static struct rte_mempool *mbuf_mp;
> +static uint32_t nb_lcores;
> +static uint32_t flows_count;
> +static uint32_t iterations_number;

Global static variables again.

>  static void usage(char *progname)
>  {
>  	printf("\nusage: %s", progname);
> +	printf("\nControl configurations:\n");
> +	printf("  --flows-count=N: to set the number of needed"
> +		" flows to insert, default is 4,000,000\n");
> +	printf("  --dump-iterations: To print rates for each"
> +		" iteration\n");
> +
> +	printf("To set flow attributes:\n");
> +	printf("  --ingress: set ingress attribute in flows\n");
> +	printf("  --egress: set egress attribute in flows\n");
> +	printf("  --transfer: set transfer attribute in flows\n");
> +	printf("  --group=N: set group for all flows,"
> +		" default is 0\n");
> +
> +	printf("To set flow items:\n");
> +	printf("  --ether: add ether layer in flow items\n");
> +	printf("  --vlan: add vlan layer in flow items\n");
> +	printf("  --ipv4: add ipv4 layer in flow items\n");
> +	printf("  --ipv6: add ipv6 layer in flow items\n");
> +	printf("  --tcp: add tcp layer in flow items\n");
> +	printf("  --udp: add udp layer in flow items\n");
> +	printf("  --vxlan: add vxlan layer in flow items\n");
> +	printf("  --vxlan-gpe: add vxlan-gpe layer in flow items\n");
> +	printf("  --gre: add gre layer in flow items\n");
> +	printf("  --geneve: add geneve layer in flow items\n");
> +	printf("  --gtp: add gtp layer in flow items\n");
> +	printf("  --meta: add meta layer in flow items\n");
> +	printf("  --tag: add tag layer in flow items\n");
> +
> +	printf("To set flow actions:\n");
> +	printf("  --port-id: add port-id action in flow actions\n");
> +	printf("  --rss: add rss action in flow actions\n");
> +	printf("  --queue: add queue action in flow actions\n");
> +	printf("  --jump: add jump action in flow actions\n");
> +	printf("  --mark: add mark action in flow actions\n");
> +	printf("  --count: add count action in flow actions\n");
> +	printf("  --set-meta: add set meta action in flow actions\n");
> +	printf("  --set-tag: add set tag action in flow actions\n");
> +	printf("  --drop: add drop action in flow actions\n");
> +	printf("  --hairpin-queue: add hairpin-queue action in flow actions\n");
> +	printf("  --hairpin-rss: add hairping-rss action in flow actions\n");
>  }
>  
>  static void
>  args_parse(int argc, char **argv)
>  {
>  	char **argvopt;
> -	int opt;
> +	int n, opt;
>  	int opt_idx;
>  	static struct option lgopts[] = {
>  		/* Control */
>  		{ "help",                       0, 0, 0 },
> +		{ "flows-count",                1, 0, 0 },
> +		{ "dump-iterations",            0, 0, 0 },
> +		/* Attributes */
> +		{ "ingress",                    0, 0, 0 },
> +		{ "egress",                     0, 0, 0 },
> +		{ "transfer",                   0, 0, 0 },
> +		{ "group",                      1, 0, 0 },
> +		/* Items */
> +		{ "ether",                      0, 0, 0 },
> +		{ "vlan",                       0, 0, 0 },
> +		{ "ipv4",                       0, 0, 0 },
> +		{ "ipv6",                       0, 0, 0 },
> +		{ "tcp",                        0, 0, 0 },
> +		{ "udp",                        0, 0, 0 },
> +		{ "vxlan",                      0, 0, 0 },
> +		{ "vxlan-gpe",                  0, 0, 0 },
> +		{ "gre",                        0, 0, 0 },
> +		{ "geneve",                     0, 0, 0 },
> +		{ "gtp",                        0, 0, 0 },
> +		{ "meta",                       0, 0, 0 },
> +		{ "tag",                        0, 0, 0 },
> +		/* Actions */
> +		{ "port-id",                    0, 0, 0 },
> +		{ "rss",                        0, 0, 0 },
> +		{ "queue",                      0, 0, 0 },
> +		{ "jump",                       0, 0, 0 },
> +		{ "mark",                       0, 0, 0 },
> +		{ "count",                      0, 0, 0 },
> +		{ "set-meta",                   0, 0, 0 },
> +		{ "set-tag",                    0, 0, 0 },
> +		{ "drop",                       0, 0, 0 },
> +		{ "hairpin-queue",              0, 0, 0 },
> +		{ "hairpin-rss",                0, 0, 0 },
>  	};
>  
> +	flow_items = 0;
> +	flow_actions = 0;
> +	flow_attrs = 0;
>  	argvopt = argv;
>  
> +	printf(":: Flow -> ");
>  	while ((opt = getopt_long(argc, argvopt, "",
>  				lgopts, &opt_idx)) != EOF) {
>  		switch (opt) {
> @@ -80,6 +170,140 @@ args_parse(int argc, char **argv)
>  				usage(argv[0]);
>  				rte_exit(EXIT_SUCCESS, "Displayed help\n");
>  			}
> +			/* Attributes */
> +			if (!strcmp(lgopts[opt_idx].name, "ingress")) {
> +				flow_attrs |= INGRESS;
> +				printf("ingress ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "egress")) {
> +				flow_attrs |= EGRESS;
> +				printf("egress ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "transfer")) {
> +				flow_attrs |= TRANSFER;
> +				printf("transfer ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "group")) {
> +				n = atoi(optarg);
> +				if (n >= 0)
> +					flow_group = n;
> +				else
> +					rte_exit(EXIT_SUCCESS,
> +						"flow group should be >= 0");
> +				printf("group %d ", flow_group);
> +			}
> +			/* Items */
> +			if (!strcmp(lgopts[opt_idx].name, "ether")) {
> +				flow_items |= ETH_ITEM;
> +				printf("ether / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "ipv4")) {
> +				flow_items |= IPV4_ITEM;
> +				printf("ipv4 / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "vlan")) {
> +				flow_items |= VLAN_ITEM;
> +				printf("vlan / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "ipv6")) {
> +				flow_items |= IPV6_ITEM;
> +				printf("ipv6 / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "tcp")) {
> +				flow_items |= TCP_ITEM;
> +				printf("tcp / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "udp")) {
> +				flow_items |= UDP_ITEM;
> +				printf("udp / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "vxlan")) {
> +				flow_items |= VXLAN_ITEM;
> +				printf("vxlan / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "vxlan-gpe")) {
> +				flow_items |= VXLAN_GPE_ITEM;
> +				printf("vxlan-gpe / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "gre")) {
> +				flow_items |= GRE_ITEM;
> +				printf("gre / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "geneve")) {
> +				flow_items |= GENEVE_ITEM;
> +				printf("geneve / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "gtp")) {
> +				flow_items |= GTP_ITEM;
> +				printf("gtp / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "meta")) {
> +				flow_items |= META_ITEM;
> +				printf("meta / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "tag")) {
> +				flow_items |= TAG_ITEM;
> +				printf("tag / ");
> +			}

I think it is doable using helper structure statically
filled in, array of it and a loop.

> +			/* Actions */
> +			if (!strcmp(lgopts[opt_idx].name, "port-id")) {
> +				flow_actions |= PORT_ID_ACTION;
> +				printf("port-id / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "rss")) {
> +				flow_actions |= RSS_ACTION;
> +				printf("rss / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "hairpin-rss")) {
> +				flow_actions |= HAIRPIN_RSS_ACTION;
> +				printf("hairpin-rss / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "queue")) {
> +				flow_actions |= QUEUE_ACTION;
> +				printf("queue / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "hairpin-queue")) {
> +				flow_actions |= HAIRPIN_QUEUE_ACTION;
> +				printf("hairpin-queue / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "jump")) {
> +				flow_actions |= JUMP_ACTION;
> +				printf("jump / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "mark")) {
> +				flow_actions |= MARK_ACTION;
> +				printf("mark / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "count")) {
> +				flow_actions |= COUNT_ACTION;
> +				printf("count / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "set-meta")) {
> +				flow_actions |= META_ACTION;
> +				printf("set-meta / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "set-tag")) {
> +				flow_actions |= TAG_ACTION;
> +				printf("set-tag / ");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "drop")) {
> +				flow_actions |= DROP_ACTION;
> +				printf("drop / ");
> +			}

Same.

> +			/* Control */
> +			if (!strcmp(lgopts[opt_idx].name, "flows-count")) {
> +				n = atoi(optarg);
> +				if (n > (int) iterations_number)
> +					flows_count = n;
> +				else {
> +					printf("\n\nflows_count should be > %d",
> +						iterations_number);
> +					rte_exit(EXIT_SUCCESS, " ");
> +				}
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "dump-iterations"))
> +				dump_iterations = true;
>  			break;
>  		default:
>  			usage(argv[0]);
> @@ -88,6 +312,128 @@ args_parse(int argc, char **argv)
>  			break;
>  		}
>  	}
> +	printf("end_flow\n");
> +}
> +
> +static void
> +print_flow_error(struct rte_flow_error error)
> +{
> +	printf("Flow can't be created %d message: %s\n",
> +		error.type,
> +		error.message ? error.message : "(no stated reason)");
> +}
> +
> +static inline void
> +flows_handler(void)
> +{
> +	struct rte_flow_error error;
> +	clock_t start_iter, end_iter;
> +	double cpu_time_used;
> +	double flows_rate;
> +	double cpu_time_per_iter[MAX_ITERATIONS];
> +	double delta;
> +	uint16_t nr_ports;
> +	uint32_t i;
> +	int port_id;
> +	int iter_id;
> +	uint32_t eagain_counter = 0;
> +
> +	nr_ports = rte_eth_dev_count_avail();
> +
> +	for (i = 0; i < MAX_ITERATIONS; i++)
> +		cpu_time_per_iter[i] = -1;
> +
> +	if (iterations_number > flows_count)
> +		iterations_number = flows_count;
> +
> +	printf(":: Flows Count per port: %d\n", flows_count);
> +
> +	for (port_id = 0; port_id < nr_ports; port_id++) {
> +		cpu_time_used = 0;
> +		if (flow_group > 0) {
> +			/*
> +			 * Create global rule to jumo into flow_group
> +			 * This way the app will avoid the default rules
> +			 *
> +			 * Golbal rule:
> +			 * group 0 eth / end actions jump group <flow_group>
> +			 *
> +			 */
> +			flow = generate_flow(port_id, 0, flow_attrs, ETH_ITEM,
> +				JUMP_ACTION, flow_group, 0, &error);
> +
> +			if (!flow) {
> +				print_flow_error(error);
> +				rte_exit(EXIT_FAILURE, "error in creating flow");
> +			}
> +		}
> +
> +		/* Insertion Rate */
> +		printf("Flows insertion on port = %d\n", port_id);
> +		start_iter = clock();
> +		for (i = 0; i < flows_count; i++) {
> +			do {
> +				rte_errno = 0;
> +				flow = generate_flow(port_id, flow_group,
> +					flow_attrs, flow_items, flow_actions,
> +					JUMP_ACTION_TABLE, i,  &error);
> +				if (!flow)
> +					eagain_counter++;
> +			} while (rte_errno == EAGAIN);
> +
> +			if (force_quit)
> +				i = flows_count;
> +
> +			if (!flow) {
> +				print_flow_error(error);
> +				rte_exit(EXIT_FAILURE, "error in creating flow");
> +			}
> +
> +			if (i && !((i + 1) % iterations_number)) {
> +				/* Save the insertion rate of each iter */
> +				end_iter = clock();
> +				delta = (double) (end_iter - start_iter);
> +				iter_id = ((i + 1) / iterations_number) - 1;
> +				cpu_time_per_iter[iter_id] =
> +					delta / CLOCKS_PER_SEC;
> +				cpu_time_used += cpu_time_per_iter[iter_id];
> +				start_iter = clock();
> +			}
> +		}
> +
> +		/* Iteration rate per iteration */
> +		if (dump_iterations)
> +			for (i = 0; i < MAX_ITERATIONS; i++) {
> +				if (cpu_time_per_iter[i] == -1)
> +					continue;
> +				delta = (double)(iterations_number /
> +					cpu_time_per_iter[i]);
> +				flows_rate = delta / 1000;
> +				printf(":: Iteration #%d: %d flows "
> +					"in %f sec[ Rate = %f K/Sec ]\n",
> +					i, iterations_number,
> +					cpu_time_per_iter[i], flows_rate);
> +			}
> +
> +		/* Insertion rate for all flows */
> +		flows_rate = ((double) (flows_count / cpu_time_used) / 1000);
> +		printf("\n:: Total flow insertion rate -> %f K/Sec\n",
> +						flows_rate);
> +		printf(":: The time for creating %d in flows %f seconds\n",
> +						flows_count, cpu_time_used);
> +		printf(":: EAGAIN counter = %d\n", eagain_counter);
> +	}
> +}
> +
> +static void
> +signal_handler(int signum)
> +{
> +	if (signum == SIGINT || signum == SIGTERM) {
> +		printf("\n\nSignal %d received, preparing to exit...\n",
> +					signum);
> +		printf("Error: Stats are wrong due to sudden signal!\n\n");
> +		force_quit = true;
> +	}
>  }
>  
>  static void
> @@ -96,6 +442,8 @@ init_port(void)
>  	int ret;
>  	uint16_t i, j;
>  	uint16_t port_id;
> +	uint16_t nr_queues;
> +	bool hairpin_flag = false;
>  	uint16_t nr_ports = rte_eth_dev_count_avail();
>  	struct rte_eth_hairpin_conf hairpin_conf = {
>  			.peer_count = 1,
> @@ -115,6 +463,13 @@ init_port(void)
>  	struct rte_eth_rxconf rxq_conf;
>  	struct rte_eth_dev_info dev_info;
>  
> +	nr_queues = RXQs;
> +	if (flow_actions & HAIRPIN_QUEUE_ACTION ||
> +		flow_actions & HAIRPIN_RSS_ACTION) {
> +		nr_queues = RXQs + HAIRPIN_QUEUES;
> +		hairpin_flag = true;
> +	}
> +
>  	if (nr_ports == 0)
>  		rte_exit(EXIT_FAILURE, "Error: no port detected\n");
>  	mbuf_mp = rte_pktmbuf_pool_create("mbuf_pool",
> @@ -134,8 +489,8 @@ init_port(void)
>  
>  		port_conf.txmode.offloads &= dev_info.tx_offload_capa;
>  		printf(":: initializing port: %d\n", port_id);
> -		ret = rte_eth_dev_configure(port_id, RXQs + HAIRPIN_QUEUES,
> -				TXQs + HAIRPIN_QUEUES, &port_conf);
> +		ret = rte_eth_dev_configure(port_id, nr_queues,
> +				nr_queues, &port_conf);
>  		if (ret < 0)
>  			rte_exit(EXIT_FAILURE,
>  					":: cannot configure device: err=%d, port=%u\n",
> @@ -173,26 +528,30 @@ init_port(void)
>  					":: promiscuous mode enable failed: err=%s, port=%u\n",
>  					rte_strerror(-ret), port_id);
>  
> -		for (i = RXQs, j = 0; i < RXQs + HAIRPIN_QUEUES; i++, j++) {
> -			hairpin_conf.peers[0].port = port_id;
> -			hairpin_conf.peers[0].queue = j + TXQs;
> -			ret = rte_eth_rx_hairpin_queue_setup(port_id, i,
> -							NR_RXD, &hairpin_conf);
> -			if (ret != 0)
> -				rte_exit(EXIT_FAILURE,
> -					":: Hairpin rx queue setup failed: err=%d, port=%u\n",
> -					ret, port_id);
> -		}
> +		if (hairpin_flag) {
> +			for (i = RXQs, j = 0;
> +					i < RXQs + HAIRPIN_QUEUES; i++, j++) {
> +				hairpin_conf.peers[0].port = port_id;
> +				hairpin_conf.peers[0].queue = j + TXQs;
> +				ret = rte_eth_rx_hairpin_queue_setup(port_id, i,
> +					NR_RXD, &hairpin_conf);
> +				if (ret != 0)
> +					rte_exit(EXIT_FAILURE,
> +						":: Hairpin rx queue setup failed: err=%d, port=%u\n",
> +						ret, port_id);
> +			}

I think it means that the code should be in a helper
function from the very beginning.


> -		for (i = TXQs, j = 0; i < TXQs + HAIRPIN_QUEUES; i++, j++) {
> -			hairpin_conf.peers[0].port = port_id;
> -			hairpin_conf.peers[0].queue = j + RXQs;
> -			ret = rte_eth_tx_hairpin_queue_setup(port_id, i,
> -							NR_TXD, &hairpin_conf);
> -			if (ret != 0)
> -				rte_exit(EXIT_FAILURE,
> -					":: Hairpin tx queue setup failed: err=%d, port=%u\n",
> -					ret, port_id);
> +			for (i = TXQs, j = 0;
> +					i < TXQs + HAIRPIN_QUEUES; i++, j++) {
> +				hairpin_conf.peers[0].port = port_id;
> +				hairpin_conf.peers[0].queue = j + RXQs;
> +				ret = rte_eth_tx_hairpin_queue_setup(port_id, i,
> +					NR_TXD, &hairpin_conf);
> +				if (ret != 0)
> +					rte_exit(EXIT_FAILURE,
> +						":: Hairpin tx queue setup failed: err=%d, port=%u\n",
> +						ret, port_id);
> +			}


I think it means that the code should be in a helper
function from the very beginning.

>  		}
>  
>  		ret = rte_eth_dev_start(port_id);
> @@ -219,6 +578,15 @@ main(int argc, char **argv)
>  	if (ret < 0)
>  		rte_exit(EXIT_FAILURE, "EAL init failed\n");
>  
> +	force_quit = false;
> +	dump_iterations = false;
> +	flows_count = 4000000;
> +	iterations_number = 100000;
> +	flow_group = 0;
> +
> +	signal(SIGINT, signal_handler);
> +	signal(SIGTERM, signal_handler);
> +
>  	argc -= ret;
>  	argv += ret;
>  
> @@ -232,6 +600,8 @@ main(int argc, char **argv)
>  	if (nb_lcores <= 1)
>  		rte_exit(EXIT_FAILURE, "This app needs at least two cores\n");
>  
> +	flows_handler();
> +
>  	RTE_LCORE_FOREACH_SLAVE(lcore_id)
>  
>  	if (rte_eal_wait_lcore(lcore_id) < 0)
> diff --git a/app/test-flow-perf/meson.build b/app/test-flow-perf/meson.build
> index ec9bb3b3aa..b3941f5c2d 100644
> --- a/app/test-flow-perf/meson.build
> +++ b/app/test-flow-perf/meson.build
> @@ -5,7 +5,15 @@
>  #
>  # To build this example as a standalone application with an already-installed
>  # DPDK instance, use 'make'
> +name = 'flow_perf'
> +allow_experimental_apis = true
> +cflags += '-Wno-deprecated-declarations'
> +cflags += '-Wunused-function'
>  
>  sources = files(
> +	'actions_gen.c',
> +	'flow_gen.c',
> +	'items_gen.c',
>  	'main.c',
>  )
> +deps += ['ethdev']
> diff --git a/app/test-flow-perf/user_parameters.h b/app/test-flow-perf/user_parameters.h
> index 56ec7f47b5..1d157430b6 100644
> --- a/app/test-flow-perf/user_parameters.h
> +++ b/app/test-flow-perf/user_parameters.h
> @@ -14,3 +14,18 @@
>  #define MBUF_CACHE_SIZE 512
>  #define NR_RXD  256
>  #define NR_TXD  256
> +
> +/** Items/Actions parameters **/
> +#define JUMP_ACTION_TABLE 2
> +#define VLAN_VALUE 1
> +#define VNI_VALUE 1
> +#define GRE_PROTO  0x6558
> +#define META_DATA 1
> +#define TAG_INDEX 0
> +#define PORT_ID_DST 1
> +#define MARK_ID 1
> +#define TEID_VALUE 1
> +
> +/** Flow items/acctions max size **/
> +#define MAX_ITEMS_NUM 20
> +#define MAX_ACTIONS_NUM 20
> diff --git a/doc/guides/tools/flow-perf.rst b/doc/guides/tools/flow-perf.rst
> index 30ce1b6cc0..62e038c430 100644
> --- a/doc/guides/tools/flow-perf.rst
> +++ b/doc/guides/tools/flow-perf.rst
> @@ -4,7 +4,19 @@
>  RTE Flow performance tool
>  =========================
>  
> -Application for rte_flow performance testing.
> +Application for rte_flow performance testing. The application provide the
> +ability to test insertion rate of specific rte_flow rule, by stressing it
> +to the NIC, and calculate the insertion rate.
> +
> +The application offers some options in the command line, to configure
> +which rule to apply.
> +
> +After that the application will start producing rules with same pattern
> +but increasing the outer IP source address by 1 each time, thus it will
> +give different flow each time, and all other items will have open masks.
> +
> +The current design have single core insertion rate. In the future we may
> +have a multi core insertion rate measurement support in the app.
>  
>  
>  Compiling the Application
> @@ -61,9 +73,179 @@ a ``--`` separator:
>  
>  .. code-block:: console
>  
> -	sudo ./test-flow-perf -n 4 -w 08:00.0,dv_flow_en=1 --
> +	sudo ./flow_perf -n 4 -w 08:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --queue --flows-count=1000000
>  
>  The command line options are:
>  
>  *	``--help``
>  	Display a help message and quit.
> +
> +*	``--flows-count=N``
> +	Set the number of needed flows to insert,
> +	where 1 <= N <= "number of flows".
> +	The default value is 4,000,000.
> +
> +*	``--dump-iterations``
> +	Print rates for each iteration of flows.
> +	Default iteration is 1,00,000.
> +
> +
> +Attributes:
> +
> +*	``--ingress``
> +	Set Ingress attribute to all flows attributes.
> +
> +*	``--egress``
> +	Set Egress attribute to all flows attributes.
> +
> +*	``--transfer``
> +	Set Transfer attribute to all flows attributes.
> +
> +*	``--group=N``
> +	Set group for all flows, where N >= 0.
> +	Default group is 0.
> +
> +Items:
> +
> +*	``--ether``
> +	Add Ether item to all flows items, This item have open mask.
> +
> +*	``--vlan``
> +	Add VLAN item to all flows items,
> +	This item have VLAN value defined in user_parameters.h
> +	under ``VNI_VALUE`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--ipv4``
> +	Add IPv4 item to all flows items,
> +	This item have incremental source IP, with full mask.
> +	Other fields are open mask.
> +
> +*	``--ipv6``
> +	Add IPv6 item to all flows item,
> +	This item have incremental source IP, with full mask.
> +	Other fields are open mask.
> +
> +*	``--tcp``
> +	Add TCP item to all flows items, This item have open mask.
> +
> +*	``--udp``
> +	Add UDP item to all flows items, This item have open mask.
> +
> +*	``--vxlan``
> +	Add VXLAN item to all flows items,
> +	This item have VNI value defined in user_parameters.h
> +	under ``VNI_VALUE`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--vxlan-gpe``
> +	Add VXLAN-GPE item to all flows items,
> +	This item have VNI value defined in user_parameters.h
> +	under ``VNI_VALUE`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--gre``
> +	Add GRE item to all flows items,
> +	This item have protocol value defined in user_parameters.h
> +	under ``GRE_PROTO`` with full mask, default protocol = 0x6558 "Ether"
> +	Other fields are open mask.
> +
> +*	``--geneve``
> +	Add GENEVE item to all flows items,
> +	This item have VNI value defined in user_parameters.h
> +	under ``VNI_VALUE`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--gtp``
> +	Add GTP item to all flows items,
> +	This item have TEID value defined in user_parameters.h
> +	under ``TEID_VALUE`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--meta``
> +	Add Meta item to all flows items,
> +	This item have data value defined in user_parameters.h
> +	under ``META_DATA`` with full mask, default value = 1.
> +	Other fields are open mask.
> +
> +*	``--tag``
> +	Add Tag item to all flows items,
> +	This item have data value defined in user_parameters.h
> +	under ``META_DATA`` with full mask, default value = 1.
> +
> +	Also it have tag value defined in user_parameters.h
> +	under ``TAG_INDEX`` with full mask, default value = 0.
> +	Other fields are open mask.
> +
> +
> +Actions:
> +
> +*	``--port-id``
> +	Add port redirection action to all flows actions.
> +	Port redirection destination is defined in user_parameters.h
> +	under PORT_ID_DST, default value = 1.
> +
> +*	``--rss``
> +	Add RSS action to all flows actions,
> +	The queues in RSS action will be all queues configured
> +	in the app.
> +
> +*	``--queue``
> +	Add queue action to all flows items,
> +	The queue will change in round robin state for each flow.
> +
> +	For example:
> +		The app running with 4 RX queues
> +		Flow #0: queue index 0
> +		Flow #1: queue index 1
> +		Flow #2: queue index 2
> +		Flow #3: queue index 3
> +		Flow #4: queue index 0
> +		...
> +
> +*	``--jump``
> +	Add jump action to all flows actions.
> +	Jump action destination is defined in user_parameters.h
> +	under ``JUMP_ACTION_TABLE``, default value = 2.
> +
> +*	``--mark``
> +	Add mark action to all flows actions.
> +	Mark action id is defined in user_parameters.h
> +	under ``MARK_ID``, default value = 1.
> +
> +*	``--count``
> +	Add count action to all flows actions.
> +
> +*	``--set-meta``
> +	Add set-meta action to all flows actions.
> +	Meta data is defined in user_parameters.h under ``META_DATA``
> +	with full mask, default value = 1.
> +
> +*	``--set-tag``
> +	Add set-tag action to all flows actions.
> +	Meta data is defined in user_parameters.h under ``META_DATA``
> +	with full mask, default value = 1.
> +
> +	Tag index is defined in user_parameters.h under ``TAG_INDEX``
> +	with full mask, default value = 0.
> +
> +*	``--drop``
> +	Add drop action to all flows actions.
> +
> +*	``--hairpin-queue``
> +	Add hairpin queue action to all flows actions.
> +	The queue will change in round robin state for each flow.
> +
> +	For example:
> +		The app running with 4 RX hairpin queues and 4 normal RX queues
> +		Flow #0: queue index 4
> +		Flow #1: queue index 5
> +		Flow #2: queue index 6
> +		Flow #3: queue index 7
> +		Flow #4: queue index 4
> +		...
> +
> +*	``--hairpin-rss``
> +	Add hairpin RSS action to all flows actions.
> +	The queues in RSS action will be all hairpin queues configured
> +	in the app.
>