[dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Anoob Joseph]

This series introduces event-mode additions to ipsec-secgw. This effort
is based on the proposed changes for l2fwd-event and the additions in
l3fwd for event support.

With this series, ipsec-secgw would be able to run in eventmode. The
worker thread (executing loop) would be receiving events and would be
submitting it back to the eventdev after the processing. This way,
multicore scaling and h/w assisted scheduling is achieved by making use
of the eventdev capabilities.

Since the underlying event device would be having varying capabilities,
the worker thread could be drafted differently to maximize performance.
This series introduces usage of multiple worker threads, among which the
one to be used will be determined by the operating conditions and the
underlying device capabilities.

For example, if an event device - eth device pair has Tx internal port,
then application can do tx_adapter_enqueue() instead of regular
event_enqueue(). So a thread making an assumption that the device pair
has internal port will not be the right solution for another pair. The
infrastructure added with these patches aims to help application to have
multiple worker threads, there by extracting maximum performance from
every device without affecting existing paths/use cases.

The eventmode configuration is predefined. All packets reaching one eth
port will hit one event queue. All event queues will be mapped to all
event ports. So all cores will be able to receive traffic from all ports.
When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event device
will ensure the ordering. Ordering would be lost when tried in PARALLEL.

Following command line options are introduced,

--transfer-mode: to choose between poll mode & event mode
--schedule-type: to specify the scheduling type
                 (RTE_SCHED_TYPE_ORDERED/
                  RTE_SCHED_TYPE_ATOMIC/
                  RTE_SCHED_TYPE_PARALLEL)
--process-dir: outbound/inbound
--process-mode: app mode /driver mode

The two s/w config options added to ipsec-secgw can be used in
benchmarking h/w performance,

1. process-dir : states whether the direction is outbound/inbound.
This option aims to avoid an unnecessary check of determining whether
inbound/outbound processing need to be done on the packet. For each
option a different light weight worker thread would be executed.

2. process-mode: states whether the application has to run in driver
                 mode or app mode.

Driver-mode: This mode will have bare minimum changes in the application
             to support ipsec. There woudn't be any lookup etc done in
             the application. And for inline-protocol use case, the
             thread would resemble l2fwd as the ipsec processing would be
             done entirely in the h/w. This mode can be used to benchmark
             the raw performance of the h/w. All the application side
             steps (like lookup) can be redone based on the requirement
             of the end user. Hence the need for a mode which would
             report the raw performance.

App-mode: This mode will have all the features currently implemented with
          ipsec-secgw (non librte_ipsec mode). All the lookups etc
          would follow the existing methods and would report numbers
          that can be compared against regular ipsec-secgw benchmark
          numbers.

Example commands to execute ipsec-secgw in various modes on OCTEONTX2 platform,

#Inbound driver mode
./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7 – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f dpdk_internal/100g_4.3.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 1

#Inbound app mode
./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x3f – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f dpdk_internal/100g_4.3.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 1

#Outbound driver mode
./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1f – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 0

#Outbound app mode
./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7f – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 0

This series is targeted for next release (20.02). This series doesn't introduce
any library change. And the decision to add eventmode additions in ipsec-secgw
was approved by the Tech Board.

Following are missing in the RFC. Will add it when sending patches.
1. Documentation.
2. More cleanup is needed. There are options that are added so that future
   expansion is not hindered. Need inputs from the community if there is use
   case for them.

Following are planned features,
1. Add burst mode workers.
2. Add non tx internal port worker.
3. Verify support for Rx core (the support is added but lack of h/w to verify).
4. Add lookaside protocol support.

Following are features that Marvell won't be attempting.
1. Inline crypto support.
2. Lookaside crypto support.

For the features that Marvell won't be attempting, new workers can be
introduced by the respective stake holders.

Anoob Joseph (13):
  examples/ipsec-secgw: add framework for eventmode helper
  examples/ipsec-secgw: add eventdev port-lcore link
  examples/ipsec-secgw: add Rx adapter support
  examples/ipsec-secgw: add Tx adapter support
  examples/ipsec-secgw: add routines to display config
  examples/ipsec-secgw: add routines to launch workers
  examples/ipsec-secgw: add support for internal ports
  examples/ipsec-secgw: add eventmode to ipsec-secgw
  examples/ipsec-secgw: add app inbound worker
  examples/ipsec-secgw: add app processing code
  examples/ipsec-secgw: add driver outbound worker
  examples/ipsec-secgw: add app outbound worker
  examples/ipsec-secgw: add cmd line option for bufs

 examples/ipsec-secgw/Makefile       |    2 +
 examples/ipsec-secgw/event_helper.c | 1757 +++++++++++++++++++++++++++++++++++
 examples/ipsec-secgw/event_helper.h |  334 +++++++
 examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
 examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
 examples/ipsec-secgw/ipsec.c        |    4 +
 examples/ipsec-secgw/ipsec.h        |   30 +-
 examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
 examples/ipsec-secgw/ipsec_worker.h |   39 +
 examples/ipsec-secgw/meson.build    |    4 +-
 examples/ipsec-secgw/sa.c           |   11 -
 11 files changed, 3360 insertions(+), 104 deletions(-)
 create mode 100644 examples/ipsec-secgw/event_helper.c
 create mode 100644 examples/ipsec-secgw/event_helper.h
 create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
 create mode 100644 examples/ipsec-secgw/ipsec_worker.c
 create mode 100644 examples/ipsec-secgw/ipsec_worker.h

-- 
2.7.4

[dpdk-dev] [RFC PATCH 01/13] examples/ipsec-secgw: add framework for eventmode helper

[Anoob Joseph]

Add framework for eventmode helper. Event mode would involve
initialization of multiple devices, like eventdev, ethdev etc.
Add routines to initialize and uninitialize event devices.
Generate a default config for event devices if it is not
specified in the configuration. The routine will iterate
over available event devices and their properties and will
set the config accordingly.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/Makefile       |   1 +
 examples/ipsec-secgw/event_helper.c | 322 ++++++++++++++++++++++++++++++++++++
 examples/ipsec-secgw/event_helper.h | 115 +++++++++++++
 examples/ipsec-secgw/meson.build    |   4 +-
 4 files changed, 440 insertions(+), 2 deletions(-)
 create mode 100644 examples/ipsec-secgw/event_helper.c
 create mode 100644 examples/ipsec-secgw/event_helper.h

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 851123b..2a76900 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -15,6 +15,7 @@ SRCS-y += sa.c
 SRCS-y += rt.c
 SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
+SRCS-y += event_helper.c
 
 CFLAGS += -gdwarf-2
 
diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
new file mode 100644
index 0000000..af7c7e2
--- /dev/null
+++ b/examples/ipsec-secgw/event_helper.c
@@ -0,0 +1,322 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2019 Marvell International Ltd.
+ */
+#include <rte_eventdev.h>
+#include <rte_ethdev.h>
+
+#include "event_helper.h"
+
+static int
+eh_validate_user_params(struct eventmode_conf *em_conf)
+{
+	/* TODO */
+	/* Check sanity of the conf requested by user */
+
+	RTE_SET_USED(em_conf);
+
+	return 0;
+}
+
+static int
+eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
+{
+	int i, ret;
+	int nb_eventdev;
+	struct eventdev_params *eventdev_config;
+	struct rte_event_dev_info dev_info;
+
+	/* Get the number of event devices */
+	nb_eventdev = rte_event_dev_count();
+
+	if (nb_eventdev == 0) {
+		EH_LOG_ERR("No event devices detected");
+		return -EINVAL;
+	}
+
+	for (i = 0; i < nb_eventdev; i++) {
+
+		/* Get the event dev conf */
+		eventdev_config = &(em_conf->eventdev_config[i]);
+
+		/* Read event device info */
+		ret = rte_event_dev_info_get(i, &dev_info);
+
+		if (ret < 0) {
+			EH_LOG_ERR("Failed reading event device info (err:%d)",
+				ret);
+			return ret;
+		}
+
+		/* Check if enough ports are available */
+		if (dev_info.max_event_ports < 2) {
+			EH_LOG_ERR("Not enough event ports available");
+			return -EINVAL;
+		}
+
+		/* Save number of queues & ports available */
+		eventdev_config->eventdev_id = i;
+		eventdev_config->nb_eventqueue = dev_info.max_event_queues;
+		eventdev_config->nb_eventport = dev_info.max_event_ports;
+		eventdev_config->ev_queue_mode =
+				RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+
+		/* One port is required for eth Rx adapter */
+		eventdev_config->nb_eventport -= 1;
+
+		/* One port is reserved for eth Tx adapter */
+		eventdev_config->nb_eventport -= 1;
+
+		/* Update the number of eventdevs */
+		em_conf->nb_eventdev++;
+	}
+
+	return 0;
+}
+
+static int
+eh_validate_conf(struct eventmode_conf *em_conf)
+{
+	int ret;
+
+	/* After parsing all args, verify that the conf can be allowed */
+	ret = eh_validate_user_params(em_conf);
+	if (ret != 0)
+		return ret;
+
+	/*
+	 * See if event devs are specified. Else probe the event devices
+	 * and initialize the conf with all ports & queues available
+	 */
+	if (em_conf->nb_eventdev == 0) {
+		ret = eh_set_default_conf_eventdev(em_conf);
+		if (ret != 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int
+eh_initialize_eventdev(struct eventmode_conf *em_conf)
+{
+	int ret = -1;
+	uint8_t i, j;
+	struct rte_event_dev_config eventdev_conf;
+	struct rte_event_dev_info evdev_default_conf;
+	struct rte_event_queue_conf eventq_conf = {0};
+	struct eventdev_params *eventdev_config;
+	int nb_eventdev = em_conf->nb_eventdev;
+	int nb_eventqueue;
+	uint8_t eventdev_id;
+
+	for (i = 0; i < nb_eventdev; i++) {
+
+		/* Get eventdev config */
+		eventdev_config = &(em_conf->eventdev_config[i]);
+
+		/* Get event dev ID */
+		eventdev_id = eventdev_config->eventdev_id;
+
+		/* Get the number of queues */
+		nb_eventqueue = eventdev_config->nb_eventqueue;
+
+		/* One queue is reserved for the final stage (doing eth tx) */
+		/* TODO handles only one Tx adapter. Fix this */
+		nb_eventqueue += 1;
+
+		/* Reset the default conf */
+		memset(&evdev_default_conf, 0,
+			sizeof(struct rte_event_dev_info));
+
+		/* Get default conf of eventdev */
+		ret = rte_event_dev_info_get(eventdev_id, &evdev_default_conf);
+		if (ret < 0) {
+			EH_LOG_ERR(
+				"Error in getting event device info[devID:%d]",
+				eventdev_id);
+			return ret;
+		}
+
+		memset(&eventdev_conf, 0, sizeof(struct rte_event_dev_config));
+		eventdev_conf.nb_events_limit =
+				evdev_default_conf.max_num_events;
+		eventdev_conf.nb_event_queues = nb_eventqueue;
+		eventdev_conf.nb_event_ports =
+				eventdev_config->nb_eventport;
+		eventdev_conf.nb_event_queue_flows =
+				evdev_default_conf.max_event_queue_flows;
+		eventdev_conf.nb_event_port_dequeue_depth =
+				evdev_default_conf.max_event_port_dequeue_depth;
+		eventdev_conf.nb_event_port_enqueue_depth =
+				evdev_default_conf.max_event_port_enqueue_depth;
+
+		/* Configure event device */
+		ret = rte_event_dev_configure(eventdev_id, &eventdev_conf);
+		if (ret < 0) {
+			EH_LOG_ERR("Error in configuring event device");
+			return ret;
+		}
+
+		/* Configure event queues */
+		for (j = 0; j < nb_eventqueue; j++) {
+
+			memset(&eventq_conf, 0,
+					sizeof(struct rte_event_queue_conf));
+
+			/* Read the requested conf */
+
+			/* Per event dev queues can be ATQ or SINGLE LINK */
+			eventq_conf.event_queue_cfg =
+					eventdev_config->ev_queue_mode;
+			/*
+			 * All queues need to be set with sched_type as
+			 * schedule type for the application stage. One queue
+			 * would be reserved for the final eth tx stage. This
+			 * will be an atomic queue.
+			 */
+			if (j == nb_eventqueue-1) {
+				eventq_conf.schedule_type =
+					RTE_SCHED_TYPE_ATOMIC;
+			} else {
+				eventq_conf.schedule_type =
+					em_conf->ext_params.sched_type;
+			}
+
+			/* Set max atomic flows to 1024 */
+			eventq_conf.nb_atomic_flows = 1024;
+			eventq_conf.nb_atomic_order_sequences = 1024;
+
+			/* Setup the queue */
+			ret = rte_event_queue_setup(eventdev_id, j,
+					&eventq_conf);
+			if (ret < 0) {
+				EH_LOG_ERR("Error in event queue setup");
+				return ret;
+			}
+		}
+
+		/* Configure event ports */
+		for (j = 0; j <  eventdev_config->nb_eventport; j++) {
+			ret = rte_event_port_setup(eventdev_id, j, NULL);
+			if (ret < 0) {
+				EH_LOG_ERR("Error setting up event port");
+				return ret;
+			}
+		}
+	}
+
+	/* Start event devices */
+	for (i = 0; i < nb_eventdev; i++) {
+
+		/* Get eventdev config */
+		eventdev_config = &(em_conf->eventdev_config[i]);
+
+		ret = rte_event_dev_start(eventdev_config->eventdev_id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error in starting event device[devID: %d]",
+				eventdev_config->eventdev_id);
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int32_t
+eh_devs_init(struct eh_conf *mode_conf)
+{
+	int ret;
+	uint16_t portid;
+	struct eventmode_conf *em_conf;
+
+	if (mode_conf == NULL) {
+		EH_LOG_ERR("Invalid conf");
+		return -EINVAL;
+	}
+
+	if (mode_conf->mode != EH_PKT_TRANSFER_MODE_EVENT)
+		return 0;
+
+	if (mode_conf->mode_params == NULL) {
+		EH_LOG_ERR("Invalid mode params");
+		return -EINVAL;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	/* Validate the conf requested */
+	if (eh_validate_conf(em_conf) != 0) {
+		EH_LOG_ERR("Failed while validating the conf requested");
+		return -EINVAL;
+	}
+
+	/* Stop eth devices before setting up adapter */
+	RTE_ETH_FOREACH_DEV(portid) {
+
+		/* Use only the ports enabled */
+		if ((mode_conf->eth_portmask & (1 << portid)) == 0)
+			continue;
+
+		rte_eth_dev_stop(portid);
+	}
+
+	/* Setup eventdev */
+	ret = eh_initialize_eventdev(em_conf);
+	if (ret != 0)
+		return ret;
+
+	/* Start eth devices after setting up adapter */
+	RTE_ETH_FOREACH_DEV(portid) {
+
+		/* Use only the ports enabled */
+		if ((mode_conf->eth_portmask & (1 << portid)) == 0)
+			continue;
+
+		ret = rte_eth_dev_start(portid);
+		if (ret < 0) {
+			EH_LOG_ERR("Error starting eth dev %d", portid);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int32_t
+eh_devs_uninit(struct eh_conf *mode_conf)
+{
+	int ret, i;
+	uint16_t id;
+	struct eventmode_conf *em_conf;
+
+	if (mode_conf == NULL) {
+		EH_LOG_ERR("Invalid conf");
+		return -EINVAL;
+	}
+
+	if (mode_conf->mode != EH_PKT_TRANSFER_MODE_EVENT)
+		return 0;
+
+	if (mode_conf->mode_params == NULL) {
+		EH_LOG_ERR("Invalid mode params");
+		return -EINVAL;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	/* Stop and release event devices */
+	for (i = 0; i < em_conf->nb_eventdev; i++) {
+
+		id = em_conf->eventdev_config[i].eventdev_id;
+		rte_event_dev_stop(id);
+
+		ret = rte_event_dev_close(id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error closing event dev %d", id);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
new file mode 100644
index 0000000..71990f9
--- /dev/null
+++ b/examples/ipsec-secgw/event_helper.h
@@ -0,0 +1,115 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (C) 2019 Marvell International Ltd.
+ */
+#ifndef _EVENT_HELPER_H_
+#define _EVENT_HELPER_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_log.h>
+
+#define RTE_LOGTYPE_EH  RTE_LOGTYPE_USER4
+
+#define EH_LOG_ERR(...) \
+	RTE_LOG(ERR, EH, \
+		RTE_FMT("%s() line %u: " RTE_FMT_HEAD(__VA_ARGS__ ,) "\n", \
+			__func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__ ,)))
+
+/* Max event devices supported */
+#define EVENT_MODE_MAX_EVENT_DEVS RTE_EVENT_MAX_DEVS
+
+/**
+ * Packet transfer mode of the application
+ */
+enum eh_pkt_transfer_mode {
+	EH_PKT_TRANSFER_MODE_POLL = 0,
+	EH_PKT_TRANSFER_MODE_EVENT,
+};
+
+/* Event dev params */
+struct eventdev_params {
+	uint8_t eventdev_id;
+	uint8_t nb_eventqueue;
+	uint8_t nb_eventport;
+	uint8_t ev_queue_mode;
+};
+
+/* Eventmode conf data */
+struct eventmode_conf {
+	int nb_eventdev;
+		/**< No of event devs */
+	struct eventdev_params eventdev_config[EVENT_MODE_MAX_EVENT_DEVS];
+		/**< Per event dev conf */
+	union {
+		RTE_STD_C11
+		struct {
+			uint64_t sched_type			: 2;
+		/**< Schedule type */
+		};
+		uint64_t u64;
+	} ext_params;
+		/**< 64 bit field to specify extended params */
+};
+
+/**
+ * Event helper configuration
+ */
+struct eh_conf {
+	enum eh_pkt_transfer_mode mode;
+		/**< Packet transfer mode of the application */
+	uint32_t eth_portmask;
+		/**<
+		 * Mask of the eth ports to be used. This portmask would be
+		 * checked while initializing devices using helper routines.
+		 */
+	void *mode_params;
+		/**< Mode specific parameters */
+};
+
+/**
+ * Initialize event mode devices
+ *
+ * Application can call this function to get the event devices, eth devices
+ * and eth rx & tx adapters initialized according to the default config or
+ * config populated using the command line args.
+ *
+ * Application is expected to initialize the eth devices and then the event
+ * mode helper subsystem will stop & start eth devices according to its
+ * requirement. Call to this function should be done after the eth devices
+ * are successfully initialized.
+ *
+ * @param mode_conf
+ *   Configuration of the mode in which app is doing packet handling
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int32_t
+eh_devs_init(struct eh_conf *mode_conf);
+
+/**
+ * Release event mode devices
+ *
+ * Application could call this function to release event devices,
+ * eth rx & tx adapters according to the conf.
+ *
+ * Call to this function should be done before application stops
+ * and closes eth devices. This function will not close and stop
+ * eth devices.
+ *
+ * @param mode_conf
+ *   Configuration of the mode in which app is doing packet handling
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int32_t
+eh_devs_uninit(struct eh_conf *mode_conf);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _EVENT_HELPER_H_ */
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 9ece345..20f4064 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,9 +6,9 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash', 'ip_frag', 'ipsec']
+deps += ['security', 'lpm', 'acl', 'hash', 'ip_frag', 'ipsec', 'eventdev']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
-	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c', 'event_helper.c'
 )
-- 
2.7.4

[dpdk-dev] [RFC PATCH 02/13] examples/ipsec-secgw: add eventdev port-lcore link

[Anoob Joseph]

Add event device port-lcore link and specify which event queues should
be connected to the event port. Generate a default config for event
port-lcore links if it is not specified in the configuration. This
routine will check the number of available ports and then create links
according to the number of cores available. This patch also adds a new
entry in the eventmode conf to denote that all queues are to be linked
with every port. This enables one core to receive packets from every port.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 132 ++++++++++++++++++++++++++++++++++++
 examples/ipsec-secgw/event_helper.h |  33 +++++++++
 2 files changed, 165 insertions(+)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index af7c7e2..3080625 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -3,9 +3,34 @@
  */
 #include <rte_eventdev.h>
 #include <rte_ethdev.h>
+#include <rte_bitmap.h>
+#include <rte_malloc.h>
 
 #include "event_helper.h"
 
+static inline unsigned int
+eh_get_next_active_core(struct eventmode_conf *em_conf,
+		unsigned int prev_core)
+{
+	unsigned int next_core;
+
+get_next_core:
+	/* Get the next core */
+	next_core = rte_get_next_lcore(prev_core, 0, 0);
+
+	/* Check if we have reached max lcores */
+	if (next_core == RTE_MAX_LCORE)
+		return next_core;
+
+	/* Some cores would be reserved as rx cores. Skip them */
+	if (rte_bitmap_get(em_conf->eth_core_mask, next_core)) {
+		prev_core = next_core;
+		goto get_next_core;
+	}
+
+	return next_core;
+}
+
 static int
 eh_validate_user_params(struct eventmode_conf *em_conf)
 {
@@ -74,6 +99,74 @@ eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
 }
 
 static int
+eh_set_default_conf_link(struct eventmode_conf *em_conf)
+{
+	int i, j;
+	struct eventdev_params *eventdev_config;
+	unsigned int lcore_id = -1;
+	int link_index;
+	struct eh_event_link_info *link;
+
+	/*
+	 * Create a 1:1 mapping from event ports to cores. If the number
+	 * of event ports is lesser than the cores, some cores won't
+	 * execute worker. If event ports are more, then some ports won't
+	 * be used.
+	 *
+	 */
+
+	/*
+	 * The event queue-port mapping is done according to the link. Since
+	 * we are falling back to the default link conf, enabling
+	 * "all_ev_queue_to_ev_port" mode flag. This will map all queues to the
+	 * port.
+	 */
+	em_conf->ext_params.all_ev_queue_to_ev_port = 1;
+
+	for (i = 0; i < em_conf->nb_eventdev; i++) {
+
+		/* Get event dev conf */
+		eventdev_config = &(em_conf->eventdev_config[i]);
+
+		/* Loop through the ports */
+		for (j = 0; j < eventdev_config->nb_eventport; j++) {
+
+			/* Get next active core id */
+			lcore_id = eh_get_next_active_core(em_conf,
+					lcore_id);
+
+			if (lcore_id == RTE_MAX_LCORE) {
+				/* Reached max cores */
+				return 0;
+			}
+
+			/* Save the current combination as one link */
+
+			/* Get the index */
+			link_index = em_conf->nb_link;
+
+			/* Get the corresponding link */
+			link = &(em_conf->link[link_index]);
+
+			/* Save link */
+			link->eventdev_id = eventdev_config->eventdev_id;
+			link->event_portid = j;
+			link->lcore_id = lcore_id;
+
+			/*
+			 * Not setting eventq_id as by default all queues
+			 * need to be mapped to the port, and is controlled
+			 * by the operating mode.
+			 */
+
+			/* Update number of links */
+			em_conf->nb_link++;
+		}
+	}
+	return 0;
+}
+
+static int
 eh_validate_conf(struct eventmode_conf *em_conf)
 {
 	int ret;
@@ -93,6 +186,16 @@ eh_validate_conf(struct eventmode_conf *em_conf)
 			return ret;
 	}
 
+	/*
+	 * See if links are specified. Else generate a default conf for
+	 * the event ports used.
+	 */
+	if (em_conf->nb_link == 0) {
+		ret = eh_set_default_conf_link(em_conf);
+		if (ret != 0)
+			return ret;
+	}
+
 	return 0;
 }
 
@@ -104,10 +207,12 @@ eh_initialize_eventdev(struct eventmode_conf *em_conf)
 	struct rte_event_dev_config eventdev_conf;
 	struct rte_event_dev_info evdev_default_conf;
 	struct rte_event_queue_conf eventq_conf = {0};
+	struct eh_event_link_info *link;
 	struct eventdev_params *eventdev_config;
 	int nb_eventdev = em_conf->nb_eventdev;
 	int nb_eventqueue;
 	uint8_t eventdev_id;
+	uint8_t *queue = NULL;
 
 	for (i = 0; i < nb_eventdev; i++) {
 
@@ -205,6 +310,33 @@ eh_initialize_eventdev(struct eventmode_conf *em_conf)
 		}
 	}
 
+	/* Make event queue - event port link */
+	for (j = 0; j <  em_conf->nb_link; j++) {
+
+		/* Get link info */
+		link = &(em_conf->link[j]);
+
+		/* Get event dev ID */
+		eventdev_id = link->eventdev_id;
+
+		/*
+		 * If "all_ev_queue_to_ev_port" params flag is selected, all
+		 * queues need to be mapped to the port.
+		 */
+		if (em_conf->ext_params.all_ev_queue_to_ev_port)
+			queue = NULL;
+		else
+			queue = &(link->eventq_id);
+
+		/* Link queue to port */
+		ret = rte_event_port_link(eventdev_id, link->event_portid,
+				queue, NULL, 1);
+		if (ret < 0) {
+			EH_LOG_ERR("Error in event port linking");
+			return ret;
+		}
+	}
+
 	/* Start event devices */
 	for (i = 0; i < nb_eventdev; i++) {
 
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index 71990f9..052ff25 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -20,6 +20,13 @@ extern "C" {
 /* Max event devices supported */
 #define EVENT_MODE_MAX_EVENT_DEVS RTE_EVENT_MAX_DEVS
 
+/* Max event queues supported per event device */
+#define EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV RTE_EVENT_MAX_QUEUES_PER_DEV
+
+/* Max event-lcore links */
+#define EVENT_MODE_MAX_LCORE_LINKS \
+	(EVENT_MODE_MAX_EVENT_DEVS * EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV)
+
 /**
  * Packet transfer mode of the application
  */
@@ -36,17 +43,43 @@ struct eventdev_params {
 	uint8_t ev_queue_mode;
 };
 
+/**
+ * Event-lcore link configuration
+ */
+struct eh_event_link_info {
+	uint8_t eventdev_id;
+		/**< Event device ID */
+	uint8_t event_portid;
+		/**< Event port ID */
+	uint8_t eventq_id;
+		/**< Event queue to be linked to the port */
+	uint8_t lcore_id;
+		/**< Lcore to be polling on this port */
+};
+
 /* Eventmode conf data */
 struct eventmode_conf {
 	int nb_eventdev;
 		/**< No of event devs */
 	struct eventdev_params eventdev_config[EVENT_MODE_MAX_EVENT_DEVS];
 		/**< Per event dev conf */
+	uint8_t nb_link;
+		/**< No of links */
+	struct eh_event_link_info
+		link[EVENT_MODE_MAX_LCORE_LINKS];
+		/**< Per link conf */
+	struct rte_bitmap *eth_core_mask;
+		/**< Core mask of cores to be used for software Rx and Tx */
 	union {
 		RTE_STD_C11
 		struct {
 			uint64_t sched_type			: 2;
 		/**< Schedule type */
+			uint64_t all_ev_queue_to_ev_port	: 1;
+		/**<
+		 * When enabled, all event queues need to be mapped to
+		 * each event port
+		 */
 		};
 		uint64_t u64;
 	} ext_params;
-- 
2.7.4

[dpdk-dev] [RFC PATCH 03/13] examples/ipsec-secgw: add Rx adapter support

[Anoob Joseph]

Add Rx adapter support. The event helper init routine will initialize
the Rx adapter according to the configuration. If Rx adapter config
is not present it will generate a default config. It will check the
available eth ports and event queues and map them 1:1. So one eth port
will be connected to one event queue. This way event queue ID could
be used to figure out the port on which a packet came in.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 267 +++++++++++++++++++++++++++++++++++-
 examples/ipsec-secgw/event_helper.h |  29 ++++
 2 files changed, 294 insertions(+), 2 deletions(-)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index 3080625..b5417a0 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright (C) 2019 Marvell International Ltd.
  */
-#include <rte_eventdev.h>
+#include <rte_event_eth_rx_adapter.h>
 #include <rte_ethdev.h>
 #include <rte_bitmap.h>
 #include <rte_malloc.h>
@@ -9,6 +9,29 @@
 #include "event_helper.h"
 
 static inline unsigned int
+eh_get_next_rx_core(struct eventmode_conf *em_conf,
+		unsigned int prev_core)
+{
+	unsigned int next_core;
+
+get_next_core:
+	/* Get the next core */
+	next_core = rte_get_next_lcore(prev_core, 0, 0);
+
+	/* Check if we have reached max lcores */
+	if (next_core == RTE_MAX_LCORE)
+		return next_core;
+
+	/* Only some cores would be marked as rx cores. Skip others */
+	if (!(rte_bitmap_get(em_conf->eth_core_mask, next_core))) {
+		prev_core = next_core;
+		goto get_next_core;
+	}
+
+	return next_core;
+}
+
+static inline unsigned int
 eh_get_next_active_core(struct eventmode_conf *em_conf,
 		unsigned int prev_core)
 {
@@ -167,6 +190,87 @@ eh_set_default_conf_link(struct eventmode_conf *em_conf)
 }
 
 static int
+eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
+{
+	int nb_eth_dev;
+	int i;
+	int adapter_id;
+	int eventdev_id;
+	int conn_id;
+	struct rx_adapter_conf *adapter;
+	struct rx_adapter_connection_info *conn;
+	struct eventdev_params *eventdev_config;
+
+	/* Create one adapter with all eth queues mapped to event queues 1:1 */
+
+	if (em_conf->nb_eventdev == 0) {
+		EH_LOG_ERR("No event devs registered\n");
+		return -EINVAL;
+	}
+
+	/* Get the number of eth devs */
+	nb_eth_dev = rte_eth_dev_count_avail();
+
+	/* Use the first event dev */
+	eventdev_config = &(em_conf->eventdev_config[0]);
+
+	/* Get eventdev ID */
+	eventdev_id = eventdev_config->eventdev_id;
+	adapter_id = 0;
+
+	/* Get adapter conf */
+	adapter = &(em_conf->rx_adapter[adapter_id]);
+
+	/* Set adapter conf */
+	adapter->eventdev_id = eventdev_id;
+	adapter->adapter_id = adapter_id;
+	adapter->rx_core_id = eh_get_next_rx_core(em_conf, -1);
+
+	/*
+	 * All queues of one eth device (port) will be mapped to one event
+	 * queue. Each port will have an individual connection.
+	 *
+	 */
+
+	/* Make sure there is enough event queues for 1:1 mapping */
+	if (nb_eth_dev > eventdev_config->nb_eventqueue) {
+		EH_LOG_ERR("Not enough event queues for 1:1 mapping "
+			"[eth devs: %d, event queues: %d]\n",
+			nb_eth_dev, eventdev_config->nb_eventqueue);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < nb_eth_dev; i++) {
+
+		/* Use only the ports enabled */
+		if ((em_conf->eth_portmask & (1 << i)) == 0)
+			continue;
+
+		/* Get the connection id */
+		conn_id = adapter->nb_connections;
+
+		/* Get the connection */
+		conn = &(adapter->conn[conn_id]);
+
+		/* Set 1:1 mapping between eth ports & event queues*/
+		conn->ethdev_id = i;
+		conn->eventq_id = i;
+
+		/* Add all eth queues of one eth port to one event queue */
+		conn->ethdev_rx_qid = -1;
+
+		/* Update no of connections */
+		adapter->nb_connections++;
+
+	}
+
+	/* We have setup one adapter */
+	em_conf->nb_rx_adapter = 1;
+
+	return 0;
+}
+
+static int
 eh_validate_conf(struct eventmode_conf *em_conf)
 {
 	int ret;
@@ -196,6 +300,16 @@ eh_validate_conf(struct eventmode_conf *em_conf)
 			return ret;
 	}
 
+	/*
+	 * See if rx adapters are specified. Else generate a default conf
+	 * with one rx adapter and all eth queue - event queue mapped.
+	 */
+	if (em_conf->nb_rx_adapter == 0) {
+		ret = eh_set_default_conf_rx_adapter(em_conf);
+		if (ret != 0)
+			return ret;
+	}
+
 	return 0;
 }
 
@@ -353,6 +467,118 @@ eh_initialize_eventdev(struct eventmode_conf *em_conf)
 	return 0;
 }
 
+static int
+eh_rx_adapter_configure(struct eventmode_conf *em_conf,
+	struct rx_adapter_conf *adapter)
+{
+	int j;
+	int ret;
+	uint8_t eventdev_id;
+	uint32_t service_id;
+	struct rx_adapter_connection_info *conn;
+	struct rte_event_port_conf port_conf = {0};
+	struct rte_event_eth_rx_adapter_queue_conf queue_conf = {0};
+	struct rte_event_dev_info evdev_default_conf = {0};
+
+	/* Get event dev ID */
+	eventdev_id = adapter->eventdev_id;
+
+	/* Get default configuration of event dev */
+	ret = rte_event_dev_info_get(eventdev_id, &evdev_default_conf);
+	if (ret < 0) {
+		EH_LOG_ERR("Error in getting event device info[devID:%d]",
+			eventdev_id);
+		return ret;
+	}
+
+	/* Setup port conf */
+	port_conf.new_event_threshold = 1200;
+	port_conf.dequeue_depth =
+			evdev_default_conf.max_event_port_dequeue_depth;
+	port_conf.enqueue_depth =
+			evdev_default_conf.max_event_port_enqueue_depth;
+
+	/* Create Rx adapter */
+	ret = rte_event_eth_rx_adapter_create(adapter->adapter_id,
+			adapter->eventdev_id,
+			&port_conf);
+	if (ret < 0) {
+		EH_LOG_ERR("Error in rx adapter creation");
+		return ret;
+	}
+
+	/* Setup various connections in the adapter */
+
+	queue_conf.rx_queue_flags =
+			RTE_EVENT_ETH_RX_ADAPTER_QUEUE_FLOW_ID_VALID;
+
+	for (j = 0; j < adapter->nb_connections; j++) {
+		/* Get connection */
+		conn = &(adapter->conn[j]);
+
+		/* Setup queue conf */
+		queue_conf.ev.queue_id = conn->eventq_id;
+		queue_conf.ev.sched_type = em_conf->ext_params.sched_type;
+
+		/* Set flow ID as ethdev ID */
+		queue_conf.ev.flow_id = conn->ethdev_id;
+
+		/* Add queue to the adapter */
+		ret = rte_event_eth_rx_adapter_queue_add(
+				adapter->adapter_id,
+				conn->ethdev_id,
+				conn->ethdev_rx_qid,
+				&queue_conf);
+		if (ret < 0) {
+			EH_LOG_ERR("Error in adding eth queue in Rx adapter");
+			return ret;
+		}
+	}
+
+	/* Get the service ID used by rx adapter */
+	ret = rte_event_eth_rx_adapter_service_id_get(adapter->adapter_id,
+						      &service_id);
+	if (ret != -ESRCH && ret != 0) {
+		EH_LOG_ERR("Error getting service ID used by Rx adapter");
+		return ret;
+	}
+
+	/*
+	 * TODO
+	 * Rx core will invoke the service when required. The runstate check
+	 * is not required.
+	 *
+	 */
+	rte_service_set_runstate_mapped_check(service_id, 0);
+
+	/* Start adapter */
+	ret = rte_event_eth_rx_adapter_start(adapter->adapter_id);
+	if (ret) {
+		EH_LOG_ERR("Error in starting rx adapter");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int
+eh_initialize_rx_adapter(struct eventmode_conf *em_conf)
+{
+	int i, ret;
+	struct rx_adapter_conf *adapter;
+
+	/* Configure rx adapters */
+	for (i = 0; i < em_conf->nb_rx_adapter; i++) {
+		adapter = &(em_conf->rx_adapter[i]);
+		ret = eh_rx_adapter_configure(em_conf, adapter);
+		if (ret < 0) {
+			EH_LOG_ERR("Rx adapter configuration failed");
+			return ret;
+		}
+	}
+	return 0;
+}
+
 int32_t
 eh_devs_init(struct eh_conf *mode_conf)
 {
@@ -376,6 +602,9 @@ eh_devs_init(struct eh_conf *mode_conf)
 	/* Get eventmode conf */
 	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
 
+	/* Eventmode conf would need eth portmask */
+	em_conf->eth_portmask = mode_conf->eth_portmask;
+
 	/* Validate the conf requested */
 	if (eh_validate_conf(em_conf) != 0) {
 		EH_LOG_ERR("Failed while validating the conf requested");
@@ -397,6 +626,11 @@ eh_devs_init(struct eh_conf *mode_conf)
 	if (ret != 0)
 		return ret;
 
+	/* Setup Rx adapter */
+	ret = eh_initialize_rx_adapter(em_conf);
+	if (ret != 0)
+		return ret;
+
 	/* Start eth devices after setting up adapter */
 	RTE_ETH_FOREACH_DEV(portid) {
 
@@ -417,7 +651,7 @@ eh_devs_init(struct eh_conf *mode_conf)
 int32_t
 eh_devs_uninit(struct eh_conf *mode_conf)
 {
-	int ret, i;
+	int ret, i, j;
 	uint16_t id;
 	struct eventmode_conf *em_conf;
 
@@ -437,6 +671,35 @@ eh_devs_uninit(struct eh_conf *mode_conf)
 	/* Get eventmode conf */
 	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
 
+	/* Stop and release rx adapters */
+	for (i = 0; i < em_conf->nb_rx_adapter; i++) {
+
+		id = em_conf->rx_adapter[i].adapter_id;
+		ret = rte_event_eth_rx_adapter_stop(id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error stopping rx adapter %d", id);
+			return ret;
+		}
+
+		for (j = 0; j < em_conf->rx_adapter[i].nb_connections; j++) {
+
+			ret = rte_event_eth_rx_adapter_queue_del(id,
+				em_conf->rx_adapter[i].conn[j].ethdev_id, -1);
+			if (ret < 0) {
+				EH_LOG_ERR(
+					"Error deleting rx adapter queues %d",
+					id);
+				return ret;
+			}
+		}
+
+		ret = rte_event_eth_rx_adapter_free(id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error freeing rx adapter %d", id);
+			return ret;
+		}
+	}
+
 	/* Stop and release event devices */
 	for (i = 0; i < em_conf->nb_eventdev; i++) {
 
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index 052ff25..4233b42 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -20,6 +20,12 @@ extern "C" {
 /* Max event devices supported */
 #define EVENT_MODE_MAX_EVENT_DEVS RTE_EVENT_MAX_DEVS
 
+/* Max Rx adapters supported */
+#define EVENT_MODE_MAX_RX_ADAPTERS RTE_EVENT_MAX_DEVS
+
+/* Max Rx adapter connections */
+#define EVENT_MODE_MAX_CONNECTIONS_PER_ADAPTER 16
+
 /* Max event queues supported per event device */
 #define EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV RTE_EVENT_MAX_QUEUES_PER_DEV
 
@@ -57,12 +63,33 @@ struct eh_event_link_info {
 		/**< Lcore to be polling on this port */
 };
 
+/* Rx adapter connection info */
+struct rx_adapter_connection_info {
+	uint8_t ethdev_id;
+	uint8_t eventq_id;
+	int32_t ethdev_rx_qid;
+};
+
+/* Rx adapter conf */
+struct rx_adapter_conf {
+	int32_t eventdev_id;
+	int32_t adapter_id;
+	uint32_t rx_core_id;
+	uint8_t nb_connections;
+	struct rx_adapter_connection_info
+			conn[EVENT_MODE_MAX_CONNECTIONS_PER_ADAPTER];
+};
+
 /* Eventmode conf data */
 struct eventmode_conf {
 	int nb_eventdev;
 		/**< No of event devs */
 	struct eventdev_params eventdev_config[EVENT_MODE_MAX_EVENT_DEVS];
 		/**< Per event dev conf */
+	uint8_t nb_rx_adapter;
+		/**< No of Rx adapters */
+	struct rx_adapter_conf rx_adapter[EVENT_MODE_MAX_RX_ADAPTERS];
+		/**< Rx adapter conf */
 	uint8_t nb_link;
 		/**< No of links */
 	struct eh_event_link_info
@@ -70,6 +97,8 @@ struct eventmode_conf {
 		/**< Per link conf */
 	struct rte_bitmap *eth_core_mask;
 		/**< Core mask of cores to be used for software Rx and Tx */
+	uint32_t eth_portmask;
+		/**< Mask of the eth ports to be used */
 	union {
 		RTE_STD_C11
 		struct {
-- 
2.7.4

[dpdk-dev] [RFC PATCH 04/13] examples/ipsec-secgw: add Tx adapter support

[Anoob Joseph]

Add Tx adapter support. The event helper init routine will initialize
the Tx adapter according to the configuration. If Tx adapter config
is not present it will generate a default config.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 375 +++++++++++++++++++++++++++++++++++-
 examples/ipsec-secgw/event_helper.h |  58 ++++++
 2 files changed, 424 insertions(+), 9 deletions(-)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index b5417a0..7745fbb 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -2,31 +2,58 @@
  * Copyright (C) 2019 Marvell International Ltd.
  */
 #include <rte_event_eth_rx_adapter.h>
+#include <rte_event_eth_tx_adapter.h>
 #include <rte_ethdev.h>
 #include <rte_bitmap.h>
 #include <rte_malloc.h>
 
 #include "event_helper.h"
 
+static int
+eh_get_enabled_cores(struct rte_bitmap *eth_core_mask)
+{
+	int i;
+	int count = 0;
+
+	RTE_LCORE_FOREACH(i) {
+		/* Check if this core is enabled in core_mask*/
+		if (rte_bitmap_get(eth_core_mask, i)) {
+			/* We have enabled core */
+			count++;
+		}
+	}
+	return count;
+}
+
 static inline unsigned int
-eh_get_next_rx_core(struct eventmode_conf *em_conf,
-		unsigned int prev_core)
+eh_get_next_eth_core(struct eventmode_conf *em_conf)
 {
 	unsigned int next_core;
+	static unsigned int prev_core = -1;
+
+	/*
+	 * Make sure we have atleast one eth core running, else the following
+	 * logic would lead to an infinite loop.
+	 */
+	if (eh_get_enabled_cores(em_conf->eth_core_mask) == 0) {
+		EH_LOG_ERR("No enabled eth core found");
+		return RTE_MAX_LCORE;
+	}
 
 get_next_core:
 	/* Get the next core */
-	next_core = rte_get_next_lcore(prev_core, 0, 0);
+	next_core = rte_get_next_lcore(prev_core, 0, 1);
 
 	/* Check if we have reached max lcores */
 	if (next_core == RTE_MAX_LCORE)
 		return next_core;
 
-	/* Only some cores would be marked as rx cores. Skip others */
-	if (!(rte_bitmap_get(em_conf->eth_core_mask, next_core))) {
-		prev_core = next_core;
+	/* Update prev_core */
+	prev_core = next_core;
+
+	/* Only some cores would be marked as eth cores. Skip others */
+	if (!(rte_bitmap_get(em_conf->eth_core_mask, next_core)))
 		goto get_next_core;
-	}
 
 	return next_core;
 }
@@ -45,7 +72,7 @@ eh_get_next_active_core(struct eventmode_conf *em_conf,
 	if (next_core == RTE_MAX_LCORE)
 		return next_core;
 
-	/* Some cores would be reserved as rx cores. Skip them */
+	/* Some cores would be reserved as eth cores. Skip them */
 	if (rte_bitmap_get(em_conf->eth_core_mask, next_core)) {
 		prev_core = next_core;
 		goto get_next_core;
@@ -54,6 +81,23 @@ eh_get_next_active_core(struct eventmode_conf *em_conf,
 	return next_core;
 }
 
+static struct eventdev_params *
+eh_get_eventdev_params(struct eventmode_conf *em_conf,
+		uint8_t eventdev_id)
+{
+	int i;
+
+	for (i = 0; i < em_conf->nb_eventdev; i++) {
+		if (em_conf->eventdev_config[i].eventdev_id == eventdev_id)
+			break;
+	}
+
+	/* No match */
+	if (i == em_conf->nb_eventdev)
+		return NULL;
+
+	return &(em_conf->eventdev_config[i]);
+}
 static int
 eh_validate_user_params(struct eventmode_conf *em_conf)
 {
@@ -224,7 +268,7 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 	/* Set adapter conf */
 	adapter->eventdev_id = eventdev_id;
 	adapter->adapter_id = adapter_id;
-	adapter->rx_core_id = eh_get_next_rx_core(em_conf, -1);
+	adapter->rx_core_id = eh_get_next_eth_core(em_conf);
 
 	/*
 	 * All queues of one eth device (port) will be mapped to one event
@@ -271,6 +315,100 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 }
 
 static int
+eh_set_default_conf_tx_adapter(struct eventmode_conf *em_conf)
+{
+	int nb_eth_dev;
+	int eventdev_id;
+	int adapter_id;
+	int i;
+	int conn_id;
+	struct eventdev_params *eventdev_config;
+	struct tx_adapter_conf *tx_adapter;
+	struct tx_adapter_connection_info *conn;
+
+	/*
+	 * Create one Tx adapter with all eth queues mapped to event queues
+	 * 1:1.
+	 */
+
+	if (em_conf->nb_eventdev == 0) {
+		EH_LOG_ERR("No event devs registered");
+		return -EINVAL;
+	}
+
+	/* Get the number of eth devs */
+	nb_eth_dev = rte_eth_dev_count_avail();
+
+	/* Use the first event dev */
+	eventdev_config = &(em_conf->eventdev_config[0]);
+
+	/* Get eventdev ID */
+	eventdev_id = eventdev_config->eventdev_id;
+	adapter_id = 0;
+
+	/* Get adapter conf */
+	tx_adapter = &(em_conf->tx_adapter[adapter_id]);
+
+	/* Set adapter conf */
+	tx_adapter->eventdev_id = eventdev_id;
+	tx_adapter->adapter_id = adapter_id;
+
+	/* TODO: Tx core is required only when internal port is not present */
+
+	tx_adapter->tx_core_id = eh_get_next_eth_core(em_conf);
+
+	/*
+	 * Application would need to use one event queue per adapter for
+	 * submitting packets for Tx. Reserving the last queue available
+	 * and decrementing the total available event queues for this
+	 */
+
+	/* Queue numbers start at 0 */
+	tx_adapter->tx_ev_queue = eventdev_config->nb_eventqueue - 1;
+
+	/* Update the number of event queues available in eventdev */
+	eventdev_config->nb_eventqueue--;
+
+	/*
+	 * All Tx queues of the eth device (port) will be mapped to the event
+	 * device.
+	 */
+
+	/* Set defaults for connections */
+
+	/*
+	 * One eth device (port) would be one connection. All Tx queues of
+	 * the device would be mapped to the Tx adapter.
+	 */
+
+	for (i = 0; i < nb_eth_dev; i++) {
+
+		/* Use only the ports enabled */
+		if ((em_conf->eth_portmask & (1 << i)) == 0)
+			continue;
+
+		/* Get the connection id */
+		conn_id = tx_adapter->nb_connections;
+
+		/* Get the connection */
+		conn = &(tx_adapter->conn[conn_id]);
+
+		/* Add ethdev to connections */
+		conn->ethdev_id = i;
+
+		/* Add all eth tx queues to adapter */
+		conn->ethdev_tx_qid = -1;
+
+		/* Update no of connections */
+		tx_adapter->nb_connections++;
+	}
+
+	/* We have setup one adapter */
+	em_conf->nb_tx_adapter = 1;
+	return 0;
+}
+
+static int
 eh_validate_conf(struct eventmode_conf *em_conf)
 {
 	int ret;
@@ -310,6 +448,16 @@ eh_validate_conf(struct eventmode_conf *em_conf)
 			return ret;
 	}
 
+	/*
+	 * See if tx adapters are specified. Else generate a default conf
+	 * with one tx adapter.
+	 */
+	if (em_conf->nb_tx_adapter == 0) {
+		ret = eh_set_default_conf_tx_adapter(em_conf);
+		if (ret != 0)
+			return ret;
+	}
+
 	return 0;
 }
 
@@ -579,6 +727,147 @@ eh_initialize_rx_adapter(struct eventmode_conf *em_conf)
 	return 0;
 }
 
+static int
+eh_tx_adapter_configure(struct eventmode_conf *em_conf,
+	struct tx_adapter_conf *adapter)
+{
+	int ret, j;
+	uint8_t tx_port_id = 0;
+	uint8_t eventdev_id;
+	uint32_t service_id;
+	struct rte_event_port_conf port_conf = {0};
+	struct rte_event_dev_info evdev_default_conf = {0};
+	struct tx_adapter_connection_info *conn;
+	struct eventdev_params *eventdev_config;
+
+	/* Get event dev ID */
+	eventdev_id = adapter->eventdev_id;
+
+	/* Get event device conf */
+	eventdev_config = eh_get_eventdev_params(em_conf, eventdev_id);
+
+	/* Create Tx adapter */
+
+	/* Get default configuration of event dev */
+	ret = rte_event_dev_info_get(eventdev_id, &evdev_default_conf);
+	if (ret < 0) {
+		EH_LOG_ERR("Error in getting event device info[devID:%d]",
+			eventdev_id);
+		return ret;
+	}
+
+	/* Setup port conf */
+	port_conf.new_event_threshold =
+			evdev_default_conf.max_num_events;
+	port_conf.dequeue_depth =
+			evdev_default_conf.max_event_port_dequeue_depth;
+	port_conf.enqueue_depth =
+			evdev_default_conf.max_event_port_enqueue_depth;
+
+	/* Create Tx adapter */
+	ret = rte_event_eth_tx_adapter_create(adapter->adapter_id,
+			adapter->eventdev_id,
+			&port_conf);
+	if (ret < 0) {
+		EH_LOG_ERR("Error in Tx adapter creation");
+		return ret;
+	}
+
+	/* Setup various connections in the adapter */
+	for (j = 0; j < adapter->nb_connections; j++) {
+
+		/* Get connection */
+		conn = &(adapter->conn[j]);
+
+		/* Add queue to the adapter */
+		ret = rte_event_eth_tx_adapter_queue_add(
+				adapter->adapter_id,
+				conn->ethdev_id,
+				conn->ethdev_tx_qid);
+		if (ret < 0) {
+			EH_LOG_ERR("Error in adding eth queue in Tx adapter");
+			return ret;
+		}
+	}
+
+	/* Get event port used by the adapter */
+	ret = rte_event_eth_tx_adapter_event_port_get(
+			adapter->adapter_id,
+			&tx_port_id);
+	if (ret) {
+		EH_LOG_ERR("Failed to get Tx adapter port ID");
+		return ret;
+	}
+
+	/*
+	 * TODO: event queue for Tx adapter is required only if the
+	 * INTERNAL PORT is not present.
+	 */
+
+	/*
+	 * Tx event queue would be reserved for Tx adapter. Need to unlink
+	 * this queue from all other ports
+	 *
+	 */
+	for (j = 0; j < eventdev_config->nb_eventport; j++) {
+		rte_event_port_unlink(eventdev_id, j,
+				      &(adapter->tx_ev_queue), 1);
+	}
+
+	ret = rte_event_port_link(
+			eventdev_id,
+			tx_port_id,
+			&(adapter->tx_ev_queue),
+			NULL, 1);
+	if (ret != 1) {
+		EH_LOG_ERR("Failed to link event queue to port");
+		return ret;
+	}
+
+	/* Get the service ID used by Tx adapter */
+	ret = rte_event_eth_tx_adapter_service_id_get(adapter->adapter_id,
+						      &service_id);
+	if (ret != -ESRCH && ret != 0) {
+		EH_LOG_ERR("Error getting service ID used by adapter");
+		return ret;
+	}
+
+	/*
+	 * TODO
+	 * Tx core will invoke the service when required. The runstate check
+	 * is not required.
+	 *
+	 */
+	rte_service_set_runstate_mapped_check(service_id, 0);
+
+	/* Start adapter */
+	ret = rte_event_eth_tx_adapter_start(adapter->adapter_id);
+	if (ret) {
+		EH_LOG_ERR("Error in starting Tx adapter");
+		return ret;
+	}
+
+	return 0;
+}
+
+static int
+eh_initialize_tx_adapter(struct eventmode_conf *em_conf)
+{
+	int i, ret;
+	struct tx_adapter_conf *adapter;
+
+	/* Configure Tx adapters */
+	for (i = 0; i < em_conf->nb_tx_adapter; i++) {
+		adapter = &(em_conf->tx_adapter[i]);
+		ret = eh_tx_adapter_configure(em_conf, adapter);
+		if (ret < 0) {
+			EH_LOG_ERR("Tx adapter configuration failed");
+			return ret;
+		}
+	}
+	return 0;
+}
+
 int32_t
 eh_devs_init(struct eh_conf *mode_conf)
 {
@@ -631,6 +920,11 @@ eh_devs_init(struct eh_conf *mode_conf)
 	if (ret != 0)
 		return ret;
 
+	/* Setup Tx adapter */
+	ret = eh_initialize_tx_adapter(em_conf);
+	if (ret != 0)
+		return ret;
+
 	/* Start eth devices after setting up adapter */
 	RTE_ETH_FOREACH_DEV(portid) {
 
@@ -713,5 +1007,68 @@ eh_devs_uninit(struct eh_conf *mode_conf)
 		}
 	}
 
+	/* Stop and release tx adapters */
+	for (i = 0; i < em_conf->nb_tx_adapter; i++) {
+
+		id = em_conf->tx_adapter[i].adapter_id;
+		ret = rte_event_eth_tx_adapter_stop(id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error stopping tx adapter %d", id);
+			return ret;
+		}
+
+		for (j = 0; j < em_conf->tx_adapter[i].nb_connections; j++) {
+
+			ret = rte_event_eth_tx_adapter_queue_del(id,
+				em_conf->tx_adapter[i].conn[j].ethdev_id, -1);
+			if (ret < 0) {
+				EH_LOG_ERR(
+					"Error deleting tx adapter queues %d",
+					id);
+				return ret;
+			}
+		}
+
+		ret = rte_event_eth_tx_adapter_free(id);
+		if (ret < 0) {
+			EH_LOG_ERR("Error freeing tx adapter %d", id);
+			return ret;
+		}
+	}
+
 	return 0;
 }
+
+uint8_t
+eh_get_tx_queue(struct eh_conf *mode_conf, uint8_t eventdev_id)
+{
+	struct eventdev_params *eventdev_config;
+	struct eventmode_conf *em_conf;
+
+	if (mode_conf == NULL) {
+		EH_LOG_ERR("Invalid conf");
+		return -EINVAL;
+	}
+
+	if (mode_conf->mode_params == NULL) {
+		EH_LOG_ERR("Invalid mode params");
+		return -EINVAL;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	/* Get event device conf */
+	eventdev_config = eh_get_eventdev_params(em_conf, eventdev_id);
+
+	if (eventdev_config == NULL) {
+		EH_LOG_ERR("Error reading eventdev conf");
+		return -EINVAL;
+	}
+
+	/*
+	 * The last queue would be reserved to be used as atomic queue for the
+	 * last stage (eth packet tx stage)
+	 */
+	return eventdev_config->nb_eventqueue - 1;
+}
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index 4233b42..01e2aca 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -10,6 +10,16 @@ extern "C" {
 
 #include <rte_log.h>
 
+/**
+ * Flag to indicate that the event device used by all adapters is same
+ */
+#define EH_TX_EV_LINK_COMMON_EVENT_DEV		(1 << 0)
+
+/**
+ * Flag to indicate that the event queue to be used for all adapters is same
+ */
+#define EH_TX_EV_LINK_COMMON_EVENT_QUEUE	(1 << 1)
+
 #define RTE_LOGTYPE_EH  RTE_LOGTYPE_USER4
 
 #define EH_LOG_ERR(...) \
@@ -23,9 +33,15 @@ extern "C" {
 /* Max Rx adapters supported */
 #define EVENT_MODE_MAX_RX_ADAPTERS RTE_EVENT_MAX_DEVS
 
+/* Max Tx adapters supported */
+#define EVENT_MODE_MAX_TX_ADAPTERS RTE_EVENT_MAX_DEVS
+
 /* Max Rx adapter connections */
 #define EVENT_MODE_MAX_CONNECTIONS_PER_ADAPTER 16
 
+/* Max Tx adapter connections */
+#define EVENT_MODE_MAX_CONNECTIONS_PER_TX_ADAPTER 16
+
 /* Max event queues supported per event device */
 #define EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV RTE_EVENT_MAX_QUEUES_PER_DEV
 
@@ -33,6 +49,9 @@ extern "C" {
 #define EVENT_MODE_MAX_LCORE_LINKS \
 	(EVENT_MODE_MAX_EVENT_DEVS * EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV)
 
+/* Max adapters that one Tx core can handle */
+#define EVENT_MODE_MAX_ADAPTERS_PER_TX_CORE EVENT_MODE_MAX_TX_ADAPTERS
+
 /**
  * Packet transfer mode of the application
  */
@@ -80,6 +99,23 @@ struct rx_adapter_conf {
 			conn[EVENT_MODE_MAX_CONNECTIONS_PER_ADAPTER];
 };
 
+/* Tx adapter connection info */
+struct tx_adapter_connection_info {
+	uint8_t ethdev_id;
+	int32_t ethdev_tx_qid;
+};
+
+/* Tx adapter conf */
+struct tx_adapter_conf {
+	int32_t eventdev_id;
+	int32_t adapter_id;
+	uint32_t tx_core_id;
+	uint8_t nb_connections;
+	struct tx_adapter_connection_info
+			conn[EVENT_MODE_MAX_CONNECTIONS_PER_TX_ADAPTER];
+	uint8_t tx_ev_queue;
+};
+
 /* Eventmode conf data */
 struct eventmode_conf {
 	int nb_eventdev;
@@ -90,6 +126,10 @@ struct eventmode_conf {
 		/**< No of Rx adapters */
 	struct rx_adapter_conf rx_adapter[EVENT_MODE_MAX_RX_ADAPTERS];
 		/**< Rx adapter conf */
+	uint8_t nb_tx_adapter;
+		/**< No of Tx adapters */
+	struct tx_adapter_conf tx_adapter[EVENT_MODE_MAX_TX_ADAPTERS];
+		/** Tx adapter conf */
 	uint8_t nb_link;
 		/**< No of links */
 	struct eh_event_link_info
@@ -170,6 +210,24 @@ eh_devs_init(struct eh_conf *mode_conf);
 int32_t
 eh_devs_uninit(struct eh_conf *mode_conf);
 
+/**
+ * Get eventdev tx queue
+ *
+ * If the application uses event device which does not support internal port
+ * then it needs to submit the events to a Tx queue before final transmission.
+ * This Tx queue will be created internally by the eventmode helper subsystem,
+ * and application will need its queue ID when it runs the execution loop.
+ *
+ * @param mode_conf
+ *   Configuration of the mode in which app is doing packet handling
+ * @param eventdev_id
+ *   Event device ID
+ * @return
+ *   Tx queue ID
+ */
+uint8_t
+eh_get_tx_queue(struct eh_conf *mode_conf, uint8_t eventdev_id);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.7.4

[dpdk-dev] [RFC PATCH 05/13] examples/ipsec-secgw: add routines to display config

[Anoob Joseph]

Add routines to display the eventmode configuration. This gives an
overview of the devices used.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 207 ++++++++++++++++++++++++++++++++++++
 examples/ipsec-secgw/event_helper.h |  14 +++
 2 files changed, 221 insertions(+)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index 7745fbb..c43360e 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -868,6 +868,210 @@ eh_initialize_tx_adapter(struct eventmode_conf *em_conf)
 	return 0;
 }
 
+static void
+eh_display_operating_mode(struct eventmode_conf *em_conf)
+{
+	char sched_types[][32] = {
+		"RTE_SCHED_TYPE_ORDERED",
+		"RTE_SCHED_TYPE_ATOMIC",
+		"RTE_SCHED_TYPE_PARALLEL",
+	};
+	EH_LOG_INFO("Operating mode:");
+
+	EH_LOG_INFO("\tScheduling type: \t%s",
+		sched_types[em_conf->ext_params.sched_type]);
+
+	EH_LOG_INFO("");
+}
+
+static void
+eh_display_event_dev_conf(struct eventmode_conf *em_conf)
+{
+	int i;
+	char print_buf[256] = { 0 };
+	char queue_mode[][32] = {
+		"",
+		"ATQ (ALL TYPE QUEUE)",
+		"SINGLE LINK",
+	};
+
+	EH_LOG_INFO("Event Device Configuration:");
+
+	for (i = 0; i < em_conf->nb_eventdev; i++) {
+		sprintf(print_buf,
+			"\tDev ID: %-2d \tQueues: %-2d \tPorts: %-2d",
+			em_conf->eventdev_config[i].eventdev_id,
+			em_conf->eventdev_config[i].nb_eventqueue,
+			em_conf->eventdev_config[i].nb_eventport);
+		sprintf(print_buf + strlen(print_buf),
+			"\tQueue mode: %s",
+			queue_mode[em_conf->eventdev_config[i].ev_queue_mode]);
+		EH_LOG_INFO("%s", print_buf);
+	}
+	EH_LOG_INFO("");
+}
+
+static void
+eh_display_rx_adapter_conf(struct eventmode_conf *em_conf)
+{
+	int i, j;
+	int nb_rx_adapter = em_conf->nb_rx_adapter;
+	struct rx_adapter_conf *adapter;
+	struct rx_adapter_connection_info *conn;
+	char print_buf[256] = { 0 };
+
+	EH_LOG_INFO("Rx adapters configured: %d", nb_rx_adapter);
+
+	for (i = 0; i < nb_rx_adapter; i++) {
+		adapter = &(em_conf->rx_adapter[i]);
+		EH_LOG_INFO(
+			"\tRx adaper ID: %-2d\tConnections: %-2d\tEvent dev ID: %-2d"
+			"\tRx core: %-2d",
+			adapter->adapter_id,
+			adapter->nb_connections,
+			adapter->eventdev_id,
+			adapter->rx_core_id);
+
+		for (j = 0; j < adapter->nb_connections; j++) {
+			conn = &(adapter->conn[j]);
+
+			sprintf(print_buf,
+				"\t\tEthdev ID: %-2d", conn->ethdev_id);
+
+			if (conn->ethdev_rx_qid == -1)
+				sprintf(print_buf + strlen(print_buf),
+					"\tEth rx queue: %-2s", "ALL");
+			else
+				sprintf(print_buf + strlen(print_buf),
+					"\tEth rx queue: %-2d",
+					conn->ethdev_rx_qid);
+
+			sprintf(print_buf + strlen(print_buf),
+				"\tEvent queue: %-2d", conn->eventq_id);
+			EH_LOG_INFO("%s", print_buf);
+		}
+	}
+	EH_LOG_INFO("");
+}
+
+static void
+eh_display_tx_adapter_conf(struct eventmode_conf *em_conf)
+{
+	int i, j;
+	int nb_tx_adapter = em_conf->nb_tx_adapter;
+	struct tx_adapter_conf *adapter;
+	struct tx_adapter_connection_info *conn;
+	char print_buf[256] = { 0 };
+
+	EH_LOG_INFO("Tx adapters configured: %d", nb_tx_adapter);
+
+	for (i = 0; i < nb_tx_adapter; i++) {
+		adapter = &(em_conf->tx_adapter[i]);
+		sprintf(print_buf,
+			"\tTx adapter ID: %-2d\tConnections: %-2d\tEvent dev ID: %-2d",
+			adapter->adapter_id,
+			adapter->nb_connections,
+			adapter->eventdev_id);
+		if (adapter->tx_core_id == (uint32_t)-1)
+			sprintf(print_buf + strlen(print_buf),
+				"\tTx core: %-2s", "[INTERNAL PORT]");
+		else if (adapter->tx_core_id == RTE_MAX_LCORE)
+			sprintf(print_buf + strlen(print_buf),
+				"\tTx core: %-2s", "[NONE]");
+		else
+			sprintf(print_buf + strlen(print_buf),
+				"\tTx core: %-2d,\tInput event queue: %-2d",
+				adapter->tx_core_id, adapter->tx_ev_queue);
+
+		EH_LOG_INFO("%s", print_buf);
+
+		for (j = 0; j < adapter->nb_connections; j++) {
+			conn = &(adapter->conn[j]);
+
+			sprintf(print_buf,
+				"\t\tEthdev ID: %-2d", conn->ethdev_id);
+
+			if (conn->ethdev_tx_qid == -1)
+				sprintf(print_buf + strlen(print_buf),
+					"\tEth tx queue: %-2s", "ALL");
+			else
+				sprintf(print_buf + strlen(print_buf),
+					"\tEth tx queue: %-2d",
+					conn->ethdev_tx_qid);
+			EH_LOG_INFO("%s", print_buf);
+		}
+	}
+	EH_LOG_INFO("");
+}
+
+static void
+eh_display_link_conf(struct eventmode_conf *em_conf)
+{
+	int i;
+	struct eh_event_link_info *link;
+	char print_buf[256] = { 0 };
+
+	EH_LOG_INFO("Links configured: %d", em_conf->nb_link);
+
+	for (i = 0; i < em_conf->nb_link; i++) {
+		link = &(em_conf->link[i]);
+
+		sprintf(print_buf,
+			"\tEvent dev ID: %-2d\tEvent port: %-2d",
+			link->eventdev_id,
+			link->event_portid);
+
+		if (em_conf->ext_params.all_ev_queue_to_ev_port)
+			sprintf(print_buf + strlen(print_buf),
+				"Event queue: %-2s\t", "ALL");
+		else
+			sprintf(print_buf + strlen(print_buf),
+				"Event queue: %-2d\t", link->eventq_id);
+
+		sprintf(print_buf + strlen(print_buf),
+			"Lcore: %-2d", link->lcore_id);
+		EH_LOG_INFO("%s", print_buf);
+	}
+	EH_LOG_INFO("");
+}
+
+void
+eh_display_conf(struct eh_conf *mode_conf)
+{
+	struct eventmode_conf *em_conf;
+
+	if (mode_conf == NULL) {
+		EH_LOG_INFO("Invalid conf");
+		return;
+	}
+
+	if (mode_conf->mode != EH_PKT_TRANSFER_MODE_EVENT)
+		return;
+
+	if (mode_conf->mode_params == NULL) {
+		EH_LOG_INFO("Invalid mode params");
+		return;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	/* Display user exposed operating modes */
+	eh_display_operating_mode(em_conf);
+
+	/* Display event device conf */
+	eh_display_event_dev_conf(em_conf);
+
+	/* Display Rx adapter conf */
+	eh_display_rx_adapter_conf(em_conf);
+
+	/* Display Tx adapter conf */
+	eh_display_tx_adapter_conf(em_conf);
+
+	/* Display event-lcore link */
+	eh_display_link_conf(em_conf);
+}
+
 int32_t
 eh_devs_init(struct eh_conf *mode_conf)
 {
@@ -900,6 +1104,9 @@ eh_devs_init(struct eh_conf *mode_conf)
 		return -EINVAL;
 	}
 
+	/* Display the current configuration */
+	eh_display_conf(mode_conf);
+
 	/* Stop eth devices before setting up adapter */
 	RTE_ETH_FOREACH_DEV(portid) {
 
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index 01e2aca..c7fdd8f 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -27,6 +27,11 @@ extern "C" {
 		RTE_FMT("%s() line %u: " RTE_FMT_HEAD(__VA_ARGS__ ,) "\n", \
 			__func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__ ,)))
 
+#define EH_LOG_INFO(...) \
+	RTE_LOG(INFO, EH, \
+		RTE_FMT("%s() line %u: " RTE_FMT_HEAD(__VA_ARGS__ ,) "\n", \
+			__func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__ ,)))
+
 /* Max event devices supported */
 #define EVENT_MODE_MAX_EVENT_DEVS RTE_EVENT_MAX_DEVS
 
@@ -228,6 +233,15 @@ eh_devs_uninit(struct eh_conf *mode_conf);
 uint8_t
 eh_get_tx_queue(struct eh_conf *mode_conf, uint8_t eventdev_id);
 
+/**
+ * Display event mode configuration
+ *
+ * @param mode_conf
+ *   Configuration of the mode in which app is doing packet handling
+ */
+void
+eh_display_conf(struct eh_conf *mode_conf);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.7.4

[dpdk-dev] [RFC PATCH 06/13] examples/ipsec-secgw: add routines to launch workers

[Anoob Joseph]

With eventmode, workers can be drafted differently according to the
capabilities of the underlying event device. The added functions will
receive an array of such workers and probe the eventmode properties to
choose the worker.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 347 +++++++++++++++++++++++++++++++++++-
 examples/ipsec-secgw/event_helper.h |  48 +++++
 2 files changed, 391 insertions(+), 4 deletions(-)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index c43360e..6993092 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -1,6 +1,7 @@
 /* SPDX-License-Identifier: BSD-3-Clause
  * Copyright (C) 2019 Marvell International Ltd.
  */
+#include <stdbool.h>
 #include <rte_event_eth_rx_adapter.h>
 #include <rte_event_eth_tx_adapter.h>
 #include <rte_ethdev.h>
@@ -9,6 +10,8 @@
 
 #include "event_helper.h"
 
+static volatile bool eth_core_running;
+
 static int
 eh_get_enabled_cores(struct rte_bitmap *eth_core_mask)
 {
@@ -98,6 +101,16 @@ eh_get_eventdev_params(struct eventmode_conf *em_conf,
 
 	return &(em_conf->eventdev_config[i]);
 }
+static inline bool
+eh_dev_has_burst_mode(uint8_t dev_id)
+{
+	struct rte_event_dev_info dev_info;
+
+	rte_event_dev_info_get(dev_id, &dev_info);
+	return (dev_info.event_dev_cap & RTE_EVENT_DEV_CAP_BURST_MODE) ?
+			true : false;
+}
+
 static int
 eh_validate_user_params(struct eventmode_conf *em_conf)
 {
@@ -692,10 +705,8 @@ eh_rx_adapter_configure(struct eventmode_conf *em_conf,
 	}
 
 	/*
-	 * TODO
 	 * Rx core will invoke the service when required. The runstate check
 	 * is not required.
-	 *
 	 */
 	rte_service_set_runstate_mapped_check(service_id, 0);
 
@@ -727,6 +738,262 @@ eh_initialize_rx_adapter(struct eventmode_conf *em_conf)
 	return 0;
 }
 
+static int32_t
+eh_start_worker_eth_core(struct eventmode_conf *em_conf, uint32_t lcore_id)
+{
+	uint32_t service_id[EVENT_MODE_MAX_ADAPTERS_PER_RX_CORE];
+	struct rx_adapter_conf *rx_adapter;
+	struct tx_adapter_conf *tx_adapter;
+	int service_count = 0;
+	int adapter_id;
+	int32_t ret;
+	int i;
+
+	EH_LOG_INFO("Entering eth_core processing on lcore %u", lcore_id);
+
+	/*
+	 * Need to parse adapter conf to see which of all Rx adapters need
+	 * to be handled by this core.
+	 */
+	for (i = 0; i < em_conf->nb_rx_adapter; i++) {
+		/* Check if we have exceeded the max allowed */
+		if (service_count > EVENT_MODE_MAX_ADAPTERS_PER_RX_CORE) {
+			EH_LOG_ERR(
+			      "Exceeded the max allowed adapters per rx core");
+			break;
+		}
+
+		rx_adapter = &(em_conf->rx_adapter[i]);
+		if (rx_adapter->rx_core_id != lcore_id)
+			continue;
+
+		/* Adapter need to be handled by this core */
+		adapter_id = rx_adapter->adapter_id;
+
+		/* Get the service ID for the adapters */
+		ret = rte_event_eth_rx_adapter_service_id_get(adapter_id,
+				&(service_id[service_count]));
+
+		if (ret != -ESRCH && ret != 0) {
+			EH_LOG_ERR(
+				"Error getting service ID used by Rx adapter");
+			return ret;
+		}
+
+		/* Update service count */
+		service_count++;
+	}
+
+	/*
+	 * Need to parse adapter conf to see which all Tx adapters need to be
+	 * handled this core.
+	 */
+	for (i = 0; i < em_conf->nb_tx_adapter; i++) {
+		/* Check if we have exceeded the max allowed */
+		if (service_count > EVENT_MODE_MAX_ADAPTERS_PER_TX_CORE) {
+			EH_LOG_ERR(
+				"Exceeded the max allowed adapters per Tx core");
+			break;
+		}
+
+		tx_adapter = &(em_conf->tx_adapter[i]);
+		if (tx_adapter->tx_core_id != lcore_id)
+			continue;
+
+		/* Adapter need to be handled by this core */
+		adapter_id = tx_adapter->adapter_id;
+
+		/* Get the service ID for the adapters */
+		ret = rte_event_eth_tx_adapter_service_id_get(adapter_id,
+				&(service_id[service_count]));
+
+		if (ret != -ESRCH && ret != 0) {
+			EH_LOG_ERR(
+				"Error getting service ID used by Tx adapter");
+			return ret;
+		}
+
+		/* Update service count */
+		service_count++;
+	}
+
+	eth_core_running = true;
+
+	while (eth_core_running) {
+		for (i = 0; i < service_count; i++) {
+			/* Initiate adapter service */
+			rte_service_run_iter_on_app_lcore(service_id[i], 0);
+		}
+	}
+
+	return 0;
+}
+
+static int32_t
+eh_stop_worker_eth_core(void)
+{
+	if (eth_core_running) {
+		EH_LOG_INFO("Stopping eth cores");
+		eth_core_running = false;
+	}
+	return 0;
+}
+
+static struct eh_app_worker_params *
+eh_find_worker(uint32_t lcore_id, struct eh_conf *conf,
+		struct eh_app_worker_params *app_wrkrs, uint8_t nb_wrkr_param)
+{
+	struct eh_event_link_info *link = NULL;
+	struct eventmode_conf *em_conf;
+	uint8_t eventdev_id;
+	int i;
+	struct eh_app_worker_params curr_conf = {
+			{{0} }, NULL};
+	struct eh_app_worker_params *tmp_wrkr;
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(conf->mode_params);
+
+	/*
+	 * Event device to be used will be derived from the first lcore-event
+	 * link.
+	 *
+	 * Assumption: All lcore-event links tied to a core would be using the
+	 * same event device. in other words, one core would be polling on
+	 * queues of a single event device only.
+	 */
+
+	/* Get a link for this lcore */
+	for (i = 0; i < em_conf->nb_link; i++) {
+		link = &(em_conf->link[i]);
+		if (link->lcore_id == lcore_id)
+			break;
+	}
+
+	if (link == NULL) {
+		EH_LOG_ERR(
+			"No valid link found for lcore(%d)", lcore_id);
+		return NULL;
+	}
+
+	/* Get event dev ID */
+	eventdev_id = link->eventdev_id;
+
+	/* Populate the curr_conf with the capabilities */
+
+	/* Check for burst mode */
+	if (eh_dev_has_burst_mode(eventdev_id))
+		curr_conf.cap.burst = EH_RX_TYPE_BURST;
+	else
+		curr_conf.cap.burst = EH_RX_TYPE_NON_BURST;
+
+	/* Now parse the passed list and see if we have matching capabilities */
+
+	/* Initialize the pointer used to traverse the list */
+	tmp_wrkr = app_wrkrs;
+
+	for (i = 0; i < nb_wrkr_param; i++, tmp_wrkr++) {
+
+		/* Skip this if capabilities are not matching */
+		if (tmp_wrkr->cap.u64 != curr_conf.cap.u64)
+			continue;
+
+		/* If the checks pass, we have a match */
+		return tmp_wrkr;
+	}
+
+	return NULL;
+}
+
+static int
+eh_verify_match_worker(
+	struct eh_app_worker_params *match_wrkr)
+{
+	/* Verify registered worker */
+	if (match_wrkr->worker_thread == NULL) {
+		EH_LOG_ERR("No worker registered for second stage");
+		return 0;
+	}
+
+	/* Success */
+	return 1;
+}
+
+static uint8_t
+eh_get_event_lcore_links(uint32_t lcore_id, struct eh_conf *mode_conf,
+		struct eh_event_link_info **links)
+{
+	int i;
+	int index = 0;
+	uint8_t lcore_nb_link = 0;
+	struct eh_event_link_info *link;
+	struct eh_event_link_info *link_cache;
+	struct eventmode_conf *em_conf = NULL;
+	size_t cache_size;
+	size_t single_link_size;
+
+	if (mode_conf == NULL || links == NULL) {
+		EH_LOG_ERR("Invalid args");
+		return -EINVAL;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	if (em_conf == NULL) {
+		EH_LOG_ERR("Invalid event mode conf");
+		return -EINVAL;
+	}
+
+	/* Get the number of links registered */
+	for (i = 0; i < em_conf->nb_link; i++) {
+
+		/* Get link */
+		link = &(em_conf->link[i]);
+
+		/* Check if we have link intended for this lcore */
+		if (link->lcore_id == lcore_id) {
+
+			/* Update the number of links for this core */
+			lcore_nb_link++;
+
+		}
+	}
+
+	/* Compute size of one entry to be copied */
+	single_link_size = sizeof(struct eh_event_link_info);
+
+	/* Compute size of the buffer required */
+	cache_size = lcore_nb_link *
+			sizeof(struct eh_event_link_info);
+
+	/* Compute size of the buffer required */
+	link_cache = calloc(1, cache_size);
+
+	/* Get the number of links registered */
+	for (i = 0; i < em_conf->nb_link; i++) {
+
+		/* Get link */
+		link = &(em_conf->link[i]);
+
+		/* Check if we have link intended for this lcore */
+		if (link->lcore_id == lcore_id) {
+
+			/* Cache the link */
+			memcpy(&link_cache[index], link, single_link_size);
+
+			/* Update index */
+			index++;
+		}
+	}
+
+	/* Update the links for application to use the cached links */
+	*links = link_cache;
+
+	/* Return the number of cached links */
+	return lcore_nb_link;
+}
+
 static int
 eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 	struct tx_adapter_conf *adapter)
@@ -833,10 +1100,8 @@ eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 	}
 
 	/*
-	 * TODO
 	 * Tx core will invoke the service when required. The runstate check
 	 * is not required.
-	 *
 	 */
 	rte_service_set_runstate_mapped_check(service_id, 0);
 
@@ -1246,6 +1511,80 @@ eh_devs_uninit(struct eh_conf *mode_conf)
 	return 0;
 }
 
+void
+eh_launch_worker(struct eh_conf *mode_conf,
+		struct eh_app_worker_params *app_wrkr, uint8_t nb_wrkr_param)
+{
+	struct eh_app_worker_params *match_wrkr;
+	struct eh_event_link_info *links = NULL;
+	struct eventmode_conf *em_conf;
+	uint32_t lcore_id;
+	uint8_t nb_links;
+
+	if (mode_conf == NULL) {
+		EH_LOG_ERR("Invalid conf");
+		return;
+	}
+
+	if (mode_conf->mode_params == NULL) {
+		EH_LOG_ERR("Invalid mode params");
+		return;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(mode_conf->mode_params);
+
+	/* Get core ID */
+	lcore_id = rte_lcore_id();
+
+	/* Check if this is eth core */
+	if (rte_bitmap_get(em_conf->eth_core_mask, lcore_id)) {
+		eh_start_worker_eth_core(em_conf, lcore_id);
+		return;
+	}
+
+	if (app_wrkr == NULL || nb_wrkr_param == 0) {
+		EH_LOG_ERR("Invalid args");
+		return;
+	}
+
+	/*
+	 * This is a regular worker thread. The application would be
+	 * registering multiple workers with various capabilities. The
+	 * worker to be run will be selected by the capabilities of the
+	 * event device configured.
+	 */
+
+	/* Get the first matching worker for the event device */
+	match_wrkr = eh_find_worker(lcore_id, mode_conf, app_wrkr,
+				    nb_wrkr_param);
+	if (match_wrkr == NULL) {
+		EH_LOG_ERR(
+		       "No matching worker registered for lcore %d", lcore_id);
+		goto clean_and_exit;
+	}
+
+	/* Verify sanity of the matched worker */
+	if (eh_verify_match_worker(match_wrkr) != 1) {
+		EH_LOG_ERR("Error in validating the matched worker\n");
+		goto clean_and_exit;
+	}
+
+	/* Get worker links */
+	nb_links = eh_get_event_lcore_links(lcore_id, mode_conf, &links);
+
+	/* Launch the worker thread */
+	match_wrkr->worker_thread(mode_conf, links, nb_links);
+
+	/* Free links info memory */
+	free(links);
+
+clean_and_exit:
+
+	/* Flag eth_cores to stop, if started */
+	eh_stop_worker_eth_core();
+}
+
 uint8_t
 eh_get_tx_queue(struct eh_conf *mode_conf, uint8_t eventdev_id)
 {
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index c7fdd8f..1d5a087 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -54,6 +54,9 @@ extern "C" {
 #define EVENT_MODE_MAX_LCORE_LINKS \
 	(EVENT_MODE_MAX_EVENT_DEVS * EVENT_MODE_MAX_EVENT_QUEUES_PER_DEV)
 
+/* Max adapters that one Rx core can handle */
+#define EVENT_MODE_MAX_ADAPTERS_PER_RX_CORE EVENT_MODE_MAX_RX_ADAPTERS
+
 /* Max adapters that one Tx core can handle */
 #define EVENT_MODE_MAX_ADAPTERS_PER_TX_CORE EVENT_MODE_MAX_TX_ADAPTERS
 
@@ -65,6 +68,14 @@ enum eh_pkt_transfer_mode {
 	EH_PKT_TRANSFER_MODE_EVENT,
 };
 
+/**
+ * Event mode packet rx types
+ */
+enum eh_rx_types {
+	EH_RX_TYPE_NON_BURST = 0,
+	EH_RX_TYPE_BURST
+};
+
 /* Event dev params */
 struct eventdev_params {
 	uint8_t eventdev_id;
@@ -175,6 +186,22 @@ struct eh_conf {
 		/**< Mode specific parameters */
 };
 
+/* Workers registered by the application */
+struct eh_app_worker_params {
+	union {
+		RTE_STD_C11
+		struct {
+			uint64_t burst : 1;
+			/**< Specify status of rx type burst */
+		};
+		uint64_t u64;
+	} cap;
+			/**< Capabilities of this worker */
+	void (*worker_thread)(struct eh_conf *mode_conf,
+			struct eh_event_link_info *links, uint8_t nb_links);
+			/**< Worker thread */
+};
+
 /**
  * Initialize event mode devices
  *
@@ -242,6 +269,27 @@ eh_get_tx_queue(struct eh_conf *mode_conf, uint8_t eventdev_id);
 void
 eh_display_conf(struct eh_conf *mode_conf);
 
+
+/**
+ * Launch eventmode worker
+ *
+ * The application can request the eventmode helper subsystem to launch the
+ * worker based on the capabilities of event device and the options selected
+ * while initializing the eventmode.
+ *
+ * @param mode_conf
+ *   Configuration of the mode in which app is doing packet handling
+ * @param app_wrkr
+ *   List of all the workers registered by application, along with its
+ *   capabilities
+ * @param nb_wrkr_param
+ *   Number of workers passed by the application
+ *
+ */
+void
+eh_launch_worker(struct eh_conf *mode_conf,
+		struct eh_app_worker_params *app_wrkr, uint8_t nb_wrkr_param);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.7.4

[dpdk-dev] [RFC PATCH 07/13] examples/ipsec-secgw: add support for internal ports

[Anoob Joseph]

Add support for Rx and Tx internal ports. When internal ports are
available then a packet can be received from eth port and forwarded
to event queue by HW without any software intervention. The same
applies to Tx side where a packet sent to an event queue can by
forwarded by HW to eth port without any software intervention.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/event_helper.c | 234 ++++++++++++++++++++++++++++--------
 examples/ipsec-secgw/event_helper.h |  11 ++
 2 files changed, 195 insertions(+), 50 deletions(-)

diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index 6993092..858b855 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -102,7 +102,38 @@ eh_get_eventdev_params(struct eventmode_conf *em_conf,
 	return &(em_conf->eventdev_config[i]);
 }
 static inline bool
-eh_dev_has_burst_mode(uint8_t dev_id)
+eh_dev_has_rx_internal_port(uint8_t eventdev_id)
+{
+	int j;
+	bool flag = true;
+
+	RTE_ETH_FOREACH_DEV(j) {
+		uint32_t caps = 0;
+
+		rte_event_eth_rx_adapter_caps_get(eventdev_id, j, &caps);
+		if (!(caps & RTE_EVENT_ETH_RX_ADAPTER_CAP_INTERNAL_PORT))
+			flag = false;
+	}
+	return flag;
+}
+
+static inline bool
+eh_dev_has_tx_internal_port(uint8_t eventdev_id)
+{
+	int j;
+	bool flag = true;
+
+	RTE_ETH_FOREACH_DEV(j) {
+		uint32_t caps = 0;
+
+		rte_event_eth_tx_adapter_caps_get(eventdev_id, j, &caps);
+		if (!(caps & RTE_EVENT_ETH_TX_ADAPTER_CAP_INTERNAL_PORT))
+			flag = false;
+	}
+	return flag;
+}
+
+static inline bool eh_dev_has_burst_mode(uint8_t dev_id)
 {
 	struct rte_event_dev_info dev_info;
 
@@ -127,6 +158,8 @@ eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
 {
 	int i, ret;
 	int nb_eventdev;
+	int nb_eth_dev;
+	int lcore_count;
 	struct eventdev_params *eventdev_config;
 	struct rte_event_dev_info dev_info;
 
@@ -138,6 +171,17 @@ eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
 		return -EINVAL;
 	}
 
+	/* Get the number of eth devs */
+	nb_eth_dev = rte_eth_dev_count_avail();
+
+	if (nb_eth_dev == 0) {
+		EH_LOG_ERR("No eth devices detected");
+		return -EINVAL;
+	}
+
+	/* Get the number of lcores */
+	lcore_count = rte_lcore_count();
+
 	for (i = 0; i < nb_eventdev; i++) {
 
 		/* Get the event dev conf */
@@ -163,13 +207,19 @@ eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
 		eventdev_config->nb_eventqueue = dev_info.max_event_queues;
 		eventdev_config->nb_eventport = dev_info.max_event_ports;
 		eventdev_config->ev_queue_mode =
-				RTE_EVENT_QUEUE_CFG_SINGLE_LINK;
+				RTE_EVENT_QUEUE_CFG_ALL_TYPES;
 
-		/* One port is required for eth Rx adapter */
-		eventdev_config->nb_eventport -= 1;
+		/* Check if there are more queues than required */
+		if (eventdev_config->nb_eventqueue > nb_eth_dev + 1) {
+			/* One queue is reserved for Tx */
+			eventdev_config->nb_eventqueue = nb_eth_dev + 1;
+		}
 
-		/* One port is reserved for eth Tx adapter */
-		eventdev_config->nb_eventport -= 1;
+		/* Check if there are more ports than required */
+		if (eventdev_config->nb_eventport > lcore_count) {
+			/* One port per lcore is enough */
+			eventdev_config->nb_eventport = lcore_count;
+		}
 
 		/* Update the number of eventdevs */
 		em_conf->nb_eventdev++;
@@ -178,6 +228,42 @@ eh_set_default_conf_eventdev(struct eventmode_conf *em_conf)
 	return 0;
 }
 
+static void
+eh_do_capability_check(struct eventmode_conf *em_conf)
+{
+	struct eventdev_params *eventdev_config;
+	uint32_t eventdev_id;
+	int all_internal_ports = 1;
+	int i;
+
+	for (i = 0; i < em_conf->nb_eventdev; i++) {
+
+		/* Get the event dev conf */
+		eventdev_config = &(em_conf->eventdev_config[i]);
+		eventdev_id = eventdev_config->eventdev_id;
+
+		/* Check if event device has internal port for Rx & Tx */
+		if (eh_dev_has_rx_internal_port(eventdev_id) &&
+		    eh_dev_has_tx_internal_port(eventdev_id)) {
+			eventdev_config->all_internal_ports = 1;
+		} else {
+			all_internal_ports = 0;
+		}
+	}
+
+	/*
+	 * If Rx & Tx internal ports are supported by all event devices then
+	 * eth cores won't be required. Override the eth core mask requested
+	 * and decrement number of event queues by one as it won't be needed
+	 * for Tx.
+	 */
+	if (all_internal_ports) {
+		rte_bitmap_reset(em_conf->eth_core_mask);
+		for (i = 0; i < em_conf->nb_eventdev; i++)
+			em_conf->eventdev_config[i].nb_eventqueue--;
+	}
+}
+
 static int
 eh_set_default_conf_link(struct eventmode_conf *em_conf)
 {
@@ -254,9 +340,12 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 	int adapter_id;
 	int eventdev_id;
 	int conn_id;
+	int nb_eventqueue;
 	struct rx_adapter_conf *adapter;
 	struct rx_adapter_connection_info *conn;
 	struct eventdev_params *eventdev_config;
+	bool rx_internal_port = true;
+	uint32_t caps = 0;
 
 	/* Create one adapter with all eth queues mapped to event queues 1:1 */
 
@@ -281,7 +370,14 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 	/* Set adapter conf */
 	adapter->eventdev_id = eventdev_id;
 	adapter->adapter_id = adapter_id;
-	adapter->rx_core_id = eh_get_next_eth_core(em_conf);
+
+	/*
+	 * If event device does not have internal ports for passing
+	 * packets then one queue is reserved for Tx path
+	 */
+	nb_eventqueue = eventdev_config->all_internal_ports ?
+			eventdev_config->nb_eventqueue :
+			eventdev_config->nb_eventqueue - 1;
 
 	/*
 	 * All queues of one eth device (port) will be mapped to one event
@@ -290,10 +386,10 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 	 */
 
 	/* Make sure there is enough event queues for 1:1 mapping */
-	if (nb_eth_dev > eventdev_config->nb_eventqueue) {
+	if (nb_eth_dev > nb_eventqueue) {
 		EH_LOG_ERR("Not enough event queues for 1:1 mapping "
 			"[eth devs: %d, event queues: %d]\n",
-			nb_eth_dev, eventdev_config->nb_eventqueue);
+			nb_eth_dev, nb_eventqueue);
 		return -EINVAL;
 	}
 
@@ -316,11 +412,24 @@ eh_set_default_conf_rx_adapter(struct eventmode_conf *em_conf)
 		/* Add all eth queues of one eth port to one event queue */
 		conn->ethdev_rx_qid = -1;
 
+		/* Get Rx adapter capabilities */
+		rte_event_eth_rx_adapter_caps_get(eventdev_id, i, &caps);
+		if (!(caps & RTE_EVENT_ETH_RX_ADAPTER_CAP_INTERNAL_PORT))
+			rx_internal_port = false;
+
 		/* Update no of connections */
 		adapter->nb_connections++;
 
 	}
 
+	if (rx_internal_port) {
+		/* Rx core is not required */
+		adapter->rx_core_id = -1;
+	} else {
+		/* Rx core is required */
+		adapter->rx_core_id = eh_get_next_eth_core(em_conf);
+	}
+
 	/* We have setup one adapter */
 	em_conf->nb_rx_adapter = 1;
 
@@ -338,6 +447,8 @@ eh_set_default_conf_tx_adapter(struct eventmode_conf *em_conf)
 	struct eventdev_params *eventdev_config;
 	struct tx_adapter_conf *tx_adapter;
 	struct tx_adapter_connection_info *conn;
+	bool tx_internal_port = true;
+	uint32_t caps = 0;
 
 	/*
 	 * Create one Tx adapter with all eth queues mapped to event queues
@@ -366,22 +477,6 @@ eh_set_default_conf_tx_adapter(struct eventmode_conf *em_conf)
 	tx_adapter->eventdev_id = eventdev_id;
 	tx_adapter->adapter_id = adapter_id;
 
-	/* TODO: Tx core is required only when internal port is not present */
-
-	tx_adapter->tx_core_id = eh_get_next_eth_core(em_conf);
-
-	/*
-	 * Application would need to use one event queue per adapter for
-	 * submitting packets for Tx. Reserving the last queue available
-	 * and decrementing the total available event queues for this
-	 */
-
-	/* Queue numbers start at 0 */
-	tx_adapter->tx_ev_queue = eventdev_config->nb_eventqueue - 1;
-
-	/* Update the number of event queues available in eventdev */
-	eventdev_config->nb_eventqueue--;
-
 	/*
 	 * All Tx queues of the eth device (port) will be mapped to the event
 	 * device.
@@ -412,10 +507,30 @@ eh_set_default_conf_tx_adapter(struct eventmode_conf *em_conf)
 		/* Add all eth tx queues to adapter */
 		conn->ethdev_tx_qid = -1;
 
+		/* Get Rx adapter capabilities */
+		rte_event_eth_tx_adapter_caps_get(eventdev_id, i, &caps);
+		if (!(caps & RTE_EVENT_ETH_TX_ADAPTER_CAP_INTERNAL_PORT))
+			tx_internal_port = false;
+
 		/* Update no of connections */
 		tx_adapter->nb_connections++;
 	}
 
+	if (tx_internal_port) {
+		/* Tx core is not required */
+		tx_adapter->tx_core_id = -1;
+	} else {
+		/* Tx core is required */
+		tx_adapter->tx_core_id = eh_get_next_eth_core(em_conf);
+
+		/*
+		 * Application would need to use one event queue per adapter for
+		 * submitting packets for Tx. Reserving the last queue available
+		 */
+		/* Queue numbers start at 0 */
+		tx_adapter->tx_ev_queue = eventdev_config->nb_eventqueue - 1;
+	}
+
 	/* We have setup one adapter */
 	em_conf->nb_tx_adapter = 1;
 	return 0;
@@ -441,6 +556,9 @@ eh_validate_conf(struct eventmode_conf *em_conf)
 			return ret;
 	}
 
+	/* Perform capability check for the selected event devices*/
+	eh_do_capability_check(em_conf);
+
 	/*
 	 * See if links are specified. Else generate a default conf for
 	 * the event ports used.
@@ -500,10 +618,6 @@ eh_initialize_eventdev(struct eventmode_conf *em_conf)
 		/* Get the number of queues */
 		nb_eventqueue = eventdev_config->nb_eventqueue;
 
-		/* One queue is reserved for the final stage (doing eth tx) */
-		/* TODO handles only one Tx adapter. Fix this */
-		nb_eventqueue += 1;
-
 		/* Reset the default conf */
 		memset(&evdev_default_conf, 0,
 			sizeof(struct rte_event_dev_info));
@@ -550,11 +664,13 @@ eh_initialize_eventdev(struct eventmode_conf *em_conf)
 					eventdev_config->ev_queue_mode;
 			/*
 			 * All queues need to be set with sched_type as
-			 * schedule type for the application stage. One queue
-			 * would be reserved for the final eth tx stage. This
-			 * will be an atomic queue.
+			 * schedule type for the application stage. One
+			 * queue would be reserved for the final eth tx
+			 * stage if event device does not have internal
+			 * ports. This will be an atomic queue.
 			 */
-			if (j == nb_eventqueue-1) {
+			if (!eventdev_config->all_internal_ports &&
+			    j == nb_eventqueue-1) {
 				eventq_conf.schedule_type =
 					RTE_SCHED_TYPE_ATOMIC;
 			} else {
@@ -669,10 +785,6 @@ eh_rx_adapter_configure(struct eventmode_conf *em_conf,
 	}
 
 	/* Setup various connections in the adapter */
-
-	queue_conf.rx_queue_flags =
-			RTE_EVENT_ETH_RX_ADAPTER_QUEUE_FLOW_ID_VALID;
-
 	for (j = 0; j < adapter->nb_connections; j++) {
 		/* Get connection */
 		conn = &(adapter->conn[j]);
@@ -680,9 +792,7 @@ eh_rx_adapter_configure(struct eventmode_conf *em_conf,
 		/* Setup queue conf */
 		queue_conf.ev.queue_id = conn->eventq_id;
 		queue_conf.ev.sched_type = em_conf->ext_params.sched_type;
-
-		/* Set flow ID as ethdev ID */
-		queue_conf.ev.flow_id = conn->ethdev_id;
+		queue_conf.ev.event_type = RTE_EVENT_TYPE_ETHDEV;
 
 		/* Add queue to the adapter */
 		ret = rte_event_eth_rx_adapter_queue_add(
@@ -881,6 +991,12 @@ eh_find_worker(uint32_t lcore_id, struct eh_conf *conf,
 
 	/* Populate the curr_conf with the capabilities */
 
+	/* Check for Tx internal port */
+	if (eh_dev_has_tx_internal_port(eventdev_id))
+		curr_conf.cap.tx_internal_port = EH_TX_TYPE_INTERNAL_PORT;
+	else
+		curr_conf.cap.tx_internal_port = EH_TX_TYPE_NO_INTERNAL_PORT;
+
 	/* Check for burst mode */
 	if (eh_dev_has_burst_mode(eventdev_id))
 		curr_conf.cap.burst = EH_RX_TYPE_BURST;
@@ -1057,6 +1173,18 @@ eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 		}
 	}
 
+	/*
+	 * Check if Tx core is assigned. If Tx core is not assigned, then
+	 * the adapter would be having internal port for submitting packets
+	 * for Tx and so Tx event queue & port setup is not required
+	 */
+	if (adapter->tx_core_id == (uint32_t) (-1)) {
+		/* Internal port is present */
+		goto skip_tx_queue_port_setup;
+	}
+
+	/* Setup Tx queue & port */
+
 	/* Get event port used by the adapter */
 	ret = rte_event_eth_tx_adapter_event_port_get(
 			adapter->adapter_id,
@@ -1067,11 +1195,6 @@ eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 	}
 
 	/*
-	 * TODO: event queue for Tx adapter is required only if the
-	 * INTERNAL PORT is not present.
-	 */
-
-	/*
 	 * Tx event queue would be reserved for Tx adapter. Need to unlink
 	 * this queue from all other ports
 	 *
@@ -1081,6 +1204,7 @@ eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 				      &(adapter->tx_ev_queue), 1);
 	}
 
+	/* Link Tx event queue to Tx port */
 	ret = rte_event_port_link(
 			eventdev_id,
 			tx_port_id,
@@ -1105,6 +1229,7 @@ eh_tx_adapter_configure(struct eventmode_conf *em_conf,
 	 */
 	rte_service_set_runstate_mapped_check(service_id, 0);
 
+skip_tx_queue_port_setup:
 	/* Start adapter */
 	ret = rte_event_eth_tx_adapter_start(adapter->adapter_id);
 	if (ret) {
@@ -1189,13 +1314,22 @@ eh_display_rx_adapter_conf(struct eventmode_conf *em_conf)
 
 	for (i = 0; i < nb_rx_adapter; i++) {
 		adapter = &(em_conf->rx_adapter[i]);
-		EH_LOG_INFO(
-			"\tRx adaper ID: %-2d\tConnections: %-2d\tEvent dev ID: %-2d"
-			"\tRx core: %-2d",
+		sprintf(print_buf,
+			"\tRx adaper ID: %-2d\tConnections: %-2d\tEvent dev ID: %-2d",
 			adapter->adapter_id,
 			adapter->nb_connections,
-			adapter->eventdev_id,
-			adapter->rx_core_id);
+			adapter->eventdev_id);
+		if (adapter->rx_core_id == (uint32_t)-1)
+			sprintf(print_buf + strlen(print_buf),
+				"\tRx core: %-2s", "[INTERNAL PORT]");
+		else if (adapter->rx_core_id == RTE_MAX_LCORE)
+			sprintf(print_buf + strlen(print_buf),
+				"\tRx core: %-2s", "[NONE]");
+		else
+			sprintf(print_buf + strlen(print_buf),
+				"\tRx core: %-2d", adapter->rx_core_id);
+
+		EH_LOG_INFO("%s", print_buf);
 
 		for (j = 0; j < adapter->nb_connections; j++) {
 			conn = &(adapter->conn[j]);
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index 1d5a087..a66bcb3 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -76,12 +76,21 @@ enum eh_rx_types {
 	EH_RX_TYPE_BURST
 };
 
+/**
+ * Event mode packet tx types
+ */
+enum eh_tx_types {
+	EH_TX_TYPE_INTERNAL_PORT = 0,
+	EH_TX_TYPE_NO_INTERNAL_PORT
+};
+
 /* Event dev params */
 struct eventdev_params {
 	uint8_t eventdev_id;
 	uint8_t nb_eventqueue;
 	uint8_t nb_eventport;
 	uint8_t ev_queue_mode;
+	uint8_t all_internal_ports;
 };
 
 /**
@@ -193,6 +202,8 @@ struct eh_app_worker_params {
 		struct {
 			uint64_t burst : 1;
 			/**< Specify status of rx type burst */
+			uint64_t tx_internal_port : 1;
+			/**< Specify whether tx internal port is available */
 		};
 		uint64_t u64;
 	} cap;
-- 
2.7.4

[dpdk-dev] [RFC PATCH 08/13] examples/ipsec-secgw: add eventmode to ipsec-secgw

[Anoob Joseph]

Add eventmode support to ipsec-secgw. This uses event helper to setup
and use the eventmode capabilities. Add driver inbound worker.

Example command:
./ipsec-secgw -c 0x1 -w 0002:02:00.0,nb_ipsec_in_sa=100 -w 0002:07:00.0
 -w 0002:0e:00.0 -w 0002:10:00.1 -- -P -p 0x3 -u 0x1
 --config "(0,0,0),(1,0,0)" -f a-aes-gcm-msa.cfg --transfer-mode 1
 --schedule-type 2 --process-mode 1 --process-dir 1

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/Makefile       |   1 +
 examples/ipsec-secgw/event_helper.c |   3 +
 examples/ipsec-secgw/event_helper.h |  26 +++
 examples/ipsec-secgw/ipsec-secgw.c  | 315 +++++++++++++++++++++++++++++++++++-
 examples/ipsec-secgw/ipsec.h        |   7 +
 examples/ipsec-secgw/ipsec_worker.c | 181 +++++++++++++++++++++
 examples/ipsec-secgw/meson.build    |   2 +-
 7 files changed, 527 insertions(+), 8 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_worker.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 2a76900..83983ba 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -15,6 +15,7 @@ SRCS-y += sa.c
 SRCS-y += rt.c
 SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
+SRCS-y += ipsec_worker.c
 SRCS-y += event_helper.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c
index 858b855..3ca9e2f 100644
--- a/examples/ipsec-secgw/event_helper.c
+++ b/examples/ipsec-secgw/event_helper.c
@@ -1003,6 +1003,9 @@ eh_find_worker(uint32_t lcore_id, struct eh_conf *conf,
 	else
 		curr_conf.cap.burst = EH_RX_TYPE_NON_BURST;
 
+	curr_conf.cap.ipsec_mode = conf->ipsec_mode;
+	curr_conf.cap.ipsec_dir = conf->ipsec_dir;
+
 	/* Now parse the passed list and see if we have matching capabilities */
 
 	/* Initialize the pointer used to traverse the list */
diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h
index a66bcb3..392e073 100644
--- a/examples/ipsec-secgw/event_helper.h
+++ b/examples/ipsec-secgw/event_helper.h
@@ -84,6 +84,22 @@ enum eh_tx_types {
 	EH_TX_TYPE_NO_INTERNAL_PORT
 };
 
+/**
+ * Event mode ipsec mode types
+ */
+enum eh_ipsec_mode_types {
+	EH_IPSEC_MODE_TYPE_APP = 0,
+	EH_IPSEC_MODE_TYPE_DRIVER
+};
+
+/**
+ * Event mode ipsec direction types
+ */
+enum eh_ipsec_dir_types {
+	EH_IPSEC_DIR_TYPE_OUTBOUND = 0,
+	EH_IPSEC_DIR_TYPE_INBOUND,
+};
+
 /* Event dev params */
 struct eventdev_params {
 	uint8_t eventdev_id;
@@ -193,6 +209,12 @@ struct eh_conf {
 		 */
 	void *mode_params;
 		/**< Mode specific parameters */
+
+		/** Application specific params */
+	enum eh_ipsec_mode_types ipsec_mode;
+		/**< Mode of ipsec run */
+	enum eh_ipsec_dir_types ipsec_dir;
+		/**< Direction of ipsec processing */
 };
 
 /* Workers registered by the application */
@@ -204,6 +226,10 @@ struct eh_app_worker_params {
 			/**< Specify status of rx type burst */
 			uint64_t tx_internal_port : 1;
 			/**< Specify whether tx internal port is available */
+			uint64_t ipsec_mode : 1;
+			/**< Specify ipsec processing level */
+			uint64_t ipsec_dir : 1;
+			/**< Specify direction of ipsec */
 		};
 		uint64_t u64;
 	} cap;
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 345c8e5..cb6da25 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -2,6 +2,7 @@
  * Copyright(c) 2016 Intel Corporation
  */
 
+#include <stdbool.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <stdint.h>
@@ -14,6 +15,7 @@
 #include <sys/queue.h>
 #include <stdarg.h>
 #include <errno.h>
+#include <signal.h>
 #include <getopt.h>
 
 #include <rte_common.h>
@@ -41,12 +43,17 @@
 #include <rte_jhash.h>
 #include <rte_cryptodev.h>
 #include <rte_security.h>
+#include <rte_bitmap.h>
+#include <rte_eventdev.h>
 #include <rte_ip.h>
 #include <rte_ip_frag.h>
 
+#include "event_helper.h"
 #include "ipsec.h"
 #include "parser.h"
 
+volatile bool force_quit;
+
 #define RTE_LOGTYPE_IPSEC RTE_LOGTYPE_USER1
 
 #define MAX_JUMBO_PKT_LEN  9600
@@ -131,6 +138,10 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
+#define CMD_LINE_OPT_TRANSFER_MODE	"transfer-mode"
+#define CMD_LINE_OPT_SCHEDULE_TYPE	"schedule-type"
+#define CMD_LINE_OPT_IPSEC_MODE		"process-mode"
+#define CMD_LINE_OPT_IPSEC_DIR		"process-dir"
 #define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
 #define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
 #define CMD_LINE_OPT_REASSEMBLE		"reassemble"
@@ -146,7 +157,11 @@ enum {
 	CMD_LINE_OPT_CONFIG_NUM,
 	CMD_LINE_OPT_SINGLE_SA_NUM,
 	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
+	CMD_LINE_OPT_TRANSFER_MODE_NUM,
+	CMD_LINE_OPT_SCHEDULE_TYPE_NUM,
 	CMD_LINE_OPT_RX_OFFLOAD_NUM,
+	CMD_LINE_OPT_IPSEC_MODE_NUM,
+	CMD_LINE_OPT_IPSEC_DIR_NUM,
 	CMD_LINE_OPT_TX_OFFLOAD_NUM,
 	CMD_LINE_OPT_REASSEMBLE_NUM,
 	CMD_LINE_OPT_MTU_NUM,
@@ -156,6 +171,10 @@ static const struct option lgopts[] = {
 	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
 	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
+	{CMD_LINE_OPT_TRANSFER_MODE, 1, 0, CMD_LINE_OPT_TRANSFER_MODE_NUM},
+	{CMD_LINE_OPT_SCHEDULE_TYPE, 1, 0, CMD_LINE_OPT_SCHEDULE_TYPE_NUM},
+	{CMD_LINE_OPT_IPSEC_MODE, 1, 0, CMD_LINE_OPT_IPSEC_MODE_NUM},
+	{CMD_LINE_OPT_IPSEC_DIR, 1, 0, CMD_LINE_OPT_IPSEC_DIR_NUM},
 	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
 	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
 	{CMD_LINE_OPT_REASSEMBLE, 1, 0, CMD_LINE_OPT_REASSEMBLE_NUM},
@@ -1087,8 +1106,8 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 }
 
 /* main processing loop */
-static int32_t
-main_loop(__attribute__((unused)) void *dummy)
+void
+ipsec_poll_mode_worker(void)
 {
 	struct rte_mbuf *pkts[MAX_PKT_BURST];
 	uint32_t lcore_id;
@@ -1130,7 +1149,7 @@ main_loop(__attribute__((unused)) void *dummy)
 	if (qconf->nb_rx_queue == 0) {
 		RTE_LOG(DEBUG, IPSEC, "lcore %u has nothing to do\n",
 			lcore_id);
-		return 0;
+		return;
 	}
 
 	RTE_LOG(INFO, IPSEC, "entering main loop on lcore %u\n", lcore_id);
@@ -1143,7 +1162,7 @@ main_loop(__attribute__((unused)) void *dummy)
 			lcore_id, portid, queueid);
 	}
 
-	while (1) {
+	while (!force_quit) {
 		cur_tsc = rte_rdtsc();
 
 		/* TX queue buffer drain */
@@ -1270,6 +1289,10 @@ print_usage(const char *prgname)
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
 		" [--cryptodev_mask MASK]"
+		" [--transfer-mode MODE]"
+		" [--schedule-type TYPE]"
+		" [--process-mode MODE]"
+		" [--process-dir DIR]"
 		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
 		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
 		" [--" CMD_LINE_OPT_REASSEMBLE " REASSEMBLE_TABLE_SIZE]"
@@ -1291,6 +1314,22 @@ print_usage(const char *prgname)
 		"                     bypassing the SP\n"
 		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
 		"                         devices to configure\n"
+		"  --transfer-mode MODE\n"
+		"               0: Packet transfer via polling (default)\n"
+		"               1: Packet transfer via eventdev\n"
+		"  --schedule-type TYPE queue schedule type, used only when\n"
+		"                       transfer mode is set to eventdev\n"
+		"               0: Ordered (default)\n"
+		"               1: Atomic\n"
+		"               2: Parallel\n"
+		"  --process-mode MODE processing mode, used only when\n"
+		"		       transfer mode is set to eventdev\n"
+		"               0: application mode (default)\n"
+		"               1: driver mode\n"
+		"  --process-dir DIR processing direction, used only when\n"
+		"                    transfer mode is set to eventdev\n"
+		"               0: outbound (default)\n"
+		"               1: inbound\n"
 		"  --" CMD_LINE_OPT_RX_OFFLOAD
 		": bitmask of the RX HW offload capabilities to enable/use\n"
 		"                         (DEV_RX_OFFLOAD_*)\n"
@@ -1424,7 +1463,85 @@ print_app_sa_prm(const struct app_sa_prm *prm)
 }
 
 static int32_t
-parse_args(int32_t argc, char **argv)
+eh_parse_decimal(const char *str)
+{
+	char *end = NULL;
+	unsigned long num;
+
+	num = strtoul(str, &end, 10);
+	if ((str[0] == '\0') || (end == NULL) || (*end != '\0'))
+		return -EINVAL;
+
+	return num;
+}
+
+static int
+parse_transfer_mode(struct eh_conf *conf, const char *optarg)
+{
+	int32_t parsed_dec;
+
+	parsed_dec = eh_parse_decimal(optarg);
+	if (parsed_dec != EH_PKT_TRANSFER_MODE_POLL &&
+	    parsed_dec != EH_PKT_TRANSFER_MODE_EVENT) {
+		printf("Unsupported packet transfer mode");
+		return -EINVAL;
+	}
+	conf->mode = parsed_dec;
+	return 0;
+}
+
+static int
+parse_schedule_type(struct eh_conf *conf, const char *optarg)
+{
+	struct eventmode_conf *em_conf = NULL;
+	int32_t parsed_dec;
+
+	parsed_dec = eh_parse_decimal(optarg);
+	if (parsed_dec != RTE_SCHED_TYPE_ORDERED &&
+	    parsed_dec != RTE_SCHED_TYPE_ATOMIC &&
+	    parsed_dec != RTE_SCHED_TYPE_PARALLEL)
+		return -EINVAL;
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(conf->mode_params);
+
+	em_conf->ext_params.sched_type = parsed_dec;
+
+	return 0;
+}
+
+static int
+parse_ipsec_mode(struct eh_conf *conf, const char *optarg)
+{
+	int32_t parsed_dec;
+
+	parsed_dec = eh_parse_decimal(optarg);
+	if (parsed_dec != EH_IPSEC_MODE_TYPE_APP &&
+	    parsed_dec != EH_IPSEC_MODE_TYPE_DRIVER) {
+		printf("Unsupported ipsec mode");
+		return -EINVAL;
+	}
+	conf->ipsec_mode = parsed_dec;
+	return 0;
+}
+
+static int
+parse_ipsec_dir(struct eh_conf *conf, const char *optarg)
+{
+	int32_t parsed_dec;
+
+	parsed_dec = eh_parse_decimal(optarg);
+	if (parsed_dec != EH_IPSEC_DIR_TYPE_INBOUND &&
+	    parsed_dec != EH_IPSEC_DIR_TYPE_OUTBOUND) {
+		printf("Unsupported ipsec mode");
+		return -EINVAL;
+	}
+	conf->ipsec_dir = parsed_dec;
+	return 0;
+}
+
+static int32_t
+parse_args(int32_t argc, char **argv, struct eh_conf *eh_conf)
 {
 	int32_t opt, ret;
 	char **argvopt;
@@ -1533,6 +1650,43 @@ parse_args(int32_t argc, char **argv)
 			/* else */
 			enabled_cryptodev_mask = ret;
 			break;
+
+		case CMD_LINE_OPT_TRANSFER_MODE_NUM:
+			ret = parse_transfer_mode(eh_conf, optarg);
+			if (ret < 0) {
+				printf("Invalid packet transfer mode\n");
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+
+		case CMD_LINE_OPT_SCHEDULE_TYPE_NUM:
+			ret = parse_schedule_type(eh_conf, optarg);
+			if (ret < 0) {
+				printf("Invalid queue schedule type\n");
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+
+		case CMD_LINE_OPT_IPSEC_MODE_NUM:
+			ret = parse_ipsec_mode(eh_conf, optarg);
+			if (ret < 0) {
+				printf("Invalid ipsec mode\n");
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+
+		case CMD_LINE_OPT_IPSEC_DIR_NUM:
+			ret = parse_ipsec_dir(eh_conf, optarg);
+			if (ret < 0) {
+				printf("Invalid ipsec direction\n");
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+
 		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
 			ret = parse_mask(optarg, &dev_rx_offload);
 			if (ret != 0) {
@@ -2378,6 +2532,114 @@ reassemble_init(void)
 	return rc;
 }
 
+static struct eh_conf *
+eh_conf_init(void)
+{
+	struct eh_conf *conf = NULL;
+	struct eventmode_conf *em_conf = NULL;
+	unsigned int eth_core_id;
+	uint32_t nb_bytes;
+	void *mem = NULL;
+
+	/* Allocate memory for config */
+	conf = calloc(1, sizeof(struct eh_conf));
+	if (conf == NULL) {
+		printf("Failed allocating memory for eventmode helper conf");
+		goto err;
+	}
+
+	/* Set default conf */
+
+	/* Packet transfer mode: poll */
+	conf->mode = EH_PKT_TRANSFER_MODE_POLL;
+	conf->ipsec_mode = EH_IPSEC_MODE_TYPE_APP;
+	conf->ipsec_dir = EH_IPSEC_DIR_TYPE_OUTBOUND;
+
+	/* Keep all ethernet ports enabled by default */
+	conf->eth_portmask = -1;
+
+	/* Allocate memory for event mode params */
+	conf->mode_params =
+		calloc(1, sizeof(struct eventmode_conf));
+	if (conf->mode_params == NULL) {
+		printf("Failed allocating memory for event mode params");
+		goto err;
+	}
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(conf->mode_params);
+
+	/* Allocate and initialize bitmap for eth cores */
+	nb_bytes = rte_bitmap_get_memory_footprint(RTE_MAX_LCORE);
+	if (!nb_bytes) {
+		printf("Failed getting bitmap footprint");
+		goto err;
+	}
+
+	mem = rte_zmalloc("event-helper-ethcore-bitmap", nb_bytes,
+			  RTE_CACHE_LINE_SIZE);
+	if (!mem) {
+		printf("Failed allocating memory for eth cores bitmap\n");
+		goto err;
+	}
+
+	em_conf->eth_core_mask = rte_bitmap_init(RTE_MAX_LCORE, mem, nb_bytes);
+	if (!em_conf->eth_core_mask) {
+		printf("Failed initializing bitmap");
+		goto err;
+	}
+
+	/* Schedule type: ordered */
+	em_conf->ext_params.sched_type = RTE_SCHED_TYPE_ORDERED;
+
+	/* Set two cores as eth cores for Rx & Tx */
+
+	/* Use first core other than master core as Rx core */
+	eth_core_id = rte_get_next_lcore(0,	/* curr core */
+					 1,	/* skip master core */
+					 0	/* wrap */);
+
+	rte_bitmap_set(em_conf->eth_core_mask, eth_core_id);
+
+	/* Use next core as Tx core */
+	eth_core_id = rte_get_next_lcore(eth_core_id,	/* curr core */
+					 1,		/* skip master core */
+					 0		/* wrap */);
+
+	rte_bitmap_set(em_conf->eth_core_mask, eth_core_id);
+
+	return conf;
+err:
+	rte_free(mem);
+	free(em_conf);
+	free(conf);
+	return NULL;
+}
+
+static void
+eh_conf_uninit(struct eh_conf *conf)
+{
+	struct eventmode_conf *em_conf = NULL;
+
+	/* Get eventmode conf */
+	em_conf = (struct eventmode_conf *)(conf->mode_params);
+
+	/* Free evenmode configuration memory */
+	rte_free(em_conf->eth_core_mask);
+	free(em_conf);
+	free(conf);
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\n\nSignal %d received, preparing to exit...\n",
+				signum);
+		force_quit = true;
+	}
+}
+
 int32_t
 main(int32_t argc, char **argv)
 {
@@ -2387,6 +2649,7 @@ main(int32_t argc, char **argv)
 	uint8_t socket_id;
 	uint16_t portid;
 	uint64_t req_rx_offloads, req_tx_offloads;
+	struct eh_conf *eh_conf = NULL;
 	size_t sess_sz;
 
 	/* init EAL */
@@ -2396,8 +2659,17 @@ main(int32_t argc, char **argv)
 	argc -= ret;
 	argv += ret;
 
+	force_quit = false;
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	/* initialize event helper configuration */
+	eh_conf = eh_conf_init();
+	if (eh_conf == NULL)
+		rte_exit(EXIT_FAILURE, "Failed to init event helper config");
+
 	/* parse application arguments (after the EAL ones) */
-	ret = parse_args(argc, argv);
+	ret = parse_args(argc, argv, eh_conf);
 	if (ret < 0)
 		rte_exit(EXIT_FAILURE, "Invalid parameters\n");
 
@@ -2495,12 +2767,41 @@ main(int32_t argc, char **argv)
 
 	check_all_ports_link_status(enabled_port_mask);
 
+	/*
+	 * Set the enabled port mask in helper conf to be used by helper
+	 * sub-system. This would be used while intializing devices using
+	 * helper sub-system.
+	 */
+	eh_conf->eth_portmask = enabled_port_mask;
+
+	/* Initialize eventmode components */
+	eh_devs_init(eh_conf);
+
+
 	/* launch per-lcore init on every lcore */
-	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
+	rte_eal_mp_remote_launch(ipsec_launch_one_lcore, eh_conf,
+				CALL_MASTER);
+
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
 		if (rte_eal_wait_lcore(lcore_id) < 0)
 			return -1;
 	}
 
+	/* Uninitialize eventmode components */
+	eh_devs_uninit(eh_conf);
+
+	/* Free eventmode configuration memory */
+	eh_conf_uninit(eh_conf);
+
+	RTE_ETH_FOREACH_DEV(portid) {
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+		printf("Closing port %d...", portid);
+		rte_eth_dev_stop(portid);
+		rte_eth_dev_close(portid);
+		printf(" Done\n");
+	}
+	printf("Bye...\n");
+
 	return 0;
 }
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 1efa6e4..1ab52da 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -228,6 +228,13 @@ struct ipsec_traffic {
 	struct traffic_type ip6;
 };
 
+
+void
+ipsec_poll_mode_worker(void);
+
+int
+ipsec_launch_one_lcore(void *args);
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
new file mode 100644
index 0000000..6839e25
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2016 Intel Corporation
+ * Copyright (C) 2019 Marvell International Ltd.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <netinet/in.h>
+#include <setjmp.h>
+#include <stdarg.h>
+#include <ctype.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_log.h>
+#include <rte_memcpy.h>
+#include <rte_atomic.h>
+#include <rte_cycles.h>
+#include <rte_prefetch.h>
+#include <rte_lcore.h>
+#include <rte_branch_prediction.h>
+#include <rte_event_eth_tx_adapter.h>
+#include <rte_ether.h>
+#include <rte_ethdev.h>
+#include <rte_eventdev.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+#include "event_helper.h"
+
+extern volatile bool force_quit;
+
+static inline void
+ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int portid)
+{
+	/* Save the destination port in the mbuf */
+	m->port = portid;
+
+	/* Save eth queue for Tx */
+	rte_event_eth_tx_adapter_txq_set(m, 0);
+}
+
+/*
+ * Event mode exposes various operating modes depending on the
+ * capabilities of the event device and the operating mode
+ * selected.
+ */
+
+/* Workers registered */
+#define IPSEC_EVENTMODE_WORKERS		1
+
+/*
+ * Event mode worker
+ * Operating parameters : non-burst - Tx internal port - driver mode - inbound
+ */
+static void
+ipsec_wrkr_non_burst_int_port_drvr_mode_inb(struct eh_conf *mode_conf,
+		struct eh_event_link_info *links, uint8_t nb_links)
+{
+	struct rte_event ev;
+	struct rte_mbuf *pkt;
+	uint32_t lcore_id;
+	unsigned int nb_rx = 0;
+	unsigned int portid;
+
+	RTE_SET_USED(mode_conf);
+
+	/* Check if we have links registered for this lcore */
+	if (nb_links == 0) {
+		/* No links registered - exit */
+		goto exit;
+	}
+
+	/* Get core ID */
+	lcore_id = rte_lcore_id();
+
+	RTE_LOG(INFO, IPSEC,
+		"Launching event mode worker (non-burst - Tx internal port - "
+		"driver mode - inbound) on lcore %d\n", lcore_id);
+
+	/* We have valid links */
+
+	/* See if it's single link */
+	if (nb_links != 1) {
+		RTE_LOG(INFO, IPSEC,
+			"Multiple links not supported. Using first link\n");
+	}
+
+	RTE_LOG(INFO, IPSEC, " -- lcoreid=%u event_port_id=%u\n", lcore_id,
+			links[0].event_portid);
+	while (!force_quit) {
+		/* Read packet from event queues */
+		nb_rx = rte_event_dequeue_burst(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1,	/* nb_events */
+				0	/* timeout_ticks */);
+
+		if (nb_rx == 0)
+			continue;
+
+		portid = ev.queue_id;
+		pkt = ev.mbuf;
+
+		rte_prefetch0(rte_pktmbuf_mtod(pkt, void *));
+
+		/* Process packet */
+		ipsec_event_pre_forward(pkt, portid);
+
+		/*
+		 * Since tx internal port is available, events can be
+		 * directly enqueued to the adapter and it would be
+		 * internally submitted to the eth device.
+		 */
+		rte_event_eth_tx_adapter_enqueue(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1	/* nb_events */);
+	}
+
+exit:
+	return;
+}
+
+static uint8_t
+ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
+{
+	uint8_t nb_wrkr_param = 0;
+	struct eh_app_worker_params *wrkr;
+
+	/* Save workers */
+	wrkr = wrkrs;
+
+	/* Non-burst - Tx internal port - driver mode - inbound */
+	wrkr->cap.burst = EH_RX_TYPE_NON_BURST;
+	wrkr->cap.tx_internal_port = EH_TX_TYPE_INTERNAL_PORT;
+	wrkr->cap.ipsec_mode = EH_IPSEC_MODE_TYPE_DRIVER;
+	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_INBOUND;
+	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_drvr_mode_inb;
+
+	nb_wrkr_param++;
+	return nb_wrkr_param;
+}
+
+static void
+ipsec_eventmode_worker(struct eh_conf *mode_conf)
+{
+	struct eh_app_worker_params ipsec_wrkr[IPSEC_EVENTMODE_WORKERS] = {
+					{{{0} }, NULL } };
+	uint8_t nb_wrkr_param;
+
+	/* Populate l2fwd_wrkr params */
+	nb_wrkr_param = ipsec_eventmode_populate_wrkr_params(ipsec_wrkr);
+
+	/*
+	 * The helper function will launch the correct worker after checking the
+	 * event device's capabilities.
+	 */
+	eh_launch_worker(mode_conf, ipsec_wrkr, nb_wrkr_param);
+}
+
+int ipsec_launch_one_lcore(void *args)
+{
+	struct eh_conf *mode_conf;
+
+	mode_conf = (struct eh_conf *)args;
+
+	if (mode_conf->mode == EH_PKT_TRANSFER_MODE_POLL) {
+		/* App is initialized to run in poll mode */
+		ipsec_poll_mode_worker();
+	} else if (mode_conf->mode == EH_PKT_TRANSFER_MODE_EVENT) {
+		/* App is initialized to run in event mode */
+		ipsec_eventmode_worker(mode_conf);
+	}
+	return 0;
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 20f4064..ab40ca5 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -10,5 +10,5 @@ deps += ['security', 'lpm', 'acl', 'hash', 'ip_frag', 'ipsec', 'eventdev']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
-	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c', 'event_helper.c'
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c', 'event_helper.c', 'ipsec_worker.c'
 )
-- 
2.7.4

[dpdk-dev] [RFC PATCH 09/13] examples/ipsec-secgw: add app inbound worker

[Anoob Joseph]

Add application inbound worker thread.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/ipsec_worker.c | 86 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 85 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 6839e25..b0bf29f 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -52,7 +52,7 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int portid)
  */
 
 /* Workers registered */
-#define IPSEC_EVENTMODE_WORKERS		1
+#define IPSEC_EVENTMODE_WORKERS		2
 
 /*
  * Event mode worker
@@ -127,6 +127,80 @@ ipsec_wrkr_non_burst_int_port_drvr_mode_inb(struct eh_conf *mode_conf,
 	return;
 }
 
+/*
+ * Event mode worker
+ * Operating parameters : non-burst - Tx internal port - app mode - inbound
+ */
+static void
+ipsec_wrkr_non_burst_int_port_app_mode_inb(struct eh_conf *mode_conf,
+		struct eh_event_link_info *links, uint8_t nb_links)
+{
+	struct rte_event ev;
+	struct rte_mbuf *pkt;
+	uint32_t lcore_id;
+	unsigned int nb_rx = 0;
+	unsigned int portid;
+
+	RTE_SET_USED(mode_conf);
+
+	/* Check if we have links registered for this lcore */
+	if (nb_links == 0) {
+		/* No links registered - exit */
+		goto exit;
+	}
+
+	/* We have valid links */
+
+	/* Get core ID */
+	lcore_id = rte_lcore_id();
+
+	RTE_LOG(INFO, IPSEC,
+		"Launching event mode worker (non-burst - Tx internal port - "
+		"app mode - inbound) on lcore %d\n", lcore_id);
+
+	/* See if it's single link */
+	if (nb_links != 1) {
+		RTE_LOG(INFO, IPSEC,
+			"Multiple links not supported. Using first link\n");
+	}
+
+	RTE_LOG(INFO, IPSEC, " -- lcoreid=%u event_port_id=%u\n", lcore_id,
+		links[0].event_portid);
+
+	while (!force_quit) {
+		/* Read packet from event queues */
+		nb_rx = rte_event_dequeue_burst(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,     /* events */
+				1,       /* nb_events */
+				0        /* timeout_ticks */);
+
+		if (nb_rx == 0)
+			continue;
+
+		portid = ev.queue_id;
+		pkt = ev.mbuf;
+
+		rte_prefetch0(rte_pktmbuf_mtod(pkt, void *));
+
+		/* Process packet */
+		ipsec_event_pre_forward(pkt, portid);
+
+		/*
+		 * Since tx internal port is available, events can be
+		 * directly enqueued to the adapter and it would be
+		 * internally submitted to the eth device.
+		 */
+		rte_event_eth_tx_adapter_enqueue(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1	/* nb_events */);
+	}
+
+exit:
+	return;
+}
+
 static uint8_t
 ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 {
@@ -143,6 +217,16 @@ ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_INBOUND;
 	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_drvr_mode_inb;
 
+	wrkr++;
+	nb_wrkr_param++;
+
+	/* Non-burst - Tx internal port - app mode - inbound */
+	wrkr->cap.burst = EH_RX_TYPE_NON_BURST;
+	wrkr->cap.tx_internal_port = EH_TX_TYPE_INTERNAL_PORT;
+	wrkr->cap.ipsec_mode = EH_IPSEC_MODE_TYPE_APP;
+	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_INBOUND;
+	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_app_mode_inb;
+
 	nb_wrkr_param++;
 	return nb_wrkr_param;
 }
-- 
2.7.4

[dpdk-dev] [RFC PATCH 10/13] examples/ipsec-secgw: add app processing code

[Anoob Joseph]

Add IPsec application processing code for event mode.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  | 108 ++++++----------
 examples/ipsec-secgw/ipsec-secgw.h  |  81 ++++++++++++
 examples/ipsec-secgw/ipsec.h        |  37 +++---
 examples/ipsec-secgw/ipsec_worker.c | 242 ++++++++++++++++++++++++++++++++++--
 examples/ipsec-secgw/ipsec_worker.h |  39 ++++++
 examples/ipsec-secgw/sa.c           |  11 --
 6 files changed, 403 insertions(+), 115 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
 create mode 100644 examples/ipsec-secgw/ipsec_worker.h

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cb6da25..1c102cf 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -50,12 +50,11 @@
 
 #include "event_helper.h"
 #include "ipsec.h"
+#include "ipsec_worker.h"
 #include "parser.h"
 
 volatile bool force_quit;
 
-#define RTE_LOGTYPE_IPSEC RTE_LOGTYPE_USER1
-
 #define MAX_JUMBO_PKT_LEN  9600
 
 #define MEMPOOL_CACHE_SIZE 256
@@ -70,8 +69,6 @@ volatile bool force_quit;
 
 #define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */
 
-#define NB_SOCKETS 4
-
 /* Configure how many packets ahead to prefetch, when reading packets */
 #define PREFETCH_OFFSET	3
 
@@ -79,8 +76,6 @@ volatile bool force_quit;
 
 #define MAX_LCORE_PARAMS 1024
 
-#define UNPROTECTED_PORT(port) (unprotected_port_mask & (1 << portid))
-
 /*
  * Configurable number of RX/TX ring descriptors
  */
@@ -89,29 +84,6 @@ volatile bool force_quit;
 static uint16_t nb_rxd = IPSEC_SECGW_RX_DESC_DEFAULT;
 static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 
-#if RTE_BYTE_ORDER != RTE_LITTLE_ENDIAN
-#define __BYTES_TO_UINT64(a, b, c, d, e, f, g, h) \
-	(((uint64_t)((a) & 0xff) << 56) | \
-	((uint64_t)((b) & 0xff) << 48) | \
-	((uint64_t)((c) & 0xff) << 40) | \
-	((uint64_t)((d) & 0xff) << 32) | \
-	((uint64_t)((e) & 0xff) << 24) | \
-	((uint64_t)((f) & 0xff) << 16) | \
-	((uint64_t)((g) & 0xff) << 8)  | \
-	((uint64_t)(h) & 0xff))
-#else
-#define __BYTES_TO_UINT64(a, b, c, d, e, f, g, h) \
-	(((uint64_t)((h) & 0xff) << 56) | \
-	((uint64_t)((g) & 0xff) << 48) | \
-	((uint64_t)((f) & 0xff) << 40) | \
-	((uint64_t)((e) & 0xff) << 32) | \
-	((uint64_t)((d) & 0xff) << 24) | \
-	((uint64_t)((c) & 0xff) << 16) | \
-	((uint64_t)((b) & 0xff) << 8) | \
-	((uint64_t)(a) & 0xff))
-#endif
-#define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
-
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
 		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
 		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
@@ -123,18 +95,6 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 
 #define MTU_TO_FRAMELEN(x)	((x) + RTE_ETHER_HDR_LEN + RTE_ETHER_CRC_LEN)
 
-/* port/source ethernet addr and destination ethernet addr */
-struct ethaddr_info {
-	uint64_t src, dst;
-};
-
-struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
-	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x7e, 0x94, 0x9a) },
-	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x22, 0xa1, 0xd9) },
-	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x08, 0x69, 0x26) },
-	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x49, 0x9e, 0xdd) }
-};
-
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
@@ -182,10 +142,16 @@ static const struct option lgopts[] = {
 	{NULL, 0, 0, 0}
 };
 
+struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
+	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x7e, 0x94, 0x9a) },
+	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x22, 0xa1, 0xd9) },
+	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x08, 0x69, 0x26) },
+	{ 0, ETHADDR(0x00, 0x16, 0x3e, 0x49, 0x9e, 0xdd) }
+};
+
 /* mask of enabled ports */
 static uint32_t enabled_port_mask;
 static uint64_t enabled_cryptodev_mask = UINT64_MAX;
-static uint32_t unprotected_port_mask;
 static int32_t promiscuous_on = 1;
 static int32_t numa_on = 1; /**< NUMA is enabled by default. */
 static uint32_t nb_lcores;
@@ -271,8 +237,6 @@ static struct rte_eth_conf port_conf = {
 	},
 };
 
-static struct socket_ctx socket_ctx[NB_SOCKETS];
-
 /*
  * Determine is multi-segment support required:
  *  - either frame buffer size is smaller then mtu
@@ -2718,33 +2682,6 @@ main(int32_t argc, char **argv)
 
 	cryptodevs_init();
 
-	/* start ports */
-	RTE_ETH_FOREACH_DEV(portid) {
-		if ((enabled_port_mask & (1 << portid)) == 0)
-			continue;
-
-		/*
-		 * Start device
-		 * note: device must be started before a flow rule
-		 * can be installed.
-		 */
-		ret = rte_eth_dev_start(portid);
-		if (ret < 0)
-			rte_exit(EXIT_FAILURE, "rte_eth_dev_start: "
-					"err=%d, port=%d\n", ret, portid);
-		/*
-		 * If enabled, put device in promiscuous mode.
-		 * This allows IO forwarding mode to forward packets
-		 * to itself through 2 cross-connected  ports of the
-		 * target machine.
-		 */
-		if (promiscuous_on)
-			rte_eth_promiscuous_enable(portid);
-
-		rte_eth_dev_callback_register(portid,
-			RTE_ETH_EVENT_IPSEC, inline_ipsec_event_callback, NULL);
-	}
-
 	/* fragment reassemble is enabled */
 	if (frag_tbl_sz != 0) {
 		ret = reassemble_init();
@@ -2765,8 +2702,6 @@ main(int32_t argc, char **argv)
 		}
 	}
 
-	check_all_ports_link_status(enabled_port_mask);
-
 	/*
 	 * Set the enabled port mask in helper conf to be used by helper
 	 * sub-system. This would be used while intializing devices using
@@ -2777,6 +2712,33 @@ main(int32_t argc, char **argv)
 	/* Initialize eventmode components */
 	eh_devs_init(eh_conf);
 
+	/* start each port */
+	RTE_ETH_FOREACH_DEV(portid) {
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+		/*
+		 * Start device
+		 * note: device must be started before a flow rule
+		 * can be installed.
+		 */
+		ret = rte_eth_dev_start(portid);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE, "rte_eth_dev_start: "
+					"err=%d, port=%d\n", ret, portid);
+		/*
+		 * If enabled, put device in promiscuous mode.
+		 * This allows IO forwarding mode to forward packets
+		 * to itself through 2 cross-connected  ports of the
+		 * target machine.
+		 */
+		if (promiscuous_on)
+			rte_eth_promiscuous_enable(portid);
+
+		rte_eth_dev_callback_register(portid,
+			RTE_ETH_EVENT_IPSEC, inline_ipsec_event_callback, NULL);
+	}
+
+	check_all_ports_link_status(enabled_port_mask);
 
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(ipsec_launch_one_lcore, eh_conf,
diff --git a/examples/ipsec-secgw/ipsec-secgw.h b/examples/ipsec-secgw/ipsec-secgw.h
new file mode 100644
index 0000000..445e05e
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec-secgw.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Cavium, Inc
+ */
+#ifndef _IPSEC_SECGW_H_
+#define _IPSEC_SECGW_H_
+
+#include <rte_hash.h>
+
+#define MAX_PKT_BURST 32
+
+#define RTE_LOGTYPE_IPSEC RTE_LOGTYPE_USER1
+
+#define NB_SOCKETS 4
+
+#define UNPROTECTED_PORT(port) (unprotected_port_mask & (1 << portid))
+
+#if RTE_BYTE_ORDER != RTE_LITTLE_ENDIAN
+#define __BYTES_TO_UINT64(a, b, c, d, e, f, g, h) \
+	(((uint64_t)((a) & 0xff) << 56) | \
+	((uint64_t)((b) & 0xff) << 48) | \
+	((uint64_t)((c) & 0xff) << 40) | \
+	((uint64_t)((d) & 0xff) << 32) | \
+	((uint64_t)((e) & 0xff) << 24) | \
+	((uint64_t)((f) & 0xff) << 16) | \
+	((uint64_t)((g) & 0xff) << 8)  | \
+	((uint64_t)(h) & 0xff))
+#else
+#define __BYTES_TO_UINT64(a, b, c, d, e, f, g, h) \
+	(((uint64_t)((h) & 0xff) << 56) | \
+	((uint64_t)((g) & 0xff) << 48) | \
+	((uint64_t)((f) & 0xff) << 40) | \
+	((uint64_t)((e) & 0xff) << 32) | \
+	((uint64_t)((d) & 0xff) << 24) | \
+	((uint64_t)((c) & 0xff) << 16) | \
+	((uint64_t)((b) & 0xff) << 8) | \
+	((uint64_t)(a) & 0xff))
+#endif
+
+#define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
+
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
+/* Fields optimized for devices without burst */
+struct traffic_type_nb {
+	const uint8_t *data;
+	struct rte_mbuf *pkt;
+	uint32_t res;
+	uint32_t num;
+};
+
+struct ipsec_traffic_nb {
+	struct traffic_type_nb ipsec;
+	struct traffic_type_nb ip4;
+	struct traffic_type_nb ip6;
+};
+
+/* port/source ethernet addr and destination ethernet addr */
+struct ethaddr_info {
+	uint64_t src, dst;
+};
+
+struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS];
+
+/* TODO: All var definitions need to be part of a .c file */
+
+/* Port mask to identify the unprotected ports */
+uint32_t unprotected_port_mask;
+
+#endif /* _IPSEC_SECGW_H_ */
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 1ab52da..c468824 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -13,11 +13,11 @@
 #include <rte_flow.h>
 #include <rte_ipsec.h>
 
-#define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
+#include "ipsec-secgw.h"
+
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
 #define RTE_LOGTYPE_IPSEC_IPIP  RTE_LOGTYPE_USER3
 
-#define MAX_PKT_BURST 32
 #define MAX_INFLIGHT 128
 #define MAX_QP_PER_LCORE 256
 
@@ -134,6 +134,17 @@ struct ipsec_sa {
 	struct rte_security_session_conf sess_conf;
 } __rte_cache_aligned;
 
+struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
+	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
+	union {
+		struct {
+			struct rte_crypto_sym_xform a;
+			struct rte_crypto_sym_xform b;
+		};
+	} xf[IPSEC_SA_MAX_ENTRIES];
+};
+
 struct ipsec_mbuf_metadata {
 	struct ipsec_sa *sa;
 	struct rte_crypto_op cop;
@@ -214,26 +225,8 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
-
-void
-ipsec_poll_mode_worker(void);
-
-int
-ipsec_launch_one_lcore(void *args);
+/* Socket ctx */
+struct socket_ctx socket_ctx[NB_SOCKETS];
 
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index b0bf29f..f93da4d 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -15,6 +15,7 @@
 #include <ctype.h>
 #include <stdbool.h>
 
+#include <rte_acl.h>
 #include <rte_common.h>
 #include <rte_log.h>
 #include <rte_memcpy.h>
@@ -29,12 +30,51 @@
 #include <rte_eventdev.h>
 #include <rte_malloc.h>
 #include <rte_mbuf.h>
+#include <rte_lpm.h>
+#include <rte_lpm6.h>
 
 #include "ipsec.h"
+#include "ipsec_worker.h"
 #include "event_helper.h"
 
 extern volatile bool force_quit;
 
+static inline enum pkt_type
+process_ipsec_get_pkt_type(struct rte_mbuf *pkt, uint8_t **nlp)
+{
+	struct rte_ether_hdr *eth;
+
+	eth = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *);
+	if (eth->ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4)) {
+		*nlp = RTE_PTR_ADD(eth, RTE_ETHER_HDR_LEN +
+				offsetof(struct ip, ip_p));
+		if (**nlp == IPPROTO_ESP)
+			return PKT_TYPE_IPSEC_IPV4;
+		else
+			return PKT_TYPE_PLAIN_IPV4;
+	} else if (eth->ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV6)) {
+		*nlp = RTE_PTR_ADD(eth, RTE_ETHER_HDR_LEN +
+				offsetof(struct ip6_hdr, ip6_nxt));
+		if (**nlp == IPPROTO_ESP)
+			return PKT_TYPE_IPSEC_IPV6;
+		else
+			return PKT_TYPE_PLAIN_IPV6;
+	}
+
+	/* Unknown/Unsupported type */
+	return PKT_TYPE_INVALID;
+}
+
+static inline void
+update_mac_addrs(struct rte_mbuf *pkt, uint16_t portid)
+{
+	struct rte_ether_hdr *ethhdr;
+
+	ethhdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *);
+	memcpy(&ethhdr->s_addr, &ethaddr_tbl[portid].src, RTE_ETHER_ADDR_LEN);
+	memcpy(&ethhdr->d_addr, &ethaddr_tbl[portid].dst, RTE_ETHER_ADDR_LEN);
+}
+
 static inline void
 ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int portid)
 {
@@ -45,6 +85,177 @@ ipsec_event_pre_forward(struct rte_mbuf *m, unsigned int portid)
 	rte_event_eth_tx_adapter_txq_set(m, 0);
 }
 
+static inline int
+check_sp(struct sp_ctx *sp, const uint8_t *nlp, uint32_t *sa_idx)
+{
+	uint32_t res;
+
+	if (unlikely(sp == NULL))
+		return 0;
+
+	rte_acl_classify((struct rte_acl_ctx *)sp, &nlp, &res, 1,
+			DEFAULT_MAX_CATEGORIES);
+
+	if (unlikely(res == 0)) {
+		/* No match */
+		return 0;
+	}
+
+	if (res == DISCARD)
+		return 0;
+	else if (res == BYPASS) {
+		*sa_idx = 0;
+		return 1;
+	}
+
+	*sa_idx = SPI2IDX(res);
+	if (*sa_idx < IPSEC_SA_MAX_ENTRIES)
+		return 1;
+
+	/* Invalid SA IDX */
+	return 0;
+}
+
+static inline uint16_t
+route4_pkt(struct rte_mbuf *pkt, struct rt_ctx *rt_ctx)
+{
+	uint32_t hop;
+	uint32_t dst_ip;
+	uint16_t offset;
+	int ret;
+
+	offset = RTE_ETHER_HDR_LEN + offsetof(struct ip, ip_dst);
+	dst_ip = *rte_pktmbuf_mtod_offset(pkt, uint32_t *, offset);
+	dst_ip = rte_be_to_cpu_32(dst_ip);
+
+	ret = rte_lpm_lookup((struct rte_lpm *)rt_ctx, dst_ip, &hop);
+
+	if (ret == 0) {
+		/* We have a hit */
+		return hop;
+	}
+
+	/* else */
+	return RTE_MAX_ETHPORTS;
+}
+
+/* TODO: To be tested */
+static inline uint16_t
+route6_pkt(struct rte_mbuf *pkt, struct rt_ctx *rt_ctx)
+{
+	uint32_t hop;
+	uint8_t dst_ip[16];
+	uint8_t *ip6_dst;
+	uint16_t offset;
+	int ret;
+
+	offset = RTE_ETHER_HDR_LEN + offsetof(struct ip6_hdr, ip6_dst);
+	ip6_dst = rte_pktmbuf_mtod_offset(pkt, uint8_t *, offset);
+	memcpy(&dst_ip[0], ip6_dst, 16);
+
+	ret = rte_lpm6_lookup((struct rte_lpm6 *)rt_ctx, dst_ip, &hop);
+
+	if (ret == 0) {
+		/* We have a hit */
+		return hop;
+	}
+
+	/* else */
+	return RTE_MAX_ETHPORTS;
+}
+
+static inline uint16_t
+get_route(struct rte_mbuf *pkt, struct route_table *rt, enum pkt_type type)
+{
+	if (type == PKT_TYPE_PLAIN_IPV4 || type == PKT_TYPE_IPSEC_IPV4)
+		return route4_pkt(pkt, rt->rt4_ctx);
+	else if (type == PKT_TYPE_PLAIN_IPV6 || type == PKT_TYPE_IPSEC_IPV6)
+		return route6_pkt(pkt, rt->rt6_ctx);
+
+	return RTE_MAX_ETHPORTS;
+}
+
+static inline int
+process_ipsec_ev_inbound(struct ipsec_ctx *ctx, struct route_table *rt,
+		struct rte_event *ev)
+{
+	struct rte_mbuf *pkt;
+	enum pkt_type type;
+	uint8_t *nlp;
+	struct ipsec_sa *sa = NULL;
+	uint16_t portid = 0;
+	uint32_t sa_idx;
+
+	/* Get pkt from event */
+	pkt = ev->mbuf;
+
+	/* Check the packet type */
+	type = process_ipsec_get_pkt_type(pkt, &nlp);
+
+	switch (type) {
+	case PKT_TYPE_PLAIN_IPV4:
+		if (pkt->ol_flags & PKT_RX_SEC_OFFLOAD)
+			sa = (struct ipsec_sa *) pkt->udata64;
+
+		/* Check if we have a match */
+		if (check_sp(ctx->sp4_ctx, nlp, &sa_idx) == 0) {
+			/* No valid match */
+			goto drop_pkt_and_exit;
+		}
+		break;
+
+	case PKT_TYPE_PLAIN_IPV6:
+		if (pkt->ol_flags & PKT_RX_SEC_OFFLOAD)
+			sa = (struct ipsec_sa *) pkt->udata64;
+
+		/* Check if we have a match */
+		if (check_sp(ctx->sp6_ctx, nlp, &sa_idx) == 0) {
+			/* No valid match */
+			goto drop_pkt_and_exit;
+		}
+		break;
+
+	default:
+		RTE_LOG(ERR, IPSEC, "Unsupported packet type = %d\n", type);
+		goto drop_pkt_and_exit;
+	}
+
+	/* Check if the packet has to be bypassed */
+	if (sa_idx == 0)
+		goto route_and_send_pkt;
+
+	/* Else the packet has to be protected with SA */
+
+	/* If the packet was IPsec processed, then SA pointer would be set */
+	if (sa == NULL)
+		goto drop_pkt_and_exit;
+
+	/* SPI came on the packet should match with the one in SA */
+	if (unlikely(sa->spi != sa_idx))
+		goto drop_pkt_and_exit;
+
+route_and_send_pkt:
+	portid = get_route(pkt, rt, type);
+	if (unlikely(portid == RTE_MAX_ETHPORTS)) {
+		/* no match */
+		goto drop_pkt_and_exit;
+	}
+	/* else, we have a matching route */
+
+	/* Update mac addresses */
+	update_mac_addrs(pkt, portid);
+
+	/* Update the event with the dest port */
+	ipsec_event_pre_forward(pkt, portid);
+	return 1;
+
+drop_pkt_and_exit:
+	RTE_LOG(ERR, IPSEC, "Inbound packet dropped\n");
+	rte_pktmbuf_free(pkt);
+	ev->mbuf = NULL;
+	return 0;
+}
+
 /*
  * Event mode exposes various operating modes depending on the
  * capabilities of the event device and the operating mode
@@ -135,11 +346,11 @@ static void
 ipsec_wrkr_non_burst_int_port_app_mode_inb(struct eh_conf *mode_conf,
 		struct eh_event_link_info *links, uint8_t nb_links)
 {
+	struct lcore_conf_ev_tx_int_port_wrkr conf;
 	struct rte_event ev;
-	struct rte_mbuf *pkt;
 	uint32_t lcore_id;
 	unsigned int nb_rx = 0;
-	unsigned int portid;
+	int32_t socket_id;
 
 	RTE_SET_USED(mode_conf);
 
@@ -154,6 +365,21 @@ ipsec_wrkr_non_burst_int_port_app_mode_inb(struct eh_conf *mode_conf,
 	/* Get core ID */
 	lcore_id = rte_lcore_id();
 
+	/* Get socket ID */
+	socket_id = rte_lcore_to_socket_id(lcore_id);
+
+	/* Save routing table */
+	conf.rt.rt4_ctx = socket_ctx[socket_id].rt_ip4;
+	conf.rt.rt6_ctx = socket_ctx[socket_id].rt_ip6;
+	conf.inbound.sp4_ctx = socket_ctx[socket_id].sp_ip4_in;
+	conf.inbound.sp6_ctx = socket_ctx[socket_id].sp_ip6_in;
+	conf.inbound.sa_ctx = socket_ctx[socket_id].sa_in;
+	conf.inbound.session_pool = socket_ctx[socket_id].session_pool;
+	conf.outbound.sp4_ctx = socket_ctx[socket_id].sp_ip4_out;
+	conf.outbound.sp6_ctx = socket_ctx[socket_id].sp_ip6_out;
+	conf.outbound.sa_ctx = socket_ctx[socket_id].sa_out;
+	conf.outbound.session_pool = socket_ctx[socket_id].session_pool;
+
 	RTE_LOG(INFO, IPSEC,
 		"Launching event mode worker (non-burst - Tx internal port - "
 		"app mode - inbound) on lcore %d\n", lcore_id);
@@ -178,13 +404,11 @@ ipsec_wrkr_non_burst_int_port_app_mode_inb(struct eh_conf *mode_conf,
 		if (nb_rx == 0)
 			continue;
 
-		portid = ev.queue_id;
-		pkt = ev.mbuf;
-
-		rte_prefetch0(rte_pktmbuf_mtod(pkt, void *));
-
-		/* Process packet */
-		ipsec_event_pre_forward(pkt, portid);
+		if (process_ipsec_ev_inbound(&conf.inbound,
+				&conf.rt, &ev) != 1) {
+			/* The pkt has been dropped */
+			continue;
+		}
 
 		/*
 		 * Since tx internal port is available, events can be
diff --git a/examples/ipsec-secgw/ipsec_worker.h b/examples/ipsec-secgw/ipsec_worker.h
new file mode 100644
index 0000000..fd18a2e
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_worker.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Cavium, Inc
+ */
+#ifndef _IPSEC_WORKER_H_
+#define _IPSEC_WORKER_H_
+
+#include "ipsec.h"
+
+enum pkt_type {
+	PKT_TYPE_PLAIN_IPV4 = 1,
+	PKT_TYPE_IPSEC_IPV4,
+	PKT_TYPE_PLAIN_IPV6,
+	PKT_TYPE_IPSEC_IPV6,
+	PKT_TYPE_INVALID
+};
+
+struct route_table {
+	struct rt_ctx *rt4_ctx;
+	struct rt_ctx *rt6_ctx;
+};
+
+/*
+ * Conf required by event mode worker with tx internal port
+ */
+struct lcore_conf_ev_tx_int_port_wrkr {
+	struct ipsec_ctx inbound;
+	struct ipsec_ctx outbound;
+	struct route_table rt;
+} __rte_cache_aligned;
+
+/* TODO
+ *
+ * Move this function to ipsec_worker.c
+ */
+void ipsec_poll_mode_worker(void);
+
+int ipsec_launch_one_lcore(void *args);
+
+#endif /* _IPSEC_WORKER_H_ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index c3cf3bd..8a41a0e 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -713,17 +713,6 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 	printf("\n");
 }
 
-struct sa_ctx {
-	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
-	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
-	union {
-		struct {
-			struct rte_crypto_sym_xform a;
-			struct rte_crypto_sym_xform b;
-		};
-	} xf[IPSEC_SA_MAX_ENTRIES];
-};
-
 static struct sa_ctx *
 sa_create(const char *name, int32_t socket_id)
 {
-- 
2.7.4

[dpdk-dev] [RFC PATCH 11/13] examples/ipsec-secgw: add driver outbound worker

[Anoob Joseph]

This patch adds the driver outbound worker thread for ipsec-secgw.
In this mode the security session is a fixed one and sa update
is not done.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c  |  6 +++
 examples/ipsec-secgw/ipsec.c        |  4 ++
 examples/ipsec-secgw/ipsec_worker.c | 91 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 100 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1c102cf..f43e1b1 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1981,6 +1981,12 @@ cryptodevs_init(void)
 			i++;
 		}
 
+		/*
+		 * Setting the queue pair to atleast the number of ethernet
+		 * devices for inline outbound.
+		 */
+		qp = RTE_MAX(rte_eth_dev_count_avail(), qp);
+
 		if (qp == 0)
 			continue;
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 1145ca1..53a868a 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -137,6 +137,8 @@ create_lookaside_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+uint16_t sa_no;
+struct rte_security_session *sec_session_fixed[10];
 int
 create_inline_session(struct socket_ctx *skt_ctx, struct ipsec_sa *sa)
 {
@@ -382,6 +384,8 @@ create_inline_session(struct socket_ctx *skt_ctx, struct ipsec_sa *sa)
 
 		sa->ol_flags = sec_cap->ol_flags;
 		sa->security_ctx = sec_ctx;
+		sec_session_fixed[sa_no] = sa->sec_session;
+		sa_no++;
 	}
 	sa->cdev_id_qp = 0;
 
diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index f93da4d..842fd5d 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -263,7 +263,7 @@ process_ipsec_ev_inbound(struct ipsec_ctx *ctx, struct route_table *rt,
  */
 
 /* Workers registered */
-#define IPSEC_EVENTMODE_WORKERS		2
+#define IPSEC_EVENTMODE_WORKERS		3
 
 /*
  * Event mode worker
@@ -425,6 +425,85 @@ ipsec_wrkr_non_burst_int_port_app_mode_inb(struct eh_conf *mode_conf,
 	return;
 }
 
+/*
+ * Event mode worker
+ * Operating parameters : non-burst - Tx internal port - driver mode - outbound
+ */
+extern struct rte_security_session *sec_session_fixed[10];
+static void
+ipsec_wrkr_non_burst_int_port_drvr_mode_outb(struct eh_conf *mode_conf,
+		struct eh_event_link_info *links, uint8_t nb_links)
+{
+	struct rte_event ev;
+	struct rte_mbuf *pkt;
+	uint32_t lcore_id;
+	unsigned int nb_rx = 0;
+	unsigned int portid;
+
+	RTE_SET_USED(mode_conf);
+
+	/* Check if we have links registered for this lcore */
+	if (nb_links == 0) {
+		/* No links registered - exit */
+		goto exit;
+	}
+
+	/* Get core ID */
+	lcore_id = rte_lcore_id();
+
+	RTE_LOG(INFO, IPSEC,
+		"Launching event mode worker (non-burst - Tx internal port - "
+		"driver mode - outbound) on lcore %d\n", lcore_id);
+
+	/* We have valid links */
+
+	/* See if it's single link */
+	if (nb_links != 1) {
+		RTE_LOG(INFO, IPSEC,
+			"Multiple links not supported. Using first link\n");
+	}
+
+	RTE_LOG(INFO, IPSEC, " -- lcoreid=%u event_port_id=%u\n", lcore_id,
+			links[0].event_portid);
+	while (!force_quit) {
+		/* Read packet from event queues */
+		nb_rx = rte_event_dequeue_burst(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1,	/* nb_events */
+				0	/* timeout_ticks */);
+
+		if (nb_rx == 0)
+			continue;
+
+		portid = ev.queue_id;
+		pkt = ev.mbuf;
+
+		rte_prefetch0(rte_pktmbuf_mtod(pkt, void *));
+
+		/* Process packet */
+		ipsec_event_pre_forward(pkt, portid);
+
+		pkt->udata64 = (uint64_t) sec_session_fixed[portid];
+
+		/* Mark the packet for Tx security offload */
+		pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+		/*
+		 * Since tx internal port is available, events can be
+		 * directly enqueued to the adapter and it would be
+		 * internally submitted to the eth device.
+		 */
+		rte_event_eth_tx_adapter_enqueue(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1	/* nb_events */);
+	}
+
+exit:
+	return;
+}
+
 static uint8_t
 ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 {
@@ -451,6 +530,16 @@ ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_INBOUND;
 	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_app_mode_inb;
 
+	wrkr++;
+	nb_wrkr_param++;
+
+	/* Non-burst - Tx internal port - driver mode - outbound */
+	wrkr->cap.burst = EH_RX_TYPE_NON_BURST;
+	wrkr->cap.tx_internal_port = EH_TX_TYPE_INTERNAL_PORT;
+	wrkr->cap.ipsec_mode = EH_IPSEC_MODE_TYPE_DRIVER;
+	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_OUTBOUND;
+	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_drvr_mode_outb;
+
 	nb_wrkr_param++;
 	return nb_wrkr_param;
 }
-- 
2.7.4

[dpdk-dev] [RFC PATCH 12/13] examples/ipsec-secgw: add app outbound worker

[Anoob Joseph]

This patch adds the app outbound worker thread.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/ipsec_worker.c | 190 +++++++++++++++++++++++++++++++++++-
 1 file changed, 189 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/ipsec_worker.c b/examples/ipsec-secgw/ipsec_worker.c
index 842fd5d..8d96170 100644
--- a/examples/ipsec-secgw/ipsec_worker.c
+++ b/examples/ipsec-secgw/ipsec_worker.c
@@ -256,6 +256,97 @@ process_ipsec_ev_inbound(struct ipsec_ctx *ctx, struct route_table *rt,
 	return 0;
 }
 
+static inline int
+process_ipsec_ev_outbound(struct ipsec_ctx *ctx, struct route_table *rt,
+		struct rte_event *ev)
+{
+	struct rte_mbuf *pkt;
+	enum pkt_type type;
+	uint8_t *nlp;
+	uint32_t sa_idx;
+	struct sa_ctx *sa_ctx;
+	struct ipsec_sa *sa;
+	uint16_t portid = 0;
+
+	/* Get pkt from event */
+	pkt = ev->mbuf;
+
+	/* Check the packet type */
+	type = process_ipsec_get_pkt_type(pkt, &nlp);
+
+	switch (type) {
+	case PKT_TYPE_PLAIN_IPV4:
+		/* Check if we have a match */
+		if (check_sp(ctx->sp4_ctx, nlp, &sa_idx) == 0) {
+			/* No valid match */
+			goto drop_pkt_and_exit;
+		}
+		break;
+	case PKT_TYPE_PLAIN_IPV6:
+		/* Check if we have a match */
+		if (check_sp(ctx->sp6_ctx, nlp, &sa_idx) == 0) {
+			/* No valid match */
+			goto drop_pkt_and_exit;
+		}
+		break;
+	default:
+		/*
+		 * Only plain IPv4 & IPv6 packets allowed on protected port.
+		 * Drop the rest.
+		 */
+		RTE_LOG(ERR, IPSEC, "Unsupported packet type = %d\n", type);
+		goto drop_pkt_and_exit;
+	}
+
+	/* Check if the packet has to be bypassed */
+	if (sa_idx == 0) {
+		portid = get_route(pkt, rt, type);
+		if (unlikely(portid == RTE_MAX_ETHPORTS)) {
+			/* no match */
+			goto drop_pkt_and_exit;
+		}
+		/* else, we have a matching route */
+		goto send_pkt;
+	}
+
+	/* Else the packet has to be protected */
+
+	/* Get SA ctx*/
+	sa_ctx = ctx->sa_ctx;
+
+	/* Get SA */
+	sa = &(sa_ctx->sa[sa_idx]);
+
+	/* Only inline protocol allowed for now */
+	if (sa->type != RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+		RTE_LOG(ERR, IPSEC, "SA type not supported\n");
+		goto drop_pkt_and_exit;
+	}
+
+	if (sa->ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA)
+		pkt->udata64 = (uint64_t) sa->sec_session;
+
+	/* Mark the packet for Tx security offload */
+	pkt->ol_flags |= PKT_TX_SEC_OFFLOAD;
+
+	/* Get the port to which this pkt need to be submitted */
+	portid = sa->portid;
+
+send_pkt:
+	/* Update mac addresses */
+	update_mac_addrs(pkt, portid);
+
+	/* Update the event with the dest port */
+	ipsec_event_pre_forward(pkt, portid);
+	return 1;
+
+drop_pkt_and_exit:
+	RTE_LOG(ERR, IPSEC, "Outbound packet dropped\n");
+	rte_pktmbuf_free(pkt);
+	ev->mbuf = NULL;
+	return 0;
+}
+
 /*
  * Event mode exposes various operating modes depending on the
  * capabilities of the event device and the operating mode
@@ -263,7 +354,7 @@ process_ipsec_ev_inbound(struct ipsec_ctx *ctx, struct route_table *rt,
  */
 
 /* Workers registered */
-#define IPSEC_EVENTMODE_WORKERS		3
+#define IPSEC_EVENTMODE_WORKERS		4
 
 /*
  * Event mode worker
@@ -504,6 +595,93 @@ ipsec_wrkr_non_burst_int_port_drvr_mode_outb(struct eh_conf *mode_conf,
 	return;
 }
 
+/*
+ * Event mode worker
+ * Operating parameters : non-burst - Tx internal port - app mode - outbound
+ */
+static void
+ipsec_wrkr_non_burst_int_port_app_mode_outb(struct eh_conf *mode_conf,
+		struct eh_event_link_info *links, uint8_t nb_links)
+{
+	struct lcore_conf_ev_tx_int_port_wrkr conf;
+	struct rte_event ev;
+	uint32_t lcore_id;
+	unsigned int nb_rx = 0;
+	int32_t socket_id;
+
+	RTE_SET_USED(mode_conf);
+
+	/* Check if we have links registered for this lcore */
+	if (nb_links == 0) {
+		/* No links registered - exit */
+		goto exit;
+	}
+
+	/* We have valid links */
+
+	/* Get core ID */
+	lcore_id = rte_lcore_id();
+
+	/* Get socket ID */
+	socket_id = rte_lcore_to_socket_id(lcore_id);
+
+	/* Save routing table */
+	conf.rt.rt4_ctx = socket_ctx[socket_id].rt_ip4;
+	conf.rt.rt6_ctx = socket_ctx[socket_id].rt_ip6;
+	conf.inbound.sp4_ctx = socket_ctx[socket_id].sp_ip4_in;
+	conf.inbound.sp6_ctx = socket_ctx[socket_id].sp_ip6_in;
+	conf.inbound.sa_ctx = socket_ctx[socket_id].sa_in;
+	conf.inbound.session_pool = socket_ctx[socket_id].session_pool;
+	conf.outbound.sp4_ctx = socket_ctx[socket_id].sp_ip4_out;
+	conf.outbound.sp6_ctx = socket_ctx[socket_id].sp_ip6_out;
+	conf.outbound.sa_ctx = socket_ctx[socket_id].sa_out;
+	conf.outbound.session_pool = socket_ctx[socket_id].session_pool;
+
+	RTE_LOG(INFO, IPSEC,
+		"Launching event mode worker (non-burst - Tx internal port - "
+		"app mode - outbound) on lcore %d\n", lcore_id);
+
+	/* See if it's single link */
+	if (nb_links != 1) {
+		RTE_LOG(INFO, IPSEC,
+			"Multiple links not supported. Using first link\n");
+	}
+
+	RTE_LOG(INFO, IPSEC, " -- lcoreid=%u event_port_id=%u\n", lcore_id,
+		links[0].event_portid);
+
+	while (!force_quit) {
+		/* Read packet from event queues */
+		nb_rx = rte_event_dequeue_burst(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,     /* events */
+				1,       /* nb_events */
+				0        /* timeout_ticks */);
+
+		if (nb_rx == 0)
+			continue;
+
+		if (process_ipsec_ev_outbound(&conf.outbound,
+				&conf.rt, &ev) != 1) {
+			/* The pkt has been dropped */
+			continue;
+		}
+
+		/*
+		 * Since tx internal port is available, events can be
+		 * directly enqueued to the adapter and it would be
+		 * internally submitted to the eth device.
+		 */
+		rte_event_eth_tx_adapter_enqueue(links[0].eventdev_id,
+				links[0].event_portid,
+				&ev,	/* events */
+				1	/* nb_events */);
+	}
+
+exit:
+	return;
+}
+
 static uint8_t
 ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 {
@@ -540,6 +718,16 @@ ipsec_eventmode_populate_wrkr_params(struct eh_app_worker_params *wrkrs)
 	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_OUTBOUND;
 	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_drvr_mode_outb;
 
+	wrkr++;
+	nb_wrkr_param++;
+
+	/* Non-burst - Tx internal port - app mode - outbound */
+	wrkr->cap.burst = EH_RX_TYPE_NON_BURST;
+	wrkr->cap.tx_internal_port = EH_TX_TYPE_INTERNAL_PORT;
+	wrkr->cap.ipsec_mode = EH_IPSEC_MODE_TYPE_APP;
+	wrkr->cap.ipsec_dir = EH_IPSEC_DIR_TYPE_OUTBOUND;
+	wrkr->worker_thread = ipsec_wrkr_non_burst_int_port_app_mode_outb;
+
 	nb_wrkr_param++;
 	return nb_wrkr_param;
 }
-- 
2.7.4

[dpdk-dev] [RFC PATCH 13/13] examples/ipsec-secgw: add cmd line option for bufs

[Anoob Joseph]

Add command line option -s which can be used to configure number
of buffers in a pool. Default number of buffers is 4000.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Lukasz Bartosik <lbartosik@marvell.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f43e1b1..4200877 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -59,8 +59,6 @@ volatile bool force_quit;
 
 #define MEMPOOL_CACHE_SIZE 256
 
-#define NB_MBUF	(32000)
-
 #define CDEV_QUEUE_DESC 2048
 #define CDEV_MAP_ENTRIES 16384
 #define CDEV_MP_NB_OBJS 1024
@@ -157,6 +155,7 @@ static int32_t numa_on = 1; /**< NUMA is enabled by default. */
 static uint32_t nb_lcores;
 static uint32_t single_sa;
 static uint32_t single_sa_idx;
+static uint32_t nb_bufs_in_pool = 8192;
 
 /*
  * RX/TX HW offload capabilities to enable/use on ethernet ports.
@@ -1249,6 +1248,7 @@ print_usage(const char *prgname)
 		" [-w REPLAY_WINDOW_SIZE]"
 		" [-e]"
 		" [-a]"
+		" [-s NUMBER_OF_MBUFS_IN_PKT_POOL]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1272,6 +1272,7 @@ print_usage(const char *prgname)
 		"     size for each SA\n"
 		"  -e enables ESN\n"
 		"  -a enables SA SQN atomic behaviour\n"
+		"  -s number of mbufs in packet pool (default 8192)\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1515,7 +1516,7 @@ parse_args(int32_t argc, char **argv, struct eh_conf *eh_conf)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:s:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1554,6 +1555,19 @@ parse_args(int32_t argc, char **argv, struct eh_conf *eh_conf)
 			}
 			f_present = 1;
 			break;
+
+		case 's':
+			ret = parse_decimal(optarg);
+			if (ret < 0) {
+				printf("Invalid number of buffers in a pool: "
+					"%s\n", optarg);
+				print_usage(prgname);
+				return -1;
+			}
+
+			nb_bufs_in_pool = ret;
+			break;
+
 		case 'j':
 			ret = parse_decimal(optarg);
 			if (ret < RTE_MBUF_DEFAULT_BUF_SIZE ||
@@ -2672,11 +2686,12 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
+		pool_init(&socket_ctx[socket_id], socket_id, nb_bufs_in_pool);
 		session_pool_init(&socket_ctx[socket_id], socket_id, sess_sz);
 		session_priv_pool_init(&socket_ctx[socket_id], socket_id,
 			sess_sz);
 	}
+	printf("Number of mbufs in packet pool %d\n", nb_bufs_in_pool);
 
 	RTE_ETH_FOREACH_DEV(portid) {
 		if ((enabled_port_mask & (1 << portid)) == 0)
-- 
2.7.4

Re: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Ananyev, Konstantin]

> This series introduces event-mode additions to ipsec-secgw. This effort
> is based on the proposed changes for l2fwd-event and the additions in
> l3fwd for event support.
> 
> With this series, ipsec-secgw would be able to run in eventmode. The
> worker thread (executing loop) would be receiving events and would be
> submitting it back to the eventdev after the processing. This way,
> multicore scaling and h/w assisted scheduling is achieved by making use
> of the eventdev capabilities.
> 
> Since the underlying event device would be having varying capabilities,
> the worker thread could be drafted differently to maximize performance.
> This series introduces usage of multiple worker threads, among which the
> one to be used will be determined by the operating conditions and the
> underlying device capabilities.
> 
> For example, if an event device - eth device pair has Tx internal port,
> then application can do tx_adapter_enqueue() instead of regular
> event_enqueue(). So a thread making an assumption that the device pair
> has internal port will not be the right solution for another pair. The
> infrastructure added with these patches aims to help application to have
> multiple worker threads, there by extracting maximum performance from
> every device without affecting existing paths/use cases.
> 
> The eventmode configuration is predefined. All packets reaching one eth
> port will hit one event queue. All event queues will be mapped to all
> event ports. So all cores will be able to receive traffic from all ports.
> When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event device
> will ensure the ordering. Ordering would be lost when tried in PARALLEL.
> 
> Following command line options are introduced,
> 
> --transfer-mode: to choose between poll mode & event mode
> --schedule-type: to specify the scheduling type
>                  (RTE_SCHED_TYPE_ORDERED/
>                   RTE_SCHED_TYPE_ATOMIC/
>                   RTE_SCHED_TYPE_PARALLEL)
> --process-dir: outbound/inbound
> --process-mode: app mode /driver mode
> 
> The two s/w config options added to ipsec-secgw can be used in
> benchmarking h/w performance,
> 

I didn't look at the actual code (yet), just cover letter,
few quick questions below. 

> 1. process-dir : states whether the direction is outbound/inbound.
> This option aims to avoid an unnecessary check of determining whether
> inbound/outbound processing need to be done on the packet. For each
> option a different light weight worker thread would be executed.

Bur right now app can do both inbound and outbound simultaneously on the same core.
I presume the default behavior will be preserved?

> 
> 2. process-mode: states whether the application has to run in driver
>                  mode or app mode.
> 
> Driver-mode: This mode will have bare minimum changes in the application
>              to support ipsec. There woudn't be any lookup etc done in
>              the application. And for inline-protocol use case, the
>              thread would resemble l2fwd as the ipsec processing would be
>              done entirely in the h/w. This mode can be used to benchmark
>              the raw performance of the h/w. All the application side
>              steps (like lookup) can be redone based on the requirement
>              of the end user. Hence the need for a mode which would
>              report the raw performance.
> 
> App-mode: This mode will have all the features currently implemented with
>           ipsec-secgw (non librte_ipsec mode). All the lookups etc
>           would follow the existing methods and would report numbers
>           that can be compared against regular ipsec-secgw benchmark
>           numbers.
> 
> Example commands to execute ipsec-secgw in various modes on OCTEONTX2 platform,
> 
> #Inbound driver mode
> ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7 – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f
> dpdk_internal/100g_4.3.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 1

For all these parameters, I think better to use names (app/driver, etc.) instead of raw numbers.

> 
> #Inbound app mode
> ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x3f – -P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -
> f dpdk_internal/100g_4.3.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 1
> 
> #Outbound driver mode
> ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1f – -
> P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 0
> 
> #Outbound app mode
> ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7f – -
> P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 0
> 
> This series is targeted for next release (20.02). This series doesn't introduce
> any library change. 

By 'library change' you mean that this new event-mode will be supported
only by legacy code-path or ...?

>And the decision to add eventmode additions in ipsec-secgw
> was approved by the Tech Board.
> 
> Following are missing in the RFC. Will add it when sending patches.
> 1. Documentation.
> 2. More cleanup is needed. There are options that are added so that future
>    expansion is not hindered. Need inputs from the community if there is use
>    case for them.
> 
> Following are planned features,
> 1. Add burst mode workers.
> 2. Add non tx internal port worker.
> 3. Verify support for Rx core (the support is added but lack of h/w to verify).
> 4. Add lookaside protocol support.
> 
> Following are features that Marvell won't be attempting.
> 1. Inline crypto support.
> 2. Lookaside crypto support.

Ok so what mode is supported right now with this RFC?

> 
> For the features that Marvell won't be attempting, new workers can be
> introduced by the respective stake holders.
> 
> Anoob Joseph (13):
>   examples/ipsec-secgw: add framework for eventmode helper
>   examples/ipsec-secgw: add eventdev port-lcore link
>   examples/ipsec-secgw: add Rx adapter support
>   examples/ipsec-secgw: add Tx adapter support
>   examples/ipsec-secgw: add routines to display config
>   examples/ipsec-secgw: add routines to launch workers
>   examples/ipsec-secgw: add support for internal ports
>   examples/ipsec-secgw: add eventmode to ipsec-secgw
>   examples/ipsec-secgw: add app inbound worker
>   examples/ipsec-secgw: add app processing code
>   examples/ipsec-secgw: add driver outbound worker
>   examples/ipsec-secgw: add app outbound worker
>   examples/ipsec-secgw: add cmd line option for bufs
> 
>  examples/ipsec-secgw/Makefile       |    2 +
>  examples/ipsec-secgw/event_helper.c | 1757 +++++++++++++++++++++++++++++++++++
>  examples/ipsec-secgw/event_helper.h |  334 +++++++
>  examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
>  examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
>  examples/ipsec-secgw/ipsec.c        |    4 +
>  examples/ipsec-secgw/ipsec.h        |   30 +-
>  examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
>  examples/ipsec-secgw/ipsec_worker.h |   39 +
>  examples/ipsec-secgw/meson.build    |    4 +-
>  examples/ipsec-secgw/sa.c           |   11 -
>  11 files changed, 3360 insertions(+), 104 deletions(-)
>  create mode 100644 examples/ipsec-secgw/event_helper.c
>  create mode 100644 examples/ipsec-secgw/event_helper.h
>  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
>  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
>  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
> 
> --
> 2.7.4

Re: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Anoob Joseph]

Hi Konstantin,

Thanks for the review. Please see inline.

Thanks,
Anoob

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Wednesday, October 16, 2019 6:33 PM
> To: Anoob Joseph <anoobj@marvell.com>; Akhil Goyal
> <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Narayana Prasad Raju Athreya
> <pathreya@marvell.com>; Lukas Bartosik <lbartosik@marvell.com>;
> dev@dpdk.org
> Subject: [EXT] RE: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-
> secgw
> 
> External Email
> 
> ----------------------------------------------------------------------
> 
> 
> > This series introduces event-mode additions to ipsec-secgw. This
> > effort is based on the proposed changes for l2fwd-event and the
> > additions in l3fwd for event support.
> >
> > With this series, ipsec-secgw would be able to run in eventmode. The
> > worker thread (executing loop) would be receiving events and would be
> > submitting it back to the eventdev after the processing. This way,
> > multicore scaling and h/w assisted scheduling is achieved by making
> > use of the eventdev capabilities.
> >
> > Since the underlying event device would be having varying
> > capabilities, the worker thread could be drafted differently to maximize
> performance.
> > This series introduces usage of multiple worker threads, among which
> > the one to be used will be determined by the operating conditions and
> > the underlying device capabilities.
> >
> > For example, if an event device - eth device pair has Tx internal
> > port, then application can do tx_adapter_enqueue() instead of regular
> > event_enqueue(). So a thread making an assumption that the device pair
> > has internal port will not be the right solution for another pair. The
> > infrastructure added with these patches aims to help application to
> > have multiple worker threads, there by extracting maximum performance
> > from every device without affecting existing paths/use cases.
> >
> > The eventmode configuration is predefined. All packets reaching one
> > eth port will hit one event queue. All event queues will be mapped to
> > all event ports. So all cores will be able to receive traffic from all ports.
> > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event
> > device will ensure the ordering. Ordering would be lost when tried in
> PARALLEL.
> >
> > Following command line options are introduced,
> >
> > --transfer-mode: to choose between poll mode & event mode
> > --schedule-type: to specify the scheduling type
> >                  (RTE_SCHED_TYPE_ORDERED/
> >                   RTE_SCHED_TYPE_ATOMIC/
> >                   RTE_SCHED_TYPE_PARALLEL)
> > --process-dir: outbound/inbound
> > --process-mode: app mode /driver mode
> >
> > The two s/w config options added to ipsec-secgw can be used in
> > benchmarking h/w performance,
> >
> 
> I didn't look at the actual code (yet), just cover letter, few quick questions
> below.
> 
> > 1. process-dir : states whether the direction is outbound/inbound.
> > This option aims to avoid an unnecessary check of determining whether
> > inbound/outbound processing need to be done on the packet. For each
> > option a different light weight worker thread would be executed.
> 
> Bur right now app can do both inbound and outbound simultaneously on the
> same core.
> I presume the default behavior will be preserved?

[Anoob] The existing behavior with poll mode thread is not touched. The current changes are only applicable when launched in event-mode.

> 
> >
> > 2. process-mode: states whether the application has to run in driver
> >                  mode or app mode.
> >
> > Driver-mode: This mode will have bare minimum changes in the application
> >              to support ipsec. There woudn't be any lookup etc done in
> >              the application. And for inline-protocol use case, the
> >              thread would resemble l2fwd as the ipsec processing would be
> >              done entirely in the h/w. This mode can be used to benchmark
> >              the raw performance of the h/w. All the application side
> >              steps (like lookup) can be redone based on the requirement
> >              of the end user. Hence the need for a mode which would
> >              report the raw performance.
> >
> > App-mode: This mode will have all the features currently implemented
> with
> >           ipsec-secgw (non librte_ipsec mode). All the lookups etc
> >           would follow the existing methods and would report numbers
> >           that can be compared against regular ipsec-secgw benchmark
> >           numbers.
> >
> > Example commands to execute ipsec-secgw in various modes on
> OCTEONTX2
> > platform,
> >
> > #Inbound driver mode
> > ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w
> > 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> > 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1
> > --log-level=8 -c 0x7 – -P -p 0xf --config
> > "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f dpdk_internal/100g_4.3.cfg
> > --transfer-mode 1 --schedule-type 2 --process-mode 1 --process-dir 1
> 
> For all these parameters, I think better to use names (app/driver, etc.)
> instead of raw numbers.

[Anoob] Agreed. Will make this change when we submit the actual patches.

> 
> >
> > #Inbound app mode
> > ./ipsec-secgw -w 0002:02:00.0,nb_ipsec_in_sa=128 -w
> > 0002:03:00.0,nb_ipsec_in_sa=128 -w 0002:04:00.0,nb_ipsec_in_sa=128 -w
> > 0002:07:00.0,nb_ipsec_in_sa=128 -w 0002:0e:00.0 -w 0002:10:00.1
> > --log-level=8 -c 0x3f – -P -p 0xf --config
> > "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" - f dpdk_internal/100g_4.3.cfg
> > --transfer-mode 1 --schedule-type 2 --process-mode 0 --process-dir 1
> >
> > #Outbound driver mode
> > ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w
> > 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1f – -
> > P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f
> > a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 1
> > --process-dir 0
> >
> > #Outbound app mode
> > ./ipsec-secgw -w 0002:02:00.0 -w 0002:03:00.0 -w 0002:04:00.0 -w
> > 0002:07:00.0 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x7f – -
> > P -p 0xf --config "(0,0,0),(1,0,0),(2,0,0),(3,0,0)" -f
> > a-aes-gcm-new.cfg --transfer-mode 1 --schedule-type 2 --process-mode 0
> > --process-dir 0
> >
> > This series is targeted for next release (20.02). This series doesn't
> > introduce any library change.
> 
> By 'library change' you mean that this new event-mode will be supported
> only by legacy code-path or ...?

[Anoob] All the changes are confined to 'examples/ipsec-secgw' directory. Right now, the worker threads make use of the existing routines in non-librte_ipsec mode. 

> 
> >And the decision to add eventmode additions in ipsec-secgw  was
> >approved by the Tech Board.
> >
> > Following are missing in the RFC. Will add it when sending patches.
> > 1. Documentation.
> > 2. More cleanup is needed. There are options that are added so that future
> >    expansion is not hindered. Need inputs from the community if there is
> use
> >    case for them.
> >
> > Following are planned features,
> > 1. Add burst mode workers.
> > 2. Add non tx internal port worker.
> > 3. Verify support for Rx core (the support is added but lack of h/w to
> verify).
> > 4. Add lookaside protocol support.
> >
> > Following are features that Marvell won't be attempting.
> > 1. Inline crypto support.
> > 2. Lookaside crypto support.
> 
> Ok so what mode is supported right now with this RFC?

[Anoob] Inline protocol support is added with the RFC.

> 
> >
> > For the features that Marvell won't be attempting, new workers can be
> > introduced by the respective stake holders.
> >
> > Anoob Joseph (13):
> >   examples/ipsec-secgw: add framework for eventmode helper
> >   examples/ipsec-secgw: add eventdev port-lcore link
> >   examples/ipsec-secgw: add Rx adapter support
> >   examples/ipsec-secgw: add Tx adapter support
> >   examples/ipsec-secgw: add routines to display config
> >   examples/ipsec-secgw: add routines to launch workers
> >   examples/ipsec-secgw: add support for internal ports
> >   examples/ipsec-secgw: add eventmode to ipsec-secgw
> >   examples/ipsec-secgw: add app inbound worker
> >   examples/ipsec-secgw: add app processing code
> >   examples/ipsec-secgw: add driver outbound worker
> >   examples/ipsec-secgw: add app outbound worker
> >   examples/ipsec-secgw: add cmd line option for bufs
> >
> >  examples/ipsec-secgw/Makefile       |    2 +
> >  examples/ipsec-secgw/event_helper.c | 1757
> > +++++++++++++++++++++++++++++++++++
> >  examples/ipsec-secgw/event_helper.h |  334 +++++++
> > examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
> >  examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
> >  examples/ipsec-secgw/ipsec.c        |    4 +
> >  examples/ipsec-secgw/ipsec.h        |   30 +-
> >  examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
> >  examples/ipsec-secgw/ipsec_worker.h |   39 +
> >  examples/ipsec-secgw/meson.build    |    4 +-
> >  examples/ipsec-secgw/sa.c           |   11 -
> >  11 files changed, 3360 insertions(+), 104 deletions(-)  create mode
> > 100644 examples/ipsec-secgw/event_helper.c
> >  create mode 100644 examples/ipsec-secgw/event_helper.h
> >  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
> >  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
> >  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
> >
> > --
> > 2.7.4

Re: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Ananyev, Konstantin]

<snip>

> > >
> > > This series is targeted for next release (20.02). This series doesn't
> > > introduce any library change.
> >
> > By 'library change' you mean that this new event-mode will be supported
> > only by legacy code-path or ...?
> 
> [Anoob] All the changes are confined to 'examples/ipsec-secgw' directory. Right now, the worker threads make use of the existing
> routines in non-librte_ipsec mode.

And as I understand, you don't plan to use/support library mode?
I suppose I need to look to the actual code to understand more here.

> 
> >
> > >And the decision to add eventmode additions in ipsec-secgw  was
> > >approved by the Tech Board.
> > >
> > > Following are missing in the RFC. Will add it when sending patches.
> > > 1. Documentation.
> > > 2. More cleanup is needed. There are options that are added so that future
> > >    expansion is not hindered. Need inputs from the community if there is
> > use
> > >    case for them.
> > >
> > > Following are planned features,
> > > 1. Add burst mode workers.
> > > 2. Add non tx internal port worker.
> > > 3. Verify support for Rx core (the support is added but lack of h/w to
> > verify).
> > > 4. Add lookaside protocol support.
> > >
> > > Following are features that Marvell won't be attempting.
> > > 1. Inline crypto support.
> > > 2. Lookaside crypto support.
> >
> > Ok so what mode is supported right now with this RFC?
> 
> [Anoob] Inline protocol support is added with the RFC.

Ok..., but  do we have within DPDK any PMD that implements inline-proto?
AFAIK we don't, but might be I am missing something?

> 
> >
> > >
> > > For the features that Marvell won't be attempting, new workers can be
> > > introduced by the respective stake holders.
> > >
> > > Anoob Joseph (13):
> > >   examples/ipsec-secgw: add framework for eventmode helper
> > >   examples/ipsec-secgw: add eventdev port-lcore link
> > >   examples/ipsec-secgw: add Rx adapter support
> > >   examples/ipsec-secgw: add Tx adapter support
> > >   examples/ipsec-secgw: add routines to display config
> > >   examples/ipsec-secgw: add routines to launch workers
> > >   examples/ipsec-secgw: add support for internal ports
> > >   examples/ipsec-secgw: add eventmode to ipsec-secgw
> > >   examples/ipsec-secgw: add app inbound worker
> > >   examples/ipsec-secgw: add app processing code
> > >   examples/ipsec-secgw: add driver outbound worker
> > >   examples/ipsec-secgw: add app outbound worker
> > >   examples/ipsec-secgw: add cmd line option for bufs
> > >
> > >  examples/ipsec-secgw/Makefile       |    2 +
> > >  examples/ipsec-secgw/event_helper.c | 1757
> > > +++++++++++++++++++++++++++++++++++
> > >  examples/ipsec-secgw/event_helper.h |  334 +++++++
> > > examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
> > >  examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
> > >  examples/ipsec-secgw/ipsec.c        |    4 +
> > >  examples/ipsec-secgw/ipsec.h        |   30 +-
> > >  examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
> > >  examples/ipsec-secgw/ipsec_worker.h |   39 +
> > >  examples/ipsec-secgw/meson.build    |    4 +-
> > >  examples/ipsec-secgw/sa.c           |   11 -
> > >  11 files changed, 3360 insertions(+), 104 deletions(-)  create mode
> > > 100644 examples/ipsec-secgw/event_helper.c
> > >  create mode 100644 examples/ipsec-secgw/event_helper.h
> > >  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
> > >  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
> > >  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
> > >
> > > --
> > > 2.7.4

Re: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-secgw

[Anoob Joseph]

Hi Konstantin,

Please see inline.

Thanks,
Anoob

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Friday, October 25, 2019 3:10 PM
> To: Anoob Joseph <anoobj@marvell.com>; Akhil Goyal
> <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Narayana Prasad Raju Athreya
> <pathreya@marvell.com>; Lukas Bartosik <lbartosik@marvell.com>;
> dev@dpdk.org
> Subject: [EXT] RE: [dpdk-dev] [RFC PATCH 00/13] add eventmode to ipsec-
> secgw
> 
> External Email
> 
> ----------------------------------------------------------------------
> 
> <snip>
> 
> > > >
> > > > This series is targeted for next release (20.02). This series
> > > > doesn't introduce any library change.
> > >
> > > By 'library change' you mean that this new event-mode will be
> > > supported only by legacy code-path or ...?
> >
> > [Anoob] All the changes are confined to 'examples/ipsec-secgw'
> > directory. Right now, the worker threads make use of the existing routines
> in non-librte_ipsec mode.
> 
> And as I understand, you don't plan to use/support library mode?
> I suppose I need to look to the actual code to understand more here.

[Anoob] Supporting library mode is also in the pipeline. But initial target would be support with non-librte_ipsec mode.
 
> 
> >
> > >
> > > >And the decision to add eventmode additions in ipsec-secgw  was
> > > >approved by the Tech Board.
> > > >
> > > > Following are missing in the RFC. Will add it when sending patches.
> > > > 1. Documentation.
> > > > 2. More cleanup is needed. There are options that are added so that
> future
> > > >    expansion is not hindered. Need inputs from the community if
> > > > there is
> > > use
> > > >    case for them.
> > > >
> > > > Following are planned features,
> > > > 1. Add burst mode workers.
> > > > 2. Add non tx internal port worker.
> > > > 3. Verify support for Rx core (the support is added but lack of
> > > > h/w to
> > > verify).
> > > > 4. Add lookaside protocol support.
> > > >
> > > > Following are features that Marvell won't be attempting.
> > > > 1. Inline crypto support.
> > > > 2. Lookaside crypto support.
> > >
> > > Ok so what mode is supported right now with this RFC?
> >
> > [Anoob] Inline protocol support is added with the RFC.
> 
> Ok..., but  do we have within DPDK any PMD that implements inline-proto?
> AFAIK we don't, but might be I am missing something?

[Anoob] Inline protocol support would be added to OCTEON TX2 eth PMD in the next cycle. This is an RFC to introduce the idea of using events in ipsec-segcw, which is required for OCTEON TX2 platform to perform inline ipsec processing.

> 
> >
> > >
> > > >
> > > > For the features that Marvell won't be attempting, new workers can
> > > > be introduced by the respective stake holders.
> > > >
> > > > Anoob Joseph (13):
> > > >   examples/ipsec-secgw: add framework for eventmode helper
> > > >   examples/ipsec-secgw: add eventdev port-lcore link
> > > >   examples/ipsec-secgw: add Rx adapter support
> > > >   examples/ipsec-secgw: add Tx adapter support
> > > >   examples/ipsec-secgw: add routines to display config
> > > >   examples/ipsec-secgw: add routines to launch workers
> > > >   examples/ipsec-secgw: add support for internal ports
> > > >   examples/ipsec-secgw: add eventmode to ipsec-secgw
> > > >   examples/ipsec-secgw: add app inbound worker
> > > >   examples/ipsec-secgw: add app processing code
> > > >   examples/ipsec-secgw: add driver outbound worker
> > > >   examples/ipsec-secgw: add app outbound worker
> > > >   examples/ipsec-secgw: add cmd line option for bufs
> > > >
> > > >  examples/ipsec-secgw/Makefile       |    2 +
> > > >  examples/ipsec-secgw/event_helper.c | 1757
> > > > +++++++++++++++++++++++++++++++++++
> > > >  examples/ipsec-secgw/event_helper.h |  334 +++++++
> > > > examples/ipsec-secgw/ipsec-secgw.c  |  436 +++++++--
> > > >  examples/ipsec-secgw/ipsec-secgw.h  |   81 ++
> > > >  examples/ipsec-secgw/ipsec.c        |    4 +
> > > >  examples/ipsec-secgw/ipsec.h        |   30 +-
> > > >  examples/ipsec-secgw/ipsec_worker.c |  766 +++++++++++++++
> > > >  examples/ipsec-secgw/ipsec_worker.h |   39 +
> > > >  examples/ipsec-secgw/meson.build    |    4 +-
> > > >  examples/ipsec-secgw/sa.c           |   11 -
> > > >  11 files changed, 3360 insertions(+), 104 deletions(-)  create
> > > > mode
> > > > 100644 examples/ipsec-secgw/event_helper.c
> > > >  create mode 100644 examples/ipsec-secgw/event_helper.h
> > > >  create mode 100644 examples/ipsec-secgw/ipsec-secgw.h
> > > >  create mode 100644 examples/ipsec-secgw/ipsec_worker.c
> > > >  create mode 100644 examples/ipsec-secgw/ipsec_worker.h
> > > >
> > > > --
> > > > 2.7.4