[dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang]

Lock-free mode is supported by hash library and LPM library.
Now we add an option for l3fwd example to enable the lock-free mode.
Necessary preparation steps are added to use lock-free LPM mode.

Patch 2/2 has dependency on RCU QSBR integration with LPM library:
http://patches.dpdk.org/project/dpdk/list/?series=6288


Ruifeng Wang (2):
  examples/l3fwd: add lock-free option for l3fwd
  examples/l3fwd: integrate RCU QSBR for LPM mode

 doc/guides/sample_app_ug/l3_forward.rst |  3 ++
 examples/l3fwd/Makefile                 |  1 +
 examples/l3fwd/l3fwd.h                  |  4 +-
 examples/l3fwd/l3fwd_em.c               | 10 +++-
 examples/l3fwd/l3fwd_lpm.c              | 72 +++++++++++++++++++++++--
 examples/l3fwd/main.c                   | 27 ++++++++--
 examples/l3fwd/meson.build              |  1 +
 7 files changed, 108 insertions(+), 10 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH 1/2] examples/l3fwd: add lock-free option for l3fwd

[Ruifeng Wang]

The new option is provided for lock-free read / write concurrency.
For EM mode, lock-free APIs will be used when this option is enabled.

Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Gavin Hu <gavin.hu@arm.com>
---
 doc/guides/sample_app_ug/l3_forward.rst |  3 +++
 examples/l3fwd/l3fwd.h                  |  4 ++--
 examples/l3fwd/l3fwd_em.c               | 10 +++++++++-
 examples/l3fwd/l3fwd_lpm.c              |  2 +-
 examples/l3fwd/main.c                   | 24 +++++++++++++++++++++---
 5 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward.rst b/doc/guides/sample_app_ug/l3_forward.rst
index 4cb4b18da..a40beeeb0 100644
--- a/doc/guides/sample_app_ug/l3_forward.rst
+++ b/doc/guides/sample_app_ug/l3_forward.rst
@@ -56,6 +56,7 @@ The application has a number of command line options::
                              [--ipv6]
                              [--parse-ptype]
                              [--per-port-pool]
+                             [--lock-free]
 
 Where,
 
@@ -86,6 +87,8 @@ Where,
 
 * ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports.
 
+* ``--lock-free:`` Optional, set to enable lock-free table read-write concurrency.
+
 For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0,
 while cores 8-15 and 24-31 appear on socket 1.
 
diff --git a/examples/l3fwd/l3fwd.h b/examples/l3fwd/l3fwd.h
index 293fb1fa2..eaf71d429 100644
--- a/examples/l3fwd/l3fwd.h
+++ b/examples/l3fwd/l3fwd.h
@@ -171,10 +171,10 @@ is_valid_ipv4_pkt(struct rte_ipv4_hdr *pkt, uint32_t link_len)
 
 /* Function pointers for LPM or EM functionality. */
 void
-setup_lpm(const int socketid);
+setup_lpm(const int socketid, const unsigned int flags);
 
 void
-setup_hash(const int socketid);
+setup_hash(const int socketid, const unsigned int flags);
 
 int
 em_check_ptype(int portid);
diff --git a/examples/l3fwd/l3fwd_em.c b/examples/l3fwd/l3fwd_em.c
index 74a7c8fa4..7263ede04 100644
--- a/examples/l3fwd/l3fwd_em.c
+++ b/examples/l3fwd/l3fwd_em.c
@@ -703,7 +703,7 @@ em_main_loop(__attribute__((unused)) void *dummy)
  * Initialize exact match (hash) parameters.
  */
 void
-setup_hash(const int socketid)
+setup_hash(const int socketid, const unsigned int flags)
 {
 	struct rte_hash_parameters ipv4_l3fwd_hash_params = {
 		.name = NULL,
@@ -727,6 +727,10 @@ setup_hash(const int socketid)
 	snprintf(s, sizeof(s), "ipv4_l3fwd_hash_%d", socketid);
 	ipv4_l3fwd_hash_params.name = s;
 	ipv4_l3fwd_hash_params.socket_id = socketid;
+	/* enable lock free hash algorithm for ipv4 forward*/
+	if (flags & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)
+		ipv4_l3fwd_hash_params.extra_flag |=
+			RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
 	ipv4_l3fwd_em_lookup_struct[socketid] =
 		rte_hash_create(&ipv4_l3fwd_hash_params);
 	if (ipv4_l3fwd_em_lookup_struct[socketid] == NULL)
@@ -738,6 +742,10 @@ setup_hash(const int socketid)
 	snprintf(s, sizeof(s), "ipv6_l3fwd_hash_%d", socketid);
 	ipv6_l3fwd_hash_params.name = s;
 	ipv6_l3fwd_hash_params.socket_id = socketid;
+	/* enable lock free hash algorithm for ipv6 forward*/
+	if (flags & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF)
+		ipv6_l3fwd_hash_params.extra_flag |=
+			RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
 	ipv6_l3fwd_em_lookup_struct[socketid] =
 		rte_hash_create(&ipv6_l3fwd_hash_params);
 	if (ipv6_l3fwd_em_lookup_struct[socketid] == NULL)
diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c
index 4143683cb..1435a5711 100644
--- a/examples/l3fwd/l3fwd_lpm.c
+++ b/examples/l3fwd/l3fwd_lpm.c
@@ -255,7 +255,7 @@ lpm_main_loop(__attribute__((unused)) void *dummy)
 }
 
 void
-setup_lpm(const int socketid)
+setup_lpm(const int socketid, __rte_unused const unsigned int flags)
 {
 	struct rte_lpm6_config config;
 	struct rte_lpm_config config_ipv4;
diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index 3800bad19..1b435e9eb 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -41,6 +41,7 @@
 #include <rte_udp.h>
 #include <rte_string_fns.h>
 #include <rte_cpuflags.h>
+#include <rte_hash.h>
 
 #include <cmdline_parse.h>
 #include <cmdline_parse_etheraddr.h>
@@ -76,6 +77,10 @@ static int parse_ptype; /**< Parse packet type using rx callback, and */
 			/**< disabled by default */
 static int per_port_pool; /**< Use separate buffer pools per port; disabled */
 			  /**< by default */
+static int rw_lf;	/**< Enable lock-free read-write concurrency, */
+			/**< disabled by default */
+
+/* Global variables. */
 
 volatile bool force_quit;
 
@@ -139,7 +144,7 @@ static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
 static uint8_t lkp_per_socket[NB_SOCKETS];
 
 struct l3fwd_lkp_mode {
-	void  (*setup)(int);
+	void  (*setup)(int, unsigned int);
 	int   (*check_ptype)(int);
 	rte_rx_callback_fn cb_parse_ptype;
 	int   (*main_loop)(void *);
@@ -290,6 +295,7 @@ print_usage(const char *prgname)
 		" [--ipv6]"
 		" [--parse-ptype]"
 		" [--per-port-pool]\n\n"
+		" [--lock-free]\n\n"
 
 		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
 		"  -P : Enable promiscuous mode\n"
@@ -304,7 +310,8 @@ print_usage(const char *prgname)
 		"  --hash-entry-num: Specify the hash entry number in hexadecimal to be setup\n"
 		"  --ipv6: Set if running ipv6 packets\n"
 		"  --parse-ptype: Set to use software to analyze packet type\n"
-		"  --per-port-pool: Use separate buffer pool per port\n\n",
+		"  --per-port-pool: Use separate buffer pool per port\n\n"
+		"  --lock-free: Set to enable lock-free table read-write concurrency\n\n",
 		prgname);
 }
 
@@ -458,6 +465,8 @@ static const char short_options[] =
 #define CMD_LINE_OPT_HASH_ENTRY_NUM "hash-entry-num"
 #define CMD_LINE_OPT_PARSE_PTYPE "parse-ptype"
 #define CMD_LINE_OPT_PER_PORT_POOL "per-port-pool"
+#define CMD_LINE_OPT_LOCK_FREE "lock-free"
+
 enum {
 	/* long options mapped to a short option */
 
@@ -472,6 +481,7 @@ enum {
 	CMD_LINE_OPT_HASH_ENTRY_NUM_NUM,
 	CMD_LINE_OPT_PARSE_PTYPE_NUM,
 	CMD_LINE_OPT_PARSE_PER_PORT_POOL,
+	CMD_LINE_OPT_LOCK_FREE_NUM,
 };
 
 static const struct option lgopts[] = {
@@ -483,6 +493,7 @@ static const struct option lgopts[] = {
 	{CMD_LINE_OPT_HASH_ENTRY_NUM, 1, 0, CMD_LINE_OPT_HASH_ENTRY_NUM_NUM},
 	{CMD_LINE_OPT_PARSE_PTYPE, 0, 0, CMD_LINE_OPT_PARSE_PTYPE_NUM},
 	{CMD_LINE_OPT_PER_PORT_POOL, 0, 0, CMD_LINE_OPT_PARSE_PER_PORT_POOL},
+	{CMD_LINE_OPT_LOCK_FREE, 0, 0, CMD_LINE_OPT_LOCK_FREE_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -607,6 +618,10 @@ parse_args(int argc, char **argv)
 			per_port_pool = 1;
 			break;
 
+		case CMD_LINE_OPT_LOCK_FREE_NUM:
+			rw_lf = 1;
+			break;
+
 		default:
 			print_usage(prgname);
 			return -1;
@@ -661,6 +676,7 @@ init_mem(uint16_t portid, unsigned int nb_mbuf)
 	int socketid;
 	unsigned lcore_id;
 	char s[64];
+	unsigned int flags = 0;
 
 	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
 		if (rte_lcore_is_enabled(lcore_id) == 0)
@@ -695,8 +711,10 @@ init_mem(uint16_t portid, unsigned int nb_mbuf)
 			/* Setup either LPM or EM(f.e Hash). But, only once per
 			 * available socket.
 			 */
+			if (rw_lf)
+				flags |= RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
 			if (!lkp_per_socket[socketid]) {
-				l3fwd_lkp.setup(socketid);
+				l3fwd_lkp.setup(socketid, flags);
 				lkp_per_socket[socketid] = 1;
 			}
 		}
-- 
2.17.1

[dpdk-dev] [PATCH 2/2] examples/l3fwd: integrate RCU QSBR for LPM mode

[Ruifeng Wang]

LPM library relies on RCU QSBR for safe tbl8 group reclaim.
l3fwd will start LPM RCU QSBR process when lock-free option
is enabled.

Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Gavin Hu <gavin.hu@arm.com>
---
 examples/l3fwd/Makefile    |  1 +
 examples/l3fwd/l3fwd_lpm.c | 70 ++++++++++++++++++++++++++++++++++++--
 examples/l3fwd/main.c      |  5 +--
 examples/l3fwd/meson.build |  1 +
 4 files changed, 73 insertions(+), 4 deletions(-)

diff --git a/examples/l3fwd/Makefile b/examples/l3fwd/Makefile
index c55f5c288..278586fa1 100644
--- a/examples/l3fwd/Makefile
+++ b/examples/l3fwd/Makefile
@@ -52,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 CFLAGS += -I$(SRCDIR)
 CFLAGS += -O3 $(USER_FLAGS)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c
index 1435a5711..33ee04c0e 100644
--- a/examples/l3fwd/l3fwd_lpm.c
+++ b/examples/l3fwd/l3fwd_lpm.c
@@ -21,6 +21,7 @@
 #include <rte_ethdev.h>
 #include <rte_cycles.h>
 #include <rte_mbuf.h>
+#include <rte_malloc.h>
 #include <rte_ip.h>
 #include <rte_tcp.h>
 #include <rte_udp.h>
@@ -77,6 +78,9 @@ static struct ipv6_l3fwd_lpm_route ipv6_l3fwd_lpm_route_array[] = {
 
 struct rte_lpm *ipv4_l3fwd_lpm_lookup_struct[NB_SOCKETS];
 struct rte_lpm6 *ipv6_l3fwd_lpm_lookup_struct[NB_SOCKETS];
+struct rte_rcu_qsbr *lpm4_qsv[NB_SOCKETS];	/* RCU QSBR variable for LPM4 */
+extern int numa_on;
+extern int rw_lf;
 
 static inline uint16_t
 lpm_get_ipv4_dst_port(void *ipv4_hdr, uint16_t portid, void *lookup_struct)
@@ -178,7 +182,7 @@ lpm_main_loop(__attribute__((unused)) void *dummy)
 	struct rte_mbuf *pkts_burst[MAX_PKT_BURST];
 	unsigned lcore_id;
 	uint64_t prev_tsc, diff_tsc, cur_tsc;
-	int i, nb_rx;
+	int i, nb_rx, socketid = 0;
 	uint16_t portid;
 	uint8_t queueid;
 	struct lcore_conf *qconf;
@@ -206,6 +210,22 @@ lpm_main_loop(__attribute__((unused)) void *dummy)
 			lcore_id, portid, queueid);
 	}
 
+	if (rw_lf) {
+		if (numa_on)
+			socketid = rte_lcore_to_socket_id(lcore_id);
+		else
+			socketid = 0;
+
+		if (rte_rcu_qsbr_thread_register(lpm4_qsv[socketid],
+						lcore_id) != 0) {
+			RTE_LOG(ERR, L3FWD,
+				"lcore %u failed RCU QSBR register\n",
+				lcore_id);
+			return -1;
+		}
+		rte_rcu_qsbr_thread_online(lpm4_qsv[socketid], lcore_id);
+	}
+
 	while (!force_quit) {
 
 		cur_tsc = rte_rdtsc();
@@ -237,8 +257,12 @@ lpm_main_loop(__attribute__((unused)) void *dummy)
 			queueid = qconf->rx_queue_list[i].queue_id;
 			nb_rx = rte_eth_rx_burst(portid, queueid, pkts_burst,
 				MAX_PKT_BURST);
-			if (nb_rx == 0)
+			if (nb_rx == 0) {
+				if (rw_lf)
+					rte_rcu_qsbr_quiescent(
+						lpm4_qsv[socketid], lcore_id);
 				continue;
+			}
 
 #if defined RTE_ARCH_X86 || defined RTE_MACHINE_CPUFLAG_NEON \
 			 || defined RTE_ARCH_PPC_64
@@ -248,6 +272,20 @@ lpm_main_loop(__attribute__((unused)) void *dummy)
 			l3fwd_lpm_no_opt_send_packets(nb_rx, pkts_burst,
 							portid, qconf);
 #endif /* X86 */
+			if (rw_lf)
+				rte_rcu_qsbr_quiescent(lpm4_qsv[socketid],
+							lcore_id);
+		}
+	}
+
+	if (rw_lf) {
+		rte_rcu_qsbr_thread_offline(lpm4_qsv[socketid], lcore_id);
+		if (rte_rcu_qsbr_thread_unregister(lpm4_qsv[socketid],
+						lcore_id) != 0) {
+			RTE_LOG(ERR, L3FWD,
+				"lcore %u failed RCU QSBR unregister\n",
+				lcore_id);
+			return -1;
 		}
 	}
 
@@ -303,6 +341,34 @@ setup_lpm(const int socketid, __rte_unused const unsigned int flags)
 			ipv4_l3fwd_lpm_route_array[i].if_out);
 	}
 
+	if (rw_lf) {
+		size_t sz;
+
+		/* create RCU QSBR variable */
+		sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+		lpm4_qsv[socketid] = (struct rte_rcu_qsbr *)rte_zmalloc_socket(
+						NULL, sz, RTE_CACHE_LINE_SIZE,
+						socketid);
+		if (lpm4_qsv[socketid] == NULL)
+			rte_exit(EXIT_FAILURE,
+				"RCU QSBR alloc fails on socket %d\n",
+				socketid);
+		else {
+			if (rte_rcu_qsbr_init(lpm4_qsv[socketid],
+						RTE_MAX_LCORE) != 0)
+				rte_exit(EXIT_FAILURE,
+					"RCU QSBR init fails on socket %d\n",
+					socketid);
+		}
+
+		/* attach RCU QSBR to LPM table */
+		if (rte_lpm_rcu_qsbr_add(ipv4_l3fwd_lpm_lookup_struct[socketid],
+					lpm4_qsv[socketid]) != 0)
+			rte_exit(EXIT_FAILURE,
+				"RCU QSBR attach to LPM fails on socket %d\n",
+				socketid);
+	}
+
 	/* create the LPM6 table */
 	snprintf(s, sizeof(s), "IPV6_L3FWD_LPM_%d", socketid);
 
diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index 1b435e9eb..797b57554 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -72,12 +72,12 @@ static int l3fwd_em_on;
 
 /* Global variables. */
 
-static int numa_on = 1; /**< NUMA is enabled by default. */
+int numa_on = 1;	/**< NUMA is enabled by default. */
 static int parse_ptype; /**< Parse packet type using rx callback, and */
 			/**< disabled by default */
 static int per_port_pool; /**< Use separate buffer pools per port; disabled */
 			  /**< by default */
-static int rw_lf;	/**< Enable lock-free read-write concurrency, */
+int rw_lf;		/**< Enable lock-free read-write concurrency, */
 			/**< disabled by default */
 
 /* Global variables. */
@@ -619,6 +619,7 @@ parse_args(int argc, char **argv)
 			break;
 
 		case CMD_LINE_OPT_LOCK_FREE_NUM:
+			printf("RCU lock-free is enabled\n");
 			rw_lf = 1;
 			break;
 
diff --git a/examples/l3fwd/meson.build b/examples/l3fwd/meson.build
index 6dd4b9022..d6e462a1f 100644
--- a/examples/l3fwd/meson.build
+++ b/examples/l3fwd/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 deps += ['hash', 'lpm']
 sources = files(
 	'l3fwd_em.c', 'l3fwd_lpm.c', 'main.c'
-- 
2.17.1

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ananyev, Konstantin]

Hi,

> 
> Lock-free mode is supported by hash library and LPM library.
> Now we add an option for l3fwd example to enable the lock-free mode.
> Necessary preparation steps are added to use lock-free LPM mode.

Can I ask about the purpose of these changes?
Right now in  l3fwd both lpm and hash tables are static and hard-coded.
we initialize them at startup and then just do read from them.
Do you plan to enhance l3fwd with ability to dynamically update tables contents?
Though fir that we first have to get rid of hard-coded values (config file or so).
Konstantin

> 
> Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> http://patches.dpdk.org/project/dpdk/list/?series=6288
> 
> 
> Ruifeng Wang (2):
>   examples/l3fwd: add lock-free option for l3fwd
>   examples/l3fwd: integrate RCU QSBR for LPM mode
> 
>  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
>  examples/l3fwd/Makefile                 |  1 +
>  examples/l3fwd/l3fwd.h                  |  4 +-
>  examples/l3fwd/l3fwd_em.c               | 10 +++-
>  examples/l3fwd/l3fwd_lpm.c              | 72 +++++++++++++++++++++++--
>  examples/l3fwd/main.c                   | 27 ++++++++--
>  examples/l3fwd/meson.build              |  1 +
>  7 files changed, 108 insertions(+), 10 deletions(-)
> 
> --
> 2.17.1

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

Hi,

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Friday, September 6, 2019 18:35
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Kantecki, Tomasz <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> Hi,
> 
> >
> > Lock-free mode is supported by hash library and LPM library.
> > Now we add an option for l3fwd example to enable the lock-free mode.
> > Necessary preparation steps are added to use lock-free LPM mode.
> 
> Can I ask about the purpose of these changes?
> Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> we initialize them at startup and then just do read from them.
> Do you plan to enhance l3fwd with ability to dynamically update tables
> contents?
> Though fir that we first have to get rid of hard-coded values (config file or so).
> Konstantin
> 
Thanks for your questions.
Currently, we have no plan to enhance l3fwd with ability to dynamically update table contents.
Lock-free method is being integrated into Hash library and LPM library.  Lock-free algorithms
are not only about control plane (adding or deleting routes), they affect the data path performance
as well.
Since l3fwd application is showcasing data path performance, we need to show the impact of
including the quiescent state reporting on data path.
This change also serves as an example of using the RCU APIs.

> >
> > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > http://patches.dpdk.org/project/dpdk/list/?series=6288
> >
> >
> > Ruifeng Wang (2):
> >   examples/l3fwd: add lock-free option for l3fwd
> >   examples/l3fwd: integrate RCU QSBR for LPM mode
> >
> >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> >  examples/l3fwd/Makefile                 |  1 +
> >  examples/l3fwd/l3fwd.h                  |  4 +-
> >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> >  examples/l3fwd/l3fwd_lpm.c              | 72 +++++++++++++++++++++++--
> >  examples/l3fwd/main.c                   | 27 ++++++++--
> >  examples/l3fwd/meson.build              |  1 +
> >  7 files changed, 108 insertions(+), 10 deletions(-)
> >
> > --
> > 2.17.1

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Honnappa Nagarahalli]

<snip>

> >
> > Hi,
> >
> > >
> > > Lock-free mode is supported by hash library and LPM library.
> > > Now we add an option for l3fwd example to enable the lock-free mode.
> > > Necessary preparation steps are added to use lock-free LPM mode.
> >
> > Can I ask about the purpose of these changes?
> > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > we initialize them at startup and then just do read from them.
> > Do you plan to enhance l3fwd with ability to dynamically update tables
> > contents?
> > Though fir that we first have to get rid of hard-coded values (config file or
> so).
> > Konstantin
> >
> Thanks for your questions.
> Currently, we have no plan to enhance l3fwd with ability to dynamically
> update table contents.
> Lock-free method is being integrated into Hash library and LPM library.  Lock-
> free algorithms are not only about control plane (adding or deleting routes),
> they affect the data path performance as well.
> Since l3fwd application is showcasing data path performance, we need to
> show the impact of including the quiescent state reporting on data path.
> This change also serves as an example of using the RCU APIs.
> 
Without the dynamic deletes the quiescent state reporting overhead is not captured completely.
I suggest that we add and delete a small set of unrelated routes (the routes that are not used currently) on a regular basis.

> > >
> > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > >
> > >
> > > Ruifeng Wang (2):
> > >   examples/l3fwd: add lock-free option for l3fwd
> > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > >
> > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > >  examples/l3fwd/Makefile                 |  1 +
> > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > >  examples/l3fwd/l3fwd_lpm.c              | 72 +++++++++++++++++++++++--
> > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > >  examples/l3fwd/meson.build              |  1 +
> > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > >
> > > --
> > > 2.17.1
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Tuesday, September 10, 2019 06:45
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd
> <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> <snip>
> 
> > >
> > > Hi,
> > >
> > > >
> > > > Lock-free mode is supported by hash library and LPM library.
> > > > Now we add an option for l3fwd example to enable the lock-free mode.
> > > > Necessary preparation steps are added to use lock-free LPM mode.
> > >
> > > Can I ask about the purpose of these changes?
> > > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > > we initialize them at startup and then just do read from them.
> > > Do you plan to enhance l3fwd with ability to dynamically update
> > > tables contents?
> > > Though fir that we first have to get rid of hard-coded values
> > > (config file or
> > so).
> > > Konstantin
> > >
> > Thanks for your questions.
> > Currently, we have no plan to enhance l3fwd with ability to
> > dynamically update table contents.
> > Lock-free method is being integrated into Hash library and LPM
> > library.  Lock- free algorithms are not only about control plane
> > (adding or deleting routes), they affect the data path performance as well.
> > Since l3fwd application is showcasing data path performance, we need
> > to show the impact of including the quiescent state reporting on data path.
> > This change also serves as an example of using the RCU APIs.
> >
> Without the dynamic deletes the quiescent state reporting overhead is not
> captured completely.
> I suggest that we add and delete a small set of unrelated routes (the routes
> that are not used currently) on a regular basis.
> 
Add and delete unrelated routes on a regular basis will simulate overhead on
control path. However, control path performance is not a l3fwd showcase, and
it is covered by LPM performance unit test.
On data path, quiescent state reporting overhead is constant (calling of rte_rcu_qsbr_quiescent).
It will not be impacted by route addition / deletion. 

> > > >
> > > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > >
> > > >
> > > > Ruifeng Wang (2):
> > > >   examples/l3fwd: add lock-free option for l3fwd
> > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > >
> > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > >  examples/l3fwd/Makefile                 |  1 +
> > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> +++++++++++++++++++++++--
> > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > >  examples/l3fwd/meson.build              |  1 +
> > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > >
> > > > --
> > > > 2.17.1
> >
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Honnappa Nagarahalli]

> >
> > <snip>
> >
> > > >
> > > > Hi,
> > > >
> > > > >
> > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > Now we add an option for l3fwd example to enable the lock-free
> mode.
> > > > > Necessary preparation steps are added to use lock-free LPM mode.
> > > >
> > > > Can I ask about the purpose of these changes?
> > > > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > > > we initialize them at startup and then just do read from them.
> > > > Do you plan to enhance l3fwd with ability to dynamically update
> > > > tables contents?
> > > > Though fir that we first have to get rid of hard-coded values
> > > > (config file or
> > > so).
> > > > Konstantin
> > > >
> > > Thanks for your questions.
> > > Currently, we have no plan to enhance l3fwd with ability to
> > > dynamically update table contents.
> > > Lock-free method is being integrated into Hash library and LPM
> > > library.  Lock- free algorithms are not only about control plane
> > > (adding or deleting routes), they affect the data path performance as well.
> > > Since l3fwd application is showcasing data path performance, we need
> > > to show the impact of including the quiescent state reporting on data
> path.
> > > This change also serves as an example of using the RCU APIs.
> > >
> > Without the dynamic deletes the quiescent state reporting overhead is
> > not captured completely.
> > I suggest that we add and delete a small set of unrelated routes (the
> > routes that are not used currently) on a regular basis.
> >
> Add and delete unrelated routes on a regular basis will simulate overhead on
> control path. However, control path performance is not a l3fwd showcase,
> and it is covered by LPM performance unit test.
> On data path, quiescent state reporting overhead is constant (calling of
> rte_rcu_qsbr_quiescent).
> It will not be impacted by route addition / deletion.
The recent changes to RCU [1] are such that the reader threads avoid a write if there are no deletes. So, without the deletion, the impact on readers is complete.

[1] https://patchwork.dpdk.org/patch/58961/
> 
> > > > >
> > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > >
> > > > >
> > > > > Ruifeng Wang (2):
> > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > >
> > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > +++++++++++++++++++++++--
> > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > >  examples/l3fwd/meson.build              |  1 +
> > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > >
> > > > > --
> > > > > 2.17.1
> > >
> >
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Wednesday, September 11, 2019 13:33
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> > >
> > > <snip>
> > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > >
> > > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > > Now we add an option for l3fwd example to enable the lock-free
> > mode.
> > > > > > Necessary preparation steps are added to use lock-free LPM mode.
> > > > >
> > > > > Can I ask about the purpose of these changes?
> > > > > Right now in  l3fwd both lpm and hash tables are static and hard-
> coded.
> > > > > we initialize them at startup and then just do read from them.
> > > > > Do you plan to enhance l3fwd with ability to dynamically update
> > > > > tables contents?
> > > > > Though fir that we first have to get rid of hard-coded values
> > > > > (config file or
> > > > so).
> > > > > Konstantin
> > > > >
> > > > Thanks for your questions.
> > > > Currently, we have no plan to enhance l3fwd with ability to
> > > > dynamically update table contents.
> > > > Lock-free method is being integrated into Hash library and LPM
> > > > library.  Lock- free algorithms are not only about control plane
> > > > (adding or deleting routes), they affect the data path performance as
> well.
> > > > Since l3fwd application is showcasing data path performance, we
> > > > need to show the impact of including the quiescent state reporting
> > > > on data
> > path.
> > > > This change also serves as an example of using the RCU APIs.
> > > >
> > > Without the dynamic deletes the quiescent state reporting overhead
> > > is not captured completely.
> > > I suggest that we add and delete a small set of unrelated routes
> > > (the routes that are not used currently) on a regular basis.
> > >
> > Add and delete unrelated routes on a regular basis will simulate
> > overhead on control path. However, control path performance is not a
> > l3fwd showcase, and it is covered by LPM performance unit test.
> > On data path, quiescent state reporting overhead is constant (calling
> > of rte_rcu_qsbr_quiescent).
> > It will not be impacted by route addition / deletion.
> The recent changes to RCU [1] are such that the reader threads avoid a write
> if there are no deletes. So, without the deletion, the impact on readers is
> complete.
> 
> [1] https://patchwork.dpdk.org/patch/58961/

OK, understand that. 
Assumptions based on current implementation of rte_rcu_qsbr_quiescent is not reliable. 
The part on control path will be added.

> >
> > > > > >
> > > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM
> library:
> > > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > > >
> > > > > >
> > > > > > Ruifeng Wang (2):
> > > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > > >
> > > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > > +++++++++++++++++++++++--
> > > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > > >  examples/l3fwd/meson.build              |  1 +
> > > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.17.1
> > > >
> > >
> >
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ananyev, Konstantin]

> >
> > >
> > > Lock-free mode is supported by hash library and LPM library.
> > > Now we add an option for l3fwd example to enable the lock-free mode.
> > > Necessary preparation steps are added to use lock-free LPM mode.
> >
> > Can I ask about the purpose of these changes?
> > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > we initialize them at startup and then just do read from them.
> > Do you plan to enhance l3fwd with ability to dynamically update tables
> > contents?
> > Though fir that we first have to get rid of hard-coded values (config file or so).
> > Konstantin
> >
> Thanks for your questions.
> Currently, we have no plan to enhance l3fwd with ability to dynamically update table contents.
> Lock-free method is being integrated into Hash library and LPM library.  Lock-free algorithms
> are not only about control plane (adding or deleting routes), they affect the data path performance
> as well.
> Since l3fwd application is showcasing data path performance, we need to show the impact of
> including the quiescent state reporting on data path.
> This change also serves as an example of using the RCU APIs.


But what you suggest doesn't provide the complete picture.
With dynamic updates in place (via control path) the data-path impact might
be completely different then without.
Again without dynamic updates how can you test that your data-path lock-free
approach does work as expected?
Also it can't even be used as a reference implementation for users,
as half of the functionality they need to implement is simply missing.
My opinion - we either need to leave l3fwd as it is (static routes),
or implement a proper control path with ability to dynamically update routes
before starting to introduce some synchronization schemes (RCU or whatever). 

Konstantin

> 
> > >
> > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > >
> > >
> > > Ruifeng Wang (2):
> > >   examples/l3fwd: add lock-free option for l3fwd
> > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > >
> > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > >  examples/l3fwd/Makefile                 |  1 +
> > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > >  examples/l3fwd/l3fwd_lpm.c              | 72 +++++++++++++++++++++++--
> > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > >  examples/l3fwd/meson.build              |  1 +
> > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > >
> > > --
> > > 2.17.1

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Tuesday, September 10, 2019 17:06
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Kantecki, Tomasz <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd
> <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> 
> 
> > >
> > > >
> > > > Lock-free mode is supported by hash library and LPM library.
> > > > Now we add an option for l3fwd example to enable the lock-free mode.
> > > > Necessary preparation steps are added to use lock-free LPM mode.
> > >
> > > Can I ask about the purpose of these changes?
> > > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > > we initialize them at startup and then just do read from them.
> > > Do you plan to enhance l3fwd with ability to dynamically update
> > > tables contents?
> > > Though fir that we first have to get rid of hard-coded values (config file or
> so).
> > > Konstantin
> > >
> > Thanks for your questions.
> > Currently, we have no plan to enhance l3fwd with ability to dynamically
> update table contents.
> > Lock-free method is being integrated into Hash library and LPM
> > library.  Lock-free algorithms are not only about control plane
> > (adding or deleting routes), they affect the data path performance as well.
> > Since l3fwd application is showcasing data path performance, we need
> > to show the impact of including the quiescent state reporting on data path.
> > This change also serves as an example of using the RCU APIs.
> 
> 
> But what you suggest doesn't provide the complete picture.
> With dynamic updates in place (via control path) the data-path impact might
> be completely different then without.
> Again without dynamic updates how can you test that your data-path lock-
> free approach does work as expected?
> Also it can't even be used as a reference implementation for users, as half of
> the functionality they need to implement is simply missing.
> My opinion - we either need to leave l3fwd as it is (static routes), or
> implement a proper control path with ability to dynamically update routes
> before starting to introduce some synchronization schemes (RCU or
> whatever).
> 
> Konstantin
> 

Agree that dynamic control path updates should be included for a whole picture.
I will add dynamic update to l3fwd and reroll the patch series.
Thanks.

> >
> > > >
> > > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > >
> > > >
> > > > Ruifeng Wang (2):
> > > >   examples/l3fwd: add lock-free option for l3fwd
> > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > >
> > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > >  examples/l3fwd/Makefile                 |  1 +
> > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> +++++++++++++++++++++++--
> > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > >  examples/l3fwd/meson.build              |  1 +
> > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > >
> > > > --
> > > > 2.17.1

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Honnappa Nagarahalli]

<snip>
> >
> >
> >
> > > >
> > > > >
> > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > Now we add an option for l3fwd example to enable the lock-free
> mode.
> > > > > Necessary preparation steps are added to use lock-free LPM mode.
> > > >
> > > > Can I ask about the purpose of these changes?
> > > > Right now in  l3fwd both lpm and hash tables are static and hard-coded.
> > > > we initialize them at startup and then just do read from them.
> > > > Do you plan to enhance l3fwd with ability to dynamically update
> > > > tables contents?
> > > > Though fir that we first have to get rid of hard-coded values
> > > > (config file or
> > so).
> > > > Konstantin
> > > >
> > > Thanks for your questions.
> > > Currently, we have no plan to enhance l3fwd with ability to
> > > dynamically
> > update table contents.
> > > Lock-free method is being integrated into Hash library and LPM
> > > library.  Lock-free algorithms are not only about control plane
> > > (adding or deleting routes), they affect the data path performance as well.
> > > Since l3fwd application is showcasing data path performance, we need
> > > to show the impact of including the quiescent state reporting on data
> path.
> > > This change also serves as an example of using the RCU APIs.
> >
> >
> > But what you suggest doesn't provide the complete picture.
> > With dynamic updates in place (via control path) the data-path impact
> > might be completely different then without.
> > Again without dynamic updates how can you test that your data-path
> > lock- free approach does work as expected?
> > Also it can't even be used as a reference implementation for users, as
> > half of the functionality they need to implement is simply missing.
> > My opinion - we either need to leave l3fwd as it is (static routes),
> > or implement a proper control path with ability to dynamically update
> > routes before starting to introduce some synchronization schemes (RCU
> > or whatever).
> >
> > Konstantin
> >
> 
> Agree that dynamic control path updates should be included for a whole
> picture.
> I will add dynamic update to l3fwd and reroll the patch series.
> Thanks.
I think we should have an agreement on what exactly we mean by 'dynamically update routes'.
IMO, we should not disturb the existing static routes as there might be automated tests running in the labs. I suggest that we should add/delete new routes/hash entries which are different from the existing routes/hash entries. This should be sufficient to showcase the functionality as well as measure the impact.

> 
> > >
> > > > >
> > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM library:
> > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > >
> > > > >
> > > > > Ruifeng Wang (2):
> > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > >
> > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > +++++++++++++++++++++++--
> > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > >  examples/l3fwd/meson.build              |  1 +
> > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > >
> > > > > --
> > > > > 2.17.1
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Wednesday, September 11, 2019 13:38
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd
> <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> <snip>
> > >
> > >
> > >
> > > > >
> > > > > >
> > > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > > Now we add an option for l3fwd example to enable the lock-free
> > mode.
> > > > > > Necessary preparation steps are added to use lock-free LPM mode.
> > > > >
> > > > > Can I ask about the purpose of these changes?
> > > > > Right now in  l3fwd both lpm and hash tables are static and hard-
> coded.
> > > > > we initialize them at startup and then just do read from them.
> > > > > Do you plan to enhance l3fwd with ability to dynamically update
> > > > > tables contents?
> > > > > Though fir that we first have to get rid of hard-coded values
> > > > > (config file or
> > > so).
> > > > > Konstantin
> > > > >
> > > > Thanks for your questions.
> > > > Currently, we have no plan to enhance l3fwd with ability to
> > > > dynamically
> > > update table contents.
> > > > Lock-free method is being integrated into Hash library and LPM
> > > > library.  Lock-free algorithms are not only about control plane
> > > > (adding or deleting routes), they affect the data path performance as
> well.
> > > > Since l3fwd application is showcasing data path performance, we
> > > > need to show the impact of including the quiescent state reporting
> > > > on data
> > path.
> > > > This change also serves as an example of using the RCU APIs.
> > >
> > >
> > > But what you suggest doesn't provide the complete picture.
> > > With dynamic updates in place (via control path) the data-path
> > > impact might be completely different then without.
> > > Again without dynamic updates how can you test that your data-path
> > > lock- free approach does work as expected?
> > > Also it can't even be used as a reference implementation for users,
> > > as half of the functionality they need to implement is simply missing.
> > > My opinion - we either need to leave l3fwd as it is (static routes),
> > > or implement a proper control path with ability to dynamically
> > > update routes before starting to introduce some synchronization
> > > schemes (RCU or whatever).
> > >
> > > Konstantin
> > >
> >
> > Agree that dynamic control path updates should be included for a whole
> > picture.
> > I will add dynamic update to l3fwd and reroll the patch series.
> > Thanks.
> I think we should have an agreement on what exactly we mean by
> 'dynamically update routes'.
> IMO, we should not disturb the existing static routes as there might be
> automated tests running in the labs. I suggest that we should add/delete
> new routes/hash entries which are different from the existing routes/hash
> entries. This should be sufficient to showcase the functionality as well as
> measure the impact.
> 
Yes, existing static routes should be kept intact.
To perform regular route/hash entries add/delete, a dedicated lcore will be needed.
An interactive prompt is not an option since we need automatic add/delete.
We can skip master core for data path main loop. And perform unrelated route/hash entries add/delete regularly on master core.
The impact is that command lines used in tests will need update since master core will no longer do data path work.
> >
> > > >
> > > > > >
> > > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM
> library:
> > > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > > >
> > > > > >
> > > > > > Ruifeng Wang (2):
> > > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > > >
> > > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > > +++++++++++++++++++++++--
> > > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > > >  examples/l3fwd/meson.build              |  1 +
> > > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.17.1
> >
> 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ananyev, Konstantin]

> -----Original Message-----
> From: Ruifeng Wang (Arm Technology China) [mailto:Ruifeng.Wang@arm.com]
> Sent: Wednesday, September 11, 2019 7:58 AM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>; nd <nd@arm.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> > -----Original Message-----
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > Sent: Wednesday, September 11, 2019 13:38
> > To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> > <tomasz.kantecki@intel.com>
> > Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> > Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd
> > <nd@arm.com>; nd <nd@arm.com>
> > Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> >
> > <snip>
> > > >
> > > >
> > > >
> > > > > >
> > > > > > >
> > > > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > > > Now we add an option for l3fwd example to enable the lock-free
> > > mode.
> > > > > > > Necessary preparation steps are added to use lock-free LPM mode.
> > > > > >
> > > > > > Can I ask about the purpose of these changes?
> > > > > > Right now in  l3fwd both lpm and hash tables are static and hard-
> > coded.
> > > > > > we initialize them at startup and then just do read from them.
> > > > > > Do you plan to enhance l3fwd with ability to dynamically update
> > > > > > tables contents?
> > > > > > Though fir that we first have to get rid of hard-coded values
> > > > > > (config file or
> > > > so).
> > > > > > Konstantin
> > > > > >
> > > > > Thanks for your questions.
> > > > > Currently, we have no plan to enhance l3fwd with ability to
> > > > > dynamically
> > > > update table contents.
> > > > > Lock-free method is being integrated into Hash library and LPM
> > > > > library.  Lock-free algorithms are not only about control plane
> > > > > (adding or deleting routes), they affect the data path performance as
> > well.
> > > > > Since l3fwd application is showcasing data path performance, we
> > > > > need to show the impact of including the quiescent state reporting
> > > > > on data
> > > path.
> > > > > This change also serves as an example of using the RCU APIs.
> > > >
> > > >
> > > > But what you suggest doesn't provide the complete picture.
> > > > With dynamic updates in place (via control path) the data-path
> > > > impact might be completely different then without.
> > > > Again without dynamic updates how can you test that your data-path
> > > > lock- free approach does work as expected?
> > > > Also it can't even be used as a reference implementation for users,
> > > > as half of the functionality they need to implement is simply missing.
> > > > My opinion - we either need to leave l3fwd as it is (static routes),
> > > > or implement a proper control path with ability to dynamically
> > > > update routes before starting to introduce some synchronization
> > > > schemes (RCU or whatever).
> > > >
> > > > Konstantin
> > > >
> > >
> > > Agree that dynamic control path updates should be included for a whole
> > > picture.
> > > I will add dynamic update to l3fwd and reroll the patch series.
> > > Thanks.
> > I think we should have an agreement on what exactly we mean by
> > 'dynamically update routes'.
> > IMO, we should not disturb the existing static routes as there might be
> > automated tests running in the labs. I suggest that we should add/delete
> > new routes/hash entries which are different from the existing routes/hash
> > entries. This should be sufficient to showcase the functionality as well as
> > measure the impact.
> >
> Yes, existing static routes should be kept intact.
> To perform regular route/hash entries add/delete, a dedicated lcore will be needed.
> An interactive prompt is not an option since we need automatic add/delete.
> We can skip master core for data path main loop. And perform unrelated route/hash entries add/delete regularly on master core.
> The impact is that command lines used in tests will need update since master core will no longer do data path work.

Not sure why it has to be  master core?
Why interrupt thread wouldn't do?
I think what we need to:
1. introduce reading routes from config file instead of having them hard-coded within the app.
2. add ability to update routes dynamically.
    Probably the easiest (and commonly used way) re-read conf file and update routes on the signal (SIGUSR1 or so).
Konstantin


> > >
> > > > >
> > > > > > >
> > > > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM
> > library:
> > > > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > > > >
> > > > > > >
> > > > > > > Ruifeng Wang (2):
> > > > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > > > >
> > > > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > > > +++++++++++++++++++++++--
> > > > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > > > >  examples/l3fwd/meson.build              |  1 +
> > > > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > > > >
> > > > > > > --
> > > > > > > 2.17.1
> > >
> >

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Wednesday, September 11, 2019 16:35
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>
> Cc: dev@dpdk.org; Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>;
> nd <nd@arm.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> 
> 
> > -----Original Message-----
> > From: Ruifeng Wang (Arm Technology China)
> > [mailto:Ruifeng.Wang@arm.com]
> > Sent: Wednesday, September 11, 2019 7:58 AM
> > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Ananyev,
> > Konstantin <konstantin.ananyev@intel.com>; Kantecki, Tomasz
> > <tomasz.kantecki@intel.com>
> > Cc: dev@dpdk.org; Gavin Hu (Arm Technology China)
> <Gavin.Hu@arm.com>;
> > nd <nd@arm.com>; nd <nd@arm.com>; nd <nd@arm.com>
> > Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> >
> > > -----Original Message-----
> > > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > > Sent: Wednesday, September 11, 2019 13:38
> > > To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>;
> > > Ananyev, Konstantin <konstantin.ananyev@intel.com>; Kantecki,
> Tomasz
> > > <tomasz.kantecki@intel.com>
> > > Cc: dev@dpdk.org; Gavin Hu (Arm Technology China)
> > > <Gavin.Hu@arm.com>; Honnappa Nagarahalli
> > > <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; nd
> <nd@arm.com>
> > > Subject: RE: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> > >
> > > <snip>
> > > > >
> > > > >
> > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Lock-free mode is supported by hash library and LPM library.
> > > > > > > > Now we add an option for l3fwd example to enable the
> > > > > > > > lock-free
> > > > mode.
> > > > > > > > Necessary preparation steps are added to use lock-free LPM
> mode.
> > > > > > >
> > > > > > > Can I ask about the purpose of these changes?
> > > > > > > Right now in  l3fwd both lpm and hash tables are static and
> > > > > > > hard-
> > > coded.
> > > > > > > we initialize them at startup and then just do read from them.
> > > > > > > Do you plan to enhance l3fwd with ability to dynamically
> > > > > > > update tables contents?
> > > > > > > Though fir that we first have to get rid of hard-coded
> > > > > > > values (config file or
> > > > > so).
> > > > > > > Konstantin
> > > > > > >
> > > > > > Thanks for your questions.
> > > > > > Currently, we have no plan to enhance l3fwd with ability to
> > > > > > dynamically
> > > > > update table contents.
> > > > > > Lock-free method is being integrated into Hash library and LPM
> > > > > > library.  Lock-free algorithms are not only about control
> > > > > > plane (adding or deleting routes), they affect the data path
> > > > > > performance as
> > > well.
> > > > > > Since l3fwd application is showcasing data path performance,
> > > > > > we need to show the impact of including the quiescent state
> > > > > > reporting on data
> > > > path.
> > > > > > This change also serves as an example of using the RCU APIs.
> > > > >
> > > > >
> > > > > But what you suggest doesn't provide the complete picture.
> > > > > With dynamic updates in place (via control path) the data-path
> > > > > impact might be completely different then without.
> > > > > Again without dynamic updates how can you test that your
> > > > > data-path
> > > > > lock- free approach does work as expected?
> > > > > Also it can't even be used as a reference implementation for
> > > > > users, as half of the functionality they need to implement is simply
> missing.
> > > > > My opinion - we either need to leave l3fwd as it is (static
> > > > > routes), or implement a proper control path with ability to
> > > > > dynamically update routes before starting to introduce some
> > > > > synchronization schemes (RCU or whatever).
> > > > >
> > > > > Konstantin
> > > > >
> > > >
> > > > Agree that dynamic control path updates should be included for a
> > > > whole picture.
> > > > I will add dynamic update to l3fwd and reroll the patch series.
> > > > Thanks.
> > > I think we should have an agreement on what exactly we mean by
> > > 'dynamically update routes'.
> > > IMO, we should not disturb the existing static routes as there might
> > > be automated tests running in the labs. I suggest that we should
> > > add/delete new routes/hash entries which are different from the
> > > existing routes/hash entries. This should be sufficient to showcase
> > > the functionality as well as measure the impact.
> > >
> > Yes, existing static routes should be kept intact.
> > To perform regular route/hash entries add/delete, a dedicated lcore will be
> needed.
> > An interactive prompt is not an option since we need automatic add/delete.
> > We can skip master core for data path main loop. And perform unrelated
> route/hash entries add/delete regularly on master core.
> > The impact is that command lines used in tests will need update since
> master core will no longer do data path work.
> 
> Not sure why it has to be  master core?
> Why interrupt thread wouldn't do?
> I think what we need to:
> 1. introduce reading routes from config file instead of having them hard-
> coded within the app.
> 2. add ability to update routes dynamically.
>     Probably the easiest (and commonly used way) re-read conf file and
> update routes on the signal (SIGUSR1 or so).
> Konstantin
> 
> 
Sorry for delayed response. Just back from vacation. 
Thanks for your suggestion.
I will try the config file based updating approach and get back with new version.

> > > >
> > > > > >
> > > > > > > >
> > > > > > > > Patch 2/2 has dependency on RCU QSBR integration with LPM
> > > library:
> > > > > > > > http://patches.dpdk.org/project/dpdk/list/?series=6288
> > > > > > > >
> > > > > > > >
> > > > > > > > Ruifeng Wang (2):
> > > > > > > >   examples/l3fwd: add lock-free option for l3fwd
> > > > > > > >   examples/l3fwd: integrate RCU QSBR for LPM mode
> > > > > > > >
> > > > > > > >  doc/guides/sample_app_ug/l3_forward.rst |  3 ++
> > > > > > > >  examples/l3fwd/Makefile                 |  1 +
> > > > > > > >  examples/l3fwd/l3fwd.h                  |  4 +-
> > > > > > > >  examples/l3fwd/l3fwd_em.c               | 10 +++-
> > > > > > > >  examples/l3fwd/l3fwd_lpm.c              | 72
> > > > > +++++++++++++++++++++++--
> > > > > > > >  examples/l3fwd/main.c                   | 27 ++++++++--
> > > > > > > >  examples/l3fwd/meson.build              |  1 +
> > > > > > > >  7 files changed, 108 insertions(+), 10 deletions(-)
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.17.1
> > > >
> > >

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[David Marchand]

Hello Ruifeng,

On Mon, Sep 16, 2019 at 4:30 AM Ruifeng Wang (Arm Technology China)
<Ruifeng.Wang@arm.com> wrote:
> > Not sure why it has to be  master core?
> > Why interrupt thread wouldn't do?
> > I think what we need to:
> > 1. introduce reading routes from config file instead of having them hard-
> > coded within the app.
> > 2. add ability to update routes dynamically.
> >     Probably the easiest (and commonly used way) re-read conf file and
> > update routes on the signal (SIGUSR1 or so).
> > Konstantin
> >
> >
> Sorry for delayed response. Just back from vacation.
> Thanks for your suggestion.
> I will try the config file based updating approach and get back with new version.

Any update on this series?


-- 
David Marchand

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, November 6, 2019 22:04
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Kantecki, Tomasz
> <tomasz.kantecki@intel.com>; dev@dpdk.org; Gavin Hu (Arm Technology
> China) <Gavin.Hu@arm.com>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> Hello Ruifeng,
> 
> On Mon, Sep 16, 2019 at 4:30 AM Ruifeng Wang (Arm Technology China)
> <Ruifeng.Wang@arm.com> wrote:
> > > Not sure why it has to be  master core?
> > > Why interrupt thread wouldn't do?
> > > I think what we need to:
> > > 1. introduce reading routes from config file instead of having them
> > > hard- coded within the app.
> > > 2. add ability to update routes dynamically.
> > >     Probably the easiest (and commonly used way) re-read conf file
> > > and update routes on the signal (SIGUSR1 or so).
> > > Konstantin
> > >
> > >
> > Sorry for delayed response. Just back from vacation.
> > Thanks for your suggestion.
> > I will try the config file based updating approach and get back with new
> version.
> 
> Any update on this series?

I heard different opinions on the approach. Currently no new version patch set available.
Will have more discussion before respin.
> 
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Stephen Hemminger]

On Fri,  6 Sep 2019 18:26:13 +0800
Ruifeng Wang <ruifeng.wang@arm.com> wrote:

> Lock-free mode is supported by hash library and LPM library.
> Now we add an option for l3fwd example to enable the lock-free mode.
> Necessary preparation steps are added to use lock-free LPM mode.

If lock-free mode works it should just do that.
Having options mean that there are two test cases; which inevitably
leads to one of them being broken.

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Ruifeng Wang (Arm Technology China)]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Saturday, September 7, 2019 01:29
> To: Ruifeng Wang (Arm Technology China) <Ruifeng.Wang@arm.com>
> Cc: tomasz.kantecki@intel.com; dev@dpdk.org; Gavin Hu (Arm Technology
> China) <Gavin.Hu@arm.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd
> 
> On Fri,  6 Sep 2019 18:26:13 +0800
> Ruifeng Wang <ruifeng.wang@arm.com> wrote:
> 
> > Lock-free mode is supported by hash library and LPM library.
> > Now we add an option for l3fwd example to enable the lock-free mode.
> > Necessary preparation steps are added to use lock-free LPM mode.
> 
> If lock-free mode works it should just do that.
> Having options mean that there are two test cases; which inevitably leads to
> one of them being broken.

Agree that having options will add scenarios that being tested.
Since these different scenarios are supported by Hash / LPM library, the tests on
them should be valid. 
As l3fwd application is always used to benchmark data path performance, make
both supported modes available can help user to easily collect data and compare.
In the long run, we can make lock-free mode the default used by l3fwd when it
is fine tuned. 

Re: [dpdk-dev] [PATCH 0/2] add lock-free mode for l3fwd

[Honnappa Nagarahalli]

<snip>

> >
> > On Fri,  6 Sep 2019 18:26:13 +0800
> > Ruifeng Wang <ruifeng.wang@arm.com> wrote:
> >
> > > Lock-free mode is supported by hash library and LPM library.
> > > Now we add an option for l3fwd example to enable the lock-free mode.
> > > Necessary preparation steps are added to use lock-free LPM mode.
> >
> > If lock-free mode works it should just do that.
> > Having options mean that there are two test cases; which inevitably
> > leads to one of them being broken.
> 
> Agree that having options will add scenarios that being tested.
> Since these different scenarios are supported by Hash / LPM library, the tests
> on them should be valid.
> As l3fwd application is always used to benchmark data path performance,
> make both supported modes available can help user to easily collect data and
> compare.
> In the long run, we can make lock-free mode the default used by l3fwd when
> it is fine tuned.
I agree that if lock-free works we should just do that. It makes L3fwd application more practical.

The LPM has always been lock free on the data plane side. The writer side parts and quiescent state reporting on the data plane are being added now.

With the optimizations in the last release, the lock free hash algorithm's performance has come very close to RW lock based algorithm both for hit and miss cases for single core. Unfortunately (😊), any further optimizations seem to increase the performance of the RW lock based algorithm as well. So, looks like there will be that small difference between the 2 algorithms. However, a practical application will use other cores in the SoC and lock-free algorithms scale better with increasing number of cores. IMO, we should bite the bullet and make hash lock-free algorithm default.