DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads
@ 2018-10-11 19:57 Ferruh Yigit
  2018-10-11 19:57 ` [dpdk-dev] [PATCH 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Ferruh Yigit @ 2018-10-11 19:57 UTC (permalink / raw)
  Cc: dev, Ferruh Yigit

It is common that sample applications call rte_eal_wait_lcore() while
waiting for worker threads to be terminated.
Mostly master lcore keeps waiting in this function.

The waiting app for termination is not a time critical task, app can
prefer a sleep version of the waiting to consume less cycles.

A sleeping version of the API, rte_eal_wait_lcore_sleep(), has been
added which gets sleeping interval as parameter.

Sample applications will be updated later to use this API.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 lib/librte_eal/common/eal_common_launch.c  | 22 ++++++++++++++++++
 lib/librte_eal/common/include/rte_launch.h | 27 ++++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  1 +
 3 files changed, 50 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_launch.c b/lib/librte_eal/common/eal_common_launch.c
index fe0ba3f0d..e804c70c3 100644
--- a/lib/librte_eal/common/eal_common_launch.c
+++ b/lib/librte_eal/common/eal_common_launch.c
@@ -5,6 +5,7 @@
 #include <errno.h>
 #include <stdint.h>
 #include <stdio.h>
+#include <unistd.h>
 #include <sys/queue.h>
 
 #include <rte_launch.h>
@@ -35,6 +36,27 @@ rte_eal_wait_lcore(unsigned slave_id)
 	return lcore_config[slave_id].ret;
 }
 
+/*
+ * Wait until a lcore finished its job by sleeping.
+ * Sleep time will be times of 'usec'
+ */
+int
+rte_eal_wait_lcore_sleep(unsigned slave_id, size_t usec)
+{
+	if (lcore_config[slave_id].state == WAIT)
+		return 0;
+
+	while (lcore_config[slave_id].state != WAIT &&
+	       lcore_config[slave_id].state != FINISHED)
+			usleep(usec);
+
+	rte_rmb();
+
+	/* we are in finished state, go to wait state */
+	lcore_config[slave_id].state = WAIT;
+	return lcore_config[slave_id].ret;
+}
+
 /*
  * Check that every SLAVE lcores are in WAIT state, then call
  * rte_eal_remote_launch() for all of them. If call_master is true
diff --git a/lib/librte_eal/common/include/rte_launch.h b/lib/librte_eal/common/include/rte_launch.h
index 06a671752..a935fd8b5 100644
--- a/lib/librte_eal/common/include/rte_launch.h
+++ b/lib/librte_eal/common/include/rte_launch.h
@@ -11,6 +11,8 @@
  * Launch tasks on other lcores
  */
 
+#include <rte_compat.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -129,6 +131,31 @@ enum rte_lcore_state_t rte_eal_get_lcore_state(unsigned slave_id);
  */
 int rte_eal_wait_lcore(unsigned slave_id);
 
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Wait until an lcore finishes its job.
+ *
+ * To be executed on the MASTER lcore only.
+ *
+ * Same as rte_eal_wait_lcore() but sleeps instead of busy wait.
+ *
+ * @param slave_id
+ *   The identifier of the lcore.
+ * @param usec
+ *   The sleep interval in microseconds
+ * @return
+ *   - 0: If the lcore identified by the slave_id is in a WAIT state.
+ *   - The value that was returned by the previous remote launch
+ *     function call if the lcore identified by the slave_id was in a
+ *     FINISHED or RUNNING state. In this case, it changes the state
+ *     of the lcore to WAIT.
+ */
+__rte_experimental int
+rte_eal_wait_lcore_sleep(unsigned slave_id, size_t usec);
+
 /**
  * Wait until all lcores finish their jobs.
  *
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index e968edc2e..6c636a65d 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -292,6 +292,7 @@ EXPERIMENTAL {
 	rte_devargs_remove;
 	rte_devargs_type_count;
 	rte_eal_cleanup;
+	rte_eal_wait_lcore_sleep;
 	rte_fbarray_attach;
 	rte_fbarray_destroy;
 	rte_fbarray_detach;
-- 
2.17.1

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [PATCH 2/2] examples: sleep while waiting worker threads to terminate
  2018-10-11 19:57 [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Ferruh Yigit
@ 2018-10-11 19:57 ` Ferruh Yigit
  2018-10-11 20:32 ` [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Stephen Hemminger
  2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
  2 siblings, 0 replies; 8+ messages in thread
From: Ferruh Yigit @ 2018-10-11 19:57 UTC (permalink / raw)
  To: Amr Mokhtar, Bruce Richardson, David Hunt, Remy Horton, Ori Kam,
	Pablo de Lara, Radu Nicolau, Akhil Goyal, Tomasz Kantecki,
	Konstantin Ananyev, Declan Doherty, Reshma Pattan, John McNamara,
	Harry van Haaren, Xiaoyun Li, Cristian Dumitrescu,
	Maxime Coquelin, Tiwei Bie, Zhihong Wang
  Cc: dev, Ferruh Yigit

This is to prevent sample applications to do busy wait while waiting for
worker threads to terminate. Should save cycles on master core.
By default a 1ms wait interval used.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 examples/bbdev_app/main.c                         | 2 +-
 examples/distributor/Makefile                     | 3 +++
 examples/distributor/main.c                       | 2 +-
 examples/distributor/meson.build                  | 1 +
 examples/ethtool/ethtool-app/Makefile             | 1 +
 examples/ethtool/ethtool-app/main.c               | 2 +-
 examples/exception_path/Makefile                  | 3 +++
 examples/exception_path/main.c                    | 2 +-
 examples/exception_path/meson.build               | 1 +
 examples/ip_fragmentation/Makefile                | 3 +++
 examples/ip_fragmentation/main.c                  | 2 +-
 examples/ip_fragmentation/meson.build             | 1 +
 examples/ip_reassembly/Makefile                   | 3 +++
 examples/ip_reassembly/main.c                     | 2 +-
 examples/ip_reassembly/meson.build                | 1 +
 examples/ipsec-secgw/ipsec-secgw.c                | 2 +-
 examples/ipv4_multicast/Makefile                  | 3 +++
 examples/ipv4_multicast/main.c                    | 2 +-
 examples/ipv4_multicast/meson.build               | 1 +
 examples/kni/Makefile                             | 3 +++
 examples/kni/main.c                               | 2 +-
 examples/kni/meson.build                          | 1 +
 examples/l2fwd-crypto/Makefile                    | 3 +++
 examples/l2fwd-crypto/main.c                      | 2 +-
 examples/l2fwd-crypto/meson.build                 | 1 +
 examples/l2fwd-jobstats/Makefile                  | 3 +++
 examples/l2fwd-jobstats/main.c                    | 2 +-
 examples/l2fwd-jobstats/meson.build               | 1 +
 examples/l2fwd-keepalive/Makefile                 | 3 +++
 examples/l2fwd-keepalive/main.c                   | 2 +-
 examples/l2fwd-keepalive/meson.build              | 1 +
 examples/l2fwd/Makefile                           | 3 +++
 examples/l2fwd/main.c                             | 2 +-
 examples/l2fwd/meson.build                        | 1 +
 examples/l3fwd-acl/Makefile                       | 3 +++
 examples/l3fwd-acl/main.c                         | 2 +-
 examples/l3fwd-acl/meson.build                    | 1 +
 examples/l3fwd-power/Makefile                     | 3 +++
 examples/l3fwd-power/main.c                       | 2 +-
 examples/l3fwd-power/meson.build                  | 1 +
 examples/l3fwd-vf/Makefile                        | 3 +++
 examples/l3fwd-vf/main.c                          | 2 +-
 examples/l3fwd-vf/meson.build                     | 1 +
 examples/l3fwd/Makefile                           | 3 +++
 examples/l3fwd/main.c                             | 2 +-
 examples/l3fwd/meson.build                        | 1 +
 examples/link_status_interrupt/Makefile           | 3 +++
 examples/link_status_interrupt/main.c             | 2 +-
 examples/link_status_interrupt/meson.build        | 1 +
 examples/load_balancer/Makefile                   | 3 +++
 examples/load_balancer/main.c                     | 2 +-
 examples/load_balancer/meson.build                | 1 +
 examples/packet_ordering/Makefile                 | 3 +++
 examples/packet_ordering/main.c                   | 2 +-
 examples/packet_ordering/meson.build              | 1 +
 examples/performance-thread/l3fwd-thread/Makefile | 1 +
 examples/performance-thread/l3fwd-thread/main.c   | 2 +-
 examples/performance-thread/pthread_shim/Makefile | 1 +
 examples/performance-thread/pthread_shim/main.c   | 2 +-
 examples/qos_meter/Makefile                       | 1 +
 examples/qos_meter/main.c                         | 2 +-
 examples/qos_meter/meson.build                    | 1 +
 examples/tep_termination/main.c                   | 2 +-
 examples/vhost/main.c                             | 2 +-
 examples/vhost_crypto/main.c                      | 2 +-
 examples/vmdq/Makefile                            | 3 +++
 examples/vmdq/main.c                              | 2 +-
 examples/vmdq/meson.build                         | 1 +
 68 files changed, 104 insertions(+), 27 deletions(-)

diff --git a/examples/bbdev_app/main.c b/examples/bbdev_app/main.c
index d68c06aef..c8faa1e38 100644
--- a/examples/bbdev_app/main.c
+++ b/examples/bbdev_app/main.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
 		stats_loop(&stats_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		ret |= rte_eal_wait_lcore(lcore_id);
+		ret |= rte_eal_wait_lcore_sleep(lcore_id, 1000);
 	}
 
 	return ret;
diff --git a/examples/distributor/Makefile b/examples/distributor/Makefile
index 05ea0bfec..5a3c5292d 100644
--- a/examples/distributor/Makefile
+++ b/examples/distributor/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -49,6 +51,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/distributor/main.c b/examples/distributor/main.c
index 03a05e3d9..63c366d90 100644
--- a/examples/distributor/main.c
+++ b/examples/distributor/main.c
@@ -801,7 +801,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/distributor/meson.build b/examples/distributor/meson.build
index 88c001f56..d036ea0f6 100644
--- a/examples/distributor/meson.build
+++ b/examples/distributor/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'distributor'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ethtool/ethtool-app/Makefile b/examples/ethtool/ethtool-app/Makefile
index 4cd9efdd5..b7092d267 100644
--- a/examples/ethtool/ethtool-app/Makefile
+++ b/examples/ethtool/ethtool-app/Makefile
@@ -18,6 +18,7 @@ SRCS-y := main.c ethapp.c
 
 CFLAGS += -O3 -D_GNU_SOURCE -pthread -I$(SRCDIR)/../lib
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 LDLIBS += -L$(subst ethtool-app,lib,$(RTE_OUTPUT))/lib
 LDLIBS += -lrte_ethtool
diff --git a/examples/ethtool/ethtool-app/main.c b/examples/ethtool/ethtool-app/main.c
index dc93adfe3..bef6682c3 100644
--- a/examples/ethtool/ethtool-app/main.c
+++ b/examples/ethtool/ethtool-app/main.c
@@ -276,7 +276,7 @@ int main(int argc, char **argv)
 
 	app_cfg.exit_now = 1;
 	RTE_LCORE_FOREACH_SLAVE(id_core) {
-		if (rte_eal_wait_lcore(id_core) < 0)
+		if (rte_eal_wait_lcore_sleep(id_core, 1000)< 0)
 			return -1;
 	}
 
diff --git a/examples/exception_path/Makefile b/examples/exception_path/Makefile
index ae74781ec..02e23d435 100644
--- a/examples/exception_path/Makefile
+++ b/examples/exception_path/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 
diff --git a/examples/exception_path/main.c b/examples/exception_path/main.c
index 4180a8689..7cbd66a6f 100644
--- a/examples/exception_path/main.c
+++ b/examples/exception_path/main.c
@@ -581,7 +581,7 @@ main(int argc, char** argv)
 	/* Launch per-lcore function on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(i) {
-		if (rte_eal_wait_lcore(i) < 0)
+		if (rte_eal_wait_lcore_sleep(i, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/exception_path/meson.build b/examples/exception_path/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/exception_path/meson.build
+++ b/examples/exception_path/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ip_fragmentation/Makefile b/examples/ip_fragmentation/Makefile
index 9e89e744c..bebcd4703 100644
--- a/examples/ip_fragmentation/Makefile
+++ b/examples/ip_fragmentation/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ip_fragmentation/main.c b/examples/ip_fragmentation/main.c
index 17a877da2..fdd6c8a94 100644
--- a/examples/ip_fragmentation/main.c
+++ b/examples/ip_fragmentation/main.c
@@ -1018,7 +1018,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/ip_fragmentation/meson.build b/examples/ip_fragmentation/meson.build
index 304203eed..9782a6a7b 100644
--- a/examples/ip_fragmentation/meson.build
+++ b/examples/ip_fragmentation/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps +=  ['ip_frag', 'lpm']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ip_reassembly/Makefile b/examples/ip_reassembly/Makefile
index 1e81315f2..7af4c2756 100644
--- a/examples/ip_reassembly/Makefile
+++ b/examples/ip_reassembly/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ip_reassembly/main.c b/examples/ip_reassembly/main.c
index 17b55d4c7..b498ad0fc 100644
--- a/examples/ip_reassembly/main.c
+++ b/examples/ip_reassembly/main.c
@@ -1175,7 +1175,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/ip_reassembly/meson.build b/examples/ip_reassembly/meson.build
index 8ebd48291..8a667c265 100644
--- a/examples/ip_reassembly/meson.build
+++ b/examples/ip_reassembly/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['lpm', 'ip_frag']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..a9af17103 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1837,7 +1837,7 @@ main(int32_t argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/ipv4_multicast/Makefile b/examples/ipv4_multicast/Makefile
index a16c62333..a1cefc50a 100644
--- a/examples/ipv4_multicast/Makefile
+++ b/examples/ipv4_multicast/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ipv4_multicast/main.c b/examples/ipv4_multicast/main.c
index 6530d4830..825f5264d 100644
--- a/examples/ipv4_multicast/main.c
+++ b/examples/ipv4_multicast/main.c
@@ -793,7 +793,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/ipv4_multicast/meson.build b/examples/ipv4_multicast/meson.build
index d9e4c7c21..6969e2c54 100644
--- a/examples/ipv4_multicast/meson.build
+++ b/examples/ipv4_multicast/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'hash'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/kni/Makefile b/examples/kni/Makefile
index 7e19d2e2a..adb921ce2 100644
--- a/examples/kni/Makefile
+++ b/examples/kni/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -55,6 +57,7 @@ endif
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/kni/main.c b/examples/kni/main.c
index 80c401c51..ac022b6ae 100644
--- a/examples/kni/main.c
+++ b/examples/kni/main.c
@@ -952,7 +952,7 @@ main(int argc, char** argv)
 	/* Launch per-lcore function on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(i) {
-		if (rte_eal_wait_lcore(i) < 0)
+		if (rte_eal_wait_lcore_sleep(i, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/kni/meson.build b/examples/kni/meson.build
index 791316394..46ac53e0d 100644
--- a/examples/kni/meson.build
+++ b/examples/kni/meson.build
@@ -9,6 +9,7 @@
 # this app can be built if-and-only-if KNI library is buildable
 build = dpdk_conf.has('RTE_LIBRTE_KNI')
 deps += ['kni', 'bus_pci']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-crypto/Makefile b/examples/l2fwd-crypto/Makefile
index 6658fd0d4..1995677df 100644
--- a/examples/l2fwd-crypto/Makefile
+++ b/examples/l2fwd-crypto/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
 ifeq ($(CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER),y)
diff --git a/examples/l2fwd-crypto/main.c b/examples/l2fwd-crypto/main.c
index f12fd266e..e8b81e998 100644
--- a/examples/l2fwd-crypto/main.c
+++ b/examples/l2fwd-crypto/main.c
@@ -2765,7 +2765,7 @@ main(int argc, char **argv)
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, (void *)&options,
 			CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-crypto/meson.build b/examples/l2fwd-crypto/meson.build
index 09438a6a0..6c852ad19 100644
--- a/examples/l2fwd-crypto/meson.build
+++ b/examples/l2fwd-crypto/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'cryptodev'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-jobstats/Makefile b/examples/l2fwd-jobstats/Makefile
index 696a8b21a..7350fa931 100644
--- a/examples/l2fwd-jobstats/Makefile
+++ b/examples/l2fwd-jobstats/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l2fwd-jobstats/main.c b/examples/l2fwd-jobstats/main.c
index a4d28e178..bf4b1560b 100644
--- a/examples/l2fwd-jobstats/main.c
+++ b/examples/l2fwd-jobstats/main.c
@@ -995,7 +995,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-jobstats/meson.build b/examples/l2fwd-jobstats/meson.build
index 1ffd484e2..3653aa7ec 100644
--- a/examples/l2fwd-jobstats/meson.build
+++ b/examples/l2fwd-jobstats/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['jobstats', 'timer']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-keepalive/Makefile b/examples/l2fwd-keepalive/Makefile
index 4ab67db44..4ca5685f3 100644
--- a/examples/l2fwd-keepalive/Makefile
+++ b/examples/l2fwd-keepalive/Makefile
@@ -25,6 +25,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -52,6 +54,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDFLAGS += -lrt
 
 include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/l2fwd-keepalive/main.c b/examples/l2fwd-keepalive/main.c
index 0bf2b5336..f3e7de572 100644
--- a/examples/l2fwd-keepalive/main.c
+++ b/examples/l2fwd-keepalive/main.c
@@ -788,7 +788,7 @@ main(int argc, char **argv)
 		}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-keepalive/meson.build b/examples/l2fwd-keepalive/meson.build
index 6f7b007e1..2dffffaaa 100644
--- a/examples/l2fwd-keepalive/meson.build
+++ b/examples/l2fwd-keepalive/meson.build
@@ -8,6 +8,7 @@
 
 ext_deps += cc.find_library('rt')
 deps += 'timer'
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'shm.c'
 )
diff --git a/examples/l2fwd/Makefile b/examples/l2fwd/Makefile
index a8a47ad4e..7030b62e0 100644
--- a/examples/l2fwd/Makefile
+++ b/examples/l2fwd/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 6c23215a5..27acf2ff7 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -728,7 +728,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0) {
 			ret = -1;
 			break;
 		}
diff --git a/examples/l2fwd/meson.build b/examples/l2fwd/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/l2fwd/meson.build
+++ b/examples/l2fwd/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd-acl/Makefile b/examples/l3fwd-acl/Makefile
index 285683f83..de10e9d12 100644
--- a/examples/l3fwd-acl/Makefile
+++ b/examples/l3fwd-acl/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-acl/main.c b/examples/l3fwd-acl/main.c
index a322ce4f2..79f4da8c6 100644
--- a/examples/l3fwd-acl/main.c
+++ b/examples/l3fwd-acl/main.c
@@ -2078,7 +2078,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-acl/meson.build b/examples/l3fwd-acl/meson.build
index 7096e00c1..68cebd6ce 100644
--- a/examples/l3fwd-acl/meson.build
+++ b/examples/l3fwd-acl/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['acl', 'lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd-power/Makefile b/examples/l3fwd-power/Makefile
index d7e39a343..7ca02935a 100644
--- a/examples/l3fwd-power/Makefile
+++ b/examples/l3fwd-power/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -56,6 +58,7 @@ else
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 68527d26f..fa111f717 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -1908,7 +1908,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-power/meson.build b/examples/l3fwd-power/meson.build
index 20c805436..d304ab7ab 100644
--- a/examples/l3fwd-power/meson.build
+++ b/examples/l3fwd-power/meson.build
@@ -10,6 +10,7 @@ if host_machine.system() != 'linux'
 	build = false
 endif
 deps += ['power', 'timer', 'lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'perf_core.c'
 )
diff --git a/examples/l3fwd-vf/Makefile b/examples/l3fwd-vf/Makefile
index dfb1d52d3..a4eb1564d 100644
--- a/examples/l3fwd-vf/Makefile
+++ b/examples/l3fwd-vf/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3 $(USER_FLAGS)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-vf/main.c b/examples/l3fwd-vf/main.c
index 41137f978..07a28290a 100644
--- a/examples/l3fwd-vf/main.c
+++ b/examples/l3fwd-vf/main.c
@@ -1088,7 +1088,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-vf/meson.build b/examples/l3fwd-vf/meson.build
index 226286e74..00f3c38f4 100644
--- a/examples/l3fwd-vf/meson.build
+++ b/examples/l3fwd-vf/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd/Makefile b/examples/l3fwd/Makefile
index cccdd9dfa..4f032f577 100644
--- a/examples/l3fwd/Makefile
+++ b/examples/l3fwd/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 CFLAGS += -I$(SRCDIR)
 CFLAGS += -O3 $(USER_FLAGS)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index e4b99efe0..729b75673 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -1022,7 +1022,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l3fwd_lkp.main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0) {
 			ret = -1;
 			break;
 		}
diff --git a/examples/l3fwd/meson.build b/examples/l3fwd/meson.build
index 6dd4b9022..cbef07f4f 100644
--- a/examples/l3fwd/meson.build
+++ b/examples/l3fwd/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['hash', 'lpm']
+allow_experimental_apis = true
 sources = files(
 	'l3fwd_em.c', 'l3fwd_lpm.c', 'main.c'
 )
diff --git a/examples/link_status_interrupt/Makefile b/examples/link_status_interrupt/Makefile
index 160682123..e6d7acfe2 100644
--- a/examples/link_status_interrupt/Makefile
+++ b/examples/link_status_interrupt/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/link_status_interrupt/main.c b/examples/link_status_interrupt/main.c
index f3346d23b..b72c39347 100644
--- a/examples/link_status_interrupt/main.c
+++ b/examples/link_status_interrupt/main.c
@@ -703,7 +703,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(lsi_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/link_status_interrupt/meson.build b/examples/link_status_interrupt/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/link_status_interrupt/meson.build
+++ b/examples/link_status_interrupt/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/load_balancer/Makefile b/examples/load_balancer/Makefile
index fc8df71e8..57b602815 100644
--- a/examples/load_balancer/Makefile
+++ b/examples/load_balancer/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3 -g
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 CFLAGS_config.o := -D_GNU_SOURCE
 
 # workaround for a gcc bug with noreturn attribute
diff --git a/examples/load_balancer/main.c b/examples/load_balancer/main.c
index d3dcb235d..f20beeedf 100644
--- a/examples/load_balancer/main.c
+++ b/examples/load_balancer/main.c
@@ -67,7 +67,7 @@ main(int argc, char **argv)
 	/* Launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(app_lcore_main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore) {
-		if (rte_eal_wait_lcore(lcore) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore, 1000) < 0) {
 			return -1;
 		}
 	}
diff --git a/examples/load_balancer/meson.build b/examples/load_balancer/meson.build
index 4f7ac3999..19708974c 100644
--- a/examples/load_balancer/meson.build
+++ b/examples/load_balancer/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'lpm'
+allow_experimental_apis = true
 sources = files(
 	'config.c', 'init.c', 'main.c', 'runtime.c'
 )
diff --git a/examples/packet_ordering/Makefile b/examples/packet_ordering/Makefile
index 3cf1ee1dc..c608d2e1c 100644
--- a/examples/packet_ordering/Makefile
+++ b/examples/packet_ordering/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/packet_ordering/main.c b/examples/packet_ordering/main.c
index 149bfdd02..e9053a756 100644
--- a/examples/packet_ordering/main.c
+++ b/examples/packet_ordering/main.c
@@ -720,7 +720,7 @@ main(int argc, char **argv)
 	rx_thread(rx_to_workers);
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/packet_ordering/meson.build b/examples/packet_ordering/meson.build
index 6c2fccdcb..a3776946f 100644
--- a/examples/packet_ordering/meson.build
+++ b/examples/packet_ordering/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'reorder'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/performance-thread/l3fwd-thread/Makefile b/examples/performance-thread/l3fwd-thread/Makefile
index 5558043f2..46d333849 100644
--- a/examples/performance-thread/l3fwd-thread/Makefile
+++ b/examples/performance-thread/l3fwd-thread/Makefile
@@ -19,6 +19,7 @@ SRCS-y := main.c
 include $(RTE_SDK)/examples/performance-thread/common/common.mk
 
 CFLAGS += -O3 -g $(USER_FLAGS) $(INCLUDES) $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/performance-thread/l3fwd-thread/main.c b/examples/performance-thread/l3fwd-thread/main.c
index 50fd1b00a..71365b4e8 100644
--- a/examples/performance-thread/l3fwd-thread/main.c
+++ b/examples/performance-thread/l3fwd-thread/main.c
@@ -3734,7 +3734,7 @@ main(int argc, char **argv)
 		/* launch per-lcore init on every lcore */
 		rte_eal_mp_remote_launch(pthread_run, NULL, CALL_MASTER);
 		RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-			if (rte_eal_wait_lcore(lcore_id) < 0)
+			if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 				return -1;
 		}
 	}
diff --git a/examples/performance-thread/pthread_shim/Makefile b/examples/performance-thread/pthread_shim/Makefile
index a6d276a49..9294e703e 100644
--- a/examples/performance-thread/pthread_shim/Makefile
+++ b/examples/performance-thread/pthread_shim/Makefile
@@ -20,6 +20,7 @@ include $(RTE_SDK)/examples/performance-thread/common/common.mk
 
 CFLAGS += -g -O3 $(USER_FLAGS) $(INCLUDES)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 LDFLAGS += -lpthread
 
diff --git a/examples/performance-thread/pthread_shim/main.c b/examples/performance-thread/pthread_shim/main.c
index 7d0d58116..80b92016b 100644
--- a/examples/performance-thread/pthread_shim/main.c
+++ b/examples/performance-thread/pthread_shim/main.c
@@ -257,7 +257,7 @@ int main(int argc, char **argv)
 
 	/* wait for threads to stop */
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id, 1000);
 	}
 	return 0;
 }
diff --git a/examples/qos_meter/Makefile b/examples/qos_meter/Makefile
index 46341b1a7..7482d7e61 100644
--- a/examples/qos_meter/Makefile
+++ b/examples/qos_meter/Makefile
@@ -52,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/qos_meter/main.c b/examples/qos_meter/main.c
index 9b0112449..cb1b43606 100644
--- a/examples/qos_meter/main.c
+++ b/examples/qos_meter/main.c
@@ -438,7 +438,7 @@ main(int argc, char **argv)
 	/* Launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/qos_meter/meson.build b/examples/qos_meter/meson.build
index ef7779f2f..10cd4bc79 100644
--- a/examples/qos_meter/meson.build
+++ b/examples/qos_meter/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'meter'
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'rte_policer.c'
 )
diff --git a/examples/tep_termination/main.c b/examples/tep_termination/main.c
index 7795d0894..bae34fe9f 100644
--- a/examples/tep_termination/main.c
+++ b/examples/tep_termination/main.c
@@ -1238,7 +1238,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id)
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id, 1000);
 
 	return 0;
 }
diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index dc9ea1018..5c571ffd3 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -1558,7 +1558,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id)
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id, 1000);
 
 	return 0;
 
diff --git a/examples/vhost_crypto/main.c b/examples/vhost_crypto/main.c
index cbb5e49d2..942911214 100644
--- a/examples/vhost_crypto/main.c
+++ b/examples/vhost_crypto/main.c
@@ -582,7 +582,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH(lcore)
-		rte_eal_wait_lcore(lcore);
+		rte_eal_wait_lcore_sleep(lcore, 1000);
 
 	free_resource();
 
diff --git a/examples/vmdq/Makefile b/examples/vmdq/Makefile
index 87abeab93..6ff8f530d 100644
--- a/examples/vmdq/Makefile
+++ b/examples/vmdq/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -49,6 +51,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 EXTRA_CFLAGS += -O3
 
diff --git a/examples/vmdq/main.c b/examples/vmdq/main.c
index 627a5da48..aa93dcece 100644
--- a/examples/vmdq/main.c
+++ b/examples/vmdq/main.c
@@ -611,7 +611,7 @@ main(int argc, char *argv[])
 	/* call lcore_main() on every lcore */
 	rte_eal_mp_remote_launch(lcore_main, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id, 1000) < 0)
 			return -1;
 	}
 
diff --git a/examples/vmdq/meson.build b/examples/vmdq/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/vmdq/meson.build
+++ b/examples/vmdq/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
-- 
2.17.1

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads
  2018-10-11 19:57 [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Ferruh Yigit
  2018-10-11 19:57 ` [dpdk-dev] [PATCH 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
@ 2018-10-11 20:32 ` Stephen Hemminger
  2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
  2 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2018-10-11 20:32 UTC (permalink / raw)
  To: Ferruh Yigit; +Cc: dev

On Thu, 11 Oct 2018 20:57:52 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> +/*
> + * Wait until a lcore finished its job by sleeping.
> + * Sleep time will be times of 'usec'
> + */
> +int
> +rte_eal_wait_lcore_sleep(unsigned slave_id, size_t usec)
> +{
> +	if (lcore_config[slave_id].state == WAIT)
> +		return 0;
> +
> +	while (lcore_config[slave_id].state != WAIT &&
> +	       lcore_config[slave_id].state != FINISHED)
> +			usleep(usec);
> +
> +	rte_rmb();
> +
> +	/* we are in finished state, go to wait state */
> +	lcore_config[slave_id].state = WAIT;
> +	return lcore_config[slave_id].ret;
> +}
> +

Since lcore threads are really pthreads you could avoid doing polling by
using a pthread condition (ie. pthread_cond_wait and/or pthread_cond_timedwait)

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads
  2018-10-11 19:57 [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Ferruh Yigit
  2018-10-11 19:57 ` [dpdk-dev] [PATCH 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
  2018-10-11 20:32 ` [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Stephen Hemminger
@ 2018-10-15 22:21 ` Ferruh Yigit
  2018-10-15 22:21   ` [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
                     ` (2 more replies)
  2 siblings, 3 replies; 8+ messages in thread
From: Ferruh Yigit @ 2018-10-15 22:21 UTC (permalink / raw)
  To: Bruce Richardson; +Cc: dev, Ferruh Yigit, stephen

It is common that sample applications call rte_eal_wait_lcore() while
waiting for worker threads to be terminated.
Mostly master lcore keeps waiting in this function.

The waiting app for termination is not a time critical task, app can
prefer a sleep version of the waiting to consume less cycles.

A sleeping version of the API, rte_eal_wait_lcore_sleep(), has been
added which uses pthread conditions.

Sample applications will be updated later to use this API.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v2:
* use pthread cond instead of usleep
---
 lib/librte_eal/bsdapp/eal/eal.c            |  3 +++
 lib/librte_eal/bsdapp/eal/eal_thread.c     |  7 ++++++
 lib/librte_eal/common/eal_common_launch.c  | 22 ++++++++++++++++++
 lib/librte_eal/common/include/rte_launch.h | 26 ++++++++++++++++++++++
 lib/librte_eal/common/include/rte_lcore.h  |  3 +++
 lib/librte_eal/linuxapp/eal/eal.c          |  3 +++
 lib/librte_eal/linuxapp/eal/eal_thread.c   |  7 ++++++
 lib/librte_eal/rte_eal_version.map         |  1 +
 8 files changed, 72 insertions(+)

diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 7735194a3..e7d676657 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -756,6 +756,9 @@ rte_eal_init(int argc, char **argv)
 		snprintf(thread_name, sizeof(thread_name),
 				"lcore-slave-%d", i);
 		rte_thread_setname(lcore_config[i].thread_id, thread_name);
+
+		pthread_mutex_init(&rte_eal_thread_mutex[i], NULL);
+		pthread_cond_init(&rte_eal_thread_cond[i], NULL);
 	}
 
 	/*
diff --git a/lib/librte_eal/bsdapp/eal/eal_thread.c b/lib/librte_eal/bsdapp/eal/eal_thread.c
index 309b58726..60db32d57 100644
--- a/lib/librte_eal/bsdapp/eal/eal_thread.c
+++ b/lib/librte_eal/bsdapp/eal/eal_thread.c
@@ -28,6 +28,9 @@ RTE_DEFINE_PER_LCORE(unsigned, _lcore_id) = LCORE_ID_ANY;
 RTE_DEFINE_PER_LCORE(unsigned, _socket_id) = (unsigned)SOCKET_ID_ANY;
 RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
 
+pthread_cond_t rte_eal_thread_cond[RTE_MAX_LCORE];
+pthread_mutex_t rte_eal_thread_mutex[RTE_MAX_LCORE];
+
 /*
  * Send a message to a slave lcore identified by slave_id to call a
  * function f with argument arg. Once the execution is done, the
@@ -154,6 +157,10 @@ eal_thread_loop(__attribute__((unused)) void *arg)
 		lcore_config[lcore_id].ret = ret;
 		rte_wmb();
 		lcore_config[lcore_id].state = FINISHED;
+
+		pthread_mutex_lock(&rte_eal_thread_mutex[lcore_id]);
+		pthread_cond_signal(&rte_eal_thread_cond[lcore_id]);
+		pthread_mutex_unlock(&rte_eal_thread_mutex[lcore_id]);
 	}
 
 	/* never reached */
diff --git a/lib/librte_eal/common/eal_common_launch.c b/lib/librte_eal/common/eal_common_launch.c
index fe0ba3f0d..ebfa65847 100644
--- a/lib/librte_eal/common/eal_common_launch.c
+++ b/lib/librte_eal/common/eal_common_launch.c
@@ -5,6 +5,7 @@
 #include <errno.h>
 #include <stdint.h>
 #include <stdio.h>
+#include <unistd.h>
 #include <sys/queue.h>
 
 #include <rte_launch.h>
@@ -35,6 +36,27 @@ rte_eal_wait_lcore(unsigned slave_id)
 	return lcore_config[slave_id].ret;
 }
 
+/*
+ * Wait until a lcore finished its job by pthread condition.
+ */
+int
+rte_eal_wait_lcore_sleep(unsigned slave_id)
+{
+	if (lcore_config[slave_id].state == WAIT)
+		return 0;
+
+	pthread_mutex_lock(&rte_eal_thread_mutex[slave_id]);
+	while (lcore_config[slave_id].state != WAIT &&
+	       lcore_config[slave_id].state != FINISHED)
+		pthread_cond_wait(&rte_eal_thread_cond[slave_id],
+				&rte_eal_thread_mutex[slave_id]);
+	pthread_mutex_unlock(&rte_eal_thread_mutex[slave_id]);
+
+	/* we are in finished state, go to wait state */
+	lcore_config[slave_id].state = WAIT;
+	return lcore_config[slave_id].ret;
+}
+
 /*
  * Check that every SLAVE lcores are in WAIT state, then call
  * rte_eal_remote_launch() for all of them. If call_master is true
diff --git a/lib/librte_eal/common/include/rte_launch.h b/lib/librte_eal/common/include/rte_launch.h
index 06a671752..0306f7c3a 100644
--- a/lib/librte_eal/common/include/rte_launch.h
+++ b/lib/librte_eal/common/include/rte_launch.h
@@ -11,6 +11,8 @@
  * Launch tasks on other lcores
  */
 
+#include <rte_compat.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -129,6 +131,30 @@ enum rte_lcore_state_t rte_eal_get_lcore_state(unsigned slave_id);
  */
 int rte_eal_wait_lcore(unsigned slave_id);
 
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Wait until an lcore finishes its job.
+ *
+ * To be executed on the MASTER lcore only.
+ *
+ * Same as rte_eal_wait_lcore() but waits using pthread conditions
+ * instead of polling in busy loop.
+ *
+ * @param slave_id
+ *   The identifier of the lcore.
+ * @return
+ *   - 0: If the lcore identified by the slave_id is in a WAIT state.
+ *   - The value that was returned by the previous remote launch
+ *     function call if the lcore identified by the slave_id was in a
+ *     FINISHED or RUNNING state. In this case, it changes the state
+ *     of the lcore to WAIT.
+ */
+__rte_experimental int
+rte_eal_wait_lcore_sleep(unsigned slave_id);
+
 /**
  * Wait until all lcores finish their jobs.
  *
diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h
index 6e09d9181..9ce8bf643 100644
--- a/lib/librte_eal/common/include/rte_lcore.h
+++ b/lib/librte_eal/common/include/rte_lcore.h
@@ -53,6 +53,9 @@ struct lcore_config {
  */
 extern struct lcore_config lcore_config[RTE_MAX_LCORE];
 
+extern pthread_cond_t rte_eal_thread_cond[RTE_MAX_LCORE];
+extern pthread_mutex_t rte_eal_thread_mutex[RTE_MAX_LCORE];
+
 RTE_DECLARE_PER_LCORE(unsigned, _lcore_id);  /**< Per thread "lcore id". */
 RTE_DECLARE_PER_LCORE(rte_cpuset_t, _cpuset); /**< Per thread "cpuset". */
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 950f33f2c..9d69a0642 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1019,6 +1019,9 @@ rte_eal_init(int argc, char **argv)
 
 		lcore_config[i].state = WAIT;
 
+		pthread_mutex_init(&rte_eal_thread_mutex[i], NULL);
+		pthread_cond_init(&rte_eal_thread_cond[i], NULL);
+
 		/* create a thread for each lcore */
 		ret = pthread_create(&lcore_config[i].thread_id, NULL,
 				     eal_thread_loop, NULL);
diff --git a/lib/librte_eal/linuxapp/eal/eal_thread.c b/lib/librte_eal/linuxapp/eal/eal_thread.c
index b496fc711..5381c7aa2 100644
--- a/lib/librte_eal/linuxapp/eal/eal_thread.c
+++ b/lib/librte_eal/linuxapp/eal/eal_thread.c
@@ -28,6 +28,9 @@ RTE_DEFINE_PER_LCORE(unsigned, _lcore_id) = LCORE_ID_ANY;
 RTE_DEFINE_PER_LCORE(unsigned, _socket_id) = (unsigned)SOCKET_ID_ANY;
 RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
 
+pthread_cond_t rte_eal_thread_cond[RTE_MAX_LCORE];
+pthread_mutex_t rte_eal_thread_mutex[RTE_MAX_LCORE];
+
 /*
  * Send a message to a slave lcore identified by slave_id to call a
  * function f with argument arg. Once the execution is done, the
@@ -161,6 +164,10 @@ eal_thread_loop(__attribute__((unused)) void *arg)
 			lcore_config[lcore_id].state = WAIT;
 		else
 			lcore_config[lcore_id].state = FINISHED;
+
+		pthread_mutex_lock(&rte_eal_thread_mutex[lcore_id]);
+		pthread_cond_signal(&rte_eal_thread_cond[lcore_id]);
+		pthread_mutex_unlock(&rte_eal_thread_mutex[lcore_id]);
 	}
 
 	/* never reached */
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index e968edc2e..6c636a65d 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -292,6 +292,7 @@ EXPERIMENTAL {
 	rte_devargs_remove;
 	rte_devargs_type_count;
 	rte_eal_cleanup;
+	rte_eal_wait_lcore_sleep;
 	rte_fbarray_attach;
 	rte_fbarray_destroy;
 	rte_fbarray_detach;
-- 
2.17.2

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate
  2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
@ 2018-10-15 22:21   ` Ferruh Yigit
  2023-06-12  2:38     ` Stephen Hemminger
  2018-10-16  8:42   ` [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads Ananyev, Konstantin
  2018-10-16  9:05   ` Ananyev, Konstantin
  2 siblings, 1 reply; 8+ messages in thread
From: Ferruh Yigit @ 2018-10-15 22:21 UTC (permalink / raw)
  To: Bruce Richardson, Amr Mokhtar, David Hunt, Remy Horton, Ori Kam,
	Pablo de Lara, Radu Nicolau, Akhil Goyal, Tomasz Kantecki,
	Konstantin Ananyev, Declan Doherty, Reshma Pattan, John McNamara,
	Harry van Haaren, Xiaoyun Li, Cristian Dumitrescu,
	Maxime Coquelin, Tiwei Bie, Zhihong Wang
  Cc: dev, Ferruh Yigit, stephen

This is to prevent sample applications to do busy wait while waiting for
worker threads to terminate. Should save cycles on master core.
By default a 1ms wait interval used.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 examples/bbdev_app/main.c                         | 2 +-
 examples/distributor/Makefile                     | 3 +++
 examples/distributor/main.c                       | 2 +-
 examples/distributor/meson.build                  | 1 +
 examples/ethtool/ethtool-app/Makefile             | 1 +
 examples/ethtool/ethtool-app/main.c               | 2 +-
 examples/exception_path/Makefile                  | 3 +++
 examples/exception_path/main.c                    | 2 +-
 examples/exception_path/meson.build               | 1 +
 examples/ip_fragmentation/Makefile                | 3 +++
 examples/ip_fragmentation/main.c                  | 2 +-
 examples/ip_fragmentation/meson.build             | 1 +
 examples/ip_reassembly/Makefile                   | 3 +++
 examples/ip_reassembly/main.c                     | 2 +-
 examples/ip_reassembly/meson.build                | 1 +
 examples/ipsec-secgw/ipsec-secgw.c                | 2 +-
 examples/ipv4_multicast/Makefile                  | 3 +++
 examples/ipv4_multicast/main.c                    | 2 +-
 examples/ipv4_multicast/meson.build               | 1 +
 examples/kni/Makefile                             | 3 +++
 examples/kni/main.c                               | 2 +-
 examples/kni/meson.build                          | 1 +
 examples/l2fwd-crypto/Makefile                    | 3 +++
 examples/l2fwd-crypto/main.c                      | 2 +-
 examples/l2fwd-crypto/meson.build                 | 1 +
 examples/l2fwd-jobstats/Makefile                  | 3 +++
 examples/l2fwd-jobstats/main.c                    | 2 +-
 examples/l2fwd-jobstats/meson.build               | 1 +
 examples/l2fwd-keepalive/Makefile                 | 3 +++
 examples/l2fwd-keepalive/main.c                   | 2 +-
 examples/l2fwd-keepalive/meson.build              | 1 +
 examples/l2fwd/Makefile                           | 3 +++
 examples/l2fwd/main.c                             | 2 +-
 examples/l2fwd/meson.build                        | 1 +
 examples/l3fwd-acl/Makefile                       | 3 +++
 examples/l3fwd-acl/main.c                         | 2 +-
 examples/l3fwd-acl/meson.build                    | 1 +
 examples/l3fwd-power/Makefile                     | 3 +++
 examples/l3fwd-power/main.c                       | 2 +-
 examples/l3fwd-power/meson.build                  | 1 +
 examples/l3fwd-vf/Makefile                        | 3 +++
 examples/l3fwd-vf/main.c                          | 2 +-
 examples/l3fwd-vf/meson.build                     | 1 +
 examples/l3fwd/Makefile                           | 3 +++
 examples/l3fwd/main.c                             | 2 +-
 examples/l3fwd/meson.build                        | 1 +
 examples/link_status_interrupt/Makefile           | 3 +++
 examples/link_status_interrupt/main.c             | 2 +-
 examples/link_status_interrupt/meson.build        | 1 +
 examples/load_balancer/Makefile                   | 3 +++
 examples/load_balancer/main.c                     | 2 +-
 examples/load_balancer/meson.build                | 1 +
 examples/packet_ordering/Makefile                 | 3 +++
 examples/packet_ordering/main.c                   | 2 +-
 examples/packet_ordering/meson.build              | 1 +
 examples/performance-thread/l3fwd-thread/Makefile | 1 +
 examples/performance-thread/l3fwd-thread/main.c   | 2 +-
 examples/performance-thread/pthread_shim/Makefile | 1 +
 examples/performance-thread/pthread_shim/main.c   | 2 +-
 examples/qos_meter/Makefile                       | 1 +
 examples/qos_meter/main.c                         | 2 +-
 examples/qos_meter/meson.build                    | 1 +
 examples/tep_termination/main.c                   | 2 +-
 examples/vhost/main.c                             | 2 +-
 examples/vhost_crypto/main.c                      | 2 +-
 examples/vmdq/Makefile                            | 3 +++
 examples/vmdq/main.c                              | 2 +-
 examples/vmdq/meson.build                         | 1 +
 68 files changed, 104 insertions(+), 27 deletions(-)

diff --git a/examples/bbdev_app/main.c b/examples/bbdev_app/main.c
index d68c06aef..2dc17bd60 100644
--- a/examples/bbdev_app/main.c
+++ b/examples/bbdev_app/main.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
 		stats_loop(&stats_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		ret |= rte_eal_wait_lcore(lcore_id);
+		ret |= rte_eal_wait_lcore_sleep(lcore_id);
 	}
 
 	return ret;
diff --git a/examples/distributor/Makefile b/examples/distributor/Makefile
index 05ea0bfec..5a3c5292d 100644
--- a/examples/distributor/Makefile
+++ b/examples/distributor/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -49,6 +51,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/distributor/main.c b/examples/distributor/main.c
index 03a05e3d9..53d6e6a17 100644
--- a/examples/distributor/main.c
+++ b/examples/distributor/main.c
@@ -801,7 +801,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/distributor/meson.build b/examples/distributor/meson.build
index 88c001f56..d036ea0f6 100644
--- a/examples/distributor/meson.build
+++ b/examples/distributor/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'distributor'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ethtool/ethtool-app/Makefile b/examples/ethtool/ethtool-app/Makefile
index 4cd9efdd5..b7092d267 100644
--- a/examples/ethtool/ethtool-app/Makefile
+++ b/examples/ethtool/ethtool-app/Makefile
@@ -18,6 +18,7 @@ SRCS-y := main.c ethapp.c
 
 CFLAGS += -O3 -D_GNU_SOURCE -pthread -I$(SRCDIR)/../lib
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 LDLIBS += -L$(subst ethtool-app,lib,$(RTE_OUTPUT))/lib
 LDLIBS += -lrte_ethtool
diff --git a/examples/ethtool/ethtool-app/main.c b/examples/ethtool/ethtool-app/main.c
index dc93adfe3..c3a72a739 100644
--- a/examples/ethtool/ethtool-app/main.c
+++ b/examples/ethtool/ethtool-app/main.c
@@ -276,7 +276,7 @@ int main(int argc, char **argv)
 
 	app_cfg.exit_now = 1;
 	RTE_LCORE_FOREACH_SLAVE(id_core) {
-		if (rte_eal_wait_lcore(id_core) < 0)
+		if (rte_eal_wait_lcore_sleep(id_core) < 0)
 			return -1;
 	}
 
diff --git a/examples/exception_path/Makefile b/examples/exception_path/Makefile
index ae74781ec..02e23d435 100644
--- a/examples/exception_path/Makefile
+++ b/examples/exception_path/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 
diff --git a/examples/exception_path/main.c b/examples/exception_path/main.c
index 4180a8689..515809725 100644
--- a/examples/exception_path/main.c
+++ b/examples/exception_path/main.c
@@ -581,7 +581,7 @@ main(int argc, char** argv)
 	/* Launch per-lcore function on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(i) {
-		if (rte_eal_wait_lcore(i) < 0)
+		if (rte_eal_wait_lcore_sleep(i) < 0)
 			return -1;
 	}
 
diff --git a/examples/exception_path/meson.build b/examples/exception_path/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/exception_path/meson.build
+++ b/examples/exception_path/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ip_fragmentation/Makefile b/examples/ip_fragmentation/Makefile
index 9e89e744c..bebcd4703 100644
--- a/examples/ip_fragmentation/Makefile
+++ b/examples/ip_fragmentation/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ip_fragmentation/main.c b/examples/ip_fragmentation/main.c
index 17a877da2..8d24d9891 100644
--- a/examples/ip_fragmentation/main.c
+++ b/examples/ip_fragmentation/main.c
@@ -1018,7 +1018,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/ip_fragmentation/meson.build b/examples/ip_fragmentation/meson.build
index 304203eed..9782a6a7b 100644
--- a/examples/ip_fragmentation/meson.build
+++ b/examples/ip_fragmentation/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps +=  ['ip_frag', 'lpm']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ip_reassembly/Makefile b/examples/ip_reassembly/Makefile
index 1e81315f2..7af4c2756 100644
--- a/examples/ip_reassembly/Makefile
+++ b/examples/ip_reassembly/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ip_reassembly/main.c b/examples/ip_reassembly/main.c
index 17b55d4c7..a60d0f6ea 100644
--- a/examples/ip_reassembly/main.c
+++ b/examples/ip_reassembly/main.c
@@ -1175,7 +1175,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/ip_reassembly/meson.build b/examples/ip_reassembly/meson.build
index 8ebd48291..8a667c265 100644
--- a/examples/ip_reassembly/meson.build
+++ b/examples/ip_reassembly/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['lpm', 'ip_frag']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..67fccfd16 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1837,7 +1837,7 @@ main(int32_t argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/ipv4_multicast/Makefile b/examples/ipv4_multicast/Makefile
index a16c62333..a1cefc50a 100644
--- a/examples/ipv4_multicast/Makefile
+++ b/examples/ipv4_multicast/Makefile
@@ -24,6 +24,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/ipv4_multicast/main.c b/examples/ipv4_multicast/main.c
index 6530d4830..21557121b 100644
--- a/examples/ipv4_multicast/main.c
+++ b/examples/ipv4_multicast/main.c
@@ -793,7 +793,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/ipv4_multicast/meson.build b/examples/ipv4_multicast/meson.build
index d9e4c7c21..6969e2c54 100644
--- a/examples/ipv4_multicast/meson.build
+++ b/examples/ipv4_multicast/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'hash'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/kni/Makefile b/examples/kni/Makefile
index 7e19d2e2a..adb921ce2 100644
--- a/examples/kni/Makefile
+++ b/examples/kni/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -55,6 +57,7 @@ endif
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/kni/main.c b/examples/kni/main.c
index 80c401c51..9c48a8bcf 100644
--- a/examples/kni/main.c
+++ b/examples/kni/main.c
@@ -952,7 +952,7 @@ main(int argc, char** argv)
 	/* Launch per-lcore function on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(i) {
-		if (rte_eal_wait_lcore(i) < 0)
+		if (rte_eal_wait_lcore_sleep(i) < 0)
 			return -1;
 	}
 
diff --git a/examples/kni/meson.build b/examples/kni/meson.build
index 791316394..46ac53e0d 100644
--- a/examples/kni/meson.build
+++ b/examples/kni/meson.build
@@ -9,6 +9,7 @@
 # this app can be built if-and-only-if KNI library is buildable
 build = dpdk_conf.has('RTE_LIBRTE_KNI')
 deps += ['kni', 'bus_pci']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-crypto/Makefile b/examples/l2fwd-crypto/Makefile
index 6658fd0d4..1995677df 100644
--- a/examples/l2fwd-crypto/Makefile
+++ b/examples/l2fwd-crypto/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
 ifeq ($(CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER),y)
diff --git a/examples/l2fwd-crypto/main.c b/examples/l2fwd-crypto/main.c
index f12fd266e..c63afdc34 100644
--- a/examples/l2fwd-crypto/main.c
+++ b/examples/l2fwd-crypto/main.c
@@ -2765,7 +2765,7 @@ main(int argc, char **argv)
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, (void *)&options,
 			CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-crypto/meson.build b/examples/l2fwd-crypto/meson.build
index 09438a6a0..6c852ad19 100644
--- a/examples/l2fwd-crypto/meson.build
+++ b/examples/l2fwd-crypto/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'cryptodev'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-jobstats/Makefile b/examples/l2fwd-jobstats/Makefile
index 696a8b21a..7350fa931 100644
--- a/examples/l2fwd-jobstats/Makefile
+++ b/examples/l2fwd-jobstats/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l2fwd-jobstats/main.c b/examples/l2fwd-jobstats/main.c
index a4d28e178..2fe3a15e5 100644
--- a/examples/l2fwd-jobstats/main.c
+++ b/examples/l2fwd-jobstats/main.c
@@ -995,7 +995,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-jobstats/meson.build b/examples/l2fwd-jobstats/meson.build
index 1ffd484e2..3653aa7ec 100644
--- a/examples/l2fwd-jobstats/meson.build
+++ b/examples/l2fwd-jobstats/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['jobstats', 'timer']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l2fwd-keepalive/Makefile b/examples/l2fwd-keepalive/Makefile
index 4ab67db44..4ca5685f3 100644
--- a/examples/l2fwd-keepalive/Makefile
+++ b/examples/l2fwd-keepalive/Makefile
@@ -25,6 +25,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -52,6 +54,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 LDFLAGS += -lrt
 
 include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/l2fwd-keepalive/main.c b/examples/l2fwd-keepalive/main.c
index 0bf2b5336..23b305781 100644
--- a/examples/l2fwd-keepalive/main.c
+++ b/examples/l2fwd-keepalive/main.c
@@ -788,7 +788,7 @@ main(int argc, char **argv)
 		}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l2fwd-keepalive/meson.build b/examples/l2fwd-keepalive/meson.build
index 6f7b007e1..2dffffaaa 100644
--- a/examples/l2fwd-keepalive/meson.build
+++ b/examples/l2fwd-keepalive/meson.build
@@ -8,6 +8,7 @@
 
 ext_deps += cc.find_library('rt')
 deps += 'timer'
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'shm.c'
 )
diff --git a/examples/l2fwd/Makefile b/examples/l2fwd/Makefile
index a8a47ad4e..7030b62e0 100644
--- a/examples/l2fwd/Makefile
+++ b/examples/l2fwd/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 6c23215a5..22707dac3 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -728,7 +728,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0) {
 			ret = -1;
 			break;
 		}
diff --git a/examples/l2fwd/meson.build b/examples/l2fwd/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/l2fwd/meson.build
+++ b/examples/l2fwd/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd-acl/Makefile b/examples/l3fwd-acl/Makefile
index 285683f83..de10e9d12 100644
--- a/examples/l3fwd-acl/Makefile
+++ b/examples/l3fwd-acl/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-acl/main.c b/examples/l3fwd-acl/main.c
index a322ce4f2..d1d222df0 100644
--- a/examples/l3fwd-acl/main.c
+++ b/examples/l3fwd-acl/main.c
@@ -2078,7 +2078,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-acl/meson.build b/examples/l3fwd-acl/meson.build
index 7096e00c1..68cebd6ce 100644
--- a/examples/l3fwd-acl/meson.build
+++ b/examples/l3fwd-acl/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['acl', 'lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd-power/Makefile b/examples/l3fwd-power/Makefile
index d7e39a343..7ca02935a 100644
--- a/examples/l3fwd-power/Makefile
+++ b/examples/l3fwd-power/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -56,6 +58,7 @@ else
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 68527d26f..66ba6a292 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -1908,7 +1908,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-power/meson.build b/examples/l3fwd-power/meson.build
index 20c805436..d304ab7ab 100644
--- a/examples/l3fwd-power/meson.build
+++ b/examples/l3fwd-power/meson.build
@@ -10,6 +10,7 @@ if host_machine.system() != 'linux'
 	build = false
 endif
 deps += ['power', 'timer', 'lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'perf_core.c'
 )
diff --git a/examples/l3fwd-vf/Makefile b/examples/l3fwd-vf/Makefile
index dfb1d52d3..a4eb1564d 100644
--- a/examples/l3fwd-vf/Makefile
+++ b/examples/l3fwd-vf/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3 $(USER_FLAGS)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/l3fwd-vf/main.c b/examples/l3fwd-vf/main.c
index 41137f978..ac901f24a 100644
--- a/examples/l3fwd-vf/main.c
+++ b/examples/l3fwd-vf/main.c
@@ -1088,7 +1088,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/l3fwd-vf/meson.build b/examples/l3fwd-vf/meson.build
index 226286e74..00f3c38f4 100644
--- a/examples/l3fwd-vf/meson.build
+++ b/examples/l3fwd-vf/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['lpm', 'hash']
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/l3fwd/Makefile b/examples/l3fwd/Makefile
index cccdd9dfa..4f032f577 100644
--- a/examples/l3fwd/Makefile
+++ b/examples/l3fwd/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -51,6 +53,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 CFLAGS += -I$(SRCDIR)
 CFLAGS += -O3 $(USER_FLAGS)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/l3fwd/main.c b/examples/l3fwd/main.c
index e4b99efe0..6c4faeb8e 100644
--- a/examples/l3fwd/main.c
+++ b/examples/l3fwd/main.c
@@ -1022,7 +1022,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(l3fwd_lkp.main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0) {
 			ret = -1;
 			break;
 		}
diff --git a/examples/l3fwd/meson.build b/examples/l3fwd/meson.build
index 6dd4b9022..cbef07f4f 100644
--- a/examples/l3fwd/meson.build
+++ b/examples/l3fwd/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += ['hash', 'lpm']
+allow_experimental_apis = true
 sources = files(
 	'l3fwd_em.c', 'l3fwd_lpm.c', 'main.c'
 )
diff --git a/examples/link_status_interrupt/Makefile b/examples/link_status_interrupt/Makefile
index 160682123..e6d7acfe2 100644
--- a/examples/link_status_interrupt/Makefile
+++ b/examples/link_status_interrupt/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/link_status_interrupt/main.c b/examples/link_status_interrupt/main.c
index f3346d23b..316a107dd 100644
--- a/examples/link_status_interrupt/main.c
+++ b/examples/link_status_interrupt/main.c
@@ -703,7 +703,7 @@ main(int argc, char **argv)
 	/* launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(lsi_launch_one_lcore, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/link_status_interrupt/meson.build b/examples/link_status_interrupt/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/link_status_interrupt/meson.build
+++ b/examples/link_status_interrupt/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/load_balancer/Makefile b/examples/load_balancer/Makefile
index fc8df71e8..57b602815 100644
--- a/examples/load_balancer/Makefile
+++ b/examples/load_balancer/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3 -g
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 CFLAGS_config.o := -D_GNU_SOURCE
 
 # workaround for a gcc bug with noreturn attribute
diff --git a/examples/load_balancer/main.c b/examples/load_balancer/main.c
index d3dcb235d..a1aa99eeb 100644
--- a/examples/load_balancer/main.c
+++ b/examples/load_balancer/main.c
@@ -67,7 +67,7 @@ main(int argc, char **argv)
 	/* Launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(app_lcore_main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore) {
-		if (rte_eal_wait_lcore(lcore) < 0) {
+		if (rte_eal_wait_lcore_sleep(lcore) < 0) {
 			return -1;
 		}
 	}
diff --git a/examples/load_balancer/meson.build b/examples/load_balancer/meson.build
index 4f7ac3999..19708974c 100644
--- a/examples/load_balancer/meson.build
+++ b/examples/load_balancer/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'lpm'
+allow_experimental_apis = true
 sources = files(
 	'config.c', 'init.c', 'main.c', 'runtime.c'
 )
diff --git a/examples/packet_ordering/Makefile b/examples/packet_ordering/Makefile
index 3cf1ee1dc..c608d2e1c 100644
--- a/examples/packet_ordering/Makefile
+++ b/examples/packet_ordering/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -50,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 include $(RTE_SDK)/mk/rte.extapp.mk
 endif
diff --git a/examples/packet_ordering/main.c b/examples/packet_ordering/main.c
index 149bfdd02..19588fc81 100644
--- a/examples/packet_ordering/main.c
+++ b/examples/packet_ordering/main.c
@@ -720,7 +720,7 @@ main(int argc, char **argv)
 	rx_thread(rx_to_workers);
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/packet_ordering/meson.build b/examples/packet_ordering/meson.build
index 6c2fccdcb..a3776946f 100644
--- a/examples/packet_ordering/meson.build
+++ b/examples/packet_ordering/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'reorder'
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
diff --git a/examples/performance-thread/l3fwd-thread/Makefile b/examples/performance-thread/l3fwd-thread/Makefile
index 5558043f2..46d333849 100644
--- a/examples/performance-thread/l3fwd-thread/Makefile
+++ b/examples/performance-thread/l3fwd-thread/Makefile
@@ -19,6 +19,7 @@ SRCS-y := main.c
 include $(RTE_SDK)/examples/performance-thread/common/common.mk
 
 CFLAGS += -O3 -g $(USER_FLAGS) $(INCLUDES) $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/performance-thread/l3fwd-thread/main.c b/examples/performance-thread/l3fwd-thread/main.c
index 50fd1b00a..b0a681103 100644
--- a/examples/performance-thread/l3fwd-thread/main.c
+++ b/examples/performance-thread/l3fwd-thread/main.c
@@ -3734,7 +3734,7 @@ main(int argc, char **argv)
 		/* launch per-lcore init on every lcore */
 		rte_eal_mp_remote_launch(pthread_run, NULL, CALL_MASTER);
 		RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-			if (rte_eal_wait_lcore(lcore_id) < 0)
+			if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 				return -1;
 		}
 	}
diff --git a/examples/performance-thread/pthread_shim/Makefile b/examples/performance-thread/pthread_shim/Makefile
index a6d276a49..9294e703e 100644
--- a/examples/performance-thread/pthread_shim/Makefile
+++ b/examples/performance-thread/pthread_shim/Makefile
@@ -20,6 +20,7 @@ include $(RTE_SDK)/examples/performance-thread/common/common.mk
 
 CFLAGS += -g -O3 $(USER_FLAGS) $(INCLUDES)
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 LDFLAGS += -lpthread
 
diff --git a/examples/performance-thread/pthread_shim/main.c b/examples/performance-thread/pthread_shim/main.c
index 7d0d58116..f831dd3f3 100644
--- a/examples/performance-thread/pthread_shim/main.c
+++ b/examples/performance-thread/pthread_shim/main.c
@@ -257,7 +257,7 @@ int main(int argc, char **argv)
 
 	/* wait for threads to stop */
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id);
 	}
 	return 0;
 }
diff --git a/examples/qos_meter/Makefile b/examples/qos_meter/Makefile
index 46341b1a7..7482d7e61 100644
--- a/examples/qos_meter/Makefile
+++ b/examples/qos_meter/Makefile
@@ -52,6 +52,7 @@ include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/examples/qos_meter/main.c b/examples/qos_meter/main.c
index 9b0112449..1f112812d 100644
--- a/examples/qos_meter/main.c
+++ b/examples/qos_meter/main.c
@@ -438,7 +438,7 @@ main(int argc, char **argv)
 	/* Launch per-lcore init on every lcore */
 	rte_eal_mp_remote_launch(main_loop, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/qos_meter/meson.build b/examples/qos_meter/meson.build
index ef7779f2f..10cd4bc79 100644
--- a/examples/qos_meter/meson.build
+++ b/examples/qos_meter/meson.build
@@ -7,6 +7,7 @@
 # DPDK instance, use 'make'
 
 deps += 'meter'
+allow_experimental_apis = true
 sources = files(
 	'main.c', 'rte_policer.c'
 )
diff --git a/examples/tep_termination/main.c b/examples/tep_termination/main.c
index 7795d0894..0197544c8 100644
--- a/examples/tep_termination/main.c
+++ b/examples/tep_termination/main.c
@@ -1238,7 +1238,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id)
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id);
 
 	return 0;
 }
diff --git a/examples/vhost/main.c b/examples/vhost/main.c
index dc9ea1018..2ec766560 100644
--- a/examples/vhost/main.c
+++ b/examples/vhost/main.c
@@ -1558,7 +1558,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH_SLAVE(lcore_id)
-		rte_eal_wait_lcore(lcore_id);
+		rte_eal_wait_lcore_sleep(lcore_id);
 
 	return 0;
 
diff --git a/examples/vhost_crypto/main.c b/examples/vhost_crypto/main.c
index cbb5e49d2..8f219fafa 100644
--- a/examples/vhost_crypto/main.c
+++ b/examples/vhost_crypto/main.c
@@ -582,7 +582,7 @@ main(int argc, char *argv[])
 	}
 
 	RTE_LCORE_FOREACH(lcore)
-		rte_eal_wait_lcore(lcore);
+		rte_eal_wait_lcore_sleep(lcore);
 
 	free_resource();
 
diff --git a/examples/vmdq/Makefile b/examples/vmdq/Makefile
index 87abeab93..6ff8f530d 100644
--- a/examples/vmdq/Makefile
+++ b/examples/vmdq/Makefile
@@ -23,6 +23,8 @@ CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
 LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
 LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
 
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
 build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
@@ -49,6 +51,7 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 EXTRA_CFLAGS += -O3
 
diff --git a/examples/vmdq/main.c b/examples/vmdq/main.c
index 627a5da48..e575df6b3 100644
--- a/examples/vmdq/main.c
+++ b/examples/vmdq/main.c
@@ -611,7 +611,7 @@ main(int argc, char *argv[])
 	/* call lcore_main() on every lcore */
 	rte_eal_mp_remote_launch(lcore_main, NULL, CALL_MASTER);
 	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
-		if (rte_eal_wait_lcore(lcore_id) < 0)
+		if (rte_eal_wait_lcore_sleep(lcore_id) < 0)
 			return -1;
 	}
 
diff --git a/examples/vmdq/meson.build b/examples/vmdq/meson.build
index c34e11e36..2b0a25036 100644
--- a/examples/vmdq/meson.build
+++ b/examples/vmdq/meson.build
@@ -6,6 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
+allow_experimental_apis = true
 sources = files(
 	'main.c'
 )
-- 
2.17.2

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads
  2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
  2018-10-15 22:21   ` [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
@ 2018-10-16  8:42   ` Ananyev, Konstantin
  2018-10-16  9:05   ` Ananyev, Konstantin
  2 siblings, 0 replies; 8+ messages in thread
From: Ananyev, Konstantin @ 2018-10-16  8:42 UTC (permalink / raw)
  To: Yigit, Ferruh, Richardson, Bruce; +Cc: dev, Yigit, Ferruh, stephen

HI Ferruh,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
> Sent: Monday, October 15, 2018 11:21 PM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org; Yigit, Ferruh <ferruh.yigit@intel.com>; stephen@networkplumber.org
> Subject: [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads
> 
> It is common that sample applications call rte_eal_wait_lcore() while
> waiting for worker threads to be terminated.
> Mostly master lcore keeps waiting in this function.
> 
> The waiting app for termination is not a time critical task, app can
> prefer a sleep version of the waiting to consume less cycles.
> 
> A sleeping version of the API, rte_eal_wait_lcore_sleep(), has been
> added which uses pthread conditions.
> 
> Sample applications will be updated later to use this API.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> v2:
> * use pthread cond instead of usleep
> ---
>  lib/librte_eal/bsdapp/eal/eal.c            |  3 +++
>  lib/librte_eal/bsdapp/eal/eal_thread.c     |  7 ++++++
>  lib/librte_eal/common/eal_common_launch.c  | 22 ++++++++++++++++++
>  lib/librte_eal/common/include/rte_launch.h | 26 ++++++++++++++++++++++
>  lib/librte_eal/common/include/rte_lcore.h  |  3 +++
>  lib/librte_eal/linuxapp/eal/eal.c          |  3 +++
>  lib/librte_eal/linuxapp/eal/eal_thread.c   |  7 ++++++
>  lib/librte_eal/rte_eal_version.map         |  1 +
>  8 files changed, 72 insertions(+)
> 
> diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
> index 7735194a3..e7d676657 100644
> --- a/lib/librte_eal/bsdapp/eal/eal.c
> +++ b/lib/librte_eal/bsdapp/eal/eal.c
> @@ -756,6 +756,9 @@ rte_eal_init(int argc, char **argv)
>  		snprintf(thread_name, sizeof(thread_name),
>  				"lcore-slave-%d", i);
>  		rte_thread_setname(lcore_config[i].thread_id, thread_name);
> +
> +		pthread_mutex_init(&rte_eal_thread_mutex[i], NULL);
> +		pthread_cond_init(&rte_eal_thread_cond[i], NULL);
>  	}
> 
>  	/*
> diff --git a/lib/librte_eal/bsdapp/eal/eal_thread.c b/lib/librte_eal/bsdapp/eal/eal_thread.c
> index 309b58726..60db32d57 100644
> --- a/lib/librte_eal/bsdapp/eal/eal_thread.c
> +++ b/lib/librte_eal/bsdapp/eal/eal_thread.c
> @@ -28,6 +28,9 @@ RTE_DEFINE_PER_LCORE(unsigned, _lcore_id) = LCORE_ID_ANY;
>  RTE_DEFINE_PER_LCORE(unsigned, _socket_id) = (unsigned)SOCKET_ID_ANY;
>  RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> 
> +pthread_cond_t rte_eal_thread_cond[RTE_MAX_LCORE];
> +pthread_mutex_t rte_eal_thread_mutex[RTE_MAX_LCORE];

I think would be better to include cond and mutex into struct lcore_config itself,
probably would help to avoid false sharing.
Though yeh, it would mean ABI breakage, I suppose. 

> +
>  /*
>   * Send a message to a slave lcore identified by slave_id to call a
>   * function f with argument arg. Once the execution is done, the
> @@ -154,6 +157,10 @@ eal_thread_loop(__attribute__((unused)) void *arg)
>  		lcore_config[lcore_id].ret = ret;
>  		rte_wmb();
>  		lcore_config[lcore_id].state = FINISHED;
> +
> +		pthread_mutex_lock(&rte_eal_thread_mutex[lcore_id]);
> +		pthread_cond_signal(&rte_eal_thread_cond[lcore_id]);
> +		pthread_mutex_unlock(&rte_eal_thread_mutex[lcore_id]);

I understand it would work that way too, but if you introduce mutex and cond around  
the state, then it is better to manipulate/access the state after grabbing the mutex.
BTW in that case we don't need wmb:

lcore_config[lcore_id].ret = ret;
pthread_mutex_lock(...);
lcore_config[lcore_id].state = FINISHED;
pthread_cond_signal(..);
pthread_mutex_unlock(...);

Konstantin

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads
  2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
  2018-10-15 22:21   ` [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
  2018-10-16  8:42   ` [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads Ananyev, Konstantin
@ 2018-10-16  9:05   ` Ananyev, Konstantin
  2 siblings, 0 replies; 8+ messages in thread
From: Ananyev, Konstantin @ 2018-10-16  9:05 UTC (permalink / raw)
  To: Yigit, Ferruh, Richardson, Bruce; +Cc: dev, Yigit, Ferruh, stephen

> 
> +/*
> + * Wait until a lcore finished its job by pthread condition.
> + */
> +int
> +rte_eal_wait_lcore_sleep(unsigned slave_id)
> +{
> +	if (lcore_config[slave_id].state == WAIT)
> +		return 0;
> +
> +	pthread_mutex_lock(&rte_eal_thread_mutex[slave_id]);
> +	while (lcore_config[slave_id].state != WAIT &&
> +	       lcore_config[slave_id].state != FINISHED)
> +		pthread_cond_wait(&rte_eal_thread_cond[slave_id],
> +				&rte_eal_thread_mutex[slave_id]);
> +	pthread_mutex_unlock(&rte_eal_thread_mutex[slave_id]);
> +
> +	/* we are in finished state, go to wait state */
> +	lcore_config[slave_id].state = WAIT;
> +	return lcore_config[slave_id].ret;
> +}
> +

Actually, another question - could that 2 or more threads wait for the same core simultaneously?
If yes, then 2-nd thread in that function might stuck forever.
In that case it is better to use cond_timed_wait() here and cond_broadcast() above.
Konstantin

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate
  2018-10-15 22:21   ` [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
@ 2023-06-12  2:38     ` Stephen Hemminger
  0 siblings, 0 replies; 8+ messages in thread
From: Stephen Hemminger @ 2023-06-12  2:38 UTC (permalink / raw)
  To: Ferruh Yigit
  Cc: Bruce Richardson, Amr Mokhtar, David Hunt, Remy Horton, Ori Kam,
	Pablo de Lara, Radu Nicolau, Akhil Goyal, Tomasz Kantecki,
	Konstantin Ananyev, Declan Doherty, Reshma Pattan, John McNamara,
	Harry van Haaren, Xiaoyun Li, Cristian Dumitrescu,
	Maxime Coquelin, Tiwei Bie, Zhihong Wang, dev

On Mon, 15 Oct 2018 23:21:10 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> This is to prevent sample applications to do busy wait while waiting for
> worker threads to terminate. Should save cycles on master core.
> By default a 1ms wait interval used.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>

This is not a bad idea, and it might make shutdown faster in some cases.
But it looks like it never made it farther along in 5 years.
Will mark it as rejected, but it would be good to have better lcore/thread
shutdown; if someone wants to work more on it, would be a good project.

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2023-06-12  2:38 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-11 19:57 [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Ferruh Yigit
2018-10-11 19:57 ` [dpdk-dev] [PATCH 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
2018-10-11 20:32 ` [dpdk-dev] [PATCH 1/2] eal: add API that sleeps while waiting for threads Stephen Hemminger
2018-10-15 22:21 ` [dpdk-dev] [PATCH v2 " Ferruh Yigit
2018-10-15 22:21   ` [dpdk-dev] [PATCH v2 2/2] examples: sleep while waiting worker threads to terminate Ferruh Yigit
2023-06-12  2:38     ` Stephen Hemminger
2018-10-16  8:42   ` [dpdk-dev] [PATCH v2 1/2] eal: add API that sleeps while waiting for threads Ananyev, Konstantin
2018-10-16  9:05   ` Ananyev, Konstantin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).