[dpdk-dev] [PATCH] example: add new service cores sample application

[dpdk-dev] [PATCH] example: add new service cores sample application

[Harry van Haaren]

This commit adds a new sample app, which showcases the value
of running services. In particular it allows the application
to dynamically schedule services to service-cores.

The sample app itself registers a number of dummy services,
and applies different profiles to them at runtime. Note that
this sample application does not forward any traffic - it
demonstrates advanced usage of the service cores API.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
---
 examples/service_cores/Makefile |  54 +++++++++
 examples/service_cores/main.c   | 245 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 299 insertions(+)
 create mode 100644 examples/service_cores/Makefile
 create mode 100644 examples/service_cores/main.c

diff --git a/examples/service_cores/Makefile b/examples/service_cores/Makefile
new file mode 100644
index 0000000..bd4a345
--- /dev/null
+++ b/examples/service_cores/Makefile
@@ -0,0 +1,54 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = service_cores
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/service_cores/main.c b/examples/service_cores/main.c
new file mode 100644
index 0000000..c1f125a
--- /dev/null
+++ b/examples/service_cores/main.c
@@ -0,0 +1,245 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+#include <rte_cycles.h>
+
+/* allow application scheduling of the services */
+#include <rte_service.h>
+
+/* Allow application registration of its own services. An application does not
+ * have to register services, but it can be useful if it wishes to run a
+ * function on a core that is otherwise in use as a service core. In this
+ * example, all services are dummy services registered by the sample app itself.
+ */
+#include <rte_service_component.h>
+
+#define PROFILE_CORES_MAX 8
+#define PROFILE_SERVICE_PER_CORE 8
+
+/* dummy function to do "work" */
+static int32_t service_func(void *args)
+{
+	RTE_SET_USED(args);
+	rte_delay_us(2000);
+	return 0;
+}
+
+static struct rte_service_spec services[] = {
+	{"service_1", service_func, NULL, 0, 0},
+	{"service_2", service_func, NULL, 0, 0},
+	{"service_3", service_func, NULL, 0, 0},
+	{"service_4", service_func, NULL, 0, 0},
+	{"service_5", service_func, NULL, 0, 0},
+};
+#define NUM_SERVICES RTE_DIM(services)
+
+/* this struct holds the mapping of a particular core to all services */
+struct profile_for_core {
+	uint32_t mapped_services[PROFILE_SERVICE_PER_CORE];
+};
+
+/* struct that can be applied as the service core mapping. Items in this
+ * struct will be passed to the ordinary rte_service_* APIs to configure the
+ * service cores at runtime, based on the requirements.
+ *
+ * These profiles can be considered a "configuration" for the service cores,
+ * where switching profile just changes the number of cores and the mappings
+ * for each of them. As a result, the core requirements and performance of the
+ * application scales.
+ */
+struct profile {
+	char name[64];
+	uint32_t num_cores;
+	struct profile_for_core cores[PROFILE_CORES_MAX];
+};
+
+static struct profile profiles[] = {
+	/* profile 0: high performance */
+	{
+		.name = "High Performance",
+		.num_cores = 5,
+		.cores[0] = {.mapped_services = {1, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {0, 1, 0, 0, 0, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 1, 0, 0, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 1, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 1, 0, 0, 0} },
+	},
+	/* profile 1: mid performance with single service priority */
+	{
+		.name = "Mid-High Performance",
+		.num_cores = 3,
+		.cores[0] = {.mapped_services = {1, 1, 0, 0, 0, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {0, 0, 1, 1, 0, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 1, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+	},
+	/* profile 2: mid performance with single service priority */
+	{
+		.name = "Mid-Low Performance",
+		.num_cores = 2,
+		.cores[0] = {.mapped_services = {1, 1, 1, 0, 0, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {1, 1, 0, 1, 1, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+	},
+	/* profile 3: scale down performance on single core */
+	{
+		.name = "Scale down performance",
+		.num_cores = 1,
+		.cores[0] = {.mapped_services = {1, 1, 1, 1, 1, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0, 0, 0, 0} },
+	},
+};
+#define NUM_PROFILES RTE_DIM(profiles)
+
+static int
+apply_profile(int profile_id)
+{
+	uint32_t i;
+	uint32_t s;
+	int ret;
+	struct profile *p = &profiles[profile_id];
+	const uint8_t core_off = 1;
+
+	for (i = 0; i < p->num_cores; i++) {
+		ret = rte_service_lcore_add(i + core_off);
+		if (ret && ret != -EALREADY)
+			printf("core %d added ret %d\n", i + core_off, ret);
+
+		ret = rte_service_lcore_start(i + core_off);
+		if (ret && ret != -EALREADY)
+			printf("core %d start ret %d\n", i + core_off, ret);
+
+		for (s = 0; s < NUM_SERVICES; s++) {
+			if (rte_service_map_lcore_set(s, i + core_off,
+					p->cores[i].mapped_services[s]))
+				rte_panic("failed to map lcore to 1\n");
+		}
+	}
+
+	for ( ; i < PROFILE_CORES_MAX; i++) {
+		for (s = 0; s < NUM_SERVICES; s++) {
+			ret = rte_service_map_lcore_set(s, i + core_off, 0);
+			if (ret && ret != -EINVAL) {
+				printf("%s %d: map lcore set = %d\n", __func__,
+						__LINE__, ret);
+			}
+		}
+		ret = rte_service_lcore_stop(i + core_off);
+		if (ret && ret != -EALREADY) {
+			printf("%s %d: lcore stop = %d\n", __func__,
+					__LINE__, ret);
+		}
+		ret = rte_service_lcore_del(i + core_off);
+		if (ret && ret != -EINVAL) {
+			printf("%s %d: lcore del = %d\n", __func__,
+					__LINE__, ret);
+		}
+	}
+}
+
+int
+main(int argc, char **argv)
+{
+	int ret;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	uint32_t i;
+	for (i = 0; i < NUM_SERVICES; i++) {
+		services[i].callback_userdata = 0;
+		uint32_t id;
+		ret = rte_service_component_register(&services[i], &id);
+		if (ret)
+			rte_exit(-1, "service register() failed");
+
+		/* set the service itself to be ready to run. In the case of
+		 * ethdev, eventdev etc PMDs, this will be set when the
+		 * appropriate configure or setup function is called.
+		 */
+		rte_service_component_runstate_set(id, 1);
+
+		/* Collect statistics for the service */
+		rte_service_set_stats_enable(id, 1);
+
+		/* the application sets the service to be active. Note that the
+		 * previous component_runstate_set() is the PMD indicating
+		 * ready, while this function is the application setting the
+		 * service to run. Applications can choose to not run a service
+		 * by setting runstate to 0 at any time.
+		 */
+		ret = rte_service_runstate_set(id, 1);
+		if (ret)
+			return -ENOEXEC;
+	}
+
+	i = 0;
+	while (1) {
+		const char clr[] = { 27, '[', '2', 'J', '\0' };
+		const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' };
+		printf("%s%s", clr, topLeft);
+
+		apply_profile(i);
+		printf("\n==> Profile: %s\n\n", profiles[i].name);
+
+		sleep(1);
+		rte_service_dump(stdout, UINT32_MAX);
+
+		sleep(5);
+		rte_service_dump(stdout, UINT32_MAX);
+
+		i++;
+		if (i >= NUM_PROFILES)
+			i = 0;
+	}
+
+	return 0;
+}
-- 
2.7.4

Re: [dpdk-dev] [PATCH] example: add new service cores sample application

[Thomas Monjalon]

25/09/2017 15:23, Harry van Haaren:
> This commit adds a new sample app, which showcases the value
> of running services. In particular it allows the application
> to dynamically schedule services to service-cores.
> 
> The sample app itself registers a number of dummy services,
> and applies different profiles to them at runtime. Note that
> this sample application does not forward any traffic - it
> demonstrates advanced usage of the service cores API.
> 
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> ---
>  examples/service_cores/Makefile |  54 +++++++++
>  examples/service_cores/main.c   | 245 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 299 insertions(+)

Please add this example in the global examples Makefile.

Re: [dpdk-dev] [PATCH] example: add new service cores sample application

[Thomas Monjalon]

14/10/2017 00:20, Thomas Monjalon:
> 25/09/2017 15:23, Harry van Haaren:
> > This commit adds a new sample app, which showcases the value
> > of running services. In particular it allows the application
> > to dynamically schedule services to service-cores.
> > 
> > The sample app itself registers a number of dummy services,
> > and applies different profiles to them at runtime. Note that
> > this sample application does not forward any traffic - it
> > demonstrates advanced usage of the service cores API.
> > 
> > Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> > ---
> >  examples/service_cores/Makefile |  54 +++++++++
> >  examples/service_cores/main.c   | 245 ++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 299 insertions(+)
> 
> Please add this example in the global examples Makefile.

And you need to add it also in MAINTAINERS.

I guess an entry in the doc is maybe required, too?

Re: [dpdk-dev] [PATCH] example: add new service cores sample application

[Van Haaren, Harry]

> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Friday, October 13, 2017 11:24 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org; Mcnamara, John <john.mcnamara@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] example: add new service cores sample
> application
> 
> 14/10/2017 00:20, Thomas Monjalon:
> > 25/09/2017 15:23, Harry van Haaren:
> > > This commit adds a new sample app, which showcases the value
> > > of running services. In particular it allows the application
> > > to dynamically schedule services to service-cores.
> > >
> > > The sample app itself registers a number of dummy services,
> > > and applies different profiles to them at runtime. Note that
> > > this sample application does not forward any traffic - it
> > > demonstrates advanced usage of the service cores API.
> > >
> > > Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> > > ---
> > >  examples/service_cores/Makefile |  54 +++++++++
> > >  examples/service_cores/main.c   | 245
> ++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 299 insertions(+)
> >
> > Please add this example in the global examples Makefile.
> 
> And you need to add it also in MAINTAINERS.
> 
> I guess an entry in the doc is maybe required, too?

Yes, yes, and yes. Will all be fixed in v2! -Harry

[dpdk-dev] [PATCH v2] example: add new service cores sample application

[Harry van Haaren]

This commit adds a new sample app, which showcases the value
of running services. In particular it allows the application
to dynamically schedule services to service-cores.

The sample app itself registers a number of dummy services,
and applies different profiles to them at runtime. Note that
this sample application does not forward any traffic - it
demonstrates advanced usage of the service cores API.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

---

v2 feedback http://dpdk.org/dev/patchwork/patch/29169/
- Add to examples Makefile (Thomas)
- Add new files to MAINTAINERS (Thomas)
- Added documentation to sample app user guides (Thomas)
- Reduce service-per-core define to 5, removes extra zeros (Gage)
- Improve error handling if service mapping fails (Gage)


---
 MAINTAINERS                                |   4 +
 doc/guides/sample_app_ug/index.rst         |   1 +
 doc/guides/sample_app_ug/service_cores.rst | 172 ++++++++++++++++++++
 examples/Makefile                          |   1 +
 examples/service_cores/Makefile            |  54 +++++++
 examples/service_cores/main.c              | 247 +++++++++++++++++++++++++++++
 6 files changed, 479 insertions(+)
 create mode 100644 doc/guides/sample_app_ug/service_cores.rst
 create mode 100644 examples/service_cores/Makefile
 create mode 100644 examples/service_cores/main.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 8df2a7f..44a8c38 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -903,6 +903,10 @@ M: John McNamara <john.mcnamara@intel.com>
 F: examples/rxtx_callbacks/
 F: doc/guides/sample_app_ug/rxtx_callbacks.rst
 
+M: Harry van Haaren <harry.van.haaren@intel.com>
+F: examples/service_cores/
+F: doc/guides/sample_app_ug/service_cores.rst
+
 M: Bruce Richardson <bruce.richardson@intel.com>
 M: John McNamara <john.mcnamara@intel.com>
 F: examples/skeleton/
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 069d4f1..0c17ebb 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -58,6 +58,7 @@ Sample Applications User Guides
     link_status_intr
     load_balancer
     server_node_efd
+    service_cores
     multi_process
     qos_metering
     qos_scheduler
diff --git a/doc/guides/sample_app_ug/service_cores.rst b/doc/guides/sample_app_ug/service_cores.rst
new file mode 100644
index 0000000..07e3929
--- /dev/null
+++ b/doc/guides/sample_app_ug/service_cores.rst
@@ -0,0 +1,172 @@
+..  BSD LICENSE
+    Copyright(c) 2017 Intel Corporation. All rights reserved.
+    All rights reserved.
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions
+    are met:
+
+    * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+    * Neither the name of Intel Corporation nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Service Cores Sample Application
+================================
+
+The service cores sample application demonstrates the service cores capabilities
+of DPDK. The service cores infrastructure is part of the DPDK EAL, and allows
+any DPDK component to register a service. A service is a work item or task, that
+requires CPU time to perform its duty.
+
+This sample application registers 5 dummy services. These 5 services are used
+to show how the service_cores API can be used to orchestrate these services to
+run on different service lcores. This orchestration is done by calling the
+service cores APIs, however the sample application introduces a "profile"
+concept to contain the service mapping details. Note that the profile concept
+is application specific, and not a part of the service cores API.
+
+
+Compiling the Application
+-------------------------
+
+#.  Go to the example directory:
+
+    .. code-block:: console
+
+        export RTE_SDK=/path/to/rte_sdk
+        cd ${RTE_SDK}/examples/service_cores
+
+#.  Set the target (a default target is used if not specified). For example:
+
+    .. code-block:: console
+
+        export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+    See the *DPDK Getting Started* Guide for possible RTE_TARGET values.
+
+#.  Build the application:
+
+    .. code-block:: console
+
+        make
+
+Running the Application
+-----------------------
+
+To run the example, just execute the binary. Since the application dynamically
+adds service cores in the application code itself, there is no requirement to
+pass a service core-mask as an EAL argument at startup time.
+
+.. code-block:: console
+
+    $ ./build/service_cores
+
+
+Explanation
+-----------
+
+The following sections provide some explanation of code focusing on
+registering applications from an applications point of view, and modifying the
+service core counts and mappings at runtime.
+
+
+Registering a Service
+~~~~~~~~~~~~~~~~~~~~~
+
+The following code section shows how to register a service as an application.
+Note that the service component header must be included by the application in
+order to register services: ``rte_service_component.h``, in addition
+to the ordinary service cores header ``rte_service.h`` which provides
+the runtime functions to add, remove and remap service cores.
+
+.. code-block:: c
+
+        struct rte_service_spec service = {
+                .name = "service_name",
+        };
+        int ret = rte_service_component_register(services, &id);
+        if (ret)
+                return -1;
+
+        /* set the service itself to be ready to run. In the case of
+        * ethdev, eventdev etc PMDs, this will be set when the
+        * appropriate configure or setup function is called.
+        */
+        rte_service_component_runstate_set(id, 1);
+
+        /* Collect statistics for the service */
+        rte_service_set_stats_enable(id, 1);
+
+        /* The application sets the service to running state. Note that this
+         * function enables the service to run - while the 'component' version
+         * of this function (as above) marks the service itself as ready */
+        ret = rte_service_runstate_set(id, 1);
+
+
+Controlling A Service Core
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This section demonstrates how to add a service core. The ``rte_service.h``
+header file provides the functions for dynamically adding and removing cores.
+The APIs to add and remove cores use lcore IDs similar to existing DPDK
+functions.
+
+These are the functions to start a service core, and have it run a service:
+
+.. code-block:: c
+
+        /* the lcore ID to use as a service core */
+        uint32_t service_core_id = 7;
+        ret = rte_service_lcore_add(service_core_id);
+        if(ret)
+                return -1;
+
+        /* service cores are in "stopped" state when added, so start it */
+        ret = rte_service_lcore_start(service_core_id);
+        if(ret)
+                return -1;
+
+        /* map a service to the service core, causing it to run the service */
+        uint32_t service_id; /* ID of a registered service */
+        uint32_t enable = 1; /* 1 maps the service, 0 unmaps */
+        ret = rte_service_map_lcore_set(service_id, service_core_id, enable);
+        if(ret)
+                return -1;
+
+
+Removing A Service Core
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To remove a service core, the steps are similar to adding but in reverse order.
+Note that it is not allowed to remove a service core if the service is running,
+and the service-core is the only core running that service (see documentation
+for ``rte_service_lcore_stop`` function for details).
+
+
+Conclusion
+~~~~~~~~~~
+
+The service cores infrastructure provides DPDK with two main features. The first
+is to abstract away hardware differences: the service core can CPU cycles to
+a software fallback implementation, allowing the application to be abstracted
+from the difference in HW / SW availability. The second feature is a flexible
+method of registering functions to be run, allowing the running of the
+functions to be scaled across multiple CPUs. 
diff --git a/examples/Makefile b/examples/Makefile
index 28354ff..266451f 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -83,6 +83,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_METER) += qos_meter
 DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched
 DIRS-y += quota_watermark
 DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks
+DIRS-y += service_cores
 DIRS-y += skeleton
 ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
diff --git a/examples/service_cores/Makefile b/examples/service_cores/Makefile
new file mode 100644
index 0000000..bd4a345
--- /dev/null
+++ b/examples/service_cores/Makefile
@@ -0,0 +1,54 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = service_cores
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/service_cores/main.c b/examples/service_cores/main.c
new file mode 100644
index 0000000..8a8f5bd
--- /dev/null
+++ b/examples/service_cores/main.c
@@ -0,0 +1,247 @@
+/*
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <sys/queue.h>
+
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+#include <rte_cycles.h>
+
+/* allow application scheduling of the services */
+#include <rte_service.h>
+
+/* Allow application registration of its own services. An application does not
+ * have to register services, but it can be useful if it wishes to run a
+ * function on a core that is otherwise in use as a service core. In this
+ * example, all services are dummy services registered by the sample app itself.
+ */
+#include <rte_service_component.h>
+
+#define PROFILE_CORES_MAX 8
+#define PROFILE_SERVICE_PER_CORE 5
+
+/* dummy function to do "work" */
+static int32_t service_func(void *args)
+{
+	RTE_SET_USED(args);
+	rte_delay_us(2000);
+	return 0;
+}
+
+static struct rte_service_spec services[] = {
+	{"service_1", service_func, NULL, 0, 0},
+	{"service_2", service_func, NULL, 0, 0},
+	{"service_3", service_func, NULL, 0, 0},
+	{"service_4", service_func, NULL, 0, 0},
+	{"service_5", service_func, NULL, 0, 0},
+};
+#define NUM_SERVICES RTE_DIM(services)
+
+/* this struct holds the mapping of a particular core to all services */
+struct profile_for_core {
+	uint32_t mapped_services[PROFILE_SERVICE_PER_CORE];
+};
+
+/* struct that can be applied as the service core mapping. Items in this
+ * struct will be passed to the ordinary rte_service_* APIs to configure the
+ * service cores at runtime, based on the requirements.
+ *
+ * These profiles can be considered a "configuration" for the service cores,
+ * where switching profile just changes the number of cores and the mappings
+ * for each of them. As a result, the core requirements and performance of the
+ * application scales.
+ */
+struct profile {
+	char name[64];
+	uint32_t num_cores;
+	struct profile_for_core cores[PROFILE_CORES_MAX];
+};
+
+static struct profile profiles[] = {
+	/* profile 0: high performance */
+	{
+		.name = "High Performance",
+		.num_cores = 5,
+		.cores[0] = {.mapped_services = {1, 0, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {0, 1, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 1, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 1, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 1} },
+	},
+	/* profile 1: mid performance with single service priority */
+	{
+		.name = "Mid-High Performance",
+		.num_cores = 3,
+		.cores[0] = {.mapped_services = {1, 1, 0, 0, 0} },
+		.cores[1] = {.mapped_services = {0, 0, 1, 1, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 1} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0} },
+	},
+	/* profile 2: mid performance with single service priority */
+	{
+		.name = "Mid-Low Performance",
+		.num_cores = 2,
+		.cores[0] = {.mapped_services = {1, 1, 1, 0, 0} },
+		.cores[1] = {.mapped_services = {1, 1, 0, 1, 1} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0} },
+	},
+	/* profile 3: scale down performance on single core */
+	{
+		.name = "Scale down performance",
+		.num_cores = 1,
+		.cores[0] = {.mapped_services = {1, 1, 1, 1, 1} },
+		.cores[1] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[2] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[3] = {.mapped_services = {0, 0, 0, 0, 0} },
+		.cores[4] = {.mapped_services = {0, 0, 0, 0, 0} },
+	},
+};
+#define NUM_PROFILES RTE_DIM(profiles)
+
+static int
+apply_profile(int profile_id)
+{
+	uint32_t i;
+	uint32_t s;
+	int ret;
+	struct profile *p = &profiles[profile_id];
+	const uint8_t core_off = 1;
+
+	for (i = 0; i < p->num_cores; i++) {
+		uint32_t core = i + core_off;
+		ret = rte_service_lcore_add(core);
+		if (ret && ret != -EALREADY)
+			printf("core %d added ret %d\n", core, ret);
+
+		ret = rte_service_lcore_start(core);
+		if (ret && ret != -EALREADY)
+			printf("core %d start ret %d\n", core, ret);
+
+		for (s = 0; s < NUM_SERVICES; s++) {
+			if (rte_service_map_lcore_set(s, core,
+					p->cores[i].mapped_services[s]))
+				printf("failed to map lcore %d\n", core);
+		}
+	}
+
+	for ( ; i < PROFILE_CORES_MAX; i++) {
+		uint32_t core = i + core_off;
+		for (s = 0; s < NUM_SERVICES; s++) {
+			ret = rte_service_map_lcore_set(s, core, 0);
+			if (ret && ret != -EINVAL) {
+				printf("%s %d: map lcore set = %d\n", __func__,
+						__LINE__, ret);
+			}
+		}
+		ret = rte_service_lcore_stop(core);
+		if (ret && ret != -EALREADY) {
+			printf("%s %d: lcore stop = %d\n", __func__,
+					__LINE__, ret);
+		}
+		ret = rte_service_lcore_del(core);
+		if (ret && ret != -EINVAL) {
+			printf("%s %d: lcore del = %d\n", __func__,
+					__LINE__, ret);
+		}
+	}
+}
+
+int
+main(int argc, char **argv)
+{
+	int ret;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	uint32_t i;
+	for (i = 0; i < NUM_SERVICES; i++) {
+		services[i].callback_userdata = 0;
+		uint32_t id;
+		ret = rte_service_component_register(&services[i], &id);
+		if (ret)
+			rte_exit(-1, "service register() failed");
+
+		/* set the service itself to be ready to run. In the case of
+		 * ethdev, eventdev etc PMDs, this will be set when the
+		 * appropriate configure or setup function is called.
+		 */
+		rte_service_component_runstate_set(id, 1);
+
+		/* Collect statistics for the service */
+		rte_service_set_stats_enable(id, 1);
+
+		/* the application sets the service to be active. Note that the
+		 * previous component_runstate_set() is the PMD indicating
+		 * ready, while this function is the application setting the
+		 * service to run. Applications can choose to not run a service
+		 * by setting runstate to 0 at any time.
+		 */
+		ret = rte_service_runstate_set(id, 1);
+		if (ret)
+			return -ENOEXEC;
+	}
+
+	i = 0;
+	while (1) {
+		const char clr[] = { 27, '[', '2', 'J', '\0' };
+		const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' };
+		printf("%s%s", clr, topLeft);
+
+		apply_profile(i);
+		printf("\n==> Profile: %s\n\n", profiles[i].name);
+
+		sleep(1);
+		rte_service_dump(stdout, UINT32_MAX);
+
+		sleep(5);
+		rte_service_dump(stdout, UINT32_MAX);
+
+		i++;
+		if (i >= NUM_PROFILES)
+			i = 0;
+	}
+
+	return 0;
+}
-- 
2.7.4

Re: [dpdk-dev] [PATCH v2] example: add new service cores sample application

[Eads, Gage]

> -----Original Message-----
> From: Van Haaren, Harry
> Sent: Thursday, October 19, 2017 5:31 AM
> To: dev@dpdk.org
> Cc: thomas@monjalon.net; Mcnamara, John <john.mcnamara@intel.com>;
> Eads, Gage <gage.eads@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: [PATCH v2] example: add new service cores sample application
> 
> This commit adds a new sample app, which showcases the value of running
> services. In particular it allows the application to dynamically schedule services
> to service-cores.
> 
> The sample app itself registers a number of dummy services, and applies
> different profiles to them at runtime. Note that this sample application does not
> forward any traffic - it demonstrates advanced usage of the service cores API.
> 
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
> 

Acked-by: Gage Eads <gage.eads@intel.com>

Re: [dpdk-dev] [PATCH v2] example: add new service cores sample application

[Thomas Monjalon]

19/10/2017 12:31, Harry van Haaren:
> This commit adds a new sample app, which showcases the value
> of running services. In particular it allows the application
> to dynamically schedule services to service-cores.
> 
> The sample app itself registers a number of dummy services,
> and applies different profiles to them at runtime. Note that
> this sample application does not forward any traffic - it
> demonstrates advanced usage of the service cores API.
> 
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Compilation error with clang:
	examples/service_cores/main.c:187:1: error:
		control reaches end of non-void function
Fixed by declaring function return as void.

Applied, thanks

Re: [dpdk-dev] [PATCH] example: add new service cores sample application

[Van Haaren, Harry]

> From: Eads, Gage 
> Sent: Monday, September 25, 2017 5:32 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] example: add new service cores sample application
> 
> Neat example. Looks good overall, I just have a few questions.
> 
> > +#define PROFILE_SERVICE_PER_CORE 8
> 
> Any reason not to use 5 here? That would remove a few zeroes from the profiles[] initializers.

Nope - good suggestion, Fixed in v2.


> > +static int
> > +apply_profile(int profile_id)
> > +{
> > +          uint32_t i;
> > +          uint32_t s;
> > +          int ret;
> > +          struct profile *p = &profiles[profile_id];
> > +          const uint8_t core_off = 1;
> > +
> > +          for (i = 0; i < p->num_cores; i++) {
> > +                         ret = rte_service_lcore_add(i + core_off);
> > +                         if (ret && ret != -EALREADY)
> > +                                        printf("core %d added ret %d\n", i + core_off, ret);
> 
> I'm wondering if this and the other printfs in this function should be rte_panics? These seem like fatal errors.

They're not fatal, they shouldn't (and don't) occur in the sample app. The error handling is there to be a good example - panic() doesn't seem a good suggestion to handle and error.. I'll leave as is.

> 
> > +
> > +                         ret = rte_service_lcore_start(i + core_off);
> > +                         if (ret && ret != -EALREADY)
> > +                                        printf("core %d start ret %d\n", i + core_off, ret);
> > +
> > +                         for (s = 0; s < NUM_SERVICES; s++) {
> > +                                        if (rte_service_map_lcore_set(s, i + core_off,
> > +                                                                      p->cores[i].mapped_services[s]))
> > +                                                       rte_panic("failed to map lcore to 1\n");
> 
> What does '1' refer to here? Perhaps it should be: rte_panic("failed to map lcore %d to %s\n", i + core_off, services[s].name);

Good catch, fixed in v2.

Re: [dpdk-dev] [PATCH] example: add new service cores sample application

[Eads, Gage]

Neat example. Looks good overall, I just have a few questions.

> +#define PROFILE_SERVICE_PER_CORE 8

Any reason not to use 5 here? That would remove a few zeroes from the profiles[] initializers.

> +static int
> +apply_profile(int profile_id)
> +{
> +          uint32_t i;
> +          uint32_t s;
> +          int ret;
> +          struct profile *p = &profiles[profile_id];
> +          const uint8_t core_off = 1;
> +
> +          for (i = 0; i < p->num_cores; i++) {
> +                         ret = rte_service_lcore_add(i + core_off);
> +                         if (ret && ret != -EALREADY)
> +                                        printf("core %d added ret %d\n", i + core_off, ret);

I'm wondering if this and the other printfs in this function should be rte_panics? These seem like fatal errors.

> +
> +                         ret = rte_service_lcore_start(i + core_off);
> +                         if (ret && ret != -EALREADY)
> +                                        printf("core %d start ret %d\n", i + core_off, ret);
> +
> +                         for (s = 0; s < NUM_SERVICES; s++) {
> +                                        if (rte_service_map_lcore_set(s, i + core_off,
> +                                                                      p->cores[i].mapped_services[s]))
> +                                                       rte_panic("failed to map lcore to 1\n");

What does '1' refer to here? Perhaps it should be: rte_panic("failed to map lcore %d to %s\n", i + core_off, services[s].name);