From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id E74DD101B for ; Thu, 19 Oct 2017 12:30:39 +0200 (CEST) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Oct 2017 03:30:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.43,401,1503385200"; d="scan'208";a="325094547" Received: from silpixa00398672.ir.intel.com ([10.237.223.128]) by fmsmga004.fm.intel.com with ESMTP; 19 Oct 2017 03:30:36 -0700 From: Harry van Haaren To: dev@dpdk.org Cc: thomas@monjalon.net, john.mcnamara@intel.com, gage.eads@intel.com, Harry van Haaren Date: Thu, 19 Oct 2017 11:31:11 +0100 Message-Id: <1508409071-17740-1-git-send-email-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1506345834-137009-1-git-send-email-harry.van.haaren@intel.com> References: <1506345834-137009-1-git-send-email-harry.van.haaren@intel.com> Subject: [dpdk-dev] [PATCH v2] example: add new service cores sample application X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Oct 2017 10:30:40 -0000 This commit adds a new sample app, which showcases the value of running services. In particular it allows the application to dynamically schedule services to service-cores. The sample app itself registers a number of dummy services, and applies different profiles to them at runtime. Note that this sample application does not forward any traffic - it demonstrates advanced usage of the service cores API. Signed-off-by: Harry van Haaren --- v2 feedback http://dpdk.org/dev/patchwork/patch/29169/ - Add to examples Makefile (Thomas) - Add new files to MAINTAINERS (Thomas) - Added documentation to sample app user guides (Thomas) - Reduce service-per-core define to 5, removes extra zeros (Gage) - Improve error handling if service mapping fails (Gage) --- MAINTAINERS | 4 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/service_cores.rst | 172 ++++++++++++++++++++ examples/Makefile | 1 + examples/service_cores/Makefile | 54 +++++++ examples/service_cores/main.c | 247 +++++++++++++++++++++++++++++ 6 files changed, 479 insertions(+) create mode 100644 doc/guides/sample_app_ug/service_cores.rst create mode 100644 examples/service_cores/Makefile create mode 100644 examples/service_cores/main.c diff --git a/MAINTAINERS b/MAINTAINERS index 8df2a7f..44a8c38 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -903,6 +903,10 @@ M: John McNamara F: examples/rxtx_callbacks/ F: doc/guides/sample_app_ug/rxtx_callbacks.rst +M: Harry van Haaren +F: examples/service_cores/ +F: doc/guides/sample_app_ug/service_cores.rst + M: Bruce Richardson M: John McNamara F: examples/skeleton/ diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index 069d4f1..0c17ebb 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -58,6 +58,7 @@ Sample Applications User Guides link_status_intr load_balancer server_node_efd + service_cores multi_process qos_metering qos_scheduler diff --git a/doc/guides/sample_app_ug/service_cores.rst b/doc/guides/sample_app_ug/service_cores.rst new file mode 100644 index 0000000..07e3929 --- /dev/null +++ b/doc/guides/sample_app_ug/service_cores.rst @@ -0,0 +1,172 @@ +.. BSD LICENSE + Copyright(c) 2017 Intel Corporation. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Service Cores Sample Application +================================ + +The service cores sample application demonstrates the service cores capabilities +of DPDK. The service cores infrastructure is part of the DPDK EAL, and allows +any DPDK component to register a service. A service is a work item or task, that +requires CPU time to perform its duty. + +This sample application registers 5 dummy services. These 5 services are used +to show how the service_cores API can be used to orchestrate these services to +run on different service lcores. This orchestration is done by calling the +service cores APIs, however the sample application introduces a "profile" +concept to contain the service mapping details. Note that the profile concept +is application specific, and not a part of the service cores API. + + +Compiling the Application +------------------------- + +#. Go to the example directory: + + .. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + cd ${RTE_SDK}/examples/service_cores + +#. Set the target (a default target is used if not specified). For example: + + .. code-block:: console + + export RTE_TARGET=x86_64-native-linuxapp-gcc + + See the *DPDK Getting Started* Guide for possible RTE_TARGET values. + +#. Build the application: + + .. code-block:: console + + make + +Running the Application +----------------------- + +To run the example, just execute the binary. Since the application dynamically +adds service cores in the application code itself, there is no requirement to +pass a service core-mask as an EAL argument at startup time. + +.. code-block:: console + + $ ./build/service_cores + + +Explanation +----------- + +The following sections provide some explanation of code focusing on +registering applications from an applications point of view, and modifying the +service core counts and mappings at runtime. + + +Registering a Service +~~~~~~~~~~~~~~~~~~~~~ + +The following code section shows how to register a service as an application. +Note that the service component header must be included by the application in +order to register services: ``rte_service_component.h``, in addition +to the ordinary service cores header ``rte_service.h`` which provides +the runtime functions to add, remove and remap service cores. + +.. code-block:: c + + struct rte_service_spec service = { + .name = "service_name", + }; + int ret = rte_service_component_register(services, &id); + if (ret) + return -1; + + /* set the service itself to be ready to run. In the case of + * ethdev, eventdev etc PMDs, this will be set when the + * appropriate configure or setup function is called. + */ + rte_service_component_runstate_set(id, 1); + + /* Collect statistics for the service */ + rte_service_set_stats_enable(id, 1); + + /* The application sets the service to running state. Note that this + * function enables the service to run - while the 'component' version + * of this function (as above) marks the service itself as ready */ + ret = rte_service_runstate_set(id, 1); + + +Controlling A Service Core +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section demonstrates how to add a service core. The ``rte_service.h`` +header file provides the functions for dynamically adding and removing cores. +The APIs to add and remove cores use lcore IDs similar to existing DPDK +functions. + +These are the functions to start a service core, and have it run a service: + +.. code-block:: c + + /* the lcore ID to use as a service core */ + uint32_t service_core_id = 7; + ret = rte_service_lcore_add(service_core_id); + if(ret) + return -1; + + /* service cores are in "stopped" state when added, so start it */ + ret = rte_service_lcore_start(service_core_id); + if(ret) + return -1; + + /* map a service to the service core, causing it to run the service */ + uint32_t service_id; /* ID of a registered service */ + uint32_t enable = 1; /* 1 maps the service, 0 unmaps */ + ret = rte_service_map_lcore_set(service_id, service_core_id, enable); + if(ret) + return -1; + + +Removing A Service Core +~~~~~~~~~~~~~~~~~~~~~~~ + +To remove a service core, the steps are similar to adding but in reverse order. +Note that it is not allowed to remove a service core if the service is running, +and the service-core is the only core running that service (see documentation +for ``rte_service_lcore_stop`` function for details). + + +Conclusion +~~~~~~~~~~ + +The service cores infrastructure provides DPDK with two main features. The first +is to abstract away hardware differences: the service core can CPU cycles to +a software fallback implementation, allowing the application to be abstracted +from the difference in HW / SW availability. The second feature is a flexible +method of registering functions to be run, allowing the running of the +functions to be scaled across multiple CPUs. diff --git a/examples/Makefile b/examples/Makefile index 28354ff..266451f 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -83,6 +83,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_METER) += qos_meter DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched DIRS-y += quota_watermark DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks +DIRS-y += service_cores DIRS-y += skeleton ifeq ($(CONFIG_RTE_LIBRTE_HASH),y) DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination diff --git a/examples/service_cores/Makefile b/examples/service_cores/Makefile new file mode 100644 index 0000000..bd4a345 --- /dev/null +++ b/examples/service_cores/Makefile @@ -0,0 +1,54 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +# binary name +APP = service_cores + +# all source are stored in SRCS-y +SRCS-y := main.c + +CFLAGS += $(WERROR_FLAGS) + +# workaround for a gcc bug with noreturn attribute +# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603 +ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y) +CFLAGS_main.o += -Wno-return-type +endif + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/service_cores/main.c b/examples/service_cores/main.c new file mode 100644 index 0000000..8a8f5bd --- /dev/null +++ b/examples/service_cores/main.c @@ -0,0 +1,247 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +/* allow application scheduling of the services */ +#include + +/* Allow application registration of its own services. An application does not + * have to register services, but it can be useful if it wishes to run a + * function on a core that is otherwise in use as a service core. In this + * example, all services are dummy services registered by the sample app itself. + */ +#include + +#define PROFILE_CORES_MAX 8 +#define PROFILE_SERVICE_PER_CORE 5 + +/* dummy function to do "work" */ +static int32_t service_func(void *args) +{ + RTE_SET_USED(args); + rte_delay_us(2000); + return 0; +} + +static struct rte_service_spec services[] = { + {"service_1", service_func, NULL, 0, 0}, + {"service_2", service_func, NULL, 0, 0}, + {"service_3", service_func, NULL, 0, 0}, + {"service_4", service_func, NULL, 0, 0}, + {"service_5", service_func, NULL, 0, 0}, +}; +#define NUM_SERVICES RTE_DIM(services) + +/* this struct holds the mapping of a particular core to all services */ +struct profile_for_core { + uint32_t mapped_services[PROFILE_SERVICE_PER_CORE]; +}; + +/* struct that can be applied as the service core mapping. Items in this + * struct will be passed to the ordinary rte_service_* APIs to configure the + * service cores at runtime, based on the requirements. + * + * These profiles can be considered a "configuration" for the service cores, + * where switching profile just changes the number of cores and the mappings + * for each of them. As a result, the core requirements and performance of the + * application scales. + */ +struct profile { + char name[64]; + uint32_t num_cores; + struct profile_for_core cores[PROFILE_CORES_MAX]; +}; + +static struct profile profiles[] = { + /* profile 0: high performance */ + { + .name = "High Performance", + .num_cores = 5, + .cores[0] = {.mapped_services = {1, 0, 0, 0, 0} }, + .cores[1] = {.mapped_services = {0, 1, 0, 0, 0} }, + .cores[2] = {.mapped_services = {0, 0, 1, 0, 0} }, + .cores[3] = {.mapped_services = {0, 0, 0, 1, 0} }, + .cores[4] = {.mapped_services = {0, 0, 0, 0, 1} }, + }, + /* profile 1: mid performance with single service priority */ + { + .name = "Mid-High Performance", + .num_cores = 3, + .cores[0] = {.mapped_services = {1, 1, 0, 0, 0} }, + .cores[1] = {.mapped_services = {0, 0, 1, 1, 0} }, + .cores[2] = {.mapped_services = {0, 0, 0, 0, 1} }, + .cores[3] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[4] = {.mapped_services = {0, 0, 0, 0, 0} }, + }, + /* profile 2: mid performance with single service priority */ + { + .name = "Mid-Low Performance", + .num_cores = 2, + .cores[0] = {.mapped_services = {1, 1, 1, 0, 0} }, + .cores[1] = {.mapped_services = {1, 1, 0, 1, 1} }, + .cores[2] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[3] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[4] = {.mapped_services = {0, 0, 0, 0, 0} }, + }, + /* profile 3: scale down performance on single core */ + { + .name = "Scale down performance", + .num_cores = 1, + .cores[0] = {.mapped_services = {1, 1, 1, 1, 1} }, + .cores[1] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[2] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[3] = {.mapped_services = {0, 0, 0, 0, 0} }, + .cores[4] = {.mapped_services = {0, 0, 0, 0, 0} }, + }, +}; +#define NUM_PROFILES RTE_DIM(profiles) + +static int +apply_profile(int profile_id) +{ + uint32_t i; + uint32_t s; + int ret; + struct profile *p = &profiles[profile_id]; + const uint8_t core_off = 1; + + for (i = 0; i < p->num_cores; i++) { + uint32_t core = i + core_off; + ret = rte_service_lcore_add(core); + if (ret && ret != -EALREADY) + printf("core %d added ret %d\n", core, ret); + + ret = rte_service_lcore_start(core); + if (ret && ret != -EALREADY) + printf("core %d start ret %d\n", core, ret); + + for (s = 0; s < NUM_SERVICES; s++) { + if (rte_service_map_lcore_set(s, core, + p->cores[i].mapped_services[s])) + printf("failed to map lcore %d\n", core); + } + } + + for ( ; i < PROFILE_CORES_MAX; i++) { + uint32_t core = i + core_off; + for (s = 0; s < NUM_SERVICES; s++) { + ret = rte_service_map_lcore_set(s, core, 0); + if (ret && ret != -EINVAL) { + printf("%s %d: map lcore set = %d\n", __func__, + __LINE__, ret); + } + } + ret = rte_service_lcore_stop(core); + if (ret && ret != -EALREADY) { + printf("%s %d: lcore stop = %d\n", __func__, + __LINE__, ret); + } + ret = rte_service_lcore_del(core); + if (ret && ret != -EINVAL) { + printf("%s %d: lcore del = %d\n", __func__, + __LINE__, ret); + } + } +} + +int +main(int argc, char **argv) +{ + int ret; + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_panic("Cannot init EAL\n"); + + uint32_t i; + for (i = 0; i < NUM_SERVICES; i++) { + services[i].callback_userdata = 0; + uint32_t id; + ret = rte_service_component_register(&services[i], &id); + if (ret) + rte_exit(-1, "service register() failed"); + + /* set the service itself to be ready to run. In the case of + * ethdev, eventdev etc PMDs, this will be set when the + * appropriate configure or setup function is called. + */ + rte_service_component_runstate_set(id, 1); + + /* Collect statistics for the service */ + rte_service_set_stats_enable(id, 1); + + /* the application sets the service to be active. Note that the + * previous component_runstate_set() is the PMD indicating + * ready, while this function is the application setting the + * service to run. Applications can choose to not run a service + * by setting runstate to 0 at any time. + */ + ret = rte_service_runstate_set(id, 1); + if (ret) + return -ENOEXEC; + } + + i = 0; + while (1) { + const char clr[] = { 27, '[', '2', 'J', '\0' }; + const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' }; + printf("%s%s", clr, topLeft); + + apply_profile(i); + printf("\n==> Profile: %s\n\n", profiles[i].name); + + sleep(1); + rte_service_dump(stdout, UINT32_MAX); + + sleep(5); + rte_service_dump(stdout, UINT32_MAX); + + i++; + if (i >= NUM_PROFILES) + i = 0; + } + + return 0; +} -- 2.7.4