* [dpdk-dev] [PATCH] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
@ 2016-02-24 13:26 Wojciech Andralojc
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
0 siblings, 1 reply; 14+ messages in thread
From: Wojciech Andralojc @ 2016-02-24 13:26 UTC (permalink / raw)
To: dev; +Cc: Marcel D Cornu, Tomasz Kantecki
Because of the feedback that we have received off the mailing list,
that extending EAL commands is not an option due to the
Intel Architecture nature of CAT,
we have changed the design of PQoS patch.
The current V2 patch implements a sample code, based on the DPDK skeleton
example app, that links against the existing 01.org PQoS library
(https://github.com/01org/intel-cmt-cat).
This eliminates the need for librte_pqos and EAL extensions introduced in
the V1 patch. The sample code implements a C module that parses
the application specific part of the command line with CAT configuration
options (--l3ca, same format as the V1 patch EAL command, but expects
CPU ids rather than lcores).
The module is easy to re-use in other applications as needed.
Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
---
Details of "--l3ca" app parameter to configure Intel CAT and CDP features:
--l3ca=bitmask@<cpu_list>
--l3ca=(code_bitmask,data_bitmask)@<cpu_list>
- makes selected CPU's use specified CAT bitmasks
CAT and CDP features allow management of the CPU's last level cache.
CAT introduces classes of service (COS) that are essentially bitmasks.
In current CAT implementations, a bit in a COS bitmask corresponds to
one cache way in the last level cache.
A CPU core is always assigned to one of the CAT classes.
By programming CPU core assignment and COS bitmasks, applications can be
given exclusive, shared, or mixed access to the CPU's last level cache.
CDP extends CAT so that there are two bitmasks per COS,
one for data and one for code.
The number of classes and number of valid bits in a COS bitmask is CPU
model specific and COS bitmasks need to be contiguous. Sample code calls
this bitmask a cbm or a capacity bitmask.
By default, after reset, all CPU cores are assigned to COS 0 and all
classes are programmed to allow fill into all cache ways.
CDP is off by default.
For more information about CAT please see
https://github.com/01org/intel-cmt-cat
Known issues and limitations:
- --l3ca must be a first app parameter
---
MAINTAINERS | 4 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/skeleton-cat.rst | 461 ++++++++++++++
examples/Makefile | 1 +
examples/skeleton-cat/Makefile | 68 +++
examples/skeleton-cat/basicfwd-cat.c | 220 +++++++
examples/skeleton-cat/cat.c | 957 ++++++++++++++++++++++++++++++
examples/skeleton-cat/cat.h | 70 +++
8 files changed, 1782 insertions(+)
create mode 100644 doc/guides/sample_app_ug/skeleton-cat.rst
create mode 100644 examples/skeleton-cat/Makefile
create mode 100644 examples/skeleton-cat/basicfwd-cat.c
create mode 100644 examples/skeleton-cat/cat.c
create mode 100644 examples/skeleton-cat/cat.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 628bc05..7a6702b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -600,3 +600,7 @@ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
M: Daniel Mrzyglod <danielx.t.mrzyglod@intel.com>
F: examples/ptpclient/
+
+M: Tomasz Kantecki <tomasz.kantecki@intel.com>
+F: examples/skeleton-cat/
+F: doc/guides/sample_app_ug/skeleton-cat.rst
\ No newline at end of file
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 8a646dd..f065e54 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -41,6 +41,7 @@ Sample Applications User Guide
exception_path
hello_world
skeleton
+ skeleton-cat
rxtx_callbacks
ip_frag
ipv4_multicast
diff --git a/doc/guides/sample_app_ug/skeleton-cat.rst b/doc/guides/sample_app_ug/skeleton-cat.rst
new file mode 100644
index 0000000..6684f61
--- /dev/null
+++ b/doc/guides/sample_app_ug/skeleton-cat.rst
@@ -0,0 +1,461 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Cache Allocation Technology (CAT) enabled Basic Forwarding Sample Application
+=============================================================================
+
+Basic Forwarding sample application is a simple *skeleton* example of
+a forwarding application. It has been extended to make use of CAT via extended
+command line options and linking against the libpqos library.
+
+It is intended as a demonstration of the basic components of a DPDK forwarding
+application and use of the libpqos library to program CAT.
+For more detailed implementations see the L2 and L3 forwarding
+sample applications.
+
+CAT and CDP features allow management of the CPU's last level cache.
+CAT introduces classes of service (COS) that are essentially bitmasks.
+In current CAT implementations, a bit in a COS bitmask corresponds to
+one cache way in last level cache.
+A CPU core is always assigned to one of the CAT classes.
+By programming CPU core assignment and COS bitmasks, applications can be given
+exclusive, shared, or mixed access to the CPU's last level cache.
+CDP extends CAT so that there are two bitmasks per COS,
+one for data and one for code.
+The number of classes and number of valid bits in a COS bitmask is CPU model
+specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
+``cbm`` or capacity bitmask.
+By default, after reset, all CPU cores are assigned to COS 0 and all classes
+are programmed to allow fill into all cache ways.
+CDP is off by default.
+
+For more information about CAT please see https://github.com/01org/intel-cmt-cat
+
+
+Compiling the Application
+-------------------------
+
+Requires ``libpqos`` from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
+hosted on GitHub repository. For installation notes, please see ``README`` file.
+
+GIT:
+
+* https://github.com/01org/intel-cmt-cat
+
+To compile the application export the path to PQoS lib (needed only if libpqos
+is not installed in the default location (/usr/local))
+and the DPDK source tree and go to the example directory:
+
+.. code-block:: console
+
+ export PQOS_INSTALL_PATH=/path/to/libpqos
+ export RTE_SDK=/path/to/rte_sdk
+
+ cd ${RTE_SDK}/examples/skeleton-cat
+
+Set the target, for example:
+
+.. code-block:: console
+
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
+
+Build the application as follows:
+
+.. code-block:: console
+
+ make
+
+
+Running the Application
+-----------------------
+
+To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=0x3@(0-2)
+
+or to enable CAT and CDP on cpus 1,3:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=(0x00C00,0x00300)@(1,3)
+
+The option to enable CAT is:
+
+* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
+
+ where ``cbm`` stands for capacity bitmask and must be expressed in
+ hexadecimal form.
+
+ ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
+ masks (``code_cbm`` and ``data_cbm``) is used.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ ``cpus`` could be a single digit/range or a group and must be expressed in
+ decimal form.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
+
+ * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
+
+ * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
+
+ * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
+
+ * cpu 7 has exclusive access to all of its 4 ways;
+
+ e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
+
+ * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
+ data ways are not overlapping.
+
+
+Refer to *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+
+To reset or list CAT configuration and control CDP please use ``pqos`` tool
+from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
+
+To enabled or disable CDP:
+
+.. code-block:: console
+
+ sudo ./pqos -S cdp-on
+
+ sudo ./pqos -S cdp-off
+
+to reset CAT configuration:
+
+.. code-block:: console
+
+ sudo ./pqos -R
+
+to list CAT config:
+
+.. code-block:: console
+
+ sudo ./pqos -s
+
+For more info about ``pqos`` tool please see its man page or
+`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
+
+
+Explanation
+-----------
+
+The following sections provide an explanation of the main components of the
+code.
+
+All DPDK library functions used in the sample code are prefixed with ``rte_``
+and are explained in detail in the *DPDK API Documentation*.
+
+
+The Main Function
+~~~~~~~~~~~~~~~~~
+
+The ``main()`` function performs the initialization and calls the execution
+threads for each lcore.
+
+The first task is to initialize the Environment Abstraction Layer (EAL). The
+``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+The next task is to initialize the PQoS library and configure CAT. The
+``argc`` and ``argv`` arguments are provided to the ``cat_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+``cat_init()`` is a wrapper function which parses the command, validates
+the requested parameters and configures CAT accordingly.
+
+Parsing of command line arguments is done in ``parse_args(...)``.
+libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
+queried for system CPU information and L3CA capabilities via
+``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
+calls. When all capability and topology information is collected, the requested
+CAT configuration is validated. A check is then performed (on per socket basis)
+for a sufficient number of unassociated COS. COS are selected and
+configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
+relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
+a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
+COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers)
+used by the application:
+
+.. code-block:: c
+
+ mbuf_pool = rte_mempool_create("MBUF_POOL",
+ NUM_MBUFS * nb_ports,
+ MBUF_SIZE,
+ MBUF_CACHE_SIZE,
+ sizeof(struct rte_pktmbuf_pool_private),
+ rte_pktmbuf_pool_init, NULL,
+ rte_pktmbuf_init, NULL,
+ rte_socket_id(),
+ 0);
+
+Mbufs are the packet buffer structures used by DPDK. They are explained in
+detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*.
+
+The ``main()`` function also initializes all the ports using the user defined
+``port_init()`` function which is explained in the next section:
+
+.. code-block:: c
+
+ for (portid = 0; portid < nb_ports; portid++) {
+ if (port_init(portid, mbuf_pool) != 0) {
+ rte_exit(EXIT_FAILURE,
+ "Cannot init port %" PRIu8 "\n", portid);
+ }
+ }
+
+
+Once the initialization is complete, the application is ready to launch a
+function on an lcore. In this example ``lcore_main()`` is called on a single
+lcore.
+
+
+.. code-block:: c
+
+ lcore_main();
+
+The ``lcore_main()`` function is explained below.
+
+
+
+The Port Initialization Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The main functional part of the port initialization used in the Basic
+Forwarding application is shown below:
+
+.. code-block:: c
+
+ static inline int
+ port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+ {
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ struct ether_addr addr;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+ }
+
+The Ethernet ports are configured with default settings using the
+``rte_eth_dev_configure()`` function and the ``port_conf_default`` struct:
+
+.. code-block:: c
+
+ static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+ };
+
+For this example the ports are set up with 1 RX and 1 TX queue using the
+``rte_eth_rx_queue_setup()`` and ``rte_eth_tx_queue_setup()`` functions.
+
+The Ethernet port is then started:
+
+.. code-block:: c
+
+ retval = rte_eth_dev_start(port);
+
+
+Finally the RX port is set in promiscuous mode:
+
+.. code-block:: c
+
+ rte_eth_promiscuous_enable(port);
+
+
+The Lcores Main
+~~~~~~~~~~~~~~~
+
+As we saw above the ``main()`` function calls an application function on the
+available lcores. For the Basic Forwarding application the lcore function
+looks like the following:
+
+.. code-block:: c
+
+ static __attribute__((noreturn)) void
+ lcore_main(void)
+ {
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+ }
+
+
+The main work of the application is done within the loop:
+
+.. code-block:: c
+
+ for (;;) {
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+
+Packets are received in bursts on the RX ports and transmitted in bursts on
+the TX ports. The ports are grouped in pairs with a simple mapping scheme
+using the an XOR on the port number::
+
+ 0 -> 1
+ 1 -> 0
+
+ 2 -> 3
+ 3 -> 2
+
+ etc.
+
+The ``rte_eth_tx_burst()`` function frees the memory buffers of packets that
+are transmitted. If packets fail to transmit, ``(nb_tx < nb_rx)``, then they
+must be freed explicitly using ``rte_pktmbuf_free()``.
+
+The forwarding loop can be interrupted and the application closed using
+``Ctrl-C``.
diff --git a/examples/Makefile b/examples/Makefile
index 1665df1..10168a9 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -74,6 +74,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched
DIRS-y += quota_watermark
DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks
DIRS-y += skeleton
+DIRS-y += skeleton-cat
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost
diff --git a/examples/skeleton-cat/Makefile b/examples/skeleton-cat/Makefile
new file mode 100644
index 0000000..e3ef642
--- /dev/null
+++ b/examples/skeleton-cat/Makefile
@@ -0,0 +1,68 @@
+# BSD LICENSE
+#
+# Copyright(c) 2016 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Default location of PQoS library and includes,
+# can be overridden by command line or environment
+PQOS_INSTALL_PATH ?= /usr/local/
+PQOS_LIBRARY_PATH = $(PQOS_INSTALL_PATH)/lib/libpqos.a
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = basicfwd-cat
+
+# all source are stored in SRCS-y
+SRCS-y := basicfwd-cat.c cat.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+EXTRA_CFLAGS += -O3 -g -Wfatal-errors
+
+CFLAGS += -I$(PQOS_INSTALL_PATH)/include
+CFLAGS_cat.o := -D_GNU_SOURCE
+
+LDLIBS += -L$(PQOS_INSTALL_PATH)/lib
+LDLIBS += $(PQOS_LIBRARY_PATH) \
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/skeleton-cat/basicfwd-cat.c b/examples/skeleton-cat/basicfwd-cat.c
new file mode 100644
index 0000000..cb68bef
--- /dev/null
+++ b/examples/skeleton-cat/basicfwd-cat.c
@@ -0,0 +1,220 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_mbuf.h>
+
+#include "cat.h"
+
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+#define BURST_SIZE 32
+
+static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+};
+
+/* basicfwd-cat.c: CAT enabled, basic DPDK skeleton forwarding example. */
+
+/*
+ * Initializes a given port using global settings and with the RX buffers
+ * coming from the mbuf_pool passed as a parameter.
+ */
+static inline int
+port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+{
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+}
+
+/*
+ * The lcore main. This is the main thread that does the work, reading from
+ * an input port and writing to an output port.
+ */
+static __attribute__((noreturn)) void
+lcore_main(void)
+{
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+}
+
+/*
+ * The main function, which does initialization and calls the per-lcore
+ * functions.
+ */
+int
+main(int argc, char *argv[])
+{
+ struct rte_mempool *mbuf_pool;
+ unsigned nb_ports;
+ uint8_t portid;
+
+ /* Initialize the Environment Abstraction Layer (EAL). */
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+ argc -= ret;
+ argv += ret;
+
+ ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /* Check that there is an even number of ports to send/receive on. */
+ nb_ports = rte_eth_dev_count();
+ if (nb_ports < 2 || (nb_ports & 1))
+ rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /* Initialize all ports. */
+ for (portid = 0; portid < nb_ports; portid++)
+ if (port_init(portid, mbuf_pool) != 0)
+ rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+ portid);
+
+ if (rte_lcore_count() > 1)
+ printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
+
+ /* Call lcore_main on the master core only. */
+ lcore_main();
+
+ return 0;
+}
diff --git a/examples/skeleton-cat/cat.c b/examples/skeleton-cat/cat.c
new file mode 100644
index 0000000..d0a3457
--- /dev/null
+++ b/examples/skeleton-cat/cat.c
@@ -0,0 +1,957 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <inttypes.h>
+#include <limits.h>
+#include <sched.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_memcpy.h>
+
+#include <pqos.h>
+
+#include "cat.h"
+
+#define BITS_PER_HEX 4
+#define PQOS_MAX_SOCKETS 8
+#define PQOS_MAX_SOCKET_CORES 64
+#define PQOS_MAX_CORES (PQOS_MAX_SOCKET_CORES * PQOS_MAX_SOCKETS)
+
+static const struct pqos_cap *m_cap;
+static const struct pqos_cpuinfo *m_cpu;
+static unsigned m_sockets[PQOS_MAX_SOCKETS];
+static unsigned m_sock_count;
+static struct cat_config m_config[PQOS_MAX_CORES];
+static unsigned m_config_count;
+
+static unsigned
+bits_count(uint64_t bitmask)
+{
+ unsigned count = 0;
+
+ for (; bitmask != 0; count++)
+ bitmask &= bitmask - 1;
+
+ return count;
+}
+
+/*
+ * Parse elem, the elem could be single number/range or '(' ')' group
+ * 1) A single number elem, it's just a simple digit. e.g. 9
+ * 2) A single range elem, two digits with a '-' between. e.g. 2-6
+ * 3) A group elem, combines multiple 1) or 2) with '( )'. e.g (0,2-4,6)
+ * Within group elem, '-' used for a range separator;
+ * ',' used for a single number.
+ */
+static int
+parse_set(const char *input, rte_cpuset_t *cpusetp)
+{
+ unsigned idx;
+ const char *str = input;
+ char *end = NULL;
+ unsigned min, max;
+ const unsigned num = PQOS_MAX_CORES;
+
+ CPU_ZERO(cpusetp);
+
+ while (isblank(*str))
+ str++;
+
+ /* only digit or left bracket is qualify for start point */
+ if ((!isdigit(*str) && *str != '(') || *str == '\0')
+ return -1;
+
+ /* process single number or single range of number */
+ if (*str != '(') {
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ while (isblank(*end))
+ end++;
+
+ min = idx;
+ max = idx;
+ if (*end == '-') {
+ /* process single <number>-<number> */
+ end++;
+ while (isblank(*end))
+ end++;
+ if (!isdigit(*end))
+ return -1;
+
+ errno = 0;
+ idx = strtoul(end, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+ max = idx;
+ while (isblank(*end))
+ end++;
+ if (*end != ',' && *end != '\0')
+ return -1;
+ }
+
+ if (*end != ',' && *end != '\0' && *end != '@')
+ return -1;
+
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ return end - input;
+ }
+
+ /* process set within bracket */
+ str++;
+ while (isblank(*str))
+ str++;
+ if (*str == '\0')
+ return -1;
+
+ min = PQOS_MAX_CORES;
+ do {
+
+ /* go ahead to the first digit */
+ while (isblank(*str))
+ str++;
+ if (!isdigit(*str))
+ return -1;
+
+ /* get the digit value */
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ /* go ahead to separator '-',',' and ')' */
+ while (isblank(*end))
+ end++;
+ if (*end == '-') {
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ else /* avoid continuous '-' */
+ return -1;
+ } else if ((*end == ',') || (*end == ')')) {
+ max = idx;
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ min = PQOS_MAX_CORES;
+ } else
+ return -1;
+
+ str = end + 1;
+ } while (*end != '\0' && *end != ')');
+
+ return str - input;
+}
+
+/* Test if bitmask is contiguous */
+static int
+is_contiguous(uint64_t bitmask)
+{
+ /* check if bitmask is contiguous */
+ unsigned i = 0;
+ unsigned j = 0;
+ const unsigned max_idx = (sizeof(bitmask) * CHAR_BIT);
+
+ if (bitmask == 0)
+ return 0;
+
+ for (i = 0; i < max_idx; i++) {
+ if (((1ULL << i) & bitmask) != 0)
+ j++;
+ else if (j > 0)
+ break;
+ }
+
+ if (bits_count(bitmask) != j)
+ return 0;
+
+ return 1;
+}
+
+/*
+ * The format pattern: --l3ca='<cbm@cpus>[,<(ccbm,dcbm)@cpus>...]'
+ * cbm could be a single mask or for a CDP enabled system, a group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - CPUs 1 and 3 share its 4 ways with CPUs 4, 5 and 6;
+ * - CPUs 4,5 and 6 share half (4 out of 8 ways) of its L3 with 1 and 3;
+ * - CPUs 4,5 and 6 have exclusive access to 4 out of 8 ways;
+ * - CPU 7 has exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for a CDP enabled system
+ * - cpus 1 and 3 have access to 2 ways for code and 2 ways for data,
+ * code and data ways are not overlapping.;
+ */
+static int
+parse_l3ca(const char *l3ca)
+{
+ unsigned idx = 0;
+ const char *cbm_start = NULL;
+ char *cbm_end = NULL;
+ const char *end = NULL;
+ int offset;
+ rte_cpuset_t cpuset;
+ uint64_t mask = 0;
+ uint64_t cmask = 0;
+ int ret = -1;
+
+ if (l3ca == NULL)
+ return -1;
+
+ /* Get cbm */
+ do {
+ CPU_ZERO(&cpuset);
+ mask = 0;
+ cmask = 0;
+
+ while (isblank(*l3ca))
+ l3ca++;
+
+ if (*l3ca == '\0') {
+ ret = -1;
+ goto err;
+ }
+
+ /* record mask_set start point */
+ cbm_start = l3ca;
+
+ /* go across a complete bracket */
+ if (*cbm_start == '(') {
+ l3ca += strcspn(l3ca, ")");
+ if (*l3ca++ == '\0') {
+ ret = -1;
+ goto err;
+ }
+ }
+
+ /* scan the separator '@', ','(next) or '\0'(finish) */
+ l3ca += strcspn(l3ca, "@,");
+
+ if (*l3ca == '@') {
+ /* explicit assign cpu_set */
+ offset = parse_set(l3ca + 1, &cpuset);
+ if (offset < 0 || CPU_COUNT(&cpuset) == 0) {
+ ret = -1;
+ goto err;
+ }
+ end = l3ca + 1 + offset;
+ } else {
+ ret = -1;
+ goto err;
+ }
+
+ if (*end != ',' && *end != '\0') {
+ ret = -1;
+ goto err;
+ }
+
+ /* parse mask_set from start point */
+ if (*cbm_start == '(') {
+ cbm_start++;
+
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start)) {
+ ret = -1;
+ goto err;
+ }
+
+ errno = 0;
+ cmask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || cmask == 0) {
+ ret = -1;
+ goto err;
+ }
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (*cbm_end != ',') {
+ ret = -1;
+ goto err;
+ }
+
+ cbm_end++;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (!isxdigit(*cbm_end)) {
+ ret = -1;
+ goto err;
+ }
+
+ errno = 0;
+ mask = strtoul(cbm_end, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0) {
+ ret = -1;
+ goto err;
+ }
+ } else {
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start)) {
+ ret = -1;
+ goto err;
+ }
+
+ errno = 0;
+ mask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0) {
+ ret = -1;
+ goto err;
+ }
+ }
+
+ if (mask == 0 || is_contiguous(mask) == 0) {
+ ret = -1;
+ goto err;
+ }
+
+ if (cmask != 0 && is_contiguous(cmask) == 0) {
+ ret = -1;
+ goto err;
+ }
+
+ rte_memcpy(&m_config[idx].cpumask,
+ &cpuset, sizeof(rte_cpuset_t));
+
+ if (cmask != 0) {
+ m_config[idx].cdp = 1;
+ m_config[idx].code_mask = cmask;
+ m_config[idx].data_mask = mask;
+ } else
+ m_config[idx].mask = mask;
+
+ m_config_count++;
+
+ l3ca = end + 1;
+ idx++;
+ } while (*end != '\0' && idx < PQOS_MAX_CORES);
+
+ if (m_config_count == 0) {
+ ret = -1;
+ goto err;
+ }
+
+ ret = 0;
+
+err:
+ return ret;
+}
+
+static int
+check_cpus_overlapping(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ rte_cpuset_t mask;
+
+ CPU_ZERO(&mask);
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = i + 1; j < m_config_count; j++) {
+ CPU_AND(&mask,
+ &m_config[i].cpumask,
+ &m_config[j].cpumask);
+
+ if (CPU_COUNT(&mask) != 0) {
+ printf("PQOS: Requested CPUs sets are "
+ "overlapping.\n");
+ return -EFAULT;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cpus(void)
+{
+ unsigned i = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (cpu_id = 0; cpu_id < PQOS_MAX_CORES; cpu_id++) {
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) != 0) {
+
+ ret = pqos_cpu_check_core(m_cpu, cpu_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: %u is not a valid "
+ "logical core id.\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS "
+ "associated to cpu %u.\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Check if COS assigned to lcore is different
+ * then default one (#0) */
+ if (cos_id != 0) {
+ printf("PQOS: cpu %u has already "
+ "associated COS#%u. "
+ "Please reset L3CA.\n",
+ cpu_id, cos_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+static int
+check_cdp(const struct pqos_capability *cap_l3ca)
+{
+ unsigned i = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1 && cap_l3ca->u.l3ca->cdp_on == 0) {
+ if (cap_l3ca->u.l3ca->cdp == 0) {
+ printf("PQOS: CDP requested but not "
+ "supported.\n");
+ } else {
+ printf("PQOS: CDP requested but not enabled. "
+ "Please enable CDP.\n");
+ }
+ return -EFAULT;
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cbm_len_and_contention(const struct pqos_capability *cap_l3ca)
+{
+ unsigned i = 0;
+ uint64_t mask = 0;
+ const uint64_t not_cbm = (UINT64_MAX << (cap_l3ca->u.l3ca->num_ways));
+ const uint64_t cbm_contention_mask = cap_l3ca->u.l3ca->way_contention;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1)
+ mask = m_config[i].code_mask | m_config[i].data_mask;
+ else
+ mask = m_config[i].mask;
+
+ if ((mask & not_cbm) != 0) {
+ printf("PQOS: One or more of requested CBM masks not "
+ "supported by system (too long).\n");
+ ret = -EFAULT;
+ break;
+ }
+
+ /* Just a warning */
+ if ((mask & cbm_contention_mask) != 0) {
+ printf("PQOS: One or more of requested CBM masks "
+ "overlap CBM contention mask.\n");
+ break;
+ }
+
+ }
+
+ return ret;
+}
+
+static int
+check_and_select_classes(const struct pqos_capability *cap_l3ca,
+ unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned phy_pkg_id = 0;
+ unsigned cos_id = 0;
+ unsigned cpu_id = 0;
+ unsigned phy_pkg_lcores[PQOS_MAX_SOCKETS][m_config_count];
+ const unsigned cos_num = cap_l3ca->u.l3ca->num_classes;
+ unsigned used_cos_table[PQOS_MAX_SOCKETS][cos_num];
+ int ret = 0;
+
+ memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+ memset(used_cos_table, 0, sizeof(used_cos_table));
+
+ /* detect currently used COS */
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS associated to "
+ "cpu %u on phy_pkg %u.\n", cpu_id, phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for cpu %u\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Mark COS as used */
+ if (used_cos_table[phy_pkg_id][cos_id] == 0)
+ used_cos_table[phy_pkg_id][cos_id]++;
+ }
+
+ /* look for avail. COS to fulfill requested config */
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Check if we already have COS selected
+ * to be used for that group on that socket */
+ if (phy_pkg_lcores[phy_pkg_id][i] != 0)
+ continue;
+
+ phy_pkg_lcores[phy_pkg_id][i]++;
+
+ /* Search for avail. COS to be used on that socket */
+ for (cos_id = 0; cos_id < cos_num; cos_id++) {
+ if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+ used_cos_table[phy_pkg_id][cos_id]++;
+ cos_id_map[i][phy_pkg_id] = cos_id;
+ break;
+ }
+ }
+
+ /* If there is no COS available ...*/
+ if (cos_id == cos_num) {
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ if (ret != 0)
+ printf("PQOS: Not enough available COS to configure "
+ "requested configuration.\n");
+
+ return ret;
+}
+
+static int
+configure_cat(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned phy_pkg_id = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ unsigned i = 0;
+ unsigned j = 0;
+ struct pqos_l3ca l3ca = {0};
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ memset(&l3ca, 0, sizeof(l3ca));
+
+ l3ca.cdp = m_config[i].cdp;
+ if (m_config[i].cdp == 1) {
+ l3ca.code_mask = m_config[i].code_mask;
+ l3ca.data_mask = m_config[i].data_mask;
+ } else
+ l3ca.ways_mask = m_config[i].mask;
+
+ for (j = 0; j < m_sock_count; j++) {
+ phy_pkg_id = m_sockets[j];
+ if (cos_id_map[i][phy_pkg_id] == 0)
+ continue;
+
+ l3ca.class_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to set COS %u on "
+ "phy_pkg %u.\n", l3ca.class_id,
+ phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ cos_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_assoc_set(cpu_id, cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS %u to "
+ "cpu %u\n", cos_id, cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ int opt = 0;
+ int retval = 0;
+ int oldopterr = 0;
+ char **argvopt = argv;
+ char *prgname = argv[0];
+
+ static struct option lgopts[] = {
+ { "l3ca", required_argument, 0, 0 },
+ { NULL, 0, 0, 0 }
+ };
+
+ /* Disable printing messages within getopt() */
+ oldopterr = opterr;
+ opterr = 0;
+
+ opt = getopt_long(argc, argvopt, "", lgopts, NULL);
+ if (opt == 0) {
+ retval = parse_l3ca(optarg);
+ if (retval != 0) {
+ printf("PQOS: Invalid L3CA parameters!\n");
+ goto exit;
+ }
+
+ argv[optind - 1] = prgname;
+ retval = optind - 1;
+ } else
+ retval = 0;
+
+exit:
+ /* reset getopt lib */
+ optind = 0;
+
+ /* Restore opterr value */
+ opterr = oldopterr;
+
+ return retval;
+}
+
+static void
+print_cmd_line_config(void)
+{
+ char cpustr[PQOS_MAX_CORES * 3] = {0};
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned len = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ len = 0;
+ memset(cpustr, 0, sizeof(cpustr));
+
+ /* Generate CPU list */
+ for (j = 0; j < PQOS_MAX_CORES; j++) {
+ if (CPU_ISSET(j, &m_config[i].cpumask) != 1)
+ continue;
+
+ len += snprintf(cpustr + len, sizeof(cpustr) - len - 1,
+ "%u,", j);
+
+ if (len >= sizeof(cpustr) - 1)
+ break;
+ }
+
+ if (m_config->cdp == 1) {
+ printf("PQOS: CPUs: %s cMASK: 0x%llx, dMASK: "
+ "0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].code_mask,
+ (unsigned long long)m_config[i].data_mask);
+ } else {
+ printf("PQOS: CPUs: %s MASK: 0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].mask);
+ }
+ }
+}
+
+/**
+ * @brief Prints CAT configuration
+ */
+static void
+print_cat_config(void)
+{
+ int ret = PQOS_RETVAL_OK;
+ unsigned i = 0;
+
+ for (i = 0; i < m_sock_count; i++) {
+ struct pqos_l3ca tab[PQOS_MAX_L3CA_COS] = {{0} };
+ unsigned num = 0;
+ unsigned n = 0;
+
+ ret = pqos_l3ca_get(m_sockets[i], PQOS_MAX_L3CA_COS, &num, tab);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving COS!\n");
+ return;
+ }
+
+ printf("PQOS: COS definitions for Socket %u:\n", m_sockets[i]);
+ for (n = 0; n < num; n++) {
+ if (tab[n].cdp == 1) {
+ printf("PQOS: COS: %u, cMASK: 0x%llx, "
+ "dMASK: 0x%llx\n", tab[n].class_id,
+ (unsigned long long)tab[n].code_mask,
+ (unsigned long long)tab[n].data_mask);
+ } else {
+ printf("PQOS: COS: %u, MASK: 0x%llx\n",
+ tab[n].class_id,
+ (unsigned long long)tab[n].ways_mask);
+ }
+ }
+ }
+
+ for (i = 0; i < m_sock_count; i++) {
+ unsigned lcores[PQOS_MAX_SOCKET_CORES] = {0};
+ unsigned lcount = 0;
+ unsigned n = 0;
+
+ ret = pqos_cpu_get_cores(m_cpu, m_sockets[i],
+ PQOS_MAX_SOCKET_CORES, &lcount, &lcores[0]);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving core information!\n");
+ return;
+ }
+
+ printf("PQOS: CPU information for socket %u:\n", m_sockets[i]);
+ for (n = 0; n < lcount; n++) {
+ unsigned class_id = 0;
+
+ ret = pqos_l3ca_assoc_get(lcores[n], &class_id);
+ if (ret == PQOS_RETVAL_OK)
+ printf("PQOS: CPU: %u, COS: %u\n", lcores[n],
+ class_id);
+ else
+ printf("PQOS: CPU: %u, ERROR\n", lcores[n]);
+ }
+ }
+
+}
+
+static int
+cat_set(void)
+{
+ int ret = 0;
+ const struct pqos_capability *cap_l3ca = NULL;
+ unsigned cos_id_map[m_config_count][PQOS_MAX_SOCKETS];
+
+ memset(cos_id_map, 0, sizeof(cos_id_map));
+
+ ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &cap_l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving PQOS_CAP_TYPE_L3CA "
+ "capabilities!\n");
+ return -EFAULT;
+ }
+
+ ret = check_cpus();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cdp(cap_l3ca);
+ if (ret != 0)
+ return ret;
+
+ ret = check_cbm_len_and_contention(cap_l3ca);
+ if (ret != 0)
+ return ret;
+
+ ret = check_cpus_overlapping();
+ if (ret != 0)
+ return ret;
+
+ ret = check_and_select_classes(cap_l3ca, cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ ret = configure_cat(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+cat_exit(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned cpu_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_l3ca_assoc_set(cpu_id, 0);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS 0 to "
+ "cpu %u\n", cpu_id);
+ }
+ }
+ }
+
+ /* deallocate all the resources */
+ if (pqos_fini() != PQOS_RETVAL_OK)
+ printf("PQOS: Error shutting down PQoS library!\n");
+}
+
+int
+cat_init(int argc, char **argv)
+{
+ int ret = 0;
+ int args_num = 0;
+ struct pqos_config cfg = {0};
+
+ /* Parse cmd line args */
+ ret = parse_args(argc, argv);
+
+ if (ret <= 0)
+ return ret;
+
+ args_num = ret;
+
+ /* PQoS Initialization - Check and initialize CAT capability */
+ cfg.fd_log = STDOUT_FILENO;
+ cfg.verbose = 0;
+ cfg.cdp_cfg = PQOS_REQUIRE_CDP_ANY;
+ ret = pqos_init(&cfg);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error initializing PQoS library!\n");
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Get capability and CPU info pointer */
+ ret = pqos_cap_get(&m_cap, &m_cpu);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving PQoS capabilities!\n");
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Get CPU socket information */
+ ret = pqos_cpu_get_sockets(m_cpu, PQOS_MAX_SOCKETS, &m_sock_count,
+ m_sockets);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving CPU socket information!\n");
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Print cmd line configuration */
+ print_cmd_line_config();
+
+ /* Validate cmd line configuration and configure system */
+ ret = cat_set();
+ if (ret != 0) {
+ printf("PQOS: Failed to configure CAT!\n");
+ goto exit;
+ }
+
+ ret = atexit(cat_exit);
+ if (ret != 0) {
+ printf("PQOS: Cannot set exit function\n");
+ goto exit;
+ }
+
+ /* Print CAT configuration */
+ print_cat_config();
+
+exit:
+ if (ret != 0) {
+ /* deallocate all the resources */
+ if (pqos_fini() != PQOS_RETVAL_OK)
+ printf("PQOS: Error shutting down PQoS library!\n");
+
+ return ret;
+ } else
+ return args_num; /* Numbers of args processed */
+}
diff --git a/examples/skeleton-cat/cat.h b/examples/skeleton-cat/cat.h
new file mode 100644
index 0000000..c742164
--- /dev/null
+++ b/examples/skeleton-cat/cat.h
@@ -0,0 +1,70 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _CAT_H
+#define _CAT_H
+
+/**
+ * @file
+ * PQoS CAT
+ */
+
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_lcore.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* L3 cache allocation class of service data structure */
+struct cat_config {
+ rte_cpuset_t cpumask; /* CPUs bitmask */
+ int cdp; /* data & code masks used if true */
+ union {
+ uint64_t mask; /* capacity bitmask (CBM) */
+ struct {
+ uint64_t data_mask; /* data capacity bitmask (CBM) */
+ uint64_t code_mask; /* code capacity bitmask (CBM) */
+ };
+ };
+};
+
+int cat_init(int argc, char **argv);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _CAT_H */
--
1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread
* [dpdk-dev] [PATCH v2] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-02-24 13:26 [dpdk-dev] [PATCH] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage Wojciech Andralojc
@ 2016-02-26 15:07 ` Wojciech Andralojc
2016-03-07 14:34 ` Van Haaren, Harry
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Wojciech Andralojc @ 2016-02-26 15:07 UTC (permalink / raw)
To: dev; +Cc: Marcel D Cornu, Tomasz Kantecki
Because of the feedback that we have received off the mailing list,
that extending EAL commands is not an option due to the
Intel Architecture nature of CAT,
we have changed the design of PQoS patch.
The current V2 patch implements a sample code, based on the DPDK skeleton
example app, that links against the existing 01.org PQoS library
(https://github.com/01org/intel-cmt-cat).
This eliminates the need for librte_pqos and EAL extensions introduced in
the V1 patch. The sample code implements a C module that parses
the application specific part of the command line with CAT configuration
options (--l3ca, same format as the V1 patch EAL command, but expects
CPU ids rather than lcores).
The module is easy to re-use in other applications as needed.
Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
---
Version 2:
* Added signal handlers to do clean-up on SIGINT and SIGTERM
* Clean-up function modified to zero globals (needed for testing)
* Init function modified to return more applicable errnos
Version 1:
* Initial version
Details of "--l3ca" app parameter to configure Intel CAT and CDP features:
--l3ca=bitmask@<cpu_list>
--l3ca=(code_bitmask,data_bitmask)@<cpu_list>
makes selected CPU's use specified CAT bitmasks, bitmasks must be
expressed in hexadecimal form
CAT and CDP features allow management of the CPU's last level cache.
CAT introduces classes of service (COS) that are essentially bitmasks.
In current CAT implementations, a bit in a COS bitmask corresponds to
one cache way in the last level cache.
A CPU core is always assigned to one of the CAT classes.
By programming CPU core assignment and COS bitmasks, applications can be
given exclusive, shared, or mixed access to the CPU's last level cache.
CDP extends CAT so that there are two bitmasks per COS,
one for data and one for code.
The number of classes and number of valid bits in a COS bitmask is CPU
model specific and COS bitmasks need to be contiguous. Sample code calls
this bitmask a cbm or a capacity bitmask.
By default, after reset, all CPU cores are assigned to COS 0 and all
classes are programmed to allow fill into all cache ways.
CDP is off by default.
For more information about CAT please see
https://github.com/01org/intel-cmt-cat
Known issues and limitations:
- --l3ca must be a first app parameter
---
MAINTAINERS | 4 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/skeleton-cat.rst | 461 ++++++++++++++
examples/Makefile | 1 +
examples/skeleton-cat/Makefile | 68 ++
examples/skeleton-cat/basicfwd-cat.c | 220 +++++++
examples/skeleton-cat/cat.c | 992 ++++++++++++++++++++++++++++++
examples/skeleton-cat/cat.h | 72 +++
8 files changed, 1819 insertions(+)
create mode 100644 doc/guides/sample_app_ug/skeleton-cat.rst
create mode 100644 examples/skeleton-cat/Makefile
create mode 100644 examples/skeleton-cat/basicfwd-cat.c
create mode 100644 examples/skeleton-cat/cat.c
create mode 100644 examples/skeleton-cat/cat.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 628bc05..7a6702b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -600,3 +600,7 @@ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
M: Daniel Mrzyglod <danielx.t.mrzyglod@intel.com>
F: examples/ptpclient/
+
+M: Tomasz Kantecki <tomasz.kantecki@intel.com>
+F: examples/skeleton-cat/
+F: doc/guides/sample_app_ug/skeleton-cat.rst
\ No newline at end of file
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 8a646dd..f065e54 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -41,6 +41,7 @@ Sample Applications User Guide
exception_path
hello_world
skeleton
+ skeleton-cat
rxtx_callbacks
ip_frag
ipv4_multicast
diff --git a/doc/guides/sample_app_ug/skeleton-cat.rst b/doc/guides/sample_app_ug/skeleton-cat.rst
new file mode 100644
index 0000000..cc174fc
--- /dev/null
+++ b/doc/guides/sample_app_ug/skeleton-cat.rst
@@ -0,0 +1,461 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Cache Allocation Technology (CAT) enabled Basic Forwarding Sample Application
+=============================================================================
+
+Basic Forwarding sample application is a simple *skeleton* example of
+a forwarding application. It has been extended to make use of CAT via extended
+command line options and linking against the libpqos library.
+
+It is intended as a demonstration of the basic components of a DPDK forwarding
+application and use of the libpqos library to program CAT.
+For more detailed implementations see the L2 and L3 forwarding
+sample applications.
+
+CAT and CDP features allow management of the CPU's last level cache.
+CAT introduces classes of service (COS) that are essentially bitmasks.
+In current CAT implementations, a bit in a COS bitmask corresponds to
+one cache way in last level cache.
+A CPU core is always assigned to one of the CAT classes.
+By programming CPU core assignment and COS bitmasks, applications can be given
+exclusive, shared, or mixed access to the CPU's last level cache.
+CDP extends CAT so that there are two bitmasks per COS,
+one for data and one for code.
+The number of classes and number of valid bits in a COS bitmask is CPU model
+specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
+``cbm`` or capacity bitmask.
+By default, after reset, all CPU cores are assigned to COS 0 and all classes
+are programmed to allow fill into all cache ways.
+CDP is off by default.
+
+For more information about CAT please see https://github.com/01org/intel-cmt-cat
+
+
+Compiling the Application
+-------------------------
+
+Requires ``libpqos`` from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
+hosted on GitHub repository. For installation notes, please see ``README`` file.
+
+GIT:
+
+* https://github.com/01org/intel-cmt-cat
+
+To compile the application export the path to PQoS lib (needed only if libpqos
+is not installed in the default location (/usr/local))
+and the DPDK source tree and go to the example directory:
+
+.. code-block:: console
+
+ export PQOS_INSTALL_PATH=/path/to/libpqos
+ export RTE_SDK=/path/to/rte_sdk
+
+ cd ${RTE_SDK}/examples/skeleton-cat
+
+Set the target, for example:
+
+.. code-block:: console
+
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
+
+Build the application as follows:
+
+.. code-block:: console
+
+ make
+
+
+Running the Application
+-----------------------
+
+To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=0x3@(0-2)
+
+or to enable CAT and CDP on cpus 1,3:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=(0x00C00,0x00300)@(1,3)
+
+The option to enable CAT is:
+
+* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
+
+ where ``cbm`` stands for capacity bitmask and must be expressed in
+ hexadecimal form.
+
+ ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
+ masks (``code_cbm`` and ``data_cbm``) is used.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ ``cpus`` could be a single digit/range or a group and must be expressed in
+ decimal form.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
+
+ * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
+
+ * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
+
+ * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
+
+ * cpu 7 has exclusive access to all of its 4 ways;
+
+ e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
+
+ * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
+ data ways are not overlapping.
+
+
+Refer to *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+
+To reset or list CAT configuration and control CDP please use ``pqos`` tool
+from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
+
+To enabled or disable CDP:
+
+.. code-block:: console
+
+ sudo ./pqos -S cdp-on
+
+ sudo ./pqos -S cdp-off
+
+to reset CAT configuration:
+
+.. code-block:: console
+
+ sudo ./pqos -R
+
+to list CAT config:
+
+.. code-block:: console
+
+ sudo ./pqos -s
+
+For more info about ``pqos`` tool please see its man page or
+`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
+
+
+Explanation
+-----------
+
+The following sections provide an explanation of the main components of the
+code.
+
+All DPDK library functions used in the sample code are prefixed with ``rte_``
+and are explained in detail in the *DPDK API Documentation*.
+
+
+The Main Function
+~~~~~~~~~~~~~~~~~
+
+The ``main()`` function performs the initialization and calls the execution
+threads for each lcore.
+
+The first task is to initialize the Environment Abstraction Layer (EAL). The
+``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+The next task is to initialize the PQoS library and configure CAT. The
+``argc`` and ``argv`` arguments are provided to the ``cat_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+``cat_init()`` is a wrapper function which parses the command, validates
+the requested parameters and configures CAT accordingly.
+
+Parsing of command line arguments is done in ``parse_args(...)``.
+libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
+queried for system CPU information and L3CA capabilities via
+``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
+calls. When all capability and topology information is collected, the requested
+CAT configuration is validated. A check is then performed (on per socket basis)
+for a sufficient number of unassociated COS. COS are selected and
+configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
+relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
+a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
+COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers)
+used by the application:
+
+.. code-block:: c
+
+ mbuf_pool = rte_mempool_create("MBUF_POOL",
+ NUM_MBUFS * nb_ports,
+ MBUF_SIZE,
+ MBUF_CACHE_SIZE,
+ sizeof(struct rte_pktmbuf_pool_private),
+ rte_pktmbuf_pool_init, NULL,
+ rte_pktmbuf_init, NULL,
+ rte_socket_id(),
+ 0);
+
+Mbufs are the packet buffer structures used by DPDK. They are explained in
+detail in the "Mbuf Library" section of the *DPDK Programmer's Guide*.
+
+The ``main()`` function also initializes all the ports using the user defined
+``port_init()`` function which is explained in the next section:
+
+.. code-block:: c
+
+ for (portid = 0; portid < nb_ports; portid++) {
+ if (port_init(portid, mbuf_pool) != 0) {
+ rte_exit(EXIT_FAILURE,
+ "Cannot init port %" PRIu8 "\n", portid);
+ }
+ }
+
+
+Once the initialization is complete, the application is ready to launch a
+function on an lcore. In this example ``lcore_main()`` is called on a single
+lcore.
+
+
+.. code-block:: c
+
+ lcore_main();
+
+The ``lcore_main()`` function is explained below.
+
+
+
+The Port Initialization Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The main functional part of the port initialization used in the Basic
+Forwarding application is shown below:
+
+.. code-block:: c
+
+ static inline int
+ port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+ {
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ struct ether_addr addr;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+ }
+
+The Ethernet ports are configured with default settings using the
+``rte_eth_dev_configure()`` function and the ``port_conf_default`` struct:
+
+.. code-block:: c
+
+ static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+ };
+
+For this example the ports are set up with 1 RX and 1 TX queue using the
+``rte_eth_rx_queue_setup()`` and ``rte_eth_tx_queue_setup()`` functions.
+
+The Ethernet port is then started:
+
+.. code-block:: c
+
+ retval = rte_eth_dev_start(port);
+
+
+Finally the RX port is set in promiscuous mode:
+
+.. code-block:: c
+
+ rte_eth_promiscuous_enable(port);
+
+
+The Lcores Main
+~~~~~~~~~~~~~~~
+
+As we saw above the ``main()`` function calls an application function on the
+available lcores. For the Basic Forwarding application the lcore function
+looks like the following:
+
+.. code-block:: c
+
+ static __attribute__((noreturn)) void
+ lcore_main(void)
+ {
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+ }
+
+
+The main work of the application is done within the loop:
+
+.. code-block:: c
+
+ for (;;) {
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+
+Packets are received in bursts on the RX ports and transmitted in bursts on
+the TX ports. The ports are grouped in pairs with a simple mapping scheme
+using the an XOR on the port number::
+
+ 0 -> 1
+ 1 -> 0
+
+ 2 -> 3
+ 3 -> 2
+
+ etc.
+
+The ``rte_eth_tx_burst()`` function frees the memory buffers of packets that
+are transmitted. If packets fail to transmit, ``(nb_tx < nb_rx)``, then they
+must be freed explicitly using ``rte_pktmbuf_free()``.
+
+The forwarding loop can be interrupted and the application closed using
+``Ctrl-C``.
diff --git a/examples/Makefile b/examples/Makefile
index 1665df1..10168a9 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -74,6 +74,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched
DIRS-y += quota_watermark
DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks
DIRS-y += skeleton
+DIRS-y += skeleton-cat
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost
diff --git a/examples/skeleton-cat/Makefile b/examples/skeleton-cat/Makefile
new file mode 100644
index 0000000..e3ef642
--- /dev/null
+++ b/examples/skeleton-cat/Makefile
@@ -0,0 +1,68 @@
+# BSD LICENSE
+#
+# Copyright(c) 2016 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Default location of PQoS library and includes,
+# can be overridden by command line or environment
+PQOS_INSTALL_PATH ?= /usr/local/
+PQOS_LIBRARY_PATH = $(PQOS_INSTALL_PATH)/lib/libpqos.a
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = basicfwd-cat
+
+# all source are stored in SRCS-y
+SRCS-y := basicfwd-cat.c cat.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+EXTRA_CFLAGS += -O3 -g -Wfatal-errors
+
+CFLAGS += -I$(PQOS_INSTALL_PATH)/include
+CFLAGS_cat.o := -D_GNU_SOURCE
+
+LDLIBS += -L$(PQOS_INSTALL_PATH)/lib
+LDLIBS += $(PQOS_LIBRARY_PATH) \
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/skeleton-cat/basicfwd-cat.c b/examples/skeleton-cat/basicfwd-cat.c
new file mode 100644
index 0000000..cb68bef
--- /dev/null
+++ b/examples/skeleton-cat/basicfwd-cat.c
@@ -0,0 +1,220 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_mbuf.h>
+
+#include "cat.h"
+
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+#define BURST_SIZE 32
+
+static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+};
+
+/* basicfwd-cat.c: CAT enabled, basic DPDK skeleton forwarding example. */
+
+/*
+ * Initializes a given port using global settings and with the RX buffers
+ * coming from the mbuf_pool passed as a parameter.
+ */
+static inline int
+port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+{
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+}
+
+/*
+ * The lcore main. This is the main thread that does the work, reading from
+ * an input port and writing to an output port.
+ */
+static __attribute__((noreturn)) void
+lcore_main(void)
+{
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+}
+
+/*
+ * The main function, which does initialization and calls the per-lcore
+ * functions.
+ */
+int
+main(int argc, char *argv[])
+{
+ struct rte_mempool *mbuf_pool;
+ unsigned nb_ports;
+ uint8_t portid;
+
+ /* Initialize the Environment Abstraction Layer (EAL). */
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+ argc -= ret;
+ argv += ret;
+
+ ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /* Check that there is an even number of ports to send/receive on. */
+ nb_ports = rte_eth_dev_count();
+ if (nb_ports < 2 || (nb_ports & 1))
+ rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /* Initialize all ports. */
+ for (portid = 0; portid < nb_ports; portid++)
+ if (port_init(portid, mbuf_pool) != 0)
+ rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+ portid);
+
+ if (rte_lcore_count() > 1)
+ printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
+
+ /* Call lcore_main on the master core only. */
+ lcore_main();
+
+ return 0;
+}
diff --git a/examples/skeleton-cat/cat.c b/examples/skeleton-cat/cat.c
new file mode 100644
index 0000000..f594b4d
--- /dev/null
+++ b/examples/skeleton-cat/cat.c
@@ -0,0 +1,992 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <inttypes.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_memcpy.h>
+
+#include <pqos.h>
+
+#include "cat.h"
+
+#define BITS_PER_HEX 4
+#define PQOS_MAX_SOCKETS 8
+#define PQOS_MAX_SOCKET_CORES 64
+#define PQOS_MAX_CORES (PQOS_MAX_SOCKET_CORES * PQOS_MAX_SOCKETS)
+
+static const struct pqos_cap *m_cap;
+static const struct pqos_cpuinfo *m_cpu;
+static const struct pqos_capability *m_cap_l3ca;
+static unsigned m_sockets[PQOS_MAX_SOCKETS];
+static unsigned m_sock_count;
+static struct cat_config m_config[PQOS_MAX_CORES];
+static unsigned m_config_count;
+
+static unsigned
+bits_count(uint64_t bitmask)
+{
+ unsigned count = 0;
+
+ for (; bitmask != 0; count++)
+ bitmask &= bitmask - 1;
+
+ return count;
+}
+
+/*
+ * Parse elem, the elem could be single number/range or '(' ')' group
+ * 1) A single number elem, it's just a simple digit. e.g. 9
+ * 2) A single range elem, two digits with a '-' between. e.g. 2-6
+ * 3) A group elem, combines multiple 1) or 2) with '( )'. e.g (0,2-4,6)
+ * Within group elem, '-' used for a range separator;
+ * ',' used for a single number.
+ */
+static int
+parse_set(const char *input, rte_cpuset_t *cpusetp)
+{
+ unsigned idx;
+ const char *str = input;
+ char *end = NULL;
+ unsigned min, max;
+ const unsigned num = PQOS_MAX_CORES;
+
+ CPU_ZERO(cpusetp);
+
+ while (isblank(*str))
+ str++;
+
+ /* only digit or left bracket is qualify for start point */
+ if ((!isdigit(*str) && *str != '(') || *str == '\0')
+ return -1;
+
+ /* process single number or single range of number */
+ if (*str != '(') {
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ while (isblank(*end))
+ end++;
+
+ min = idx;
+ max = idx;
+ if (*end == '-') {
+ /* process single <number>-<number> */
+ end++;
+ while (isblank(*end))
+ end++;
+ if (!isdigit(*end))
+ return -1;
+
+ errno = 0;
+ idx = strtoul(end, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+ max = idx;
+ while (isblank(*end))
+ end++;
+ if (*end != ',' && *end != '\0')
+ return -1;
+ }
+
+ if (*end != ',' && *end != '\0' && *end != '@')
+ return -1;
+
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ return end - input;
+ }
+
+ /* process set within bracket */
+ str++;
+ while (isblank(*str))
+ str++;
+ if (*str == '\0')
+ return -1;
+
+ min = PQOS_MAX_CORES;
+ do {
+
+ /* go ahead to the first digit */
+ while (isblank(*str))
+ str++;
+ if (!isdigit(*str))
+ return -1;
+
+ /* get the digit value */
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ /* go ahead to separator '-',',' and ')' */
+ while (isblank(*end))
+ end++;
+ if (*end == '-') {
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ else /* avoid continuous '-' */
+ return -1;
+ } else if ((*end == ',') || (*end == ')')) {
+ max = idx;
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ min = PQOS_MAX_CORES;
+ } else
+ return -1;
+
+ str = end + 1;
+ } while (*end != '\0' && *end != ')');
+
+ return str - input;
+}
+
+/* Test if bitmask is contiguous */
+static int
+is_contiguous(uint64_t bitmask)
+{
+ /* check if bitmask is contiguous */
+ unsigned i = 0;
+ unsigned j = 0;
+ const unsigned max_idx = (sizeof(bitmask) * CHAR_BIT);
+
+ if (bitmask == 0)
+ return 0;
+
+ for (i = 0; i < max_idx; i++) {
+ if (((1ULL << i) & bitmask) != 0)
+ j++;
+ else if (j > 0)
+ break;
+ }
+
+ if (bits_count(bitmask) != j) {
+ printf("PQOS: mask 0x%llx is not contiguous.\n",
+ (unsigned long long)bitmask);
+ return 0;
+ }
+
+ return 1;
+}
+
+/*
+ * The format pattern: --l3ca='<cbm@cpus>[,<(ccbm,dcbm)@cpus>...]'
+ * cbm could be a single mask or for a CDP enabled system, a group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - CPUs 1 and 3 share its 4 ways with CPUs 4, 5 and 6;
+ * - CPUs 4,5 and 6 share half (4 out of 8 ways) of its L3 with 1 and 3;
+ * - CPUs 4,5 and 6 have exclusive access to 4 out of 8 ways;
+ * - CPU 7 has exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for a CDP enabled system
+ * - cpus 1 and 3 have access to 2 ways for code and 2 ways for data,
+ * code and data ways are not overlapping.;
+ */
+static int
+parse_l3ca(const char *l3ca)
+{
+ unsigned idx = 0;
+ const char *cbm_start = NULL;
+ char *cbm_end = NULL;
+ const char *end = NULL;
+ int offset;
+ rte_cpuset_t cpuset;
+ uint64_t mask = 0;
+ uint64_t cmask = 0;
+
+ if (l3ca == NULL)
+ goto err;
+
+ /* Get cbm */
+ do {
+ CPU_ZERO(&cpuset);
+ mask = 0;
+ cmask = 0;
+
+ while (isblank(*l3ca))
+ l3ca++;
+
+ if (*l3ca == '\0')
+ goto err;
+
+ /* record mask_set start point */
+ cbm_start = l3ca;
+
+ /* go across a complete bracket */
+ if (*cbm_start == '(') {
+ l3ca += strcspn(l3ca, ")");
+ if (*l3ca++ == '\0')
+ goto err;
+ }
+
+ /* scan the separator '@', ','(next) or '\0'(finish) */
+ l3ca += strcspn(l3ca, "@,");
+
+ if (*l3ca == '@') {
+ /* explicit assign cpu_set */
+ offset = parse_set(l3ca + 1, &cpuset);
+ if (offset < 0 || CPU_COUNT(&cpuset) == 0)
+ goto err;
+
+ end = l3ca + 1 + offset;
+ } else
+ goto err;
+
+ if (*end != ',' && *end != '\0')
+ goto err;
+
+ /* parse mask_set from start point */
+ if (*cbm_start == '(') {
+ cbm_start++;
+
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ cmask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || cmask == 0)
+ goto err;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (*cbm_end != ',')
+ goto err;
+
+ cbm_end++;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (!isxdigit(*cbm_end))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_end, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+ } else {
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+
+ }
+
+ if (mask == 0 || is_contiguous(mask) == 0)
+ goto err;
+
+ if (cmask != 0 && is_contiguous(cmask) == 0)
+ goto err;
+
+ rte_memcpy(&m_config[idx].cpumask,
+ &cpuset, sizeof(rte_cpuset_t));
+
+ if (cmask != 0) {
+ m_config[idx].cdp = 1;
+ m_config[idx].code_mask = cmask;
+ m_config[idx].data_mask = mask;
+ } else
+ m_config[idx].mask = mask;
+
+ m_config_count++;
+
+ l3ca = end + 1;
+ idx++;
+ } while (*end != '\0' && idx < PQOS_MAX_CORES);
+
+ if (m_config_count == 0)
+ goto err;
+
+ return 0;
+
+err:
+ return -EINVAL;
+}
+
+static int
+check_cpus_overlapping(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ rte_cpuset_t mask;
+
+ CPU_ZERO(&mask);
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = i + 1; j < m_config_count; j++) {
+ CPU_AND(&mask,
+ &m_config[i].cpumask,
+ &m_config[j].cpumask);
+
+ if (CPU_COUNT(&mask) != 0) {
+ printf("PQOS: Requested CPUs sets are "
+ "overlapping.\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cpus(void)
+{
+ unsigned i = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (cpu_id = 0; cpu_id < PQOS_MAX_CORES; cpu_id++) {
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) != 0) {
+
+ ret = pqos_cpu_check_core(m_cpu, cpu_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: %u is not a valid "
+ "logical core id.\n", cpu_id);
+ ret = -ENODEV;
+ goto exit;
+ }
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS "
+ "associated to cpu %u.\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Check if COS assigned to lcore is different
+ * then default one (#0) */
+ if (cos_id != 0) {
+ printf("PQOS: cpu %u has already "
+ "associated COS#%u. "
+ "Please reset L3CA.\n",
+ cpu_id, cos_id);
+ ret = -EBUSY;
+ goto exit;
+ }
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+static int
+check_cdp(void)
+{
+ unsigned i = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1 && m_cap_l3ca->u.l3ca->cdp_on == 0) {
+ if (m_cap_l3ca->u.l3ca->cdp == 0) {
+ printf("PQOS: CDP requested but not "
+ "supported.\n");
+ } else {
+ printf("PQOS: CDP requested but not enabled. "
+ "Please enable CDP.\n");
+ }
+ return -ENOTSUP;
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cbm_len_and_contention(void)
+{
+ unsigned i = 0;
+ uint64_t mask = 0;
+ const uint64_t not_cbm = (UINT64_MAX << (m_cap_l3ca->u.l3ca->num_ways));
+ const uint64_t cbm_contention_mask = m_cap_l3ca->u.l3ca->way_contention;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1)
+ mask = m_config[i].code_mask | m_config[i].data_mask;
+ else
+ mask = m_config[i].mask;
+
+ if ((mask & not_cbm) != 0) {
+ printf("PQOS: One or more of requested CBM masks not "
+ "supported by system (too long).\n");
+ ret = -ENOTSUP;
+ break;
+ }
+
+ /* Just a warning */
+ if ((mask & cbm_contention_mask) != 0) {
+ printf("PQOS: One or more of requested CBM masks "
+ "overlap CBM contention mask.\n");
+ break;
+ }
+
+ }
+
+ return ret;
+}
+
+static int
+check_and_select_classes(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned phy_pkg_id = 0;
+ unsigned cos_id = 0;
+ unsigned cpu_id = 0;
+ unsigned phy_pkg_lcores[PQOS_MAX_SOCKETS][m_config_count];
+ const unsigned cos_num = m_cap_l3ca->u.l3ca->num_classes;
+ unsigned used_cos_table[PQOS_MAX_SOCKETS][cos_num];
+ int ret = 0;
+
+ memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+ memset(used_cos_table, 0, sizeof(used_cos_table));
+
+ /* detect currently used COS */
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS associated to "
+ "cpu %u on phy_pkg %u.\n", cpu_id, phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for cpu %u\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Mark COS as used */
+ if (used_cos_table[phy_pkg_id][cos_id] == 0)
+ used_cos_table[phy_pkg_id][cos_id]++;
+ }
+
+ /* look for avail. COS to fulfill requested config */
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Check if we already have COS selected
+ * to be used for that group on that socket */
+ if (phy_pkg_lcores[phy_pkg_id][i] != 0)
+ continue;
+
+ phy_pkg_lcores[phy_pkg_id][i]++;
+
+ /* Search for avail. COS to be used on that socket */
+ for (cos_id = 0; cos_id < cos_num; cos_id++) {
+ if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+ used_cos_table[phy_pkg_id][cos_id]++;
+ cos_id_map[i][phy_pkg_id] = cos_id;
+ break;
+ }
+ }
+
+ /* If there is no COS available ...*/
+ if (cos_id == cos_num) {
+ ret = -E2BIG;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ if (ret != 0)
+ printf("PQOS: Not enough available COS to configure "
+ "requested configuration.\n");
+
+ return ret;
+}
+
+static int
+configure_cat(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned phy_pkg_id = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ unsigned i = 0;
+ unsigned j = 0;
+ struct pqos_l3ca l3ca = {0};
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ memset(&l3ca, 0, sizeof(l3ca));
+
+ l3ca.cdp = m_config[i].cdp;
+ if (m_config[i].cdp == 1) {
+ l3ca.code_mask = m_config[i].code_mask;
+ l3ca.data_mask = m_config[i].data_mask;
+ } else
+ l3ca.ways_mask = m_config[i].mask;
+
+ for (j = 0; j < m_sock_count; j++) {
+ phy_pkg_id = m_sockets[j];
+ if (cos_id_map[i][phy_pkg_id] == 0)
+ continue;
+
+ l3ca.class_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to set COS %u on "
+ "phy_pkg %u.\n", l3ca.class_id,
+ phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ cos_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_assoc_set(cpu_id, cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS %u to "
+ "cpu %u\n", cos_id, cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ int opt = 0;
+ int retval = 0;
+ int oldopterr = 0;
+ char **argvopt = argv;
+ char *prgname = argv[0];
+
+ static struct option lgopts[] = {
+ { "l3ca", required_argument, 0, 0 },
+ { NULL, 0, 0, 0 }
+ };
+
+ /* Disable printing messages within getopt() */
+ oldopterr = opterr;
+ opterr = 0;
+
+ opt = getopt_long(argc, argvopt, "", lgopts, NULL);
+ if (opt == 0) {
+ retval = parse_l3ca(optarg);
+ if (retval != 0) {
+ printf("PQOS: Invalid L3CA parameters!\n");
+ goto exit;
+ }
+
+ argv[optind - 1] = prgname;
+ retval = optind - 1;
+ } else
+ retval = 0;
+
+exit:
+ /* reset getopt lib */
+ optind = 0;
+
+ /* Restore opterr value */
+ opterr = oldopterr;
+
+ return retval;
+}
+
+static void
+print_cmd_line_config(void)
+{
+ char cpustr[PQOS_MAX_CORES * 3] = {0};
+ unsigned i = 0;
+ unsigned j = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ unsigned len = 0;
+ memset(cpustr, 0, sizeof(cpustr));
+
+ /* Generate CPU list */
+ for (j = 0; j < PQOS_MAX_CORES; j++) {
+ if (CPU_ISSET(j, &m_config[i].cpumask) != 1)
+ continue;
+
+ len += snprintf(cpustr + len, sizeof(cpustr) - len - 1,
+ "%u,", j);
+
+ if (len >= sizeof(cpustr) - 1)
+ break;
+ }
+
+ if (m_config[i].cdp == 1) {
+ printf("PQOS: CPUs: %s cMASK: 0x%llx, dMASK: "
+ "0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].code_mask,
+ (unsigned long long)m_config[i].data_mask);
+ } else {
+ printf("PQOS: CPUs: %s MASK: 0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].mask);
+ }
+ }
+}
+
+/**
+ * @brief Prints CAT configuration
+ */
+static void
+print_cat_config(void)
+{
+ int ret = PQOS_RETVAL_OK;
+ unsigned i = 0;
+
+ for (i = 0; i < m_sock_count; i++) {
+ struct pqos_l3ca tab[PQOS_MAX_L3CA_COS] = {{0} };
+ unsigned num = 0;
+ unsigned n = 0;
+
+ ret = pqos_l3ca_get(m_sockets[i], PQOS_MAX_L3CA_COS, &num, tab);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving COS!\n");
+ return;
+ }
+
+ printf("PQOS: COS definitions for Socket %u:\n", m_sockets[i]);
+ for (n = 0; n < num; n++) {
+ if (tab[n].cdp == 1) {
+ printf("PQOS: COS: %u, cMASK: 0x%llx, "
+ "dMASK: 0x%llx\n", tab[n].class_id,
+ (unsigned long long)tab[n].code_mask,
+ (unsigned long long)tab[n].data_mask);
+ } else {
+ printf("PQOS: COS: %u, MASK: 0x%llx\n",
+ tab[n].class_id,
+ (unsigned long long)tab[n].ways_mask);
+ }
+ }
+ }
+
+ for (i = 0; i < m_sock_count; i++) {
+ unsigned lcores[PQOS_MAX_SOCKET_CORES] = {0};
+ unsigned lcount = 0;
+ unsigned n = 0;
+
+ ret = pqos_cpu_get_cores(m_cpu, m_sockets[i],
+ PQOS_MAX_SOCKET_CORES, &lcount, &lcores[0]);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving core information!\n");
+ return;
+ }
+
+ printf("PQOS: CPU information for socket %u:\n", m_sockets[i]);
+ for (n = 0; n < lcount; n++) {
+ unsigned class_id = 0;
+
+ ret = pqos_l3ca_assoc_get(lcores[n], &class_id);
+ if (ret == PQOS_RETVAL_OK)
+ printf("PQOS: CPU: %u, COS: %u\n", lcores[n],
+ class_id);
+ else
+ printf("PQOS: CPU: %u, ERROR\n", lcores[n]);
+ }
+ }
+
+}
+
+static int
+cat_validate(void)
+{
+ int ret = 0;
+
+ ret = check_cpus();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cdp();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cbm_len_and_contention();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cpus_overlapping();
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static int
+cat_set(void)
+{
+ int ret = 0;
+ unsigned cos_id_map[m_config_count][PQOS_MAX_SOCKETS];
+
+ memset(cos_id_map, 0, sizeof(cos_id_map));
+
+ ret = check_and_select_classes(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ ret = configure_cat(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+cat_fini(void)
+{
+ int ret = 0;
+
+ printf("PQOS: Shutting down PQoS library...\n");
+
+ /* deallocate all the resources */
+ ret = pqos_fini();
+ if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_INIT)
+ printf("PQOS: Error shutting down PQoS library!\n");
+
+ m_cap = NULL;
+ m_cpu = NULL;
+ m_cap_l3ca = NULL;
+ memset(m_sockets, 0, sizeof(m_sockets));
+ m_sock_count = 0;
+ memset(m_config, 0, sizeof(m_config));
+ m_config_count = 0;
+}
+
+void
+cat_exit(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned cpu_id = 0;
+ int ret = 0;
+
+ /* if lib is not initialized, do nothing */
+ if (m_cap == NULL && m_cpu == NULL)
+ return;
+
+ printf("PQOS: Reverting CAT configuration...\n");
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_l3ca_assoc_set(cpu_id, 0);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS 0 to "
+ "cpu %u\n", cpu_id);
+ }
+ }
+ }
+
+ cat_fini();
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\nPQOS: Signal %d received, preparing to exit...\n",
+ signum);
+
+ cat_exit();
+
+ /* exit with the expected status */
+ signal(signum, SIG_DFL);
+ kill(getpid(), signum);
+ }
+}
+
+int
+cat_init(int argc, char **argv)
+{
+ int ret = 0;
+ int args_num = 0;
+ struct pqos_config cfg = {0};
+
+ if (m_cap != NULL || m_cpu != NULL) {
+ printf("PQOS: CAT module already initialized!\n");
+ return -EEXIST;
+ }
+
+ /* Parse cmd line args */
+ ret = parse_args(argc, argv);
+
+ if (ret <= 0)
+ goto err;
+
+ args_num = ret;
+
+ /* Print cmd line configuration */
+ print_cmd_line_config();
+
+ /* PQoS Initialization - Check and initialize CAT capability */
+ cfg.fd_log = STDOUT_FILENO;
+ cfg.verbose = 0;
+ cfg.cdp_cfg = PQOS_REQUIRE_CDP_ANY;
+ ret = pqos_init(&cfg);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error initializing PQoS library!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get capability and CPU info pointer */
+ ret = pqos_cap_get(&m_cap, &m_cpu);
+ if (ret != PQOS_RETVAL_OK || m_cap == NULL || m_cpu == NULL) {
+ printf("PQOS: Error retrieving PQoS capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get L3CA capabilities */
+ ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &m_cap_l3ca);
+ if (ret != PQOS_RETVAL_OK || m_cap_l3ca == NULL) {
+ printf("PQOS: Error retrieving PQOS_CAP_TYPE_L3CA "
+ "capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get CPU socket information */
+ ret = pqos_cpu_get_sockets(m_cpu, PQOS_MAX_SOCKETS, &m_sock_count,
+ m_sockets);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving CPU socket information!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Validate cmd line configuration */
+ ret = cat_validate();
+ if (ret != 0) {
+ printf("PQOS: Requested CAT configuration is not valid!\n");
+ goto err;
+ }
+
+ /* configure system */
+ ret = cat_set();
+ if (ret != 0) {
+ printf("PQOS: Failed to configure CAT!\n");
+ goto err;
+ }
+
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ ret = atexit(cat_exit);
+ if (ret != 0) {
+ printf("PQOS: Cannot set exit function\n");
+ goto err;
+ }
+
+ /* Print CAT configuration */
+ print_cat_config();
+
+ return args_num;
+
+err:
+ /* deallocate all the resources */
+ cat_fini();
+ return ret;
+}
diff --git a/examples/skeleton-cat/cat.h b/examples/skeleton-cat/cat.h
new file mode 100644
index 0000000..aef2b76
--- /dev/null
+++ b/examples/skeleton-cat/cat.h
@@ -0,0 +1,72 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _CAT_H
+#define _CAT_H
+
+/**
+ * @file
+ * PQoS CAT
+ */
+
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_lcore.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* L3 cache allocation class of service data structure */
+struct cat_config {
+ rte_cpuset_t cpumask; /* CPUs bitmask */
+ int cdp; /* data & code masks used if true */
+ union {
+ uint64_t mask; /* capacity bitmask (CBM) */
+ struct {
+ uint64_t data_mask; /* data capacity bitmask (CBM) */
+ uint64_t code_mask; /* code capacity bitmask (CBM) */
+ };
+ };
+};
+
+int cat_init(int argc, char **argv);
+
+void cat_exit(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _CAT_H */
--
1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v2] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
@ 2016-03-07 14:34 ` Van Haaren, Harry
2016-03-07 16:54 ` [dpdk-dev] [PATCH v3] " Wojciech Andralojc
` (2 subsequent siblings)
3 siblings, 0 replies; 14+ messages in thread
From: Van Haaren, Harry @ 2016-03-07 14:34 UTC (permalink / raw)
To: Andralojc, WojciechX, dev; +Cc: Kantecki, Tomasz, Cornu, Marcel D
Hi All,
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wojciech Andralojc
> Subject: [dpdk-dev] [PATCH v2] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos
> usage
Some minor comments/suggested-fixes in-line.
I've snipped away lots of code, below only that relevant to my comments.
> diff --git a/doc/guides/sample_app_ug/skeleton-cat.rst
> b/doc/guides/sample_app_ug/skeleton-cat.rst
> +CAT and CDP features allow management of the CPU's last level cache.
> +CAT introduces classes of service (COS) that are essentially bitmasks.
CDP is not explained anywhere - add (Cache Data Partitioning) after it here,
or write it and abbreviate as COS is.
> +Running the Application
> +-----------------------
> +
> +To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
> +
> +.. code-block:: console
> +
> + ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=0x3@(0-2)
This doesn't work on my machine:
- Bash complains about syntax error near (
Quotes around it fixes --l3ca=0x3@"(0-2)"
- the -c 2 will enable core represented by bitmask 0x2, so only the 2nd core.
Is this intended?
> +or to enable CAT and CDP on cpus 1,3:
> +
> +.. code-block:: console
> +
> + ./build/basicfwd-cat -c 2 -n 4 -- --l3ca=(0x00C00,0x00300)@(1,3)
Add a note here that when running on a maschine that doesn't support CDP, the
"CDP requested but not supported" and "PQOS: L3CA init failed!" errors
will show up.
<snip very useful PQOS sections>
> +The ``main()`` also allocates a mempool to hold the mbufs (Message Buffers)
> +used by the application:
>From here until the end of the RST doc is duplicated from the basicfwd skeleton
app: remove all the duplicated content.
> +/*
> + * Initializes a given port using global settings and with the RX buffers
> + * coming from the mbuf_pool passed as a parameter.
> + */
> +static inline int
> +port_init(uint8_t port, struct rte_mempool *mbuf_pool)
> +{
Multi-line comment duplicated from basicfwd: can be removed.
> +/*
> + * The lcore main. This is the main thread that does the work, reading from
> + * an input port and writing to an output port.
> + */
> +static __attribute__((noreturn)) void
> +lcore_main(void)
> +{
Same.
> +/*
> + * The main function, which does initialization and calls the per-lcore
> + * functions.
> + */
> +int
> +main(int argc, char *argv[])
> +{
and same again :)
> + /* Initialize the Environment Abstraction Layer (EAL). */
> + int ret = rte_eal_init(argc, argv);
> + if (ret < 0)
> + rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
> +
> + argc -= ret;
> + argv += ret;
> +
> + ret = cat_init(argc, argv);
> + if (ret < 0)
> + rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
Add a short comment above cat_init() stating it initializes PQOS/CAT,
that the API is similar to rte_eal_init() and to refer to the
sample-app documentation for details.
Two checkpatch issues:
WARNING: Block comments use a trailing */ on a separate line
#1231: FILE: examples/skeleton-cat/cat.c:421:
+ * then default one (#0) */
WARNING: Block comments use a trailing */ on a separate line
#1360: FILE: examples/skeleton-cat/cat.c:550:
+ * to be used for that group on that socket */
Apart from these minor issues;
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
2016-03-07 14:34 ` Van Haaren, Harry
@ 2016-03-07 16:54 ` Wojciech Andralojc
2016-03-07 17:19 ` Van Haaren, Harry
2016-03-11 13:50 ` Thomas Monjalon
2016-03-11 16:24 ` [dpdk-dev] [PATCH v4] " Wojciech Andralojc
2016-03-14 13:46 ` [dpdk-dev] [PATCH v5] examples/l2fwd-cat: " Wojciech Andralojc
3 siblings, 2 replies; 14+ messages in thread
From: Wojciech Andralojc @ 2016-03-07 16:54 UTC (permalink / raw)
To: dev; +Cc: Tomasz Kantecki, Marcel D Cornu
This patch implements PQoS as an sample application.
PQoS allows management of the CPUs last level cache,
which can be useful for DPDK to ensure quality of service.
The sample app links against the existing 01.org PQoS library
(https://github.com/01org/intel-cmt-cat).
Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
---
Version 3:
* Updated rst docs
* Fixed checkpatch warnings
* Updated comments in code
Version 2:
* Added signal handlers to do clean-up on SIGINT and SIGTERM
* Clean-up function modified to zero globals (needed for testing)
* Init function modified to return more applicable errnos
Version 1:
* Initial version
Details of "--l3ca" app parameter to configure Intel CAT and CDP features:
--l3ca=bitmask@<cpu_list>
--l3ca=(code_bitmask,data_bitmask)@<cpu_list>
makes selected CPU's use specified CAT bitmasks, bitmasks must be
expressed in hexadecimal form
CAT and CDP features allow management of the CPU's last level cache.
CAT introduces classes of service (COS) that are essentially bitmasks.
In current CAT implementations, a bit in a COS bitmask corresponds to
one cache way in the last level cache.
A CPU core is always assigned to one of the CAT classes.
By programming CPU core assignment and COS bitmasks, applications can be
given exclusive, shared, or mixed access to the CPU's last level cache.
CDP extends CAT so that there are two bitmasks per COS,
one for data and one for code.
The number of classes and number of valid bits in a COS bitmask is CPU
model specific and COS bitmasks need to be contiguous. Sample code calls
this bitmask a cbm or a capacity bitmask.
By default, after reset, all CPU cores are assigned to COS 0 and all
classes are programmed to allow fill into all cache ways.
CDP is off by default.
For more information about CAT please see
https://github.com/01org/intel-cmt-cat
Known issues and limitations:
- --l3ca must be a first app parameter
---
MAINTAINERS | 4 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/skeleton-cat.rst | 241 ++++++++
examples/Makefile | 1 +
examples/skeleton-cat/Makefile | 68 ++
examples/skeleton-cat/basicfwd-cat.c | 224 +++++++
examples/skeleton-cat/cat.c | 996 ++++++++++++++++++++++++++++++
examples/skeleton-cat/cat.h | 72 +++
8 files changed, 1607 insertions(+)
create mode 100644 doc/guides/sample_app_ug/skeleton-cat.rst
create mode 100644 examples/skeleton-cat/Makefile
create mode 100644 examples/skeleton-cat/basicfwd-cat.c
create mode 100644 examples/skeleton-cat/cat.c
create mode 100644 examples/skeleton-cat/cat.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 628bc05..7a6702b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -600,3 +600,7 @@ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
M: Daniel Mrzyglod <danielx.t.mrzyglod@intel.com>
F: examples/ptpclient/
+
+M: Tomasz Kantecki <tomasz.kantecki@intel.com>
+F: examples/skeleton-cat/
+F: doc/guides/sample_app_ug/skeleton-cat.rst
\ No newline at end of file
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 8a646dd..f065e54 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -41,6 +41,7 @@ Sample Applications User Guide
exception_path
hello_world
skeleton
+ skeleton-cat
rxtx_callbacks
ip_frag
ipv4_multicast
diff --git a/doc/guides/sample_app_ug/skeleton-cat.rst b/doc/guides/sample_app_ug/skeleton-cat.rst
new file mode 100644
index 0000000..44c59f2
--- /dev/null
+++ b/doc/guides/sample_app_ug/skeleton-cat.rst
@@ -0,0 +1,241 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Cache Allocation Technology (CAT) enabled Basic Forwarding Sample Application
+=============================================================================
+
+Basic Forwarding sample application is a simple *skeleton* example of
+a forwarding application. It has been extended to make use of CAT via extended
+command line options and linking against the libpqos library.
+
+It is intended as a demonstration of the basic components of a DPDK forwarding
+application and use of the libpqos library to program CAT.
+For more detailed implementations see the L2 and L3 forwarding
+sample applications.
+
+CAT and Code Data Prioritization (CDP) features allow management of the CPU's
+last level cache. CAT introduces classes of service (COS) that are essentially
+bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to
+one cache way in last level cache.
+A CPU core is always assigned to one of the CAT classes.
+By programming CPU core assignment and COS bitmasks, applications can be given
+exclusive, shared, or mixed access to the CPU's last level cache.
+CDP extends CAT so that there are two bitmasks per COS,
+one for data and one for code.
+The number of classes and number of valid bits in a COS bitmask is CPU model
+specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
+``cbm`` or capacity bitmask.
+By default, after reset, all CPU cores are assigned to COS 0 and all classes
+are programmed to allow fill into all cache ways.
+CDP is off by default.
+
+For more information about CAT please see https://github.com/01org/intel-cmt-cat
+
+
+Compiling the Application
+-------------------------
+
+Requires ``libpqos`` from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
+hosted on GitHub repository. For installation notes, please see ``README`` file.
+
+GIT:
+
+* https://github.com/01org/intel-cmt-cat
+
+To compile the application export the path to PQoS lib (needed only if libpqos
+is not installed in the default location (/usr/local))
+and the DPDK source tree and go to the example directory:
+
+.. code-block:: console
+
+ export PQOS_INSTALL_PATH=/path/to/libpqos
+ export RTE_SDK=/path/to/rte_sdk
+
+ cd ${RTE_SDK}/examples/skeleton-cat
+
+Set the target, for example:
+
+.. code-block:: console
+
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
+
+Build the application as follows:
+
+.. code-block:: console
+
+ make
+
+
+Running the Application
+-----------------------
+
+To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)"
+
+or to enable CAT and CDP on cpus 1,3:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"
+
+If CDP is not supported it will fail with following error message:
+
+.. code-block:: console
+
+ PQOS: CDP requested but not supported.
+ PQOS: Requested CAT configuration is not valid!
+ PQOS: Shutting down PQoS library...
+ EAL: Error - exiting with code: 1
+ Cause: PQOS: L3CA init failed!
+
+The option to enable CAT is:
+
+* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
+
+ where ``cbm`` stands for capacity bitmask and must be expressed in
+ hexadecimal form.
+
+ ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
+ masks (``code_cbm`` and ``data_cbm``) is used.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ ``cpus`` could be a single digit/range or a group and must be expressed in
+ decimal form.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
+
+ * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
+
+ * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
+
+ * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
+
+ * cpu 7 has exclusive access to all of its 4 ways;
+
+ e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
+
+ * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
+ data ways are not overlapping.
+
+
+Refer to *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+
+To reset or list CAT configuration and control CDP please use ``pqos`` tool
+from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
+
+To enabled or disable CDP:
+
+.. code-block:: console
+
+ sudo ./pqos -S cdp-on
+
+ sudo ./pqos -S cdp-off
+
+to reset CAT configuration:
+
+.. code-block:: console
+
+ sudo ./pqos -R
+
+to list CAT config:
+
+.. code-block:: console
+
+ sudo ./pqos -s
+
+For more info about ``pqos`` tool please see its man page or
+`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
+
+
+Explanation
+-----------
+
+The following sections provide an explanation of the main components of the
+code.
+
+All DPDK library functions used in the sample code are prefixed with ``rte_``
+and are explained in detail in the *DPDK API Documentation*.
+
+
+The Main Function
+~~~~~~~~~~~~~~~~~
+
+The ``main()`` function performs the initialization and calls the execution
+threads for each lcore.
+
+The first task is to initialize the Environment Abstraction Layer (EAL). The
+``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+The next task is to initialize the PQoS library and configure CAT. The
+``argc`` and ``argv`` arguments are provided to the ``cat_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+``cat_init()`` is a wrapper function which parses the command, validates
+the requested parameters and configures CAT accordingly.
+
+Parsing of command line arguments is done in ``parse_args(...)``.
+libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
+queried for system CPU information and L3CA capabilities via
+``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
+calls. When all capability and topology information is collected, the requested
+CAT configuration is validated. A check is then performed (on per socket basis)
+for a sufficient number of unassociated COS. COS are selected and
+configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
+relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
+a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
+COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
\ No newline at end of file
diff --git a/examples/Makefile b/examples/Makefile
index 1665df1..10168a9 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -74,6 +74,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched
DIRS-y += quota_watermark
DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks
DIRS-y += skeleton
+DIRS-y += skeleton-cat
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost
diff --git a/examples/skeleton-cat/Makefile b/examples/skeleton-cat/Makefile
new file mode 100644
index 0000000..e3ef642
--- /dev/null
+++ b/examples/skeleton-cat/Makefile
@@ -0,0 +1,68 @@
+# BSD LICENSE
+#
+# Copyright(c) 2016 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Default location of PQoS library and includes,
+# can be overridden by command line or environment
+PQOS_INSTALL_PATH ?= /usr/local/
+PQOS_LIBRARY_PATH = $(PQOS_INSTALL_PATH)/lib/libpqos.a
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = basicfwd-cat
+
+# all source are stored in SRCS-y
+SRCS-y := basicfwd-cat.c cat.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+EXTRA_CFLAGS += -O3 -g -Wfatal-errors
+
+CFLAGS += -I$(PQOS_INSTALL_PATH)/include
+CFLAGS_cat.o := -D_GNU_SOURCE
+
+LDLIBS += -L$(PQOS_INSTALL_PATH)/lib
+LDLIBS += $(PQOS_LIBRARY_PATH) \
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/skeleton-cat/basicfwd-cat.c b/examples/skeleton-cat/basicfwd-cat.c
new file mode 100644
index 0000000..4f815b7
--- /dev/null
+++ b/examples/skeleton-cat/basicfwd-cat.c
@@ -0,0 +1,224 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_mbuf.h>
+
+#include "cat.h"
+
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+#define BURST_SIZE 32
+
+static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+};
+
+/* basicfwd-cat.c: CAT enabled, basic DPDK skeleton forwarding example. */
+
+/*
+ * Initializes a given port using global settings and with the RX buffers
+ * coming from the mbuf_pool passed as a parameter.
+ */
+static inline int
+port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+{
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+}
+
+/*
+ * The lcore main. This is the main thread that does the work, reading from
+ * an input port and writing to an output port.
+ */
+static __attribute__((noreturn)) void
+lcore_main(void)
+{
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+}
+
+/*
+ * The main function, which does initialization and calls the per-lcore
+ * functions.
+ */
+int
+main(int argc, char *argv[])
+{
+ struct rte_mempool *mbuf_pool;
+ unsigned nb_ports;
+ uint8_t portid;
+
+ /* Initialize the Environment Abstraction Layer (EAL). */
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /*
+ * Initialize the PQoS library and configure CAT.
+ * Please see skeleton-cat documentation for more info.
+ */
+ ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /* Check that there is an even number of ports to send/receive on. */
+ nb_ports = rte_eth_dev_count();
+ if (nb_ports < 2 || (nb_ports & 1))
+ rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /* Initialize all ports. */
+ for (portid = 0; portid < nb_ports; portid++)
+ if (port_init(portid, mbuf_pool) != 0)
+ rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+ portid);
+
+ if (rte_lcore_count() > 1)
+ printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
+
+ /* Call lcore_main on the master core only. */
+ lcore_main();
+
+ return 0;
+}
diff --git a/examples/skeleton-cat/cat.c b/examples/skeleton-cat/cat.c
new file mode 100644
index 0000000..bad3930
--- /dev/null
+++ b/examples/skeleton-cat/cat.c
@@ -0,0 +1,996 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <inttypes.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_memcpy.h>
+
+#include <pqos.h>
+
+#include "cat.h"
+
+#define BITS_PER_HEX 4
+#define PQOS_MAX_SOCKETS 8
+#define PQOS_MAX_SOCKET_CORES 64
+#define PQOS_MAX_CORES (PQOS_MAX_SOCKET_CORES * PQOS_MAX_SOCKETS)
+
+static const struct pqos_cap *m_cap;
+static const struct pqos_cpuinfo *m_cpu;
+static const struct pqos_capability *m_cap_l3ca;
+static unsigned m_sockets[PQOS_MAX_SOCKETS];
+static unsigned m_sock_count;
+static struct cat_config m_config[PQOS_MAX_CORES];
+static unsigned m_config_count;
+
+static unsigned
+bits_count(uint64_t bitmask)
+{
+ unsigned count = 0;
+
+ for (; bitmask != 0; count++)
+ bitmask &= bitmask - 1;
+
+ return count;
+}
+
+/*
+ * Parse elem, the elem could be single number/range or '(' ')' group
+ * 1) A single number elem, it's just a simple digit. e.g. 9
+ * 2) A single range elem, two digits with a '-' between. e.g. 2-6
+ * 3) A group elem, combines multiple 1) or 2) with '( )'. e.g (0,2-4,6)
+ * Within group elem, '-' used for a range separator;
+ * ',' used for a single number.
+ */
+static int
+parse_set(const char *input, rte_cpuset_t *cpusetp)
+{
+ unsigned idx;
+ const char *str = input;
+ char *end = NULL;
+ unsigned min, max;
+ const unsigned num = PQOS_MAX_CORES;
+
+ CPU_ZERO(cpusetp);
+
+ while (isblank(*str))
+ str++;
+
+ /* only digit or left bracket is qualify for start point */
+ if ((!isdigit(*str) && *str != '(') || *str == '\0')
+ return -1;
+
+ /* process single number or single range of number */
+ if (*str != '(') {
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ while (isblank(*end))
+ end++;
+
+ min = idx;
+ max = idx;
+ if (*end == '-') {
+ /* process single <number>-<number> */
+ end++;
+ while (isblank(*end))
+ end++;
+ if (!isdigit(*end))
+ return -1;
+
+ errno = 0;
+ idx = strtoul(end, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+ max = idx;
+ while (isblank(*end))
+ end++;
+ if (*end != ',' && *end != '\0')
+ return -1;
+ }
+
+ if (*end != ',' && *end != '\0' && *end != '@')
+ return -1;
+
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ return end - input;
+ }
+
+ /* process set within bracket */
+ str++;
+ while (isblank(*str))
+ str++;
+ if (*str == '\0')
+ return -1;
+
+ min = PQOS_MAX_CORES;
+ do {
+
+ /* go ahead to the first digit */
+ while (isblank(*str))
+ str++;
+ if (!isdigit(*str))
+ return -1;
+
+ /* get the digit value */
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ /* go ahead to separator '-',',' and ')' */
+ while (isblank(*end))
+ end++;
+ if (*end == '-') {
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ else /* avoid continuous '-' */
+ return -1;
+ } else if ((*end == ',') || (*end == ')')) {
+ max = idx;
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ min = PQOS_MAX_CORES;
+ } else
+ return -1;
+
+ str = end + 1;
+ } while (*end != '\0' && *end != ')');
+
+ return str - input;
+}
+
+/* Test if bitmask is contiguous */
+static int
+is_contiguous(uint64_t bitmask)
+{
+ /* check if bitmask is contiguous */
+ unsigned i = 0;
+ unsigned j = 0;
+ const unsigned max_idx = (sizeof(bitmask) * CHAR_BIT);
+
+ if (bitmask == 0)
+ return 0;
+
+ for (i = 0; i < max_idx; i++) {
+ if (((1ULL << i) & bitmask) != 0)
+ j++;
+ else if (j > 0)
+ break;
+ }
+
+ if (bits_count(bitmask) != j) {
+ printf("PQOS: mask 0x%llx is not contiguous.\n",
+ (unsigned long long)bitmask);
+ return 0;
+ }
+
+ return 1;
+}
+
+/*
+ * The format pattern: --l3ca='<cbm@cpus>[,<(ccbm,dcbm)@cpus>...]'
+ * cbm could be a single mask or for a CDP enabled system, a group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - CPUs 1 and 3 share its 4 ways with CPUs 4, 5 and 6;
+ * - CPUs 4,5 and 6 share half (4 out of 8 ways) of its L3 with 1 and 3;
+ * - CPUs 4,5 and 6 have exclusive access to 4 out of 8 ways;
+ * - CPU 7 has exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for a CDP enabled system
+ * - cpus 1 and 3 have access to 2 ways for code and 2 ways for data,
+ * code and data ways are not overlapping.;
+ */
+static int
+parse_l3ca(const char *l3ca)
+{
+ unsigned idx = 0;
+ const char *cbm_start = NULL;
+ char *cbm_end = NULL;
+ const char *end = NULL;
+ int offset;
+ rte_cpuset_t cpuset;
+ uint64_t mask = 0;
+ uint64_t cmask = 0;
+
+ if (l3ca == NULL)
+ goto err;
+
+ /* Get cbm */
+ do {
+ CPU_ZERO(&cpuset);
+ mask = 0;
+ cmask = 0;
+
+ while (isblank(*l3ca))
+ l3ca++;
+
+ if (*l3ca == '\0')
+ goto err;
+
+ /* record mask_set start point */
+ cbm_start = l3ca;
+
+ /* go across a complete bracket */
+ if (*cbm_start == '(') {
+ l3ca += strcspn(l3ca, ")");
+ if (*l3ca++ == '\0')
+ goto err;
+ }
+
+ /* scan the separator '@', ','(next) or '\0'(finish) */
+ l3ca += strcspn(l3ca, "@,");
+
+ if (*l3ca == '@') {
+ /* explicit assign cpu_set */
+ offset = parse_set(l3ca + 1, &cpuset);
+ if (offset < 0 || CPU_COUNT(&cpuset) == 0)
+ goto err;
+
+ end = l3ca + 1 + offset;
+ } else
+ goto err;
+
+ if (*end != ',' && *end != '\0')
+ goto err;
+
+ /* parse mask_set from start point */
+ if (*cbm_start == '(') {
+ cbm_start++;
+
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ cmask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || cmask == 0)
+ goto err;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (*cbm_end != ',')
+ goto err;
+
+ cbm_end++;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (!isxdigit(*cbm_end))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_end, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+ } else {
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+
+ }
+
+ if (mask == 0 || is_contiguous(mask) == 0)
+ goto err;
+
+ if (cmask != 0 && is_contiguous(cmask) == 0)
+ goto err;
+
+ rte_memcpy(&m_config[idx].cpumask,
+ &cpuset, sizeof(rte_cpuset_t));
+
+ if (cmask != 0) {
+ m_config[idx].cdp = 1;
+ m_config[idx].code_mask = cmask;
+ m_config[idx].data_mask = mask;
+ } else
+ m_config[idx].mask = mask;
+
+ m_config_count++;
+
+ l3ca = end + 1;
+ idx++;
+ } while (*end != '\0' && idx < PQOS_MAX_CORES);
+
+ if (m_config_count == 0)
+ goto err;
+
+ return 0;
+
+err:
+ return -EINVAL;
+}
+
+static int
+check_cpus_overlapping(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ rte_cpuset_t mask;
+
+ CPU_ZERO(&mask);
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = i + 1; j < m_config_count; j++) {
+ CPU_AND(&mask,
+ &m_config[i].cpumask,
+ &m_config[j].cpumask);
+
+ if (CPU_COUNT(&mask) != 0) {
+ printf("PQOS: Requested CPUs sets are "
+ "overlapping.\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cpus(void)
+{
+ unsigned i = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (cpu_id = 0; cpu_id < PQOS_MAX_CORES; cpu_id++) {
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) != 0) {
+
+ ret = pqos_cpu_check_core(m_cpu, cpu_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: %u is not a valid "
+ "logical core id.\n", cpu_id);
+ ret = -ENODEV;
+ goto exit;
+ }
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS "
+ "associated to cpu %u.\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if COS assigned to lcore is different
+ * then default one (#0)
+ */
+ if (cos_id != 0) {
+ printf("PQOS: cpu %u has already "
+ "associated COS#%u. "
+ "Please reset L3CA.\n",
+ cpu_id, cos_id);
+ ret = -EBUSY;
+ goto exit;
+ }
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+static int
+check_cdp(void)
+{
+ unsigned i = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1 && m_cap_l3ca->u.l3ca->cdp_on == 0) {
+ if (m_cap_l3ca->u.l3ca->cdp == 0) {
+ printf("PQOS: CDP requested but not "
+ "supported.\n");
+ } else {
+ printf("PQOS: CDP requested but not enabled. "
+ "Please enable CDP.\n");
+ }
+ return -ENOTSUP;
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cbm_len_and_contention(void)
+{
+ unsigned i = 0;
+ uint64_t mask = 0;
+ const uint64_t not_cbm = (UINT64_MAX << (m_cap_l3ca->u.l3ca->num_ways));
+ const uint64_t cbm_contention_mask = m_cap_l3ca->u.l3ca->way_contention;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1)
+ mask = m_config[i].code_mask | m_config[i].data_mask;
+ else
+ mask = m_config[i].mask;
+
+ if ((mask & not_cbm) != 0) {
+ printf("PQOS: One or more of requested CBM masks not "
+ "supported by system (too long).\n");
+ ret = -ENOTSUP;
+ break;
+ }
+
+ /* Just a warning */
+ if ((mask & cbm_contention_mask) != 0) {
+ printf("PQOS: One or more of requested CBM masks "
+ "overlap CBM contention mask.\n");
+ break;
+ }
+
+ }
+
+ return ret;
+}
+
+static int
+check_and_select_classes(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned phy_pkg_id = 0;
+ unsigned cos_id = 0;
+ unsigned cpu_id = 0;
+ unsigned phy_pkg_lcores[PQOS_MAX_SOCKETS][m_config_count];
+ const unsigned cos_num = m_cap_l3ca->u.l3ca->num_classes;
+ unsigned used_cos_table[PQOS_MAX_SOCKETS][cos_num];
+ int ret = 0;
+
+ memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+ memset(used_cos_table, 0, sizeof(used_cos_table));
+
+ /* detect currently used COS */
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS associated to "
+ "cpu %u on phy_pkg %u.\n", cpu_id, phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for cpu %u\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Mark COS as used */
+ if (used_cos_table[phy_pkg_id][cos_id] == 0)
+ used_cos_table[phy_pkg_id][cos_id]++;
+ }
+
+ /* look for avail. COS to fulfill requested config */
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if we already have COS selected
+ * to be used for that group on that socket
+ */
+ if (phy_pkg_lcores[phy_pkg_id][i] != 0)
+ continue;
+
+ phy_pkg_lcores[phy_pkg_id][i]++;
+
+ /* Search for avail. COS to be used on that socket */
+ for (cos_id = 0; cos_id < cos_num; cos_id++) {
+ if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+ used_cos_table[phy_pkg_id][cos_id]++;
+ cos_id_map[i][phy_pkg_id] = cos_id;
+ break;
+ }
+ }
+
+ /* If there is no COS available ...*/
+ if (cos_id == cos_num) {
+ ret = -E2BIG;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ if (ret != 0)
+ printf("PQOS: Not enough available COS to configure "
+ "requested configuration.\n");
+
+ return ret;
+}
+
+static int
+configure_cat(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned phy_pkg_id = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ unsigned i = 0;
+ unsigned j = 0;
+ struct pqos_l3ca l3ca = {0};
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ memset(&l3ca, 0, sizeof(l3ca));
+
+ l3ca.cdp = m_config[i].cdp;
+ if (m_config[i].cdp == 1) {
+ l3ca.code_mask = m_config[i].code_mask;
+ l3ca.data_mask = m_config[i].data_mask;
+ } else
+ l3ca.ways_mask = m_config[i].mask;
+
+ for (j = 0; j < m_sock_count; j++) {
+ phy_pkg_id = m_sockets[j];
+ if (cos_id_map[i][phy_pkg_id] == 0)
+ continue;
+
+ l3ca.class_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to set COS %u on "
+ "phy_pkg %u.\n", l3ca.class_id,
+ phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ cos_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_assoc_set(cpu_id, cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS %u to "
+ "cpu %u\n", cos_id, cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ int opt = 0;
+ int retval = 0;
+ int oldopterr = 0;
+ char **argvopt = argv;
+ char *prgname = argv[0];
+
+ static struct option lgopts[] = {
+ { "l3ca", required_argument, 0, 0 },
+ { NULL, 0, 0, 0 }
+ };
+
+ /* Disable printing messages within getopt() */
+ oldopterr = opterr;
+ opterr = 0;
+
+ opt = getopt_long(argc, argvopt, "", lgopts, NULL);
+ if (opt == 0) {
+ retval = parse_l3ca(optarg);
+ if (retval != 0) {
+ printf("PQOS: Invalid L3CA parameters!\n");
+ goto exit;
+ }
+
+ argv[optind - 1] = prgname;
+ retval = optind - 1;
+ } else
+ retval = 0;
+
+exit:
+ /* reset getopt lib */
+ optind = 0;
+
+ /* Restore opterr value */
+ opterr = oldopterr;
+
+ return retval;
+}
+
+static void
+print_cmd_line_config(void)
+{
+ char cpustr[PQOS_MAX_CORES * 3] = {0};
+ unsigned i = 0;
+ unsigned j = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ unsigned len = 0;
+ memset(cpustr, 0, sizeof(cpustr));
+
+ /* Generate CPU list */
+ for (j = 0; j < PQOS_MAX_CORES; j++) {
+ if (CPU_ISSET(j, &m_config[i].cpumask) != 1)
+ continue;
+
+ len += snprintf(cpustr + len, sizeof(cpustr) - len - 1,
+ "%u,", j);
+
+ if (len >= sizeof(cpustr) - 1)
+ break;
+ }
+
+ if (m_config[i].cdp == 1) {
+ printf("PQOS: CPUs: %s cMASK: 0x%llx, dMASK: "
+ "0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].code_mask,
+ (unsigned long long)m_config[i].data_mask);
+ } else {
+ printf("PQOS: CPUs: %s MASK: 0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].mask);
+ }
+ }
+}
+
+/**
+ * @brief Prints CAT configuration
+ */
+static void
+print_cat_config(void)
+{
+ int ret = PQOS_RETVAL_OK;
+ unsigned i = 0;
+
+ for (i = 0; i < m_sock_count; i++) {
+ struct pqos_l3ca tab[PQOS_MAX_L3CA_COS] = {{0} };
+ unsigned num = 0;
+ unsigned n = 0;
+
+ ret = pqos_l3ca_get(m_sockets[i], PQOS_MAX_L3CA_COS, &num, tab);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving COS!\n");
+ return;
+ }
+
+ printf("PQOS: COS definitions for Socket %u:\n", m_sockets[i]);
+ for (n = 0; n < num; n++) {
+ if (tab[n].cdp == 1) {
+ printf("PQOS: COS: %u, cMASK: 0x%llx, "
+ "dMASK: 0x%llx\n", tab[n].class_id,
+ (unsigned long long)tab[n].code_mask,
+ (unsigned long long)tab[n].data_mask);
+ } else {
+ printf("PQOS: COS: %u, MASK: 0x%llx\n",
+ tab[n].class_id,
+ (unsigned long long)tab[n].ways_mask);
+ }
+ }
+ }
+
+ for (i = 0; i < m_sock_count; i++) {
+ unsigned lcores[PQOS_MAX_SOCKET_CORES] = {0};
+ unsigned lcount = 0;
+ unsigned n = 0;
+
+ ret = pqos_cpu_get_cores(m_cpu, m_sockets[i],
+ PQOS_MAX_SOCKET_CORES, &lcount, &lcores[0]);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving core information!\n");
+ return;
+ }
+
+ printf("PQOS: CPU information for socket %u:\n", m_sockets[i]);
+ for (n = 0; n < lcount; n++) {
+ unsigned class_id = 0;
+
+ ret = pqos_l3ca_assoc_get(lcores[n], &class_id);
+ if (ret == PQOS_RETVAL_OK)
+ printf("PQOS: CPU: %u, COS: %u\n", lcores[n],
+ class_id);
+ else
+ printf("PQOS: CPU: %u, ERROR\n", lcores[n]);
+ }
+ }
+
+}
+
+static int
+cat_validate(void)
+{
+ int ret = 0;
+
+ ret = check_cpus();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cdp();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cbm_len_and_contention();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cpus_overlapping();
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static int
+cat_set(void)
+{
+ int ret = 0;
+ unsigned cos_id_map[m_config_count][PQOS_MAX_SOCKETS];
+
+ memset(cos_id_map, 0, sizeof(cos_id_map));
+
+ ret = check_and_select_classes(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ ret = configure_cat(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+cat_fini(void)
+{
+ int ret = 0;
+
+ printf("PQOS: Shutting down PQoS library...\n");
+
+ /* deallocate all the resources */
+ ret = pqos_fini();
+ if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_INIT)
+ printf("PQOS: Error shutting down PQoS library!\n");
+
+ m_cap = NULL;
+ m_cpu = NULL;
+ m_cap_l3ca = NULL;
+ memset(m_sockets, 0, sizeof(m_sockets));
+ m_sock_count = 0;
+ memset(m_config, 0, sizeof(m_config));
+ m_config_count = 0;
+}
+
+void
+cat_exit(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned cpu_id = 0;
+ int ret = 0;
+
+ /* if lib is not initialized, do nothing */
+ if (m_cap == NULL && m_cpu == NULL)
+ return;
+
+ printf("PQOS: Reverting CAT configuration...\n");
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_l3ca_assoc_set(cpu_id, 0);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS 0 to "
+ "cpu %u\n", cpu_id);
+ }
+ }
+ }
+
+ cat_fini();
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\nPQOS: Signal %d received, preparing to exit...\n",
+ signum);
+
+ cat_exit();
+
+ /* exit with the expected status */
+ signal(signum, SIG_DFL);
+ kill(getpid(), signum);
+ }
+}
+
+int
+cat_init(int argc, char **argv)
+{
+ int ret = 0;
+ int args_num = 0;
+ struct pqos_config cfg = {0};
+
+ if (m_cap != NULL || m_cpu != NULL) {
+ printf("PQOS: CAT module already initialized!\n");
+ return -EEXIST;
+ }
+
+ /* Parse cmd line args */
+ ret = parse_args(argc, argv);
+
+ if (ret <= 0)
+ goto err;
+
+ args_num = ret;
+
+ /* Print cmd line configuration */
+ print_cmd_line_config();
+
+ /* PQoS Initialization - Check and initialize CAT capability */
+ cfg.fd_log = STDOUT_FILENO;
+ cfg.verbose = 0;
+ cfg.cdp_cfg = PQOS_REQUIRE_CDP_ANY;
+ ret = pqos_init(&cfg);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error initializing PQoS library!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get capability and CPU info pointer */
+ ret = pqos_cap_get(&m_cap, &m_cpu);
+ if (ret != PQOS_RETVAL_OK || m_cap == NULL || m_cpu == NULL) {
+ printf("PQOS: Error retrieving PQoS capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get L3CA capabilities */
+ ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &m_cap_l3ca);
+ if (ret != PQOS_RETVAL_OK || m_cap_l3ca == NULL) {
+ printf("PQOS: Error retrieving PQOS_CAP_TYPE_L3CA "
+ "capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get CPU socket information */
+ ret = pqos_cpu_get_sockets(m_cpu, PQOS_MAX_SOCKETS, &m_sock_count,
+ m_sockets);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving CPU socket information!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Validate cmd line configuration */
+ ret = cat_validate();
+ if (ret != 0) {
+ printf("PQOS: Requested CAT configuration is not valid!\n");
+ goto err;
+ }
+
+ /* configure system */
+ ret = cat_set();
+ if (ret != 0) {
+ printf("PQOS: Failed to configure CAT!\n");
+ goto err;
+ }
+
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ ret = atexit(cat_exit);
+ if (ret != 0) {
+ printf("PQOS: Cannot set exit function\n");
+ goto err;
+ }
+
+ /* Print CAT configuration */
+ print_cat_config();
+
+ return args_num;
+
+err:
+ /* deallocate all the resources */
+ cat_fini();
+ return ret;
+}
diff --git a/examples/skeleton-cat/cat.h b/examples/skeleton-cat/cat.h
new file mode 100644
index 0000000..aef2b76
--- /dev/null
+++ b/examples/skeleton-cat/cat.h
@@ -0,0 +1,72 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _CAT_H
+#define _CAT_H
+
+/**
+ * @file
+ * PQoS CAT
+ */
+
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_lcore.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* L3 cache allocation class of service data structure */
+struct cat_config {
+ rte_cpuset_t cpumask; /* CPUs bitmask */
+ int cdp; /* data & code masks used if true */
+ union {
+ uint64_t mask; /* capacity bitmask (CBM) */
+ struct {
+ uint64_t data_mask; /* data capacity bitmask (CBM) */
+ uint64_t code_mask; /* code capacity bitmask (CBM) */
+ };
+ };
+};
+
+int cat_init(int argc, char **argv);
+
+void cat_exit(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _CAT_H */
--
1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-07 16:54 ` [dpdk-dev] [PATCH v3] " Wojciech Andralojc
@ 2016-03-07 17:19 ` Van Haaren, Harry
2016-03-11 13:50 ` Thomas Monjalon
1 sibling, 0 replies; 14+ messages in thread
From: Van Haaren, Harry @ 2016-03-07 17:19 UTC (permalink / raw)
To: Andralojc, WojciechX, dev; +Cc: Cornu, Marcel D, Kantecki, Tomasz
> From: Andralojc, WojciechX
> Sent: Monday, March 7, 2016 4:54 PM
> To: dev@dpdk.org
> Cc: Van Haaren, Harry <harry.van.haaren@intel.com>; Andralojc, WojciechX
> <wojciechx.andralojc@intel.com>; Kantecki, Tomasz <tomasz.kantecki@intel.com>; Cornu,
> Marcel D <marcel.d.cornu@intel.com>
> Subject: [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
>
> This patch implements PQoS as an sample application.
> PQoS allows management of the CPUs last level cache,
> which can be useful for DPDK to ensure quality of service.
> The sample app links against the existing 01.org PQoS library
> (https://github.com/01org/intel-cmt-cat).
>
> Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
> Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
> Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
Acked-by: Harry van Haaren <harry.van.haaren@intel.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-07 16:54 ` [dpdk-dev] [PATCH v3] " Wojciech Andralojc
2016-03-07 17:19 ` Van Haaren, Harry
@ 2016-03-11 13:50 ` Thomas Monjalon
2016-03-11 15:54 ` Kantecki, Tomasz
1 sibling, 1 reply; 14+ messages in thread
From: Thomas Monjalon @ 2016-03-11 13:50 UTC (permalink / raw)
To: Wojciech Andralojc; +Cc: dev, Marcel D Cornu, Tomasz Kantecki
2016-03-07 16:54, Wojciech Andralojc:
> +M: Tomasz Kantecki <tomasz.kantecki@intel.com>
> +F: examples/skeleton-cat/
> +F: doc/guides/sample_app_ug/skeleton-cat.rst
Please keep the alphabetical order.
> +Basic Forwarding sample application is a simple *skeleton* example of
> +a forwarding application. It has been extended to make use of CAT via extended
> +command line options and linking against the libpqos library.
The skeleton aims to show a minimal piece of code (but bigger than hello world).
For any other usage, I think the "l2fwd" name is more appropriate.
> +DIRS-y += skeleton-cat
It cannot be enabled everytime, because of the dependency.
We already have this problem with examples/vm_power_manager
and libvirt dependency. It must be solved by improving the build system.
In this case, we can simply use the variable PQOS_LIBRARY_PATH to
enable the build.
[...]
> +# Default location of PQoS library and includes,
> +# can be overridden by command line or environment
> +PQOS_INSTALL_PATH ?= /usr/local/
This default value is useless because we should detect the dependency
before trying to compile.
[...]
> +LDLIBS += $(PQOS_LIBRARY_PATH) \
Why this backslash?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-11 13:50 ` Thomas Monjalon
@ 2016-03-11 15:54 ` Kantecki, Tomasz
2016-03-11 16:03 ` Thomas Monjalon
0 siblings, 1 reply; 14+ messages in thread
From: Kantecki, Tomasz @ 2016-03-11 15:54 UTC (permalink / raw)
To: Thomas Monjalon, Andralojc, WojciechX; +Cc: dev, Cornu, Marcel D
> > +Basic Forwarding sample application is a simple *skeleton* example of
> > +a forwarding application. It has been extended to make use of CAT via
> > +extended command line options and linking against the libpqos library.
>
> The skeleton aims to show a minimal piece of code (but bigger than hello
> world).
> For any other usage, I think the "l2fwd" name is more appropriate.
>
> > +DIRS-y += skeleton-cat
>
We have used skeleton app as basis for this sample code. It is just to keep it as simple as possible.
Wouldn't it be confusing if we use "l2fwd" name for skeleton app functionality?
Thanks,
Tomasz
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-11 15:54 ` Kantecki, Tomasz
@ 2016-03-11 16:03 ` Thomas Monjalon
2016-03-11 16:27 ` Bruce Richardson
0 siblings, 1 reply; 14+ messages in thread
From: Thomas Monjalon @ 2016-03-11 16:03 UTC (permalink / raw)
To: Kantecki, Tomasz; +Cc: dev, Cornu, Marcel D
2016-03-11 15:54, Kantecki, Tomasz:
> > > +Basic Forwarding sample application is a simple *skeleton* example of
> > > +a forwarding application. It has been extended to make use of CAT via
> > > +extended command line options and linking against the libpqos library.
> >
> > The skeleton aims to show a minimal piece of code (but bigger than hello
> > world).
> > For any other usage, I think the "l2fwd" name is more appropriate.
> >
> > > +DIRS-y += skeleton-cat
> >
>
> We have used skeleton app as basis for this sample code. It is just to keep it as simple as possible.
> Wouldn't it be confusing if we use "l2fwd" name for skeleton app functionality?
I'm not sure.
We need more opinions.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [dpdk-dev] [PATCH v4] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
2016-03-07 14:34 ` Van Haaren, Harry
2016-03-07 16:54 ` [dpdk-dev] [PATCH v3] " Wojciech Andralojc
@ 2016-03-11 16:24 ` Wojciech Andralojc
2016-03-14 13:46 ` [dpdk-dev] [PATCH v5] examples/l2fwd-cat: " Wojciech Andralojc
3 siblings, 0 replies; 14+ messages in thread
From: Wojciech Andralojc @ 2016-03-11 16:24 UTC (permalink / raw)
To: dev; +Cc: Tomasz Kantecki, Marcel D Cornu
This patch implements PQoS as an sample application.
PQoS allows management of the CPUs last level cache,
which can be useful for DPDK to ensure quality of service.
The sample app links against the existing 01.org PQoS library
(https://github.com/01org/intel-cmt-cat).
Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
---
Version 4:
* PQOS_INSTALL_PATH must be defined to build example
* Updated example's Makefiles, MAINTAINERS and rst doc,
Version 3:
* Updated rst docs
* Fixed checkpatch warnings
* Updated comments in code
Version 2:
* Added signal handlers to do clean-up on SIGINT and SIGTERM
* Clean-up function modified to zero globals (needed for testing)
* Init function modified to return more applicable errnos
Version 1:
* Initial version
Details of "--l3ca" app parameter to configure Intel CAT and CDP features:
--l3ca=bitmask@<cpu_list>
--l3ca=(code_bitmask,data_bitmask)@<cpu_list>
makes selected CPU's use specified CAT bitmasks, bitmasks must be
expressed in hexadecimal form
CAT and CDP features allow management of the CPU's last level cache.
CAT introduces classes of service (COS) that are essentially bitmasks.
In current CAT implementations, a bit in a COS bitmask corresponds to
one cache way in the last level cache.
A CPU core is always assigned to one of the CAT classes.
By programming CPU core assignment and COS bitmasks, applications can be
given exclusive, shared, or mixed access to the CPU's last level cache.
CDP extends CAT so that there are two bitmasks per COS,
one for data and one for code.
The number of classes and number of valid bits in a COS bitmask is CPU
model specific and COS bitmasks need to be contiguous. Sample code calls
this bitmask a cbm or a capacity bitmask.
By default, after reset, all CPU cores are assigned to COS 0 and all
classes are programmed to allow fill into all cache ways.
CDP is off by default.
For more information about CAT please see
https://github.com/01org/intel-cmt-cat
Known issues and limitations:
- --l3ca must be a first app parameter
---
MAINTAINERS | 4 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/skeleton-cat.rst | 240 +++++++
examples/Makefile | 3 +
examples/skeleton-cat/Makefile | 70 +++
examples/skeleton-cat/basicfwd-cat.c | 224 +++++++
examples/skeleton-cat/cat.c | 996 ++++++++++++++++++++++++++++++
examples/skeleton-cat/cat.h | 72 +++
8 files changed, 1610 insertions(+)
create mode 100644 doc/guides/sample_app_ug/skeleton-cat.rst
create mode 100644 examples/skeleton-cat/Makefile
create mode 100644 examples/skeleton-cat/basicfwd-cat.c
create mode 100644 examples/skeleton-cat/cat.c
create mode 100644 examples/skeleton-cat/cat.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f10b26a..6aba0bf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -624,3 +624,7 @@ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
M: Daniel Mrzyglod <danielx.t.mrzyglod@intel.com>
F: examples/ptpclient/
+
+M: Tomasz Kantecki <tomasz.kantecki@intel.com>
+F: doc/guides/sample_app_ug/skeleton-cat.rst
+F: examples/skeleton-cat/
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 88375d2..01d01c8 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -41,6 +41,7 @@ Sample Applications User Guide
exception_path
hello_world
skeleton
+ skeleton-cat
rxtx_callbacks
ip_frag
ipv4_multicast
diff --git a/doc/guides/sample_app_ug/skeleton-cat.rst b/doc/guides/sample_app_ug/skeleton-cat.rst
new file mode 100644
index 0000000..ea63911
--- /dev/null
+++ b/doc/guides/sample_app_ug/skeleton-cat.rst
@@ -0,0 +1,240 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Cache Allocation Technology (CAT) enabled Basic Forwarding Sample Application
+=============================================================================
+
+Basic Forwarding sample application is a simple *skeleton* example of
+a forwarding application. It has been extended to make use of CAT via extended
+command line options and linking against the libpqos library.
+
+It is intended as a demonstration of the basic components of a DPDK forwarding
+application and use of the libpqos library to program CAT.
+For more detailed implementations see the L2 and L3 forwarding
+sample applications.
+
+CAT and Code Data Prioritization (CDP) features allow management of the CPU's
+last level cache. CAT introduces classes of service (COS) that are essentially
+bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to
+one cache way in last level cache.
+A CPU core is always assigned to one of the CAT classes.
+By programming CPU core assignment and COS bitmasks, applications can be given
+exclusive, shared, or mixed access to the CPU's last level cache.
+CDP extends CAT so that there are two bitmasks per COS,
+one for data and one for code.
+The number of classes and number of valid bits in a COS bitmask is CPU model
+specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
+``cbm`` or capacity bitmask.
+By default, after reset, all CPU cores are assigned to COS 0 and all classes
+are programmed to allow fill into all cache ways.
+CDP is off by default.
+
+For more information about CAT please see https://github.com/01org/intel-cmt-cat
+
+
+Compiling the Application
+-------------------------
+
+Requires ``libpqos`` from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
+hosted on GitHub repository. For installation notes, please see ``README`` file.
+
+GIT:
+
+* https://github.com/01org/intel-cmt-cat
+
+To compile the application export the path to PQoS lib
+and the DPDK source tree and go to the example directory:
+
+.. code-block:: console
+
+ export PQOS_INSTALL_PATH=/path/to/libpqos
+ export RTE_SDK=/path/to/rte_sdk
+
+ cd ${RTE_SDK}/examples/skeleton-cat
+
+Set the target, for example:
+
+.. code-block:: console
+
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
+
+Build the application as follows:
+
+.. code-block:: console
+
+ make
+
+
+Running the Application
+-----------------------
+
+To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)"
+
+or to enable CAT and CDP on cpus 1,3:
+
+.. code-block:: console
+
+ ./build/basicfwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"
+
+If CDP is not supported it will fail with following error message:
+
+.. code-block:: console
+
+ PQOS: CDP requested but not supported.
+ PQOS: Requested CAT configuration is not valid!
+ PQOS: Shutting down PQoS library...
+ EAL: Error - exiting with code: 1
+ Cause: PQOS: L3CA init failed!
+
+The option to enable CAT is:
+
+* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
+
+ where ``cbm`` stands for capacity bitmask and must be expressed in
+ hexadecimal form.
+
+ ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
+ masks (``code_cbm`` and ``data_cbm``) is used.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ ``cpus`` could be a single digit/range or a group and must be expressed in
+ decimal form.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
+
+ * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
+
+ * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
+
+ * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
+
+ * cpu 7 has exclusive access to all of its 4 ways;
+
+ e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
+
+ * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
+ data ways are not overlapping.
+
+
+Refer to *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+
+To reset or list CAT configuration and control CDP please use ``pqos`` tool
+from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
+
+To enabled or disable CDP:
+
+.. code-block:: console
+
+ sudo ./pqos -S cdp-on
+
+ sudo ./pqos -S cdp-off
+
+to reset CAT configuration:
+
+.. code-block:: console
+
+ sudo ./pqos -R
+
+to list CAT config:
+
+.. code-block:: console
+
+ sudo ./pqos -s
+
+For more info about ``pqos`` tool please see its man page or
+`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
+
+
+Explanation
+-----------
+
+The following sections provide an explanation of the main components of the
+code.
+
+All DPDK library functions used in the sample code are prefixed with ``rte_``
+and are explained in detail in the *DPDK API Documentation*.
+
+
+The Main Function
+~~~~~~~~~~~~~~~~~
+
+The ``main()`` function performs the initialization and calls the execution
+threads for each lcore.
+
+The first task is to initialize the Environment Abstraction Layer (EAL). The
+``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+The next task is to initialize the PQoS library and configure CAT. The
+``argc`` and ``argv`` arguments are provided to the ``cat_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+``cat_init()`` is a wrapper function which parses the command, validates
+the requested parameters and configures CAT accordingly.
+
+Parsing of command line arguments is done in ``parse_args(...)``.
+libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
+queried for system CPU information and L3CA capabilities via
+``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
+calls. When all capability and topology information is collected, the requested
+CAT configuration is validated. A check is then performed (on per socket basis)
+for a sufficient number of unassociated COS. COS are selected and
+configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
+relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
+a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
+COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
\ No newline at end of file
diff --git a/examples/Makefile b/examples/Makefile
index 4373d4d..849b916 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -78,6 +78,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_SCHED) += qos_sched
DIRS-y += quota_watermark
DIRS-$(CONFIG_RTE_ETHDEV_RXTX_CALLBACKS) += rxtx_callbacks
DIRS-y += skeleton
+ifneq ($(PQOS_INSTALL_PATH),)
+DIRS-y += skeleton-cat
+endif
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost
diff --git a/examples/skeleton-cat/Makefile b/examples/skeleton-cat/Makefile
new file mode 100644
index 0000000..2949ba0
--- /dev/null
+++ b/examples/skeleton-cat/Makefile
@@ -0,0 +1,70 @@
+# BSD LICENSE
+#
+# Copyright(c) 2016 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+ifeq ($(PQOS_INSTALL_PATH),)
+$(error "Please define PQOS_INSTALL_PATH environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Location of PQoS library and includes,
+PQOS_LIBRARY_PATH = $(PQOS_INSTALL_PATH)/libpqos.a
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = basicfwd-cat
+
+# all source are stored in SRCS-y
+SRCS-y := basicfwd-cat.c cat.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+EXTRA_CFLAGS += -O3 -g -Wfatal-errors
+
+CFLAGS += -I$(PQOS_INSTALL_PATH)/../include
+CFLAGS_cat.o := -D_GNU_SOURCE
+
+LDLIBS += -L$(PQOS_INSTALL_PATH)
+LDLIBS += $(PQOS_LIBRARY_PATH)
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/skeleton-cat/basicfwd-cat.c b/examples/skeleton-cat/basicfwd-cat.c
new file mode 100644
index 0000000..4f815b7
--- /dev/null
+++ b/examples/skeleton-cat/basicfwd-cat.c
@@ -0,0 +1,224 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_mbuf.h>
+
+#include "cat.h"
+
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+#define BURST_SIZE 32
+
+static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+};
+
+/* basicfwd-cat.c: CAT enabled, basic DPDK skeleton forwarding example. */
+
+/*
+ * Initializes a given port using global settings and with the RX buffers
+ * coming from the mbuf_pool passed as a parameter.
+ */
+static inline int
+port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+{
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+}
+
+/*
+ * The lcore main. This is the main thread that does the work, reading from
+ * an input port and writing to an output port.
+ */
+static __attribute__((noreturn)) void
+lcore_main(void)
+{
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+}
+
+/*
+ * The main function, which does initialization and calls the per-lcore
+ * functions.
+ */
+int
+main(int argc, char *argv[])
+{
+ struct rte_mempool *mbuf_pool;
+ unsigned nb_ports;
+ uint8_t portid;
+
+ /* Initialize the Environment Abstraction Layer (EAL). */
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /*
+ * Initialize the PQoS library and configure CAT.
+ * Please see skeleton-cat documentation for more info.
+ */
+ ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /* Check that there is an even number of ports to send/receive on. */
+ nb_ports = rte_eth_dev_count();
+ if (nb_ports < 2 || (nb_ports & 1))
+ rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /* Initialize all ports. */
+ for (portid = 0; portid < nb_ports; portid++)
+ if (port_init(portid, mbuf_pool) != 0)
+ rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+ portid);
+
+ if (rte_lcore_count() > 1)
+ printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
+
+ /* Call lcore_main on the master core only. */
+ lcore_main();
+
+ return 0;
+}
diff --git a/examples/skeleton-cat/cat.c b/examples/skeleton-cat/cat.c
new file mode 100644
index 0000000..bad3930
--- /dev/null
+++ b/examples/skeleton-cat/cat.c
@@ -0,0 +1,996 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <inttypes.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_memcpy.h>
+
+#include <pqos.h>
+
+#include "cat.h"
+
+#define BITS_PER_HEX 4
+#define PQOS_MAX_SOCKETS 8
+#define PQOS_MAX_SOCKET_CORES 64
+#define PQOS_MAX_CORES (PQOS_MAX_SOCKET_CORES * PQOS_MAX_SOCKETS)
+
+static const struct pqos_cap *m_cap;
+static const struct pqos_cpuinfo *m_cpu;
+static const struct pqos_capability *m_cap_l3ca;
+static unsigned m_sockets[PQOS_MAX_SOCKETS];
+static unsigned m_sock_count;
+static struct cat_config m_config[PQOS_MAX_CORES];
+static unsigned m_config_count;
+
+static unsigned
+bits_count(uint64_t bitmask)
+{
+ unsigned count = 0;
+
+ for (; bitmask != 0; count++)
+ bitmask &= bitmask - 1;
+
+ return count;
+}
+
+/*
+ * Parse elem, the elem could be single number/range or '(' ')' group
+ * 1) A single number elem, it's just a simple digit. e.g. 9
+ * 2) A single range elem, two digits with a '-' between. e.g. 2-6
+ * 3) A group elem, combines multiple 1) or 2) with '( )'. e.g (0,2-4,6)
+ * Within group elem, '-' used for a range separator;
+ * ',' used for a single number.
+ */
+static int
+parse_set(const char *input, rte_cpuset_t *cpusetp)
+{
+ unsigned idx;
+ const char *str = input;
+ char *end = NULL;
+ unsigned min, max;
+ const unsigned num = PQOS_MAX_CORES;
+
+ CPU_ZERO(cpusetp);
+
+ while (isblank(*str))
+ str++;
+
+ /* only digit or left bracket is qualify for start point */
+ if ((!isdigit(*str) && *str != '(') || *str == '\0')
+ return -1;
+
+ /* process single number or single range of number */
+ if (*str != '(') {
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ while (isblank(*end))
+ end++;
+
+ min = idx;
+ max = idx;
+ if (*end == '-') {
+ /* process single <number>-<number> */
+ end++;
+ while (isblank(*end))
+ end++;
+ if (!isdigit(*end))
+ return -1;
+
+ errno = 0;
+ idx = strtoul(end, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+ max = idx;
+ while (isblank(*end))
+ end++;
+ if (*end != ',' && *end != '\0')
+ return -1;
+ }
+
+ if (*end != ',' && *end != '\0' && *end != '@')
+ return -1;
+
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ return end - input;
+ }
+
+ /* process set within bracket */
+ str++;
+ while (isblank(*str))
+ str++;
+ if (*str == '\0')
+ return -1;
+
+ min = PQOS_MAX_CORES;
+ do {
+
+ /* go ahead to the first digit */
+ while (isblank(*str))
+ str++;
+ if (!isdigit(*str))
+ return -1;
+
+ /* get the digit value */
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ /* go ahead to separator '-',',' and ')' */
+ while (isblank(*end))
+ end++;
+ if (*end == '-') {
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ else /* avoid continuous '-' */
+ return -1;
+ } else if ((*end == ',') || (*end == ')')) {
+ max = idx;
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ min = PQOS_MAX_CORES;
+ } else
+ return -1;
+
+ str = end + 1;
+ } while (*end != '\0' && *end != ')');
+
+ return str - input;
+}
+
+/* Test if bitmask is contiguous */
+static int
+is_contiguous(uint64_t bitmask)
+{
+ /* check if bitmask is contiguous */
+ unsigned i = 0;
+ unsigned j = 0;
+ const unsigned max_idx = (sizeof(bitmask) * CHAR_BIT);
+
+ if (bitmask == 0)
+ return 0;
+
+ for (i = 0; i < max_idx; i++) {
+ if (((1ULL << i) & bitmask) != 0)
+ j++;
+ else if (j > 0)
+ break;
+ }
+
+ if (bits_count(bitmask) != j) {
+ printf("PQOS: mask 0x%llx is not contiguous.\n",
+ (unsigned long long)bitmask);
+ return 0;
+ }
+
+ return 1;
+}
+
+/*
+ * The format pattern: --l3ca='<cbm@cpus>[,<(ccbm,dcbm)@cpus>...]'
+ * cbm could be a single mask or for a CDP enabled system, a group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - CPUs 1 and 3 share its 4 ways with CPUs 4, 5 and 6;
+ * - CPUs 4,5 and 6 share half (4 out of 8 ways) of its L3 with 1 and 3;
+ * - CPUs 4,5 and 6 have exclusive access to 4 out of 8 ways;
+ * - CPU 7 has exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for a CDP enabled system
+ * - cpus 1 and 3 have access to 2 ways for code and 2 ways for data,
+ * code and data ways are not overlapping.;
+ */
+static int
+parse_l3ca(const char *l3ca)
+{
+ unsigned idx = 0;
+ const char *cbm_start = NULL;
+ char *cbm_end = NULL;
+ const char *end = NULL;
+ int offset;
+ rte_cpuset_t cpuset;
+ uint64_t mask = 0;
+ uint64_t cmask = 0;
+
+ if (l3ca == NULL)
+ goto err;
+
+ /* Get cbm */
+ do {
+ CPU_ZERO(&cpuset);
+ mask = 0;
+ cmask = 0;
+
+ while (isblank(*l3ca))
+ l3ca++;
+
+ if (*l3ca == '\0')
+ goto err;
+
+ /* record mask_set start point */
+ cbm_start = l3ca;
+
+ /* go across a complete bracket */
+ if (*cbm_start == '(') {
+ l3ca += strcspn(l3ca, ")");
+ if (*l3ca++ == '\0')
+ goto err;
+ }
+
+ /* scan the separator '@', ','(next) or '\0'(finish) */
+ l3ca += strcspn(l3ca, "@,");
+
+ if (*l3ca == '@') {
+ /* explicit assign cpu_set */
+ offset = parse_set(l3ca + 1, &cpuset);
+ if (offset < 0 || CPU_COUNT(&cpuset) == 0)
+ goto err;
+
+ end = l3ca + 1 + offset;
+ } else
+ goto err;
+
+ if (*end != ',' && *end != '\0')
+ goto err;
+
+ /* parse mask_set from start point */
+ if (*cbm_start == '(') {
+ cbm_start++;
+
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ cmask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || cmask == 0)
+ goto err;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (*cbm_end != ',')
+ goto err;
+
+ cbm_end++;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (!isxdigit(*cbm_end))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_end, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+ } else {
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+
+ }
+
+ if (mask == 0 || is_contiguous(mask) == 0)
+ goto err;
+
+ if (cmask != 0 && is_contiguous(cmask) == 0)
+ goto err;
+
+ rte_memcpy(&m_config[idx].cpumask,
+ &cpuset, sizeof(rte_cpuset_t));
+
+ if (cmask != 0) {
+ m_config[idx].cdp = 1;
+ m_config[idx].code_mask = cmask;
+ m_config[idx].data_mask = mask;
+ } else
+ m_config[idx].mask = mask;
+
+ m_config_count++;
+
+ l3ca = end + 1;
+ idx++;
+ } while (*end != '\0' && idx < PQOS_MAX_CORES);
+
+ if (m_config_count == 0)
+ goto err;
+
+ return 0;
+
+err:
+ return -EINVAL;
+}
+
+static int
+check_cpus_overlapping(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ rte_cpuset_t mask;
+
+ CPU_ZERO(&mask);
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = i + 1; j < m_config_count; j++) {
+ CPU_AND(&mask,
+ &m_config[i].cpumask,
+ &m_config[j].cpumask);
+
+ if (CPU_COUNT(&mask) != 0) {
+ printf("PQOS: Requested CPUs sets are "
+ "overlapping.\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cpus(void)
+{
+ unsigned i = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (cpu_id = 0; cpu_id < PQOS_MAX_CORES; cpu_id++) {
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) != 0) {
+
+ ret = pqos_cpu_check_core(m_cpu, cpu_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: %u is not a valid "
+ "logical core id.\n", cpu_id);
+ ret = -ENODEV;
+ goto exit;
+ }
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS "
+ "associated to cpu %u.\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if COS assigned to lcore is different
+ * then default one (#0)
+ */
+ if (cos_id != 0) {
+ printf("PQOS: cpu %u has already "
+ "associated COS#%u. "
+ "Please reset L3CA.\n",
+ cpu_id, cos_id);
+ ret = -EBUSY;
+ goto exit;
+ }
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+static int
+check_cdp(void)
+{
+ unsigned i = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1 && m_cap_l3ca->u.l3ca->cdp_on == 0) {
+ if (m_cap_l3ca->u.l3ca->cdp == 0) {
+ printf("PQOS: CDP requested but not "
+ "supported.\n");
+ } else {
+ printf("PQOS: CDP requested but not enabled. "
+ "Please enable CDP.\n");
+ }
+ return -ENOTSUP;
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cbm_len_and_contention(void)
+{
+ unsigned i = 0;
+ uint64_t mask = 0;
+ const uint64_t not_cbm = (UINT64_MAX << (m_cap_l3ca->u.l3ca->num_ways));
+ const uint64_t cbm_contention_mask = m_cap_l3ca->u.l3ca->way_contention;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1)
+ mask = m_config[i].code_mask | m_config[i].data_mask;
+ else
+ mask = m_config[i].mask;
+
+ if ((mask & not_cbm) != 0) {
+ printf("PQOS: One or more of requested CBM masks not "
+ "supported by system (too long).\n");
+ ret = -ENOTSUP;
+ break;
+ }
+
+ /* Just a warning */
+ if ((mask & cbm_contention_mask) != 0) {
+ printf("PQOS: One or more of requested CBM masks "
+ "overlap CBM contention mask.\n");
+ break;
+ }
+
+ }
+
+ return ret;
+}
+
+static int
+check_and_select_classes(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned phy_pkg_id = 0;
+ unsigned cos_id = 0;
+ unsigned cpu_id = 0;
+ unsigned phy_pkg_lcores[PQOS_MAX_SOCKETS][m_config_count];
+ const unsigned cos_num = m_cap_l3ca->u.l3ca->num_classes;
+ unsigned used_cos_table[PQOS_MAX_SOCKETS][cos_num];
+ int ret = 0;
+
+ memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+ memset(used_cos_table, 0, sizeof(used_cos_table));
+
+ /* detect currently used COS */
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS associated to "
+ "cpu %u on phy_pkg %u.\n", cpu_id, phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for cpu %u\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Mark COS as used */
+ if (used_cos_table[phy_pkg_id][cos_id] == 0)
+ used_cos_table[phy_pkg_id][cos_id]++;
+ }
+
+ /* look for avail. COS to fulfill requested config */
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if we already have COS selected
+ * to be used for that group on that socket
+ */
+ if (phy_pkg_lcores[phy_pkg_id][i] != 0)
+ continue;
+
+ phy_pkg_lcores[phy_pkg_id][i]++;
+
+ /* Search for avail. COS to be used on that socket */
+ for (cos_id = 0; cos_id < cos_num; cos_id++) {
+ if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+ used_cos_table[phy_pkg_id][cos_id]++;
+ cos_id_map[i][phy_pkg_id] = cos_id;
+ break;
+ }
+ }
+
+ /* If there is no COS available ...*/
+ if (cos_id == cos_num) {
+ ret = -E2BIG;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ if (ret != 0)
+ printf("PQOS: Not enough available COS to configure "
+ "requested configuration.\n");
+
+ return ret;
+}
+
+static int
+configure_cat(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned phy_pkg_id = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ unsigned i = 0;
+ unsigned j = 0;
+ struct pqos_l3ca l3ca = {0};
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ memset(&l3ca, 0, sizeof(l3ca));
+
+ l3ca.cdp = m_config[i].cdp;
+ if (m_config[i].cdp == 1) {
+ l3ca.code_mask = m_config[i].code_mask;
+ l3ca.data_mask = m_config[i].data_mask;
+ } else
+ l3ca.ways_mask = m_config[i].mask;
+
+ for (j = 0; j < m_sock_count; j++) {
+ phy_pkg_id = m_sockets[j];
+ if (cos_id_map[i][phy_pkg_id] == 0)
+ continue;
+
+ l3ca.class_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to set COS %u on "
+ "phy_pkg %u.\n", l3ca.class_id,
+ phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ cos_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_assoc_set(cpu_id, cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS %u to "
+ "cpu %u\n", cos_id, cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ int opt = 0;
+ int retval = 0;
+ int oldopterr = 0;
+ char **argvopt = argv;
+ char *prgname = argv[0];
+
+ static struct option lgopts[] = {
+ { "l3ca", required_argument, 0, 0 },
+ { NULL, 0, 0, 0 }
+ };
+
+ /* Disable printing messages within getopt() */
+ oldopterr = opterr;
+ opterr = 0;
+
+ opt = getopt_long(argc, argvopt, "", lgopts, NULL);
+ if (opt == 0) {
+ retval = parse_l3ca(optarg);
+ if (retval != 0) {
+ printf("PQOS: Invalid L3CA parameters!\n");
+ goto exit;
+ }
+
+ argv[optind - 1] = prgname;
+ retval = optind - 1;
+ } else
+ retval = 0;
+
+exit:
+ /* reset getopt lib */
+ optind = 0;
+
+ /* Restore opterr value */
+ opterr = oldopterr;
+
+ return retval;
+}
+
+static void
+print_cmd_line_config(void)
+{
+ char cpustr[PQOS_MAX_CORES * 3] = {0};
+ unsigned i = 0;
+ unsigned j = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ unsigned len = 0;
+ memset(cpustr, 0, sizeof(cpustr));
+
+ /* Generate CPU list */
+ for (j = 0; j < PQOS_MAX_CORES; j++) {
+ if (CPU_ISSET(j, &m_config[i].cpumask) != 1)
+ continue;
+
+ len += snprintf(cpustr + len, sizeof(cpustr) - len - 1,
+ "%u,", j);
+
+ if (len >= sizeof(cpustr) - 1)
+ break;
+ }
+
+ if (m_config[i].cdp == 1) {
+ printf("PQOS: CPUs: %s cMASK: 0x%llx, dMASK: "
+ "0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].code_mask,
+ (unsigned long long)m_config[i].data_mask);
+ } else {
+ printf("PQOS: CPUs: %s MASK: 0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].mask);
+ }
+ }
+}
+
+/**
+ * @brief Prints CAT configuration
+ */
+static void
+print_cat_config(void)
+{
+ int ret = PQOS_RETVAL_OK;
+ unsigned i = 0;
+
+ for (i = 0; i < m_sock_count; i++) {
+ struct pqos_l3ca tab[PQOS_MAX_L3CA_COS] = {{0} };
+ unsigned num = 0;
+ unsigned n = 0;
+
+ ret = pqos_l3ca_get(m_sockets[i], PQOS_MAX_L3CA_COS, &num, tab);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving COS!\n");
+ return;
+ }
+
+ printf("PQOS: COS definitions for Socket %u:\n", m_sockets[i]);
+ for (n = 0; n < num; n++) {
+ if (tab[n].cdp == 1) {
+ printf("PQOS: COS: %u, cMASK: 0x%llx, "
+ "dMASK: 0x%llx\n", tab[n].class_id,
+ (unsigned long long)tab[n].code_mask,
+ (unsigned long long)tab[n].data_mask);
+ } else {
+ printf("PQOS: COS: %u, MASK: 0x%llx\n",
+ tab[n].class_id,
+ (unsigned long long)tab[n].ways_mask);
+ }
+ }
+ }
+
+ for (i = 0; i < m_sock_count; i++) {
+ unsigned lcores[PQOS_MAX_SOCKET_CORES] = {0};
+ unsigned lcount = 0;
+ unsigned n = 0;
+
+ ret = pqos_cpu_get_cores(m_cpu, m_sockets[i],
+ PQOS_MAX_SOCKET_CORES, &lcount, &lcores[0]);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving core information!\n");
+ return;
+ }
+
+ printf("PQOS: CPU information for socket %u:\n", m_sockets[i]);
+ for (n = 0; n < lcount; n++) {
+ unsigned class_id = 0;
+
+ ret = pqos_l3ca_assoc_get(lcores[n], &class_id);
+ if (ret == PQOS_RETVAL_OK)
+ printf("PQOS: CPU: %u, COS: %u\n", lcores[n],
+ class_id);
+ else
+ printf("PQOS: CPU: %u, ERROR\n", lcores[n]);
+ }
+ }
+
+}
+
+static int
+cat_validate(void)
+{
+ int ret = 0;
+
+ ret = check_cpus();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cdp();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cbm_len_and_contention();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cpus_overlapping();
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static int
+cat_set(void)
+{
+ int ret = 0;
+ unsigned cos_id_map[m_config_count][PQOS_MAX_SOCKETS];
+
+ memset(cos_id_map, 0, sizeof(cos_id_map));
+
+ ret = check_and_select_classes(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ ret = configure_cat(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+cat_fini(void)
+{
+ int ret = 0;
+
+ printf("PQOS: Shutting down PQoS library...\n");
+
+ /* deallocate all the resources */
+ ret = pqos_fini();
+ if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_INIT)
+ printf("PQOS: Error shutting down PQoS library!\n");
+
+ m_cap = NULL;
+ m_cpu = NULL;
+ m_cap_l3ca = NULL;
+ memset(m_sockets, 0, sizeof(m_sockets));
+ m_sock_count = 0;
+ memset(m_config, 0, sizeof(m_config));
+ m_config_count = 0;
+}
+
+void
+cat_exit(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned cpu_id = 0;
+ int ret = 0;
+
+ /* if lib is not initialized, do nothing */
+ if (m_cap == NULL && m_cpu == NULL)
+ return;
+
+ printf("PQOS: Reverting CAT configuration...\n");
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_l3ca_assoc_set(cpu_id, 0);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS 0 to "
+ "cpu %u\n", cpu_id);
+ }
+ }
+ }
+
+ cat_fini();
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\nPQOS: Signal %d received, preparing to exit...\n",
+ signum);
+
+ cat_exit();
+
+ /* exit with the expected status */
+ signal(signum, SIG_DFL);
+ kill(getpid(), signum);
+ }
+}
+
+int
+cat_init(int argc, char **argv)
+{
+ int ret = 0;
+ int args_num = 0;
+ struct pqos_config cfg = {0};
+
+ if (m_cap != NULL || m_cpu != NULL) {
+ printf("PQOS: CAT module already initialized!\n");
+ return -EEXIST;
+ }
+
+ /* Parse cmd line args */
+ ret = parse_args(argc, argv);
+
+ if (ret <= 0)
+ goto err;
+
+ args_num = ret;
+
+ /* Print cmd line configuration */
+ print_cmd_line_config();
+
+ /* PQoS Initialization - Check and initialize CAT capability */
+ cfg.fd_log = STDOUT_FILENO;
+ cfg.verbose = 0;
+ cfg.cdp_cfg = PQOS_REQUIRE_CDP_ANY;
+ ret = pqos_init(&cfg);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error initializing PQoS library!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get capability and CPU info pointer */
+ ret = pqos_cap_get(&m_cap, &m_cpu);
+ if (ret != PQOS_RETVAL_OK || m_cap == NULL || m_cpu == NULL) {
+ printf("PQOS: Error retrieving PQoS capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get L3CA capabilities */
+ ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &m_cap_l3ca);
+ if (ret != PQOS_RETVAL_OK || m_cap_l3ca == NULL) {
+ printf("PQOS: Error retrieving PQOS_CAP_TYPE_L3CA "
+ "capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get CPU socket information */
+ ret = pqos_cpu_get_sockets(m_cpu, PQOS_MAX_SOCKETS, &m_sock_count,
+ m_sockets);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving CPU socket information!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Validate cmd line configuration */
+ ret = cat_validate();
+ if (ret != 0) {
+ printf("PQOS: Requested CAT configuration is not valid!\n");
+ goto err;
+ }
+
+ /* configure system */
+ ret = cat_set();
+ if (ret != 0) {
+ printf("PQOS: Failed to configure CAT!\n");
+ goto err;
+ }
+
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ ret = atexit(cat_exit);
+ if (ret != 0) {
+ printf("PQOS: Cannot set exit function\n");
+ goto err;
+ }
+
+ /* Print CAT configuration */
+ print_cat_config();
+
+ return args_num;
+
+err:
+ /* deallocate all the resources */
+ cat_fini();
+ return ret;
+}
diff --git a/examples/skeleton-cat/cat.h b/examples/skeleton-cat/cat.h
new file mode 100644
index 0000000..aef2b76
--- /dev/null
+++ b/examples/skeleton-cat/cat.h
@@ -0,0 +1,72 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _CAT_H
+#define _CAT_H
+
+/**
+ * @file
+ * PQoS CAT
+ */
+
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_lcore.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* L3 cache allocation class of service data structure */
+struct cat_config {
+ rte_cpuset_t cpumask; /* CPUs bitmask */
+ int cdp; /* data & code masks used if true */
+ union {
+ uint64_t mask; /* capacity bitmask (CBM) */
+ struct {
+ uint64_t data_mask; /* data capacity bitmask (CBM) */
+ uint64_t code_mask; /* code capacity bitmask (CBM) */
+ };
+ };
+};
+
+int cat_init(int argc, char **argv);
+
+void cat_exit(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _CAT_H */
--
1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-11 16:03 ` Thomas Monjalon
@ 2016-03-11 16:27 ` Bruce Richardson
2016-03-11 16:34 ` Kantecki, Tomasz
0 siblings, 1 reply; 14+ messages in thread
From: Bruce Richardson @ 2016-03-11 16:27 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, Cornu, Marcel D, Kantecki, Tomasz
On Fri, Mar 11, 2016 at 05:03:23PM +0100, Thomas Monjalon wrote:
> 2016-03-11 15:54, Kantecki, Tomasz:
> > > > +Basic Forwarding sample application is a simple *skeleton* example of
> > > > +a forwarding application. It has been extended to make use of CAT via
> > > > +extended command line options and linking against the libpqos library.
> > >
> > > The skeleton aims to show a minimal piece of code (but bigger than hello
> > > world).
> > > For any other usage, I think the "l2fwd" name is more appropriate.
> > >
> > > > +DIRS-y += skeleton-cat
> > >
> >
> > We have used skeleton app as basis for this sample code. It is just to keep it as simple as possible.
> > Wouldn't it be confusing if we use "l2fwd" name for skeleton app functionality?
>
> I'm not sure.
> We need more opinions.
Skeleton app does do a form of l2 forwarding, just without editing the mac addresses.
I don't think having "skeleton" in the name of derived apps is a good thing, so
I would vote for an l2fwd-based name.
/Bruce
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-11 16:27 ` Bruce Richardson
@ 2016-03-11 16:34 ` Kantecki, Tomasz
2016-03-11 16:38 ` Thomas Monjalon
0 siblings, 1 reply; 14+ messages in thread
From: Kantecki, Tomasz @ 2016-03-11 16:34 UTC (permalink / raw)
To: Richardson, Bruce, Thomas Monjalon; +Cc: dev, Cornu, Marcel D
> > > > The skeleton aims to show a minimal piece of code (but bigger than
> > > > hello world).
> > > > For any other usage, I think the "l2fwd" name is more appropriate.
> > > >
> > > > > +DIRS-y += skeleton-cat
> > > >
> > >
> > > We have used skeleton app as basis for this sample code. It is just to keep
> it as simple as possible.
> > > Wouldn't it be confusing if we use "l2fwd" name for skeleton app
> functionality?
> >
> > I'm not sure.
> > We need more opinions.
>
> Skeleton app does do a form of l2 forwarding, just without editing the mac
> addresses.
> I don't think having "skeleton" in the name of derived apps is a good thing, so
> I would vote for an l2fwd-based name.
OK I understand. We'll prepare patch v5 with changed application name.
Is Mon morning OK for the v5 patch submission?
Thanks,
Tomasz
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v3] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-11 16:34 ` Kantecki, Tomasz
@ 2016-03-11 16:38 ` Thomas Monjalon
0 siblings, 0 replies; 14+ messages in thread
From: Thomas Monjalon @ 2016-03-11 16:38 UTC (permalink / raw)
To: Kantecki, Tomasz; +Cc: dev, Cornu, Marcel D
2016-03-11 16:34, Kantecki, Tomasz:
>
> > > > > The skeleton aims to show a minimal piece of code (but bigger than
> > > > > hello world).
> > > > > For any other usage, I think the "l2fwd" name is more appropriate.
> > > > >
> > > > > > +DIRS-y += skeleton-cat
> > > > >
> > > >
> > > > We have used skeleton app as basis for this sample code. It is just to keep
> > it as simple as possible.
> > > > Wouldn't it be confusing if we use "l2fwd" name for skeleton app
> > functionality?
> > >
> > > I'm not sure.
> > > We need more opinions.
> >
> > Skeleton app does do a form of l2 forwarding, just without editing the mac
> > addresses.
> > I don't think having "skeleton" in the name of derived apps is a good thing, so
> > I would vote for an l2fwd-based name.
>
> OK I understand. We'll prepare patch v5 with changed application name.
> Is Mon morning OK for the v5 patch submission?
The deadline was yesterday.
But for a new example, it can be integrated later, even in rc2
if rc1 is closed Monday (far from probable).
^ permalink raw reply [flat|nested] 14+ messages in thread
* [dpdk-dev] [PATCH v5] examples/l2fwd-cat: PQoS CAT and CDP, example of libpqos usage
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
` (2 preceding siblings ...)
2016-03-11 16:24 ` [dpdk-dev] [PATCH v4] " Wojciech Andralojc
@ 2016-03-14 13:46 ` Wojciech Andralojc
2016-03-21 22:35 ` Thomas Monjalon
3 siblings, 1 reply; 14+ messages in thread
From: Wojciech Andralojc @ 2016-03-14 13:46 UTC (permalink / raw)
To: dev; +Cc: Wojciech Andralojc, Tomasz Kantecki, Marcel D Cornu
This patch implements PQoS as a sample application.
PQoS allows management of the CPUs last level cache,
which can be useful for DPDK to ensure quality of service.
The sample app links against the existing 01.org PQoS library
(https://github.com/01org/intel-cmt-cat).
White paper demonstrating example use case "Increasing Platform Determinism
with Platform Quality of Service for the Data Plane Development Kit"
(http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html)
Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
---
Version 5:
* Example app renamed to "l2fwd-cat"
* Updated rst doc, added link to white paper
Version 4:
* PQOS_INSTALL_PATH must be defined to build example file
* Updated example's Makefiles, MAINTAINERS and rst doc,
Version 3:
* Updated rst docs
* Fixed checkpatch warnings
* Updated comments in code
Version 2:
* Added signal handlers to do clean-up on SIGINT and SIGTERM
* Clean-up function modified to zero globals (needed for testing)
* Init function modified to return more applicable errnos
Version 1:
* Initial version
Details of "--l3ca" app parameter to configure Intel CAT and CDP features:
--l3ca=bitmask@<cpu_list>
--l3ca=(code_bitmask,data_bitmask)@<cpu_list>
makes selected CPU's use specified CAT bitmasks, bitmasks must be
expressed in hexadecimal form
CAT and CDP features allow management of the CPU's last level cache.
CAT introduces classes of service (COS) that are essentially bitmasks.
In current CAT implementations, a bit in a COS bitmask corresponds to
one cache way in the last level cache.
A CPU core is always assigned to one of the CAT classes.
By programming CPU core assignment and COS bitmasks, applications can be
given exclusive, shared, or mixed access to the CPU's last level cache.
CDP extends CAT so that there are two bitmasks per COS,
one for data and one for code.
The number of classes and number of valid bits in a COS bitmask is CPU
model specific and COS bitmasks need to be contiguous. Sample code calls
this bitmask a cbm or a capacity bitmask.
By default, after reset, all CPU cores are assigned to COS 0 and all
classes are programmed to allow fill into all cache ways.
CDP is off by default.
For more information about CAT please see
https://github.com/01org/intel-cmt-cat
Known issues and limitations:
- --l3ca must be a first app parameter
---
MAINTAINERS | 4 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/l2_forward_cat.rst | 245 +++++++
examples/Makefile | 3 +
examples/l2fwd-cat/Makefile | 70 ++
examples/l2fwd-cat/cat.c | 996 ++++++++++++++++++++++++++++
examples/l2fwd-cat/cat.h | 72 ++
examples/l2fwd-cat/l2fwd-cat.c | 224 +++++++
8 files changed, 1615 insertions(+)
create mode 100644 doc/guides/sample_app_ug/l2_forward_cat.rst
create mode 100644 examples/l2fwd-cat/Makefile
create mode 100644 examples/l2fwd-cat/cat.c
create mode 100644 examples/l2fwd-cat/cat.h
create mode 100644 examples/l2fwd-cat/l2fwd-cat.c
diff --git a/MAINTAINERS b/MAINTAINERS
index f10b26a..6d420f0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -624,3 +624,7 @@ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
M: Pablo de Lara <pablo.de.lara.guarch@intel.com>
M: Daniel Mrzyglod <danielx.t.mrzyglod@intel.com>
F: examples/ptpclient/
+
+M: Tomasz Kantecki <tomasz.kantecki@intel.com>
+F: doc/guides/sample_app_ug/l2_forward_cat.rst
+F: examples/l2wd-cat/
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 88375d2..eea6a45 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -49,6 +49,7 @@ Sample Applications User Guide
keep_alive
l2_forward_job_stats
l2_forward_real_virtual
+ l2_forward_cat
l3_forward
l3_forward_power_man
l3_forward_access_ctrl
diff --git a/doc/guides/sample_app_ug/l2_forward_cat.rst b/doc/guides/sample_app_ug/l2_forward_cat.rst
new file mode 100644
index 0000000..b6ab54d
--- /dev/null
+++ b/doc/guides/sample_app_ug/l2_forward_cat.rst
@@ -0,0 +1,245 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+L2 Forwarding Sample Application with Cache Allocation Technology (CAT)
+=======================================================================
+
+Basic Forwarding sample application is a simple *skeleton* example of
+a forwarding application. It has been extended to make use of CAT via extended
+command line options and linking against the libpqos library.
+
+It is intended as a demonstration of the basic components of a DPDK forwarding
+application and use of the libpqos library to program CAT.
+For more detailed implementations see the L2 and L3 forwarding
+sample applications.
+
+CAT and Code Data Prioritization (CDP) features allow management of the CPU's
+last level cache. CAT introduces classes of service (COS) that are essentially
+bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to
+one cache way in last level cache.
+A CPU core is always assigned to one of the CAT classes.
+By programming CPU core assignment and COS bitmasks, applications can be given
+exclusive, shared, or mixed access to the CPU's last level cache.
+CDP extends CAT so that there are two bitmasks per COS,
+one for data and one for code.
+The number of classes and number of valid bits in a COS bitmask is CPU model
+specific and COS bitmasks need to be contiguous. Sample code calls this bitmask
+``cbm`` or capacity bitmask.
+By default, after reset, all CPU cores are assigned to COS 0 and all classes
+are programmed to allow fill into all cache ways.
+CDP is off by default.
+
+For more information about CAT please see:
+
+* https://github.com/01org/intel-cmt-cat
+
+White paper demonstrating example use case:
+
+* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_
+
+Compiling the Application
+-------------------------
+
+Requires ``libpqos`` from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_
+hosted on GitHub repository. For installation notes, please see ``README`` file.
+
+GIT:
+
+* https://github.com/01org/intel-cmt-cat
+
+To compile the application export the path to PQoS lib
+and the DPDK source tree and go to the example directory:
+
+.. code-block:: console
+
+ export PQOS_INSTALL_PATH=/path/to/libpqos
+ export RTE_SDK=/path/to/rte_sdk
+
+ cd ${RTE_SDK}/examples/l2fwd-cat
+
+Set the target, for example:
+
+.. code-block:: console
+
+ export RTE_TARGET=x86_64-native-linuxapp-gcc
+
+See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values.
+
+Build the application as follows:
+
+.. code-block:: console
+
+ make
+
+
+Running the Application
+-----------------------
+
+To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2:
+
+.. code-block:: console
+
+ ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)"
+
+or to enable CAT and CDP on cpus 1,3:
+
+.. code-block:: console
+
+ ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)"
+
+If CDP is not supported it will fail with following error message:
+
+.. code-block:: console
+
+ PQOS: CDP requested but not supported.
+ PQOS: Requested CAT configuration is not valid!
+ PQOS: Shutting down PQoS library...
+ EAL: Error - exiting with code: 1
+ Cause: PQOS: L3CA init failed!
+
+The option to enable CAT is:
+
+* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``:
+
+ where ``cbm`` stands for capacity bitmask and must be expressed in
+ hexadecimal form.
+
+ ``common_cbm`` is a single mask, for a CDP enabled system, a group of two
+ masks (``code_cbm`` and ``data_cbm``) is used.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ ``cpus`` could be a single digit/range or a group and must be expressed in
+ decimal form.
+
+ ``(`` and ``)`` are necessary if it's a group.
+
+ e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'``
+
+ * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6;
+
+ * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3;
+
+ * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways;
+
+ * cpu 7 has exclusive access to all of its 4 ways;
+
+ e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system
+
+ * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and
+ data ways are not overlapping.
+
+
+Refer to *DPDK Getting Started Guide* for general information on running
+applications and the Environment Abstraction Layer (EAL) options.
+
+
+To reset or list CAT configuration and control CDP please use ``pqos`` tool
+from Intel's
+`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_.
+
+To enabled or disable CDP:
+
+.. code-block:: console
+
+ sudo ./pqos -S cdp-on
+
+ sudo ./pqos -S cdp-off
+
+to reset CAT configuration:
+
+.. code-block:: console
+
+ sudo ./pqos -R
+
+to list CAT config:
+
+.. code-block:: console
+
+ sudo ./pqos -s
+
+For more info about ``pqos`` tool please see its man page or
+`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_.
+
+
+Explanation
+-----------
+
+The following sections provide an explanation of the main components of the
+code.
+
+All DPDK library functions used in the sample code are prefixed with ``rte_``
+and are explained in detail in the *DPDK API Documentation*.
+
+
+The Main Function
+~~~~~~~~~~~~~~~~~
+
+The ``main()`` function performs the initialization and calls the execution
+threads for each lcore.
+
+The first task is to initialize the Environment Abstraction Layer (EAL). The
+``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+The next task is to initialize the PQoS library and configure CAT. The
+``argc`` and ``argv`` arguments are provided to the ``cat_init()``
+function. The value returned is the number of parsed arguments:
+
+.. code-block:: c
+
+ int ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+``cat_init()`` is a wrapper function which parses the command, validates
+the requested parameters and configures CAT accordingly.
+
+Parsing of command line arguments is done in ``parse_args(...)``.
+libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is
+queried for system CPU information and L3CA capabilities via
+``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)``
+calls. When all capability and topology information is collected, the requested
+CAT configuration is validated. A check is then performed (on per socket basis)
+for a sufficient number of unassociated COS. COS are selected and
+configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to
+relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
+
+``atexit(...)`` is used to register ``cat_exit(...)`` to be called on
+a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating
+COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.
\ No newline at end of file
diff --git a/examples/Makefile b/examples/Makefile
index 4373d4d..b5faad6 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -61,6 +61,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += l2fwd-crypto
DIRS-$(CONFIG_RTE_LIBRTE_IVSHMEM) += l2fwd-ivshmem
DIRS-$(CONFIG_RTE_LIBRTE_JOBSTATS) += l2fwd-jobstats
DIRS-y += l2fwd-keepalive
+ifneq ($(PQOS_INSTALL_PATH),)
+DIRS-y += l2fwd-cat
+endif
DIRS-$(CONFIG_RTE_LIBRTE_LPM) += l3fwd
DIRS-$(CONFIG_RTE_LIBRTE_ACL) += l3fwd-acl
ifeq ($(CONFIG_RTE_LIBRTE_LPM),y)
diff --git a/examples/l2fwd-cat/Makefile b/examples/l2fwd-cat/Makefile
new file mode 100644
index 0000000..ae921ad
--- /dev/null
+++ b/examples/l2fwd-cat/Makefile
@@ -0,0 +1,70 @@
+# BSD LICENSE
+#
+# Copyright(c) 2016 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+ifeq ($(PQOS_INSTALL_PATH),)
+$(error "Please define PQOS_INSTALL_PATH environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Location of PQoS library and includes,
+PQOS_LIBRARY_PATH = $(PQOS_INSTALL_PATH)/libpqos.a
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = l2fwd-cat
+
+# all source are stored in SRCS-y
+SRCS-y := l2fwd-cat.c cat.c
+
+CFLAGS += $(WERROR_FLAGS)
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+EXTRA_CFLAGS += -O3 -g -Wfatal-errors
+
+CFLAGS += -I$(PQOS_INSTALL_PATH)/../include
+CFLAGS_cat.o := -D_GNU_SOURCE
+
+LDLIBS += -L$(PQOS_INSTALL_PATH)
+LDLIBS += $(PQOS_LIBRARY_PATH)
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/l2fwd-cat/cat.c b/examples/l2fwd-cat/cat.c
new file mode 100644
index 0000000..bad3930
--- /dev/null
+++ b/examples/l2fwd-cat/cat.c
@@ -0,0 +1,996 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <inttypes.h>
+#include <limits.h>
+#include <sched.h>
+#include <signal.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_memcpy.h>
+
+#include <pqos.h>
+
+#include "cat.h"
+
+#define BITS_PER_HEX 4
+#define PQOS_MAX_SOCKETS 8
+#define PQOS_MAX_SOCKET_CORES 64
+#define PQOS_MAX_CORES (PQOS_MAX_SOCKET_CORES * PQOS_MAX_SOCKETS)
+
+static const struct pqos_cap *m_cap;
+static const struct pqos_cpuinfo *m_cpu;
+static const struct pqos_capability *m_cap_l3ca;
+static unsigned m_sockets[PQOS_MAX_SOCKETS];
+static unsigned m_sock_count;
+static struct cat_config m_config[PQOS_MAX_CORES];
+static unsigned m_config_count;
+
+static unsigned
+bits_count(uint64_t bitmask)
+{
+ unsigned count = 0;
+
+ for (; bitmask != 0; count++)
+ bitmask &= bitmask - 1;
+
+ return count;
+}
+
+/*
+ * Parse elem, the elem could be single number/range or '(' ')' group
+ * 1) A single number elem, it's just a simple digit. e.g. 9
+ * 2) A single range elem, two digits with a '-' between. e.g. 2-6
+ * 3) A group elem, combines multiple 1) or 2) with '( )'. e.g (0,2-4,6)
+ * Within group elem, '-' used for a range separator;
+ * ',' used for a single number.
+ */
+static int
+parse_set(const char *input, rte_cpuset_t *cpusetp)
+{
+ unsigned idx;
+ const char *str = input;
+ char *end = NULL;
+ unsigned min, max;
+ const unsigned num = PQOS_MAX_CORES;
+
+ CPU_ZERO(cpusetp);
+
+ while (isblank(*str))
+ str++;
+
+ /* only digit or left bracket is qualify for start point */
+ if ((!isdigit(*str) && *str != '(') || *str == '\0')
+ return -1;
+
+ /* process single number or single range of number */
+ if (*str != '(') {
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ while (isblank(*end))
+ end++;
+
+ min = idx;
+ max = idx;
+ if (*end == '-') {
+ /* process single <number>-<number> */
+ end++;
+ while (isblank(*end))
+ end++;
+ if (!isdigit(*end))
+ return -1;
+
+ errno = 0;
+ idx = strtoul(end, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+ max = idx;
+ while (isblank(*end))
+ end++;
+ if (*end != ',' && *end != '\0')
+ return -1;
+ }
+
+ if (*end != ',' && *end != '\0' && *end != '@')
+ return -1;
+
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ return end - input;
+ }
+
+ /* process set within bracket */
+ str++;
+ while (isblank(*str))
+ str++;
+ if (*str == '\0')
+ return -1;
+
+ min = PQOS_MAX_CORES;
+ do {
+
+ /* go ahead to the first digit */
+ while (isblank(*str))
+ str++;
+ if (!isdigit(*str))
+ return -1;
+
+ /* get the digit value */
+ errno = 0;
+ idx = strtoul(str, &end, 10);
+ if (errno || end == NULL || idx >= num)
+ return -1;
+
+ /* go ahead to separator '-',',' and ')' */
+ while (isblank(*end))
+ end++;
+ if (*end == '-') {
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ else /* avoid continuous '-' */
+ return -1;
+ } else if ((*end == ',') || (*end == ')')) {
+ max = idx;
+ if (min == PQOS_MAX_CORES)
+ min = idx;
+ for (idx = RTE_MIN(min, max); idx <= RTE_MAX(min, max);
+ idx++)
+ CPU_SET(idx, cpusetp);
+
+ min = PQOS_MAX_CORES;
+ } else
+ return -1;
+
+ str = end + 1;
+ } while (*end != '\0' && *end != ')');
+
+ return str - input;
+}
+
+/* Test if bitmask is contiguous */
+static int
+is_contiguous(uint64_t bitmask)
+{
+ /* check if bitmask is contiguous */
+ unsigned i = 0;
+ unsigned j = 0;
+ const unsigned max_idx = (sizeof(bitmask) * CHAR_BIT);
+
+ if (bitmask == 0)
+ return 0;
+
+ for (i = 0; i < max_idx; i++) {
+ if (((1ULL << i) & bitmask) != 0)
+ j++;
+ else if (j > 0)
+ break;
+ }
+
+ if (bits_count(bitmask) != j) {
+ printf("PQOS: mask 0x%llx is not contiguous.\n",
+ (unsigned long long)bitmask);
+ return 0;
+ }
+
+ return 1;
+}
+
+/*
+ * The format pattern: --l3ca='<cbm@cpus>[,<(ccbm,dcbm)@cpus>...]'
+ * cbm could be a single mask or for a CDP enabled system, a group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - CPUs 1 and 3 share its 4 ways with CPUs 4, 5 and 6;
+ * - CPUs 4,5 and 6 share half (4 out of 8 ways) of its L3 with 1 and 3;
+ * - CPUs 4,5 and 6 have exclusive access to 4 out of 8 ways;
+ * - CPU 7 has exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for a CDP enabled system
+ * - cpus 1 and 3 have access to 2 ways for code and 2 ways for data,
+ * code and data ways are not overlapping.;
+ */
+static int
+parse_l3ca(const char *l3ca)
+{
+ unsigned idx = 0;
+ const char *cbm_start = NULL;
+ char *cbm_end = NULL;
+ const char *end = NULL;
+ int offset;
+ rte_cpuset_t cpuset;
+ uint64_t mask = 0;
+ uint64_t cmask = 0;
+
+ if (l3ca == NULL)
+ goto err;
+
+ /* Get cbm */
+ do {
+ CPU_ZERO(&cpuset);
+ mask = 0;
+ cmask = 0;
+
+ while (isblank(*l3ca))
+ l3ca++;
+
+ if (*l3ca == '\0')
+ goto err;
+
+ /* record mask_set start point */
+ cbm_start = l3ca;
+
+ /* go across a complete bracket */
+ if (*cbm_start == '(') {
+ l3ca += strcspn(l3ca, ")");
+ if (*l3ca++ == '\0')
+ goto err;
+ }
+
+ /* scan the separator '@', ','(next) or '\0'(finish) */
+ l3ca += strcspn(l3ca, "@,");
+
+ if (*l3ca == '@') {
+ /* explicit assign cpu_set */
+ offset = parse_set(l3ca + 1, &cpuset);
+ if (offset < 0 || CPU_COUNT(&cpuset) == 0)
+ goto err;
+
+ end = l3ca + 1 + offset;
+ } else
+ goto err;
+
+ if (*end != ',' && *end != '\0')
+ goto err;
+
+ /* parse mask_set from start point */
+ if (*cbm_start == '(') {
+ cbm_start++;
+
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ cmask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || cmask == 0)
+ goto err;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (*cbm_end != ',')
+ goto err;
+
+ cbm_end++;
+
+ while (isblank(*cbm_end))
+ cbm_end++;
+
+ if (!isxdigit(*cbm_end))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_end, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+ } else {
+ while (isblank(*cbm_start))
+ cbm_start++;
+
+ if (!isxdigit(*cbm_start))
+ goto err;
+
+ errno = 0;
+ mask = strtoul(cbm_start, &cbm_end, 16);
+ if (errno != 0 || cbm_end == NULL || mask == 0)
+ goto err;
+
+ }
+
+ if (mask == 0 || is_contiguous(mask) == 0)
+ goto err;
+
+ if (cmask != 0 && is_contiguous(cmask) == 0)
+ goto err;
+
+ rte_memcpy(&m_config[idx].cpumask,
+ &cpuset, sizeof(rte_cpuset_t));
+
+ if (cmask != 0) {
+ m_config[idx].cdp = 1;
+ m_config[idx].code_mask = cmask;
+ m_config[idx].data_mask = mask;
+ } else
+ m_config[idx].mask = mask;
+
+ m_config_count++;
+
+ l3ca = end + 1;
+ idx++;
+ } while (*end != '\0' && idx < PQOS_MAX_CORES);
+
+ if (m_config_count == 0)
+ goto err;
+
+ return 0;
+
+err:
+ return -EINVAL;
+}
+
+static int
+check_cpus_overlapping(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ rte_cpuset_t mask;
+
+ CPU_ZERO(&mask);
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = i + 1; j < m_config_count; j++) {
+ CPU_AND(&mask,
+ &m_config[i].cpumask,
+ &m_config[j].cpumask);
+
+ if (CPU_COUNT(&mask) != 0) {
+ printf("PQOS: Requested CPUs sets are "
+ "overlapping.\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cpus(void)
+{
+ unsigned i = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ for (cpu_id = 0; cpu_id < PQOS_MAX_CORES; cpu_id++) {
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) != 0) {
+
+ ret = pqos_cpu_check_core(m_cpu, cpu_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: %u is not a valid "
+ "logical core id.\n", cpu_id);
+ ret = -ENODEV;
+ goto exit;
+ }
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS "
+ "associated to cpu %u.\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if COS assigned to lcore is different
+ * then default one (#0)
+ */
+ if (cos_id != 0) {
+ printf("PQOS: cpu %u has already "
+ "associated COS#%u. "
+ "Please reset L3CA.\n",
+ cpu_id, cos_id);
+ ret = -EBUSY;
+ goto exit;
+ }
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+static int
+check_cdp(void)
+{
+ unsigned i = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1 && m_cap_l3ca->u.l3ca->cdp_on == 0) {
+ if (m_cap_l3ca->u.l3ca->cdp == 0) {
+ printf("PQOS: CDP requested but not "
+ "supported.\n");
+ } else {
+ printf("PQOS: CDP requested but not enabled. "
+ "Please enable CDP.\n");
+ }
+ return -ENOTSUP;
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_cbm_len_and_contention(void)
+{
+ unsigned i = 0;
+ uint64_t mask = 0;
+ const uint64_t not_cbm = (UINT64_MAX << (m_cap_l3ca->u.l3ca->num_ways));
+ const uint64_t cbm_contention_mask = m_cap_l3ca->u.l3ca->way_contention;
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ if (m_config[i].cdp == 1)
+ mask = m_config[i].code_mask | m_config[i].data_mask;
+ else
+ mask = m_config[i].mask;
+
+ if ((mask & not_cbm) != 0) {
+ printf("PQOS: One or more of requested CBM masks not "
+ "supported by system (too long).\n");
+ ret = -ENOTSUP;
+ break;
+ }
+
+ /* Just a warning */
+ if ((mask & cbm_contention_mask) != 0) {
+ printf("PQOS: One or more of requested CBM masks "
+ "overlap CBM contention mask.\n");
+ break;
+ }
+
+ }
+
+ return ret;
+}
+
+static int
+check_and_select_classes(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned phy_pkg_id = 0;
+ unsigned cos_id = 0;
+ unsigned cpu_id = 0;
+ unsigned phy_pkg_lcores[PQOS_MAX_SOCKETS][m_config_count];
+ const unsigned cos_num = m_cap_l3ca->u.l3ca->num_classes;
+ unsigned used_cos_table[PQOS_MAX_SOCKETS][cos_num];
+ int ret = 0;
+
+ memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+ memset(used_cos_table, 0, sizeof(used_cos_table));
+
+ /* detect currently used COS */
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+
+ ret = pqos_l3ca_assoc_get(cpu_id, &cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to read COS associated to "
+ "cpu %u on phy_pkg %u.\n", cpu_id, phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for cpu %u\n",
+ cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /* Mark COS as used */
+ if (used_cos_table[phy_pkg_id][cos_id] == 0)
+ used_cos_table[phy_pkg_id][cos_id]++;
+ }
+
+ /* look for avail. COS to fulfill requested config */
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ /*
+ * Check if we already have COS selected
+ * to be used for that group on that socket
+ */
+ if (phy_pkg_lcores[phy_pkg_id][i] != 0)
+ continue;
+
+ phy_pkg_lcores[phy_pkg_id][i]++;
+
+ /* Search for avail. COS to be used on that socket */
+ for (cos_id = 0; cos_id < cos_num; cos_id++) {
+ if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+ used_cos_table[phy_pkg_id][cos_id]++;
+ cos_id_map[i][phy_pkg_id] = cos_id;
+ break;
+ }
+ }
+
+ /* If there is no COS available ...*/
+ if (cos_id == cos_num) {
+ ret = -E2BIG;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ if (ret != 0)
+ printf("PQOS: Not enough available COS to configure "
+ "requested configuration.\n");
+
+ return ret;
+}
+
+static int
+configure_cat(unsigned cos_id_map[][PQOS_MAX_SOCKETS])
+{
+ unsigned phy_pkg_id = 0;
+ unsigned cpu_id = 0;
+ unsigned cos_id = 0;
+ unsigned i = 0;
+ unsigned j = 0;
+ struct pqos_l3ca l3ca = {0};
+ int ret = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ memset(&l3ca, 0, sizeof(l3ca));
+
+ l3ca.cdp = m_config[i].cdp;
+ if (m_config[i].cdp == 1) {
+ l3ca.code_mask = m_config[i].code_mask;
+ l3ca.data_mask = m_config[i].data_mask;
+ } else
+ l3ca.ways_mask = m_config[i].mask;
+
+ for (j = 0; j < m_sock_count; j++) {
+ phy_pkg_id = m_sockets[j];
+ if (cos_id_map[i][phy_pkg_id] == 0)
+ continue;
+
+ l3ca.class_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to set COS %u on "
+ "phy_pkg %u.\n", l3ca.class_id,
+ phy_pkg_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_cpu_get_socketid(m_cpu, cpu_id, &phy_pkg_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to get socket for "
+ "cpu %u\n", cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+
+ cos_id = cos_id_map[i][phy_pkg_id];
+
+ ret = pqos_l3ca_assoc_set(cpu_id, cos_id);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS %u to "
+ "cpu %u\n", cos_id, cpu_id);
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+ }
+
+exit:
+ return ret;
+}
+
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ int opt = 0;
+ int retval = 0;
+ int oldopterr = 0;
+ char **argvopt = argv;
+ char *prgname = argv[0];
+
+ static struct option lgopts[] = {
+ { "l3ca", required_argument, 0, 0 },
+ { NULL, 0, 0, 0 }
+ };
+
+ /* Disable printing messages within getopt() */
+ oldopterr = opterr;
+ opterr = 0;
+
+ opt = getopt_long(argc, argvopt, "", lgopts, NULL);
+ if (opt == 0) {
+ retval = parse_l3ca(optarg);
+ if (retval != 0) {
+ printf("PQOS: Invalid L3CA parameters!\n");
+ goto exit;
+ }
+
+ argv[optind - 1] = prgname;
+ retval = optind - 1;
+ } else
+ retval = 0;
+
+exit:
+ /* reset getopt lib */
+ optind = 0;
+
+ /* Restore opterr value */
+ opterr = oldopterr;
+
+ return retval;
+}
+
+static void
+print_cmd_line_config(void)
+{
+ char cpustr[PQOS_MAX_CORES * 3] = {0};
+ unsigned i = 0;
+ unsigned j = 0;
+
+ for (i = 0; i < m_config_count; i++) {
+ unsigned len = 0;
+ memset(cpustr, 0, sizeof(cpustr));
+
+ /* Generate CPU list */
+ for (j = 0; j < PQOS_MAX_CORES; j++) {
+ if (CPU_ISSET(j, &m_config[i].cpumask) != 1)
+ continue;
+
+ len += snprintf(cpustr + len, sizeof(cpustr) - len - 1,
+ "%u,", j);
+
+ if (len >= sizeof(cpustr) - 1)
+ break;
+ }
+
+ if (m_config[i].cdp == 1) {
+ printf("PQOS: CPUs: %s cMASK: 0x%llx, dMASK: "
+ "0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].code_mask,
+ (unsigned long long)m_config[i].data_mask);
+ } else {
+ printf("PQOS: CPUs: %s MASK: 0x%llx\n", cpustr,
+ (unsigned long long)m_config[i].mask);
+ }
+ }
+}
+
+/**
+ * @brief Prints CAT configuration
+ */
+static void
+print_cat_config(void)
+{
+ int ret = PQOS_RETVAL_OK;
+ unsigned i = 0;
+
+ for (i = 0; i < m_sock_count; i++) {
+ struct pqos_l3ca tab[PQOS_MAX_L3CA_COS] = {{0} };
+ unsigned num = 0;
+ unsigned n = 0;
+
+ ret = pqos_l3ca_get(m_sockets[i], PQOS_MAX_L3CA_COS, &num, tab);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving COS!\n");
+ return;
+ }
+
+ printf("PQOS: COS definitions for Socket %u:\n", m_sockets[i]);
+ for (n = 0; n < num; n++) {
+ if (tab[n].cdp == 1) {
+ printf("PQOS: COS: %u, cMASK: 0x%llx, "
+ "dMASK: 0x%llx\n", tab[n].class_id,
+ (unsigned long long)tab[n].code_mask,
+ (unsigned long long)tab[n].data_mask);
+ } else {
+ printf("PQOS: COS: %u, MASK: 0x%llx\n",
+ tab[n].class_id,
+ (unsigned long long)tab[n].ways_mask);
+ }
+ }
+ }
+
+ for (i = 0; i < m_sock_count; i++) {
+ unsigned lcores[PQOS_MAX_SOCKET_CORES] = {0};
+ unsigned lcount = 0;
+ unsigned n = 0;
+
+ ret = pqos_cpu_get_cores(m_cpu, m_sockets[i],
+ PQOS_MAX_SOCKET_CORES, &lcount, &lcores[0]);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving core information!\n");
+ return;
+ }
+
+ printf("PQOS: CPU information for socket %u:\n", m_sockets[i]);
+ for (n = 0; n < lcount; n++) {
+ unsigned class_id = 0;
+
+ ret = pqos_l3ca_assoc_get(lcores[n], &class_id);
+ if (ret == PQOS_RETVAL_OK)
+ printf("PQOS: CPU: %u, COS: %u\n", lcores[n],
+ class_id);
+ else
+ printf("PQOS: CPU: %u, ERROR\n", lcores[n]);
+ }
+ }
+
+}
+
+static int
+cat_validate(void)
+{
+ int ret = 0;
+
+ ret = check_cpus();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cdp();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cbm_len_and_contention();
+ if (ret != 0)
+ return ret;
+
+ ret = check_cpus_overlapping();
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static int
+cat_set(void)
+{
+ int ret = 0;
+ unsigned cos_id_map[m_config_count][PQOS_MAX_SOCKETS];
+
+ memset(cos_id_map, 0, sizeof(cos_id_map));
+
+ ret = check_and_select_classes(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ ret = configure_cat(cos_id_map);
+ if (ret != 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+cat_fini(void)
+{
+ int ret = 0;
+
+ printf("PQOS: Shutting down PQoS library...\n");
+
+ /* deallocate all the resources */
+ ret = pqos_fini();
+ if (ret != PQOS_RETVAL_OK && ret != PQOS_RETVAL_INIT)
+ printf("PQOS: Error shutting down PQoS library!\n");
+
+ m_cap = NULL;
+ m_cpu = NULL;
+ m_cap_l3ca = NULL;
+ memset(m_sockets, 0, sizeof(m_sockets));
+ m_sock_count = 0;
+ memset(m_config, 0, sizeof(m_config));
+ m_config_count = 0;
+}
+
+void
+cat_exit(void)
+{
+ unsigned i = 0;
+ unsigned j = 0;
+ unsigned cpu_id = 0;
+ int ret = 0;
+
+ /* if lib is not initialized, do nothing */
+ if (m_cap == NULL && m_cpu == NULL)
+ return;
+
+ printf("PQOS: Reverting CAT configuration...\n");
+
+ for (i = 0; i < m_config_count; i++) {
+ for (j = 0; j < m_cpu->num_cores; j++) {
+ cpu_id = m_cpu->cores[j].lcore;
+ if (CPU_ISSET(cpu_id, &m_config[i].cpumask) == 0)
+ continue;
+
+ ret = pqos_l3ca_assoc_set(cpu_id, 0);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Failed to associate COS 0 to "
+ "cpu %u\n", cpu_id);
+ }
+ }
+ }
+
+ cat_fini();
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\nPQOS: Signal %d received, preparing to exit...\n",
+ signum);
+
+ cat_exit();
+
+ /* exit with the expected status */
+ signal(signum, SIG_DFL);
+ kill(getpid(), signum);
+ }
+}
+
+int
+cat_init(int argc, char **argv)
+{
+ int ret = 0;
+ int args_num = 0;
+ struct pqos_config cfg = {0};
+
+ if (m_cap != NULL || m_cpu != NULL) {
+ printf("PQOS: CAT module already initialized!\n");
+ return -EEXIST;
+ }
+
+ /* Parse cmd line args */
+ ret = parse_args(argc, argv);
+
+ if (ret <= 0)
+ goto err;
+
+ args_num = ret;
+
+ /* Print cmd line configuration */
+ print_cmd_line_config();
+
+ /* PQoS Initialization - Check and initialize CAT capability */
+ cfg.fd_log = STDOUT_FILENO;
+ cfg.verbose = 0;
+ cfg.cdp_cfg = PQOS_REQUIRE_CDP_ANY;
+ ret = pqos_init(&cfg);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error initializing PQoS library!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get capability and CPU info pointer */
+ ret = pqos_cap_get(&m_cap, &m_cpu);
+ if (ret != PQOS_RETVAL_OK || m_cap == NULL || m_cpu == NULL) {
+ printf("PQOS: Error retrieving PQoS capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get L3CA capabilities */
+ ret = pqos_cap_get_type(m_cap, PQOS_CAP_TYPE_L3CA, &m_cap_l3ca);
+ if (ret != PQOS_RETVAL_OK || m_cap_l3ca == NULL) {
+ printf("PQOS: Error retrieving PQOS_CAP_TYPE_L3CA "
+ "capabilities!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Get CPU socket information */
+ ret = pqos_cpu_get_sockets(m_cpu, PQOS_MAX_SOCKETS, &m_sock_count,
+ m_sockets);
+ if (ret != PQOS_RETVAL_OK) {
+ printf("PQOS: Error retrieving CPU socket information!\n");
+ ret = -EFAULT;
+ goto err;
+ }
+
+ /* Validate cmd line configuration */
+ ret = cat_validate();
+ if (ret != 0) {
+ printf("PQOS: Requested CAT configuration is not valid!\n");
+ goto err;
+ }
+
+ /* configure system */
+ ret = cat_set();
+ if (ret != 0) {
+ printf("PQOS: Failed to configure CAT!\n");
+ goto err;
+ }
+
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ ret = atexit(cat_exit);
+ if (ret != 0) {
+ printf("PQOS: Cannot set exit function\n");
+ goto err;
+ }
+
+ /* Print CAT configuration */
+ print_cat_config();
+
+ return args_num;
+
+err:
+ /* deallocate all the resources */
+ cat_fini();
+ return ret;
+}
diff --git a/examples/l2fwd-cat/cat.h b/examples/l2fwd-cat/cat.h
new file mode 100644
index 0000000..aef2b76
--- /dev/null
+++ b/examples/l2fwd-cat/cat.h
@@ -0,0 +1,72 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _CAT_H
+#define _CAT_H
+
+/**
+ * @file
+ * PQoS CAT
+ */
+
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_lcore.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* L3 cache allocation class of service data structure */
+struct cat_config {
+ rte_cpuset_t cpumask; /* CPUs bitmask */
+ int cdp; /* data & code masks used if true */
+ union {
+ uint64_t mask; /* capacity bitmask (CBM) */
+ struct {
+ uint64_t data_mask; /* data capacity bitmask (CBM) */
+ uint64_t code_mask; /* code capacity bitmask (CBM) */
+ };
+ };
+};
+
+int cat_init(int argc, char **argv);
+
+void cat_exit(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _CAT_H */
diff --git a/examples/l2fwd-cat/l2fwd-cat.c b/examples/l2fwd-cat/l2fwd-cat.c
new file mode 100644
index 0000000..8cce33b
--- /dev/null
+++ b/examples/l2fwd-cat/l2fwd-cat.c
@@ -0,0 +1,224 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <inttypes.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_lcore.h>
+#include <rte_mbuf.h>
+
+#include "cat.h"
+
+#define RX_RING_SIZE 128
+#define TX_RING_SIZE 512
+
+#define NUM_MBUFS 8191
+#define MBUF_CACHE_SIZE 250
+#define BURST_SIZE 32
+
+static const struct rte_eth_conf port_conf_default = {
+ .rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
+};
+
+/* l2fwd-cat.c: CAT enabled, basic DPDK skeleton forwarding example. */
+
+/*
+ * Initializes a given port using global settings and with the RX buffers
+ * coming from the mbuf_pool passed as a parameter.
+ */
+static inline int
+port_init(uint8_t port, struct rte_mempool *mbuf_pool)
+{
+ struct rte_eth_conf port_conf = port_conf_default;
+ const uint16_t rx_rings = 1, tx_rings = 1;
+ int retval;
+ uint16_t q;
+
+ if (port >= rte_eth_dev_count())
+ return -1;
+
+ /* Configure the Ethernet device. */
+ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf);
+ if (retval != 0)
+ return retval;
+
+ /* Allocate and set up 1 RX queue per Ethernet port. */
+ for (q = 0; q < rx_rings; q++) {
+ retval = rte_eth_rx_queue_setup(port, q, RX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL, mbuf_pool);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Allocate and set up 1 TX queue per Ethernet port. */
+ for (q = 0; q < tx_rings; q++) {
+ retval = rte_eth_tx_queue_setup(port, q, TX_RING_SIZE,
+ rte_eth_dev_socket_id(port), NULL);
+ if (retval < 0)
+ return retval;
+ }
+
+ /* Start the Ethernet port. */
+ retval = rte_eth_dev_start(port);
+ if (retval < 0)
+ return retval;
+
+ /* Display the port MAC address. */
+ struct ether_addr addr;
+ rte_eth_macaddr_get(port, &addr);
+ printf("Port %u MAC: %02" PRIx8 " %02" PRIx8 " %02" PRIx8
+ " %02" PRIx8 " %02" PRIx8 " %02" PRIx8 "\n",
+ (unsigned)port,
+ addr.addr_bytes[0], addr.addr_bytes[1],
+ addr.addr_bytes[2], addr.addr_bytes[3],
+ addr.addr_bytes[4], addr.addr_bytes[5]);
+
+ /* Enable RX in promiscuous mode for the Ethernet device. */
+ rte_eth_promiscuous_enable(port);
+
+ return 0;
+}
+
+/*
+ * The lcore main. This is the main thread that does the work, reading from
+ * an input port and writing to an output port.
+ */
+static __attribute__((noreturn)) void
+lcore_main(void)
+{
+ const uint8_t nb_ports = rte_eth_dev_count();
+ uint8_t port;
+
+ /*
+ * Check that the port is on the same NUMA node as the polling thread
+ * for best performance.
+ */
+ for (port = 0; port < nb_ports; port++)
+ if (rte_eth_dev_socket_id(port) > 0 &&
+ rte_eth_dev_socket_id(port) !=
+ (int)rte_socket_id())
+ printf("WARNING, port %u is on remote NUMA node to "
+ "polling thread.\n\tPerformance will "
+ "not be optimal.\n", port);
+
+ printf("\nCore %u forwarding packets. [Ctrl+C to quit]\n",
+ rte_lcore_id());
+
+ /* Run until the application is quit or killed. */
+ for (;;) {
+ /*
+ * Receive packets on a port and forward them on the paired
+ * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc.
+ */
+ for (port = 0; port < nb_ports; port++) {
+
+ /* Get burst of RX packets, from first port of pair. */
+ struct rte_mbuf *bufs[BURST_SIZE];
+ const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
+ bufs, BURST_SIZE);
+
+ if (unlikely(nb_rx == 0))
+ continue;
+
+ /* Send burst of TX packets, to second port of pair. */
+ const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
+ bufs, nb_rx);
+
+ /* Free any unsent packets. */
+ if (unlikely(nb_tx < nb_rx)) {
+ uint16_t buf;
+ for (buf = nb_tx; buf < nb_rx; buf++)
+ rte_pktmbuf_free(bufs[buf]);
+ }
+ }
+ }
+}
+
+/*
+ * The main function, which does initialization and calls the per-lcore
+ * functions.
+ */
+int
+main(int argc, char *argv[])
+{
+ struct rte_mempool *mbuf_pool;
+ unsigned nb_ports;
+ uint8_t portid;
+
+ /* Initialize the Environment Abstraction Layer (EAL). */
+ int ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /*
+ * Initialize the PQoS library and configure CAT.
+ * Please see l2fwd-cat documentation for more info.
+ */
+ ret = cat_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n");
+
+ argc -= ret;
+ argv += ret;
+
+ /* Check that there is an even number of ports to send/receive on. */
+ nb_ports = rte_eth_dev_count();
+ if (nb_ports < 2 || (nb_ports & 1))
+ rte_exit(EXIT_FAILURE, "Error: number of ports must be even\n");
+
+ /* Creates a new mempool in memory to hold the mbufs. */
+ mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports,
+ MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
+
+ if (mbuf_pool == NULL)
+ rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");
+
+ /* Initialize all ports. */
+ for (portid = 0; portid < nb_ports; portid++)
+ if (port_init(portid, mbuf_pool) != 0)
+ rte_exit(EXIT_FAILURE, "Cannot init port %"PRIu8 "\n",
+ portid);
+
+ if (rte_lcore_count() > 1)
+ printf("\nWARNING: Too many lcores enabled. Only 1 used.\n");
+
+ /* Call lcore_main on the master core only. */
+ lcore_main();
+
+ return 0;
+}
--
1.9.3
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-dev] [PATCH v5] examples/l2fwd-cat: PQoS CAT and CDP, example of libpqos usage
2016-03-14 13:46 ` [dpdk-dev] [PATCH v5] examples/l2fwd-cat: " Wojciech Andralojc
@ 2016-03-21 22:35 ` Thomas Monjalon
0 siblings, 0 replies; 14+ messages in thread
From: Thomas Monjalon @ 2016-03-21 22:35 UTC (permalink / raw)
To: Wojciech Andralojc; +Cc: dev, Tomasz Kantecki, Marcel D Cornu
2016-03-14 13:46, Wojciech Andralojc:
> This patch implements PQoS as a sample application.
> PQoS allows management of the CPUs last level cache,
> which can be useful for DPDK to ensure quality of service.
> The sample app links against the existing 01.org PQoS library
> (https://github.com/01org/intel-cmt-cat).
>
> White paper demonstrating example use case "Increasing Platform Determinism
> with Platform Quality of Service for the Data Plane Development Kit"
> (http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html)
>
> Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
> Signed-off-by: Tomasz Kantecki <tomasz.kantecki@intel.com>
> Signed-off-by: Marcel D Cornu <marcel.d.cornu@intel.com>
Applied, thanks
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2016-03-21 22:37 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-24 13:26 [dpdk-dev] [PATCH] examples/skeleton-cat: PQoS CAT and CDP, example of libpqos usage Wojciech Andralojc
2016-02-26 15:07 ` [dpdk-dev] [PATCH v2] " Wojciech Andralojc
2016-03-07 14:34 ` Van Haaren, Harry
2016-03-07 16:54 ` [dpdk-dev] [PATCH v3] " Wojciech Andralojc
2016-03-07 17:19 ` Van Haaren, Harry
2016-03-11 13:50 ` Thomas Monjalon
2016-03-11 15:54 ` Kantecki, Tomasz
2016-03-11 16:03 ` Thomas Monjalon
2016-03-11 16:27 ` Bruce Richardson
2016-03-11 16:34 ` Kantecki, Tomasz
2016-03-11 16:38 ` Thomas Monjalon
2016-03-11 16:24 ` [dpdk-dev] [PATCH v4] " Wojciech Andralojc
2016-03-14 13:46 ` [dpdk-dev] [PATCH v5] examples/l2fwd-cat: " Wojciech Andralojc
2016-03-21 22:35 ` Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).