[dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Wojciech Andralojc]

EAL extension allows CAT and CDP technologies to be
configured via "--l3ca*" parameters.
Reworking existing applications to make use of DPDK PQoS library is
a heavy lift. This EAL extension is to make it easier by adding
new command line options. These new options allow to leverage CAT and
CDP technologies with existing code base without any additional
development effort.

Signed-off-by: Wojciech Andralojc <wojciechx.andralojc@intel.com>
---

Adds EAL "--l3ca*" parameters to configure Intel CAT and CDP features
--l3ca-dump - lists current CAT configuration
--l3ca-reset=[cdp-on|cdp-off] - performs CAT reset.
  It can switch CDP on or off.
  
--l3ca=bitmask@<core_list>
--l3ca=(code_bitmask,data_bitmask)@<core_list>
	- makes selected cores use specified CAT bitmasks

Known issues:
- checkpatches.sh LONG_LINE warnings;

 config/common_linuxapp                        |   4 +
 config/defconfig_i686-native-linuxapp-gcc     |   3 +-
 config/defconfig_i686-native-linuxapp-icc     |   3 +-
 config/defconfig_x86_64-native-linuxapp-clang |   3 +-
 config/defconfig_x86_64-native-linuxapp-gcc   |   3 +-
 config/defconfig_x86_64-native-linuxapp-icc   |   3 +-
 config/defconfig_x86_x32-native-linuxapp-gcc  |   3 +-
 lib/librte_eal/common/eal_common_lcore.c      |   4 +
 lib/librte_eal/common/eal_common_options.c    | 354 ++++++++++++++--
 lib/librte_eal/common/eal_common_pqos.c       | 590 ++++++++++++++++++++++++++
 lib/librte_eal/common/eal_internal_cfg.h      |  34 ++
 lib/librte_eal/common/eal_options.h           |  11 +
 lib/librte_eal/common/eal_private.h           |  12 +
 lib/librte_eal/common/include/rte_lcore.h     |   3 +
 lib/librte_eal/linuxapp/eal/Makefile          |  10 +
 lib/librte_eal/linuxapp/eal/eal.c             |   5 +
 16 files changed, 1009 insertions(+), 36 deletions(-)
 create mode 100644 lib/librte_eal/common/eal_common_pqos.c

diff --git a/config/common_linuxapp b/config/common_linuxapp
index 553e640..bf30f1e 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -520,6 +520,10 @@ CONFIG_RTE_LIBRTE_VHOST_DEBUG=n
 # LIBRTE_PQOS is for librte_pqos only
 CONFIG_RTE_LIBRTE_PQOS=n
 CONFIG_RTE_LIBRTE_PQOS_DEBUG=n
+# LIBRTE_PQOS_OPTS is for enabling PQoS in EAL
+# (initialization, configuration and cmd line options)
+CONFIG_RTE_LIBRTE_PQOS_OPTS=n
+
 
 #
 #Compile Xen domain0 support
diff --git a/config/defconfig_i686-native-linuxapp-gcc b/config/defconfig_i686-native-linuxapp-gcc
index a73b999..ee8ff9f 100644
--- a/config/defconfig_i686-native-linuxapp-gcc
+++ b/config/defconfig_i686-native-linuxapp-gcc
@@ -51,4 +51,5 @@ CONFIG_RTE_LIBRTE_KNI=n
 CONFIG_RTE_IXGBE_INC_VECTOR=n
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/config/defconfig_i686-native-linuxapp-icc b/config/defconfig_i686-native-linuxapp-icc
index f454955..85d1f53 100644
--- a/config/defconfig_i686-native-linuxapp-icc
+++ b/config/defconfig_i686-native-linuxapp-icc
@@ -51,4 +51,5 @@ CONFIG_RTE_LIBRTE_KNI=n
 CONFIG_RTE_IXGBE_INC_VECTOR=n
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/config/defconfig_x86_64-native-linuxapp-clang b/config/defconfig_x86_64-native-linuxapp-clang
index 57a7ad5..a93f7d9 100644
--- a/config/defconfig_x86_64-native-linuxapp-clang
+++ b/config/defconfig_x86_64-native-linuxapp-clang
@@ -42,4 +42,5 @@ CONFIG_RTE_TOOLCHAIN="clang"
 CONFIG_RTE_TOOLCHAIN_CLANG=y
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/config/defconfig_x86_64-native-linuxapp-gcc b/config/defconfig_x86_64-native-linuxapp-gcc
index d4ba885..66621a5 100644
--- a/config/defconfig_x86_64-native-linuxapp-gcc
+++ b/config/defconfig_x86_64-native-linuxapp-gcc
@@ -42,4 +42,5 @@ CONFIG_RTE_TOOLCHAIN="gcc"
 CONFIG_RTE_TOOLCHAIN_GCC=y
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/config/defconfig_x86_64-native-linuxapp-icc b/config/defconfig_x86_64-native-linuxapp-icc
index 829b8f3..49b2062 100644
--- a/config/defconfig_x86_64-native-linuxapp-icc
+++ b/config/defconfig_x86_64-native-linuxapp-icc
@@ -42,4 +42,5 @@ CONFIG_RTE_TOOLCHAIN="icc"
 CONFIG_RTE_TOOLCHAIN_ICC=y
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/config/defconfig_x86_x32-native-linuxapp-gcc b/config/defconfig_x86_x32-native-linuxapp-gcc
index 4140a95..98df235 100644
--- a/config/defconfig_x86_x32-native-linuxapp-gcc
+++ b/config/defconfig_x86_x32-native-linuxapp-gcc
@@ -46,4 +46,5 @@ CONFIG_RTE_TOOLCHAIN_GCC=y
 CONFIG_RTE_LIBRTE_KNI=n
 
 # Enable PQoS
-CONFIG_RTE_LIBRTE_PQOS=y
\ No newline at end of file
+CONFIG_RTE_LIBRTE_PQOS=y
+CONFIG_RTE_LIBRTE_PQOS_OPTS=y
\ No newline at end of file
diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index 42eec8e..a57ccb8 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -100,6 +100,10 @@ rte_eal_cpu_init(void)
 #ifdef RTE_LIBRTE_PQOS
 		lcore_config[lcore_id].phy_pkg_id = eal_cpu_phy_pkg_id(lcore_id);
 
+#ifdef RTE_LIBRTE_PQOS_OPTS
+		lcore_config[lcore_id].pqos_l3ca_cos_id = 0;
+#endif
+
 		if (lcore_config[lcore_id].phy_pkg_id < RTE_MAX_PHY_PKGS)
 			phy_pkg_config[lcore_config[lcore_id].phy_pkg_id].detected = 1;
 		else
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 29942ea..8ca6300 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -50,6 +50,9 @@
 #include <rte_version.h>
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+#include <rte_pqos.h>
+#endif
 
 #include "eal_internal_cfg.h"
 #include "eal_options.h"
@@ -95,6 +98,11 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	{OPT_PQOS_L3CA,         1, NULL, OPT_PQOS_L3CA_NUM        },
+	{OPT_PQOS_L3CA_DUMP,    0, NULL, OPT_PQOS_L3CA_DUMP_NUM   },
+	{OPT_PQOS_L3CA_RESET,   2, NULL, OPT_PQOS_L3CA_RESET_NUM  },
+#endif
 	{0,                     0, NULL, 0                        }
 };
 
@@ -153,6 +161,21 @@ eal_reset_internal_config(struct internal_config *internal_cfg)
 #endif
 	internal_cfg->vmware_tsc_map = 0;
 	internal_cfg->create_uio_dev = 0;
+
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	internal_cfg->pqos_l3ca_reset = 0;
+	internal_cfg->pqos_l3ca_cdp_req = RTE_PQOS_REQUIRE_CDP_MAX;
+	internal_cfg->pqos_l3ca_dump = 0;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+		CPU_ZERO(&internal_cfg->pqos_l3ca_params[i].cpumask);
+		CBM_ZERO(&internal_cfg->pqos_l3ca_params[i].mask);
+		CBM_ZERO(&internal_cfg->pqos_l3ca_params[i].code_mask);
+		internal_cfg->pqos_l3ca_params[i].cdp = 0;
+		memset(internal_cfg->pqos_l3ca_params[i].cos_id, 0,
+				sizeof(internal_cfg->pqos_l3ca_params[i].cos_id));
+	}
+#endif
 }
 
 static int
@@ -259,32 +282,40 @@ static int xdigit2val(unsigned char c)
 }
 
 static int
-eal_parse_coremask(const char *coremask)
+xstr2cpuset(rte_cpuset_t *cpusetp, const char *xstr, unsigned xstrlen)
 {
-	struct rte_config *cfg = rte_eal_get_configuration();
 	int i, j, idx = 0;
-	unsigned count = 0;
 	char c;
 	int val;
 
-	if (coremask == NULL)
+	if (xstr == NULL || cpusetp == NULL || xstrlen == 0)
 		return -1;
+
+	i = xstrlen;
+
 	/* Remove all blank characters ahead and after .
 	 * Remove 0x/0X if exists.
 	 */
-	while (isblank(*coremask))
-		coremask++;
-	if (coremask[0] == '0' && ((coremask[1] == 'x')
-		|| (coremask[1] == 'X')))
-		coremask += 2;
-	i = strlen(coremask);
-	while ((i > 0) && isblank(coremask[i - 1]))
+	while (isblank(*xstr)) {
+		xstr++;
 		i--;
+	}
+
+	if (xstr[0] == '0' && ((xstr[1] == 'x') || (xstr[1] == 'X'))) {
+		xstr += 2;
+		i -= 2;
+	}
+
+	while ((i > 0) && isblank(xstr[i - 1]))
+		i--;
+
 	if (i == 0)
 		return -1;
 
+	CPU_ZERO(cpusetp);
+
 	for (i = i - 1; i >= 0 && idx < RTE_MAX_LCORE; i--) {
-		c = coremask[i];
+		c = xstr[i];
 		if (isxdigit(c) == 0) {
 			/* invalid characters */
 			return -1;
@@ -292,30 +323,51 @@ eal_parse_coremask(const char *coremask)
 		val = xdigit2val(c);
 		for (j = 0; j < BITS_PER_HEX && idx < RTE_MAX_LCORE; j++, idx++)
 		{
-			if ((1 << j) & val) {
-				if (!lcore_config[idx].detected) {
-					RTE_LOG(ERR, EAL, "lcore %u "
-					        "unavailable\n", idx);
-					return -1;
-				}
-				cfg->lcore_role[idx] = ROLE_RTE;
-				lcore_config[idx].core_index = count;
-				count++;
-			} else {
-				cfg->lcore_role[idx] = ROLE_OFF;
-				lcore_config[idx].core_index = -1;
-			}
+			if ((1 << j) & val)
+				CPU_SET(idx, cpusetp);
 		}
 	}
+
 	for (; i >= 0; i--)
-		if (coremask[i] != '0')
+		if (xstr[i] != '0')
 			return -1;
-	for (; idx < RTE_MAX_LCORE; idx++) {
-		cfg->lcore_role[idx] = ROLE_OFF;
-		lcore_config[idx].core_index = -1;
+
+	return 0;
+}
+
+static int
+eal_parse_coremask(const char *coremask)
+{
+	struct rte_config *cfg = rte_eal_get_configuration();
+	int i = 0;
+	unsigned count = 0;
+	int ret = -1;
+	rte_cpuset_t cpuset;
+
+	CPU_ZERO(&cpuset);
+
+	ret = xstr2cpuset(&cpuset, coremask, strlen(coremask));
+	if (ret == -1)
+		return ret;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+		if (CPU_ISSET(i, &cpuset)) {
+			if (!lcore_config[i].detected) {
+				RTE_LOG(ERR, EAL, "lcore %u unavailable\n", i);
+				return -1;
+			}
+			cfg->lcore_role[i] = ROLE_RTE;
+			lcore_config[i].core_index = count;
+			count++;
+		} else {
+			cfg->lcore_role[i] = ROLE_OFF;
+			lcore_config[i].core_index = -1;
+		}
 	}
+
 	if (count == 0)
 		return -1;
+
 	/* Update the count of enabled logical cores of the EAL configuration */
 	cfg->lcore_count = count;
 	return 0;
@@ -754,6 +806,206 @@ eal_parse_proc_type(const char *arg)
 	return RTE_PROC_INVALID;
 }
 
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+
+static int
+is_contiguous(const rte_pqos_cbm_t *mask)
+{
+	/* check if bitmask is contiguous */
+	int i = 0;
+	int j = 0;
+
+	if (CBM_COUNT(mask) == 0)
+		return 0;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+		if (CBM_ISSET(i, mask) == 1)
+			j++;
+		else if (j > 0)
+			break;
+	}
+
+	if (CBM_COUNT(mask) != j)
+		return 0;
+
+	return 1;
+}
+
+/*
+ * The format pattern: --l3ca='cbm[@cpus]>[<,(ccbm,dcbm)[@cpus]>...]'
+ * cbm could be single mask or for CDP enabled system, group of two masks
+ * ("code cbm" and "data cbm")
+ * '(' and ')' are necessary if it's a group.
+ * cpus could be a single digit/range or a group.
+ * '(' and ')' are necessary if it's a group.
+ *
+ * e.g. '0x00F00@(1,3), 0x0FF00@(4-6), 0xF0000@7'
+ * - lcores 1 and 3 share its 4 ways with lcores 4, 5 and 6;
+ * - lcores 4, 5 and 6 share half (4 out of 8 ways) of its L3 with lcores 1 and 3;
+ * - lcores 4,5 and 6 have exclusive access to 4 out of  8 ways;
+ * - lcore 7 have exclusive access to all of its 4 ways;
+ *
+ * e.g. '(0x00C00,0x00300)@(1,3)' for CDP enabled system
+ */
+static int
+eal_parse_pqos_l3ca(const char *l3ca, struct internal_config *conf)
+{
+	static uint16_t set[RTE_MAX_LCORE];
+	unsigned idx = 0;
+	const char *cbm_start = NULL;
+	unsigned cbm_len = 0;
+	const char *end = NULL;
+	int offset;
+	rte_cpuset_t cpuset;
+	rte_pqos_cbm_t maskset;
+	rte_pqos_cbm_t cmaskset;
+	int ret = -1;
+
+	if (l3ca == NULL)
+		return -1;
+
+	/* Get cbm */
+	do {
+		CPU_ZERO(&cpuset);
+		CBM_ZERO(&maskset);
+		CBM_ZERO(&cmaskset);
+
+		while (isblank(*l3ca))
+			l3ca++;
+
+		if (*l3ca == '\0') {
+			ret = -1;
+			goto err;
+		}
+
+		/* record mask_set start point */
+		cbm_start = l3ca;
+
+		/* go across a complete bracket */
+		if (*cbm_start == '(') {
+			l3ca += strcspn(l3ca, ")");
+			if (*l3ca++ == '\0') {
+				ret = -1;
+				goto err;
+			}
+		}
+
+		/* scan the separator '@', ','(next) or '\0'(finish) */
+		l3ca += strcspn(l3ca, "@,");
+
+		if (*l3ca == '@') {
+			/* explicit assign cpu_set */
+			offset = eal_parse_set(l3ca + 1, set, RTE_DIM(set));
+			if (offset < 0) {
+				ret = -1;
+				goto err;
+			}
+
+			/* prepare cpu_set and update the end cursor */
+			ret = convert_to_cpuset(&cpuset, set, RTE_DIM(set));
+			if (ret == -1 || CPU_COUNT(&cpuset) == 0) {
+				ret = -1;
+				goto err;
+			}
+			end = l3ca + 1 + offset;
+		} else {
+			ret = -1;
+			goto err;
+		}
+
+		if (*end != ',' && *end != '\0') {
+			ret = -1;
+			goto err;
+		}
+
+		/* parse mask_set from start point */
+		if (*cbm_start == '(') {
+			cbm_start++;
+			cbm_len = strcspn(cbm_start, ",");
+			ret = xstr2cpuset(&maskset, cbm_start, cbm_len);
+			if (ret == -1)
+				goto err;
+
+			cbm_start += cbm_len + 1;
+			cbm_len = strcspn(cbm_start, ")");
+			ret = xstr2cpuset(&cmaskset, cbm_start, cbm_len);
+			if (ret == -1 || CBM_COUNT(&cmaskset) == 0) {
+				ret = -1;
+				goto err;
+			}
+		} else {
+			cbm_len = strcspn(cbm_start, "@,");
+			ret = xstr2cpuset(&maskset, cbm_start, cbm_len);
+			if (ret == -1)
+				goto err;
+		}
+
+		if (CBM_COUNT(&maskset) == 0 || is_contiguous(&maskset) == 0) {
+			ret = -1;
+			goto err;
+		}
+
+		if (CBM_COUNT(&cmaskset) != 0 &&
+				is_contiguous(&cmaskset) == 0) {
+			ret = -1;
+			goto err;
+		}
+
+		rte_memcpy(&conf->pqos_l3ca_params[idx].cpumask,
+				&cpuset, sizeof(rte_cpuset_t));
+		if (CBM_COUNT(&cmaskset) != 0) {
+			conf->pqos_l3ca_params[idx].cdp = 1;
+			rte_memcpy(&conf->pqos_l3ca_params[idx].data_mask,
+					&maskset, sizeof(rte_pqos_cbm_t));
+			rte_memcpy(&conf->pqos_l3ca_params[idx].code_mask,
+					&cmaskset, sizeof(rte_pqos_cbm_t));
+		} else
+			rte_memcpy(&conf->pqos_l3ca_params[idx].mask,
+					&maskset, sizeof(rte_pqos_cbm_t));
+
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+		int i = 0;
+		RTE_LOG(INFO, PQOS, "CFG CDP: %u\n", conf->pqos_l3ca_params[idx].cdp);
+
+		for (i = 0; i < RTE_MAX_LCORE; i++) {
+			if (CBM_ISSET(i, &conf->pqos_l3ca_params[idx].cpumask) != 0)
+				RTE_LOG(INFO, PQOS, "CFG CPU: %u\n", i);
+		}
+
+		for (i = 0; i < RTE_MAX_LCORE; i++) {
+			if (CBM_ISSET(i, &conf->pqos_l3ca_params[idx].data_mask) != 0)
+				RTE_LOG(INFO, PQOS, "CFG (d)CBM bit: %u\n", i);
+
+			if (CBM_ISSET(i, &conf->pqos_l3ca_params[idx].code_mask) != 0)
+				RTE_LOG(INFO, PQOS, "CFG (c)CBM bit: %u\n", i);
+		}
+#endif
+		l3ca = end + 1;
+		idx++;
+	} while (*end != '\0');
+
+	ret = 0;
+
+err:
+	return ret;
+}
+
+static enum rte_pqos_cdp_config
+eal_parse_pqos_l3ca_cdp_state(const char *arg)
+{
+	if (arg == NULL)
+		return RTE_PQOS_REQUIRE_CDP_ANY;
+	if (strncasecmp(arg, OPT_PQOS_CDP_ON, sizeof(OPT_PQOS_CDP_ON)) == 0)
+		return RTE_PQOS_REQUIRE_CDP_ON;
+	if (strncasecmp(arg, OPT_PQOS_CDP_OFF, sizeof(OPT_PQOS_CDP_OFF)) == 0)
+		return RTE_PQOS_REQUIRE_CDP_OFF;
+	if (strncasecmp(arg, OPT_PQOS_CDP_ANY, sizeof(OPT_PQOS_CDP_ANY)) == 0)
+		return RTE_PQOS_REQUIRE_CDP_ANY;
+
+	return RTE_PQOS_REQUIRE_CDP_MAX;
+}
+#endif /* RTE_LIBRTE_PQOS && RTE_LIBRTE_PQOS_OPTS */
+
 int
 eal_parse_common_option(int opt, const char *optarg,
 			struct internal_config *conf)
@@ -897,6 +1149,24 @@ eal_parse_common_option(int opt, const char *optarg,
 		}
 		break;
 
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	case OPT_PQOS_L3CA_NUM:
+		if (eal_parse_pqos_l3ca(optarg, conf) < 0) {
+			RTE_LOG(ERR, EAL, "invalid parameter for --"
+				OPT_PQOS_L3CA "\n");
+			return -1;
+		}
+		break;
+
+	case OPT_PQOS_L3CA_DUMP_NUM:
+		conf->pqos_l3ca_dump = 1;
+		break;
+
+	case OPT_PQOS_L3CA_RESET_NUM:
+		conf->pqos_l3ca_reset = 1;
+		conf->pqos_l3ca_cdp_req = eal_parse_pqos_l3ca_cdp_state(optarg);
+		break;
+#endif
 	/* don't know what to do, leave this to caller */
 	default:
 		return 1;
@@ -969,7 +1239,13 @@ eal_check_common_options(struct internal_config *internal_cfg)
 			"cannot be used at the same time\n");
 		return -1;
 	}
-
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	if (internal_cfg->pqos_l3ca_reset == 1
+			&& internal_cfg->pqos_l3ca_cdp_req == RTE_PQOS_REQUIRE_CDP_MAX) {
+		RTE_LOG(ERR, EAL, "Invalid PQoS L3CA CDP state requested\n");
+		return -1;
+	}
+#endif
 	return 0;
 }
 
@@ -1011,6 +1287,24 @@ eal_common_usage(void)
 	       "  --"OPT_PROC_TYPE"         Type of this process (primary|secondary|auto)\n"
 	       "  --"OPT_SYSLOG"            Set syslog facility\n"
 	       "  --"OPT_LOG_LEVEL"         Set default log level\n"
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	       "  --"OPT_PQOS_L3CA" L3CA_CONFIG  Configures PQoS L3CA\n"
+	       "                      The argument format is\n"
+	       "                            '<cbm[@cpus]>[<,(ccbm,dcbm)[@cpus]>...]'\n"
+	       "                      cbm could be single mask or for CDP enabled system,\n"
+	       "                      group of two masks (code cbm and data cbm),\n"
+	       "                      '(' and ')' are necessary if it's a group\n"
+	       "                      cpus list is grouped by '(' and ')'\n"
+	       "                      Within the group, '-' is used for range separator,\n"
+	       "                      ',' is used for single number separator.\n"
+	       "                      '( )' can be omitted for single element group\n"
+	       "  --"OPT_PQOS_L3CA_DUMP"         Dumps current PQoS L3CA configuration\n"
+	       "  --"OPT_PQOS_L3CA_RESET"[=CDP]  Resets PQoS L3CA configuration\n"
+	       "                      The argument is optional and can be:\n"
+	       "                      "OPT_PQOS_CDP_ON" to enable CDP on reset\n"
+	       "                      "OPT_PQOS_CDP_OFF" to disable CDP on reset\n"
+	       "                      "OPT_PQOS_CDP_ANY" default one, leave CDP as it is\n"
+#endif
 	       "  -v                  Display version information on startup\n"
 	       "  -h, --help          This help\n"
 	       "\nEAL options for DEBUG use only:\n"
diff --git a/lib/librte_eal/common/eal_common_pqos.c b/lib/librte_eal/common/eal_common_pqos.c
new file mode 100644
index 0000000..44c61dd
--- /dev/null
+++ b/lib/librte_eal/common/eal_common_pqos.c
@@ -0,0 +1,590 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2016 Intel Corporation. All rights reserved.
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <getopt.h>
+#include <sched.h>
+#include <stdlib.h>
+
+#include <rte_log.h>
+#include <rte_memcpy.h>
+#include <rte_pqos.h>
+
+#include "eal_internal_cfg.h"
+#include "eal_options.h"
+#include "eal_private.h"
+
+static int
+check_lcores(void)
+{
+	unsigned i = 0;
+	unsigned lcore_id = 0;
+	unsigned cos_id = 0;
+	int ret = 0;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+		for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+
+			if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+				break;
+
+			if (CPU_ISSET(lcore_id, &internal_config.pqos_l3ca_params[i].cpumask) != 0) {
+				/* Check if lcore is being used by DPDK */
+				if (rte_lcore_is_enabled(lcore_id) == 0) {
+					RTE_LOG(ERR, PQOS, "lcore %u is not enabled.\n", lcore_id);
+					ret = -1;
+					goto out;
+				}
+
+				ret = rte_pqos_l3ca_assoc_get(lcore_id, &cos_id);
+				if (ret != 0) {
+					RTE_LOG(ERR, PQOS, "Failed to read COS associated to lcore %u.\n", lcore_id);
+					goto out;
+				}
+
+				/* Check if COS assigned to lcore is different then default one (#0) */
+				if (cos_id != 0) {
+					RTE_LOG(ERR, PQOS, "lcore %u has already associated COS#%u."
+						" Please use --%s option to reset L3CA.\n",
+						lcore_id, cos_id, OPT_PQOS_L3CA_RESET);
+					ret = -1;
+					goto out;
+				}
+			}
+		}
+	}
+
+out:
+
+	return ret;
+}
+
+static int
+check_cdp(const struct rte_pqos_cap *cap)
+{
+	unsigned i = 0;
+
+	RTE_PQOS_ASSERT(cap != NULL);
+	if (cap == NULL) {
+		RTE_LOG(ERR, PQOS, "%s:%u: Invalid param!\n", __func__, __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		if (internal_config.pqos_l3ca_params[i].cdp == 1
+				&& cap->cap_l3ca->cdp_on == 0) {
+			if (cap->cap_l3ca->cdp_supported == 0) {
+				RTE_LOG(ERR, PQOS, "CDP requested but not supported.\n");
+			} else {
+				RTE_LOG(ERR, PQOS, "CDP requested but not enabled."
+					" Please use --%s=%s option to reset L3CA and enable CDP.\n",
+					OPT_PQOS_L3CA_RESET, OPT_PQOS_CDP_ON);
+			}
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static int
+check_cbm_len(const struct rte_pqos_cap *cap)
+{
+	unsigned i = 0;
+	unsigned j = 0;
+	rte_pqos_cbm_t mask;
+	int ret = 0;
+
+	RTE_PQOS_ASSERT(cap != NULL);
+	if (cap == NULL) {
+		RTE_LOG(ERR, PQOS, "%s:%u: Invalid param!\n", __func__, __LINE__);
+		return -1;
+	}
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		if (internal_config.pqos_l3ca_params[i].cdp == 1) {
+			 CBM_OR(&mask, &internal_config.pqos_l3ca_params[i].data_mask,
+				 &internal_config.pqos_l3ca_params[i].code_mask);
+		} else {
+			rte_memcpy(&mask, &internal_config.pqos_l3ca_params[i].mask,
+				sizeof(rte_pqos_cbm_t));
+		}
+
+		/* check length of mask */
+		if ((unsigned)CBM_COUNT(&mask) > cap->cap_l3ca->cbm_len) {
+			ret = -1;
+			break;
+		}
+
+		/* check bits position */
+		for (j = 0; j < cap->cap_l3ca->cbm_len; j++)
+			CBM_CLR(j, &mask);
+
+		if (CBM_COUNT(&mask) != 0) {
+			ret = -1;
+			break;
+		}
+	}
+
+	if (ret != 0)
+		RTE_LOG(ERR, PQOS, "Requested CBM mask not supported by system.\n");
+
+	return ret;
+}
+
+static int
+check_cpu_sets_overlapping(void)
+{
+	unsigned i = 0;
+	unsigned j = 0;
+	rte_cpuset_t mask;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		for (j = i+1; j < RTE_MAX_LCORE; j++) {
+
+			CPU_AND(&mask, &internal_config.pqos_l3ca_params[i].cpumask,
+					&internal_config.pqos_l3ca_params[j].cpumask);
+
+			if (CPU_COUNT(&mask) != 0) {
+				RTE_LOG(ERR, PQOS, "Requested cpu sets are overlapping.\n");
+				return -1;
+			}
+		}
+	}
+
+	return 0;
+}
+
+static int
+check_cbm_contention(const struct rte_pqos_cap *cap)
+{
+	unsigned i = 0;
+	unsigned j = 0;
+	rte_pqos_cbm_t mask;
+
+	for (i = 0; i < cap->cap_l3ca->cbm_len; i++) {
+		if ((cap->cap_l3ca->cbm_contention_mask & (1 << i)) != 0) {
+			for (j = 0; j < RTE_MAX_LCORE; j++) {
+
+				if (CPU_COUNT(&internal_config.pqos_l3ca_params[j].cpumask) == 0)
+					break;
+
+				if (internal_config.pqos_l3ca_params[j].cdp == 1) {
+					 CBM_OR(&mask, &internal_config.pqos_l3ca_params[j].data_mask,
+						 &internal_config.pqos_l3ca_params[j].code_mask);
+				} else {
+					rte_memcpy(&mask, &internal_config.pqos_l3ca_params[j].mask,
+						sizeof(rte_pqos_cbm_t));
+				}
+
+				if (CBM_ISSET(i, &mask) != 0) {
+					RTE_LOG(WARNING, PQOS, "One or more of requested CBM masks"
+						" overlap CBM contention mask.\n");
+					return -1;
+				}
+			}
+		}
+	}
+
+	return 0;
+}
+
+static int
+check_and_select_coses(const struct rte_pqos_cap *cap)
+{
+	unsigned i = 0;
+
+	int phy_pkg_id = 0;
+	unsigned cos_id = 0;
+	unsigned lcore_id = 0;
+
+	unsigned phy_pkg_lcores[RTE_MAX_PHY_PKGS][RTE_MAX_LCORE];
+
+	unsigned *used_cos_table[RTE_MAX_PHY_PKGS];
+	int ret = 0;
+
+	RTE_PQOS_ASSERT(cap != NULL);
+	if (cap == NULL) {
+		RTE_LOG(ERR, PQOS, "%s:%u: Invalid param!\n", __func__, __LINE__);
+		return -1;
+	}
+
+	memset(phy_pkg_lcores, 0, sizeof(phy_pkg_lcores));
+
+	for (i = 0; i < RTE_MAX_PHY_PKGS; i++) {
+		used_cos_table[i] = (unsigned *)malloc(cap->cap_l3ca->cos_num * sizeof(unsigned));
+		memset(used_cos_table[i], 0, cap->cap_l3ca->cos_num * sizeof(unsigned));
+	}
+
+	/* detect currently used COS */
+	RTE_LCORE_FOREACH_DETECTED(lcore_id) {
+		ret = rte_pqos_l3ca_assoc_get(lcore_id, &cos_id);
+		if (ret != 0) {
+			RTE_LOG(ERR, PQOS, "Failed to read COS associated to lcore %u"
+				" on phy_pkg %u.\n", lcore_id, phy_pkg_id);
+			goto out;
+		}
+
+		phy_pkg_id = rte_lcore_to_phy_pkg_id(lcore_id);
+
+		if (phy_pkg_id >= 0 && used_cos_table[phy_pkg_id][cos_id] == 0) {
+			used_cos_table[phy_pkg_id][cos_id]++;
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+			RTE_LOG(INFO, PQOS, "USED: COS# %u on phy pkg %u.\n", cos_id, phy_pkg_id);
+#endif
+		}
+	}
+
+	/* look for avail. COS to fulfill requested config */
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		RTE_LCORE_FOREACH_DETECTED(lcore_id) {
+			if (CPU_ISSET(lcore_id, &internal_config.pqos_l3ca_params[i].cpumask) != 0) {
+				phy_pkg_id = rte_lcore_to_phy_pkg_id(lcore_id);
+				if (phy_pkg_id >= 0 && phy_pkg_lcores[phy_pkg_id][i] == 0) {
+					phy_pkg_lcores[phy_pkg_id][i]++;
+
+					for (cos_id = 0; cos_id < cap->cap_l3ca->cos_num; cos_id++) {
+						if (used_cos_table[phy_pkg_id][cos_id] == 0) {
+							used_cos_table[phy_pkg_id][cos_id]++;
+							internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id] = cos_id;
+							break;
+						}
+					}
+
+					if (cos_id == cap->cap_l3ca->cos_num) {
+						ret = -1;
+						goto out;
+					}
+				}
+			}
+		}
+	}
+
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+	/* For debug only, dump configuration */
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		RTE_PHY_PKG_FOREACH(phy_pkg_id) {
+			if (internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id] != 0)
+				RTE_LOG(INFO, PQOS, "COS# %u will be used for group %u on phy pkg %u.\n",
+					internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id], i, phy_pkg_id);
+		}
+	}
+#endif
+
+out:
+
+	for (i = 0; i < RTE_MAX_PHY_PKGS; i++)
+		free(used_cos_table[i]);
+
+	if (ret != 0)
+		RTE_LOG(ERR, PQOS, "Not enough available COS to configure requested"
+			" configuration.\n");
+
+	return ret;
+}
+
+static uint64_t
+cbm2uint64(rte_pqos_cbm_t *cpusetp)
+{
+	uint64_t result = 0;
+
+	if (CPU_COUNT(cpusetp) != 0) {
+		unsigned i = 0;
+		for (i = 0; i < sizeof(uint64_t) * 8; i++) {
+			if (CPU_ISSET(i, cpusetp) == 1)
+				result |= 1 << i;
+		}
+	}
+
+	return result;
+}
+
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+static void
+print_rte_pqos_l3ca(const struct rte_pqos_l3ca *l3ca)
+{
+	if (l3ca == NULL) {
+		printf("NULL pointer!\n");
+		return;
+	}
+
+	if (l3ca->cdp == 1) {
+		RTE_LOG(INFO, PQOS, "cos_id: %u, code_mask : %#"PRIx64", data_mask : %#"PRIx64", cdp: %u\n",
+			l3ca->cos_id, l3ca->code_mask, l3ca->data_mask, l3ca->cdp);
+	} else {
+		RTE_LOG(INFO, PQOS, "cos_id: %u, mask: %#"PRIx64", cdp: %u\n",
+			l3ca->cos_id, l3ca->mask, l3ca->cdp);
+	}
+}
+#endif
+
+static int
+configure_pqos_l3ca(void)
+{
+	unsigned lcore_id = 0;
+	int phy_pkg_id = 0;
+	unsigned i = 0;
+	struct rte_pqos_l3ca l3ca;
+	int ret = -1;
+
+	RTE_LCORE_FOREACH(lcore_id) {
+
+		phy_pkg_id = rte_lcore_to_phy_pkg_id(lcore_id);
+		if (phy_pkg_id < 0) {
+			ret = -1;
+			RTE_LOG(ERR, PQOS, "%s:%u: Invalid phy_pkg_id for lcore %u.\n",
+				__func__, __LINE__, lcore_id);
+			goto out;
+		}
+
+		for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+			if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+				break;
+
+			if (CPU_ISSET(lcore_id, &internal_config.pqos_l3ca_params[i].cpumask) == 1) {
+				lcore_config[lcore_id].pqos_l3ca_cos_id =
+					internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id];
+			}
+		}
+	}
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		memset(&l3ca, 0, sizeof(l3ca));
+
+		l3ca.cdp = internal_config.pqos_l3ca_params[i].cdp;
+		if (l3ca.cdp == 1) {
+			l3ca.code_mask =
+				cbm2uint64(&internal_config.pqos_l3ca_params[i].code_mask);
+			l3ca.data_mask =
+				cbm2uint64(&internal_config.pqos_l3ca_params[i].data_mask);
+		} else {
+			l3ca.mask  = cbm2uint64(&internal_config.pqos_l3ca_params[i].mask);
+		}
+
+		RTE_PHY_PKG_FOREACH(phy_pkg_id) {
+
+			if (internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id] == 0)
+				continue;
+
+			l3ca.cos_id = internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id];
+
+			ret = rte_pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+			if (ret == -1) {
+				RTE_LOG(ERR, PQOS, "%s:%u: Failed to set COS %u on phy_pkg %u.\n",
+					__func__, __LINE__, l3ca.cos_id, phy_pkg_id);
+				goto out;
+			}
+		}
+
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+		print_rte_pqos_l3ca(&l3ca);
+#endif
+	}
+
+	/* Associate COS to lcores */
+	RTE_LCORE_FOREACH(lcore_id) {
+		if (CPU_ISSET(lcore_id, &lcore_config[lcore_id].cpuset) == 1
+				&& CPU_COUNT(&lcore_config[lcore_id].cpuset) == 1) {
+
+			if (lcore_config[lcore_id].pqos_l3ca_cos_id != 0) {
+				ret = rte_pqos_l3ca_assoc_set(lcore_id,
+					lcore_config[lcore_id].pqos_l3ca_cos_id);
+				if (ret == -1) {
+					RTE_LOG(ERR, PQOS, "%s:%u: Failed to associate COS %u to lcore %u.\n",
+						__func__, __LINE__, l3ca.cos_id, lcore_id);
+					goto out;
+				}
+#ifdef RTE_LIBRTE_PQOS_DEBUG
+				RTE_LOG(DEBUG, PQOS, "COS %u associated to lcore %u.\n",
+					lcore_config[lcore_id].pqos_l3ca_cos_id, lcore_id);
+#endif
+			}
+		} else
+			RTE_LOG(DEBUG, PQOS, "Ignoring PQoS configuration for lcore %u "
+				"because of cpuset configuration other then \"1:1 map to cpu id\"\n",
+				lcore_id);
+	}
+
+out:
+	return ret;
+}
+
+static void
+pqos_l3ca_cleanup(void)
+{
+	unsigned lcore_id = 0;
+	int phy_pkg_id = 0;
+	unsigned i = 0;
+	struct rte_pqos_l3ca l3ca;
+	const struct rte_pqos_cap *cap = NULL;
+	int ret = -1;
+
+	RTE_LCORE_FOREACH(lcore_id) {
+		if (lcore_config[lcore_id].pqos_l3ca_cos_id != 0) {
+			ret = rte_pqos_l3ca_assoc_set(lcore_id, 0);
+			if (ret == -1) {
+				RTE_LOG(ERR, PQOS, "%s:%u: Failed to set COS# 0 on lcore %u.\n",
+					__func__, __LINE__, lcore_id);
+				goto out;
+			}
+		}
+	}
+
+	ret = rte_pqos_cap_get(&cap);
+	if (ret != 0 || cap == NULL || cap->cap_l3ca == NULL) {
+		ret = -1;
+		goto out;
+	}
+
+	memset(&l3ca, 0, sizeof(l3ca));
+	l3ca.mask = (1ULL << (cap->cap_l3ca->cbm_len)) - 1ULL;
+
+	for (i = 0; i < RTE_MAX_LCORE; i++) {
+
+		if (CPU_COUNT(&internal_config.pqos_l3ca_params[i].cpumask) == 0)
+			break;
+
+		RTE_PHY_PKG_FOREACH(phy_pkg_id) {
+			if (internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id] == 0)
+				continue;
+
+			l3ca.cos_id = internal_config.pqos_l3ca_params[i].cos_id[phy_pkg_id];
+
+			ret = rte_pqos_l3ca_set(phy_pkg_id, 1, &l3ca);
+			if (ret != 0) {
+				RTE_LOG(ERR, PQOS, "%s:%u: Failed to reset COS %u on phy_pkg %u.\n",
+					__func__, __LINE__, l3ca.cos_id, phy_pkg_id);
+				goto out;
+			}
+		}
+	}
+
+out:
+	if (ret != 0)
+		RTE_LOG(ERR, PQOS, "Clean-up failed\n");
+}
+
+int
+rte_eal_pqos_init(void)
+{
+	const struct rte_pqos_cap *cap = NULL;
+	int ret = -1;
+
+	ret = rte_pqos_init();
+	if (ret == -1)
+		goto out;
+	else if (ret == -2)
+		return 0;
+
+	ret = rte_pqos_cap_get(&cap);
+	if (ret != 0)
+		goto out;
+
+	/* Reset requested... */
+	if (internal_config.pqos_l3ca_reset == 1) {
+		ret = rte_pqos_l3ca_reset(cap, internal_config.pqos_l3ca_cdp_req);
+		if (ret != 0)
+			goto out;
+	}
+
+	/* no "--l3ca=" configuration, quit now */
+	if (CPU_COUNT(&internal_config.pqos_l3ca_params[0].cpumask) == 0)
+		goto out;
+
+	ret = check_lcores();
+	if (ret != 0)
+		goto out;
+
+	ret = check_cbm_len(cap);
+	if (ret != 0)
+		goto out;
+
+	ret = check_cpu_sets_overlapping();
+	if (ret != 0)
+		goto out;
+
+	/* Just a warning.... */
+	check_cbm_contention(cap);
+
+	ret = check_cdp(cap);
+	if (ret != 0)
+		goto out;
+
+	ret = check_and_select_coses(cap);
+	if (ret != 0)
+		goto out;
+
+	ret = configure_pqos_l3ca();
+	if (ret != 0)
+		goto out;
+
+	ret = atexit(pqos_l3ca_cleanup);
+	if (ret != 0) {
+		ret = -1;
+		RTE_LOG(ERR, PQOS, "Cannot set exit function\n");
+		goto out;
+	}
+
+out:
+	/* dump of current config requested... */
+	if (internal_config.pqos_l3ca_dump == 1) {
+		RTE_LOG(INFO, PQOS, "Current L3CA configuration:\n");
+		rte_pqos_l3ca_print_cfg();
+	}
+
+	return ret;
+}
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..c2095b7 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -41,6 +41,18 @@
 
 #include <rte_eal.h>
 #include <rte_pci_dev_feature_defs.h>
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+#include <rte_lcore.h>
+#include <rte_pqos.h>
+typedef rte_cpuset_t rte_pqos_cbm_t;
+#define CBM_SET(cpu, cpusetp) CPU_SET(cpu, cpusetp)
+#define CBM_CLR(cpu, cpusetp) CPU_CLR(cpu, cpusetp)
+#define CBM_ISSET(cpu, cpusetp) CPU_ISSET(cpu, cpusetp)
+#define CBM_ZERO(cpusetp) CPU_ZERO(cpusetp)
+#define CBM_COUNT(cpusetp) CPU_COUNT(cpusetp)
+#define CBM_AND(destset, srcset1, srcset2) CPU_AND(destset, srcset1, srcset2)
+#define CBM_OR(destset, srcset1, srcset2) CPU_OR(destset, srcset1, srcset2)
+#endif
 
 #define MAX_HUGEPAGE_SIZES 3  /**< support up to 3 page sizes */
 
@@ -56,6 +68,21 @@ struct hugepage_info {
 	int lock_descriptor;    /**< file descriptor for hugepage dir */
 };
 
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+struct l3ca_params {
+	rte_cpuset_t cpumask;
+	unsigned cdp;
+	unsigned cos_id[RTE_MAX_PHY_PKGS];
+	union {
+		rte_pqos_cbm_t mask; /* capacity bitmask (CBM) for L3 cache */
+		struct {
+			rte_pqos_cbm_t data_mask;
+			rte_pqos_cbm_t code_mask;
+		};
+	};
+};
+#endif
+
 /**
  * internal configuration
  */
@@ -86,6 +113,13 @@ struct internal_config {
 
 	unsigned num_hugepage_sizes;      /**< how many sizes on this system */
 	struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES];
+
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	volatile unsigned pqos_l3ca_reset;/**< true to reset L3CA functionality*/
+	volatile enum rte_pqos_cdp_config pqos_l3ca_cdp_req; /**< requested CDP state */
+	volatile unsigned pqos_l3ca_dump; /**< true to dump current L3CA config */
+	struct l3ca_params pqos_l3ca_params[RTE_MAX_LCORE]; /**< --l3ca= param data */
+#endif
 };
 extern struct internal_config internal_config; /**< Global EAL configuration. */
 
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..0f69548 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,17 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+#define OPT_PQOS_L3CA         "l3ca"
+	OPT_PQOS_L3CA_NUM,
+#define OPT_PQOS_L3CA_DUMP    "l3ca-dump"
+	OPT_PQOS_L3CA_DUMP_NUM,
+#define OPT_PQOS_L3CA_RESET   "l3ca-reset"
+#define OPT_PQOS_CDP_ON       "cdp_on"
+#define OPT_PQOS_CDP_OFF      "cdp_off"
+#define OPT_PQOS_CDP_ANY      "cdp_any"
+	OPT_PQOS_L3CA_RESET_NUM,
+#endif
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 4d3d283..0b1e3b0 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -353,6 +353,18 @@ int rte_eal_hugepage_attach(void);
  * This function is private to the EAL.
  */
 unsigned eal_cpu_phy_pkg_id(unsigned lcore_id);
+
+#ifdef RTE_LIBRTE_PQOS_OPTS
+/**
+ * Init PQoS.
+ *
+ * This function is private to EAL.
+ *
+ * @return
+ *  0 on success, negative on error
+ */
+int rte_eal_pqos_init(void);
+#endif /* RTE_LIBRTE_PQOS_OPTS */
 #endif
 
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h
index ef4235b..aadc48a 100644
--- a/lib/librte_eal/common/include/rte_lcore.h
+++ b/lib/librte_eal/common/include/rte_lcore.h
@@ -75,6 +75,9 @@ struct lcore_config {
 	rte_cpuset_t cpuset;       /**< cpu set which the lcore affinity to */
 #ifdef RTE_LIBRTE_PQOS
 	unsigned phy_pkg_id;       /**< physical packaged/socket number for this lcore */
+#ifdef RTE_LIBRTE_PQOS_OPTS
+	unsigned pqos_l3ca_cos_id; /**< L3CA Class-of-Service id to be set for lcore*/
+#endif /* RTE_LIBRTE_PQOS_OPTS */
 #endif
 };
 
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 97bd360..cc1cb2d 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -94,6 +94,11 @@ SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += rte_malloc.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += rte_keepalive.c
+ifeq ($(CONFIG_RTE_LIBRTE_PQOS),y)
+ifeq ($(CONFIG_RTE_LIBRTE_PQOS_OPTS),y)
+SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_pqos.c
+endif
+endif
 
 CFLAGS_eal.o := -D_GNU_SOURCE
 CFLAGS_eal_interrupts.o := -D_GNU_SOURCE
@@ -111,6 +116,11 @@ CFLAGS_eal_common_whitelist.o := -D_GNU_SOURCE
 CFLAGS_eal_common_options.o := -D_GNU_SOURCE
 CFLAGS_eal_common_thread.o := -D_GNU_SOURCE
 CFLAGS_eal_common_lcore.o := -D_GNU_SOURCE
+ifeq ($(CONFIG_RTE_LIBRTE_PQOS),y)
+ifeq ($(CONFIG_RTE_LIBRTE_PQOS_OPTS),y)
+CFLAGS_eal_common_pqos.o := -D_GNU_SOURCE
+endif
+endif
 
 # workaround for a gcc bug with noreturn attribute
 # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 15e806c..5f564b9 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -824,6 +824,11 @@ rte_eal_init(int argc, char **argv)
 	if (rte_eal_timer_init() < 0)
 		rte_panic("Cannot init HPET or TSC timers\n");
 
+#if defined(RTE_LIBRTE_PQOS) && defined(RTE_LIBRTE_PQOS_OPTS)
+	if (rte_eal_pqos_init() < 0)
+		rte_panic("Cannot init PQoS\n");
+#endif
+
 	eal_check_mem_on_local_socket();
 
 	rte_eal_mcfg_complete();
-- 
1.9.3

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-01-29 13:29, Wojciech Andralojc:
> EAL extension allows CAT and CDP technologies to be
> configured via "--l3ca*" parameters.

EAL should not depend on the PQoS library.
Please could you hook the command line parsing and help to
add these new options from outside?

> Reworking existing applications to make use of DPDK PQoS library is
> a heavy lift. This EAL extension is to make it easier by adding
> new command line options. These new options allow to leverage CAT and
> CDP technologies with existing code base without any additional
> development effort.

The command line options may be convenient in some cases but not
straightforward for some applications.
>From now, the runtime options must be also configurable with an API.
Please consider a clean configuration API first in the library.
Then it can be hooked in the EAL command line.
A thought: should we move the EAL command line in a separate library
with a hook API?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-24 15:14, Ananyev, Konstantin:
> 
> > > > 2016-02-24 10:22, Ananyev, Konstantin:
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > > > > Sent: Wednesday, February 24, 2016 10:10 AM
> > > > > > To: Thomas Monjalon
> > > > > > Cc: dev@dpdk.org; Kantecki, Tomasz
> > > > > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > > > > >
> > > > > > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > > > > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > > > > > DPDK?
> > > > > > > > The example makes it much easier to use the technology with DPDK.
> > > > > > > >
> > > > > > > > > Maybe the example should be better in the library itself.
> > > > > > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to
> > DPDK.
> > > > > > > >
> > > > > > > > > I suggest to mention the library in
> > > > > > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > > > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > > > > >
> > > > > > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > > > > > So yes, I would vote -1.
> > > > > > >
> > > > > > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > > > > > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > > > > > can be of great use to packet processing applications for helping to ensure that
> > > > > > the app gets the resources it needed - especially in a virtualised world - and
> > > > > > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > > > > > is well worthwhile having. It's more effective than a simple doc update in
> > > > > > raising awareness of the existence of the feature, and also provides for DPDK
> > > > > > users a readily available app for the user to start playing with to evaluate
> > > > > > PQoS for their own use-cases.
> > > > >
> > > > > +1
> > > > > I also think it is a good thing to have.
> > > > > Again user don't have to trust the whitepapers - instead he can run the app
> > > > > and measure performance gain on his particular platform.
> > > >
> > > > I totally agree the example is good to have.
> > > > Konstantin, are you thinking it must be hosted in the PQoS lib repository?
> > >
> > > Personally I prefer it to be part of dpdk samples.
> > > DPDK IO code path is a bit different from what the 'classical' user app usually does -
> > > a lot of polling, avoid system calls, etc.
> > > Also it would probably have much better visibility here.
> > > Again, as Bruce already mentioned,  we have QAT & TAP samples, why we can't have PQoS too.
> > 
> > Indeed the DPDK policies are really flexible.
> > How would you suggest to decide which examples can enter in DPDK?
> 
> That's a good question, for which I don't have an exact answer.
> Probably a good opportunity for the TB to show itself :)
> My input would be - to justify new sample for dpdk+third-party-lib it has to demonstrate one of:
> a) clear performance gain for the existing dpdk application,
> i.e under scenario X with library Y dpdk app Z shows N% better performance.
> (PQos example).
> b) how to integrate dpdk based app with some well-known and widely used technology.
> (tap example, using fuse to implement vhost example).
> c) How to expand packet processing with the functionality that is not part of dpdk project. 
> So yes, if tomorrow someone will come up with example that does packet compression,
> or encryption or DPI using some third party library, I think we at least have to consider to
> include it inside dpdk.org/examples.
> 
>  As a restriction I would put that the example has to be relatively small and simple 
> and demonstrate particular feature usage. 
> Plus I think that this third-party library has to be freely available and open-sourced. 

It looks reasonnable.
I'd like we have such description in the doc (doc/guides/sample_app_ug/).
If everybody agree on such doc patch, we would have an official policy.

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Ananyev, Konstantin]

> > > 2016-02-24 10:22, Ananyev, Konstantin:
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > > > Sent: Wednesday, February 24, 2016 10:10 AM
> > > > > To: Thomas Monjalon
> > > > > Cc: dev@dpdk.org; Kantecki, Tomasz
> > > > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > > > >
> > > > > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > > > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > > > > DPDK?
> > > > > > > The example makes it much easier to use the technology with DPDK.
> > > > > > >
> > > > > > > > Maybe the example should be better in the library itself.
> > > > > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to
> DPDK.
> > > > > > >
> > > > > > > > I suggest to mention the library in
> > > > > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > > > >
> > > > > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > > > > So yes, I would vote -1.
> > > > > >
> > > > > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > > > > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > > > > can be of great use to packet processing applications for helping to ensure that
> > > > > the app gets the resources it needed - especially in a virtualised world - and
> > > > > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > > > > is well worthwhile having. It's more effective than a simple doc update in
> > > > > raising awareness of the existence of the feature, and also provides for DPDK
> > > > > users a readily available app for the user to start playing with to evaluate
> > > > > PQoS for their own use-cases.
> > > >
> > > > +1
> > > > I also think it is a good thing to have.
> > > > Again user don't have to trust the whitepapers - instead he can run the app
> > > > and measure performance gain on his particular platform.
> > >
> > > I totally agree the example is good to have.
> > > Konstantin, are you thinking it must be hosted in the PQoS lib repository?
> >
> > Personally I prefer it to be part of dpdk samples.
> > DPDK IO code path is a bit different from what the 'classical' user app usually does -
> > a lot of polling, avoid system calls, etc.
> > Also it would probably have much better visibility here.
> > Again, as Bruce already mentioned,  we have QAT & TAP samples, why we can't have PQoS too.
> 
> Indeed the DPDK policies are really flexible.
> How would you suggest to decide which examples can enter in DPDK?

That's a good question, for which I don't have an exact answer.
Probably a good opportunity for the TB to show itself :)
My input would be - to justify new sample for dpdk+third-party-lib it has to demonstrate one of:
a) clear performance gain for the existing dpdk application,
i.e under scenario X with library Y dpdk app Z shows N% better performance.
(PQos example).
b) how to integrate dpdk based app with some well-known and widely used technology.
(tap example, using fuse to implement vhost example).
c) How to expand packet processing with the functionality that is not part of dpdk project. 
So yes, if tomorrow someone will come up with example that does packet compression,
or encryption or DPI using some third party library, I think we at least have to consider to
include it inside dpdk.org/examples.

 As a restriction I would put that the example has to be relatively small and simple 
and demonstrate particular feature usage. 
Plus I think that this third-party library has to be freely available and open-sourced. 

Konstantin

> Examples: what about a zip compression in the forwarding plane?
> What about a VM2VM failover synchronization?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Ananyev, Konstantin]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, February 24, 2016 11:46 AM
> To: Ananyev, Konstantin
> Cc: Richardson, Bruce; dev@dpdk.org; Kantecki, Tomasz
> Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> 
> 2016-02-24 11:21, Ananyev, Konstantin:
> >
> > > -----Original Message-----
> > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > Sent: Wednesday, February 24, 2016 10:35 AM
> > > To: Ananyev, Konstantin
> > > Cc: Richardson, Bruce; dev@dpdk.org; Kantecki, Tomasz
> > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > >
> > > 2016-02-24 10:22, Ananyev, Konstantin:
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > > > Sent: Wednesday, February 24, 2016 10:10 AM
> > > > > To: Thomas Monjalon
> > > > > Cc: dev@dpdk.org; Kantecki, Tomasz
> > > > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > > > >
> > > > > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > > > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > > > > DPDK?
> > > > > > > The example makes it much easier to use the technology with DPDK.
> > > > > > >
> > > > > > > > Maybe the example should be better in the library itself.
> > > > > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to
> DPDK.
> > > > > > >
> > > > > > > > I suggest to mention the library in
> > > > > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > > > >
> > > > > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > > > > So yes, I would vote -1.
> > > > > >
> > > > > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > > > > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > > > > can be of great use to packet processing applications for helping to ensure that
> > > > > the app gets the resources it needed - especially in a virtualised world - and
> > > > > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > > > > is well worthwhile having. It's more effective than a simple doc update in
> > > > > raising awareness of the existence of the feature, and also provides for DPDK
> > > > > users a readily available app for the user to start playing with to evaluate
> > > > > PQoS for their own use-cases.
> > > >
> > > > +1
> > > > I also think it is a good thing to have.
> > > > Again user don't have to trust the whitepapers - instead he can run the app
> > > > and measure performance gain on his particular platform.
> > >
> > > I totally agree the example is good to have.
> > > Konstantin, are you thinking it must be hosted in the PQoS lib repository?
> >
> > Personally I prefer it to be part of dpdk samples.
> > DPDK IO code path is a bit different from what the 'classical' user app usually does -
> > a lot of polling, avoid system calls, etc.
> > Also it would probably have much better visibility here.
> > Again, as Bruce already mentioned,  we have QAT & TAP samples, why we can't have PQoS too.
> 
> Indeed the DPDK policies are really flexible.
> How would you suggest to decide which examples can enter in DPDK?

That's a good question, for which I probably don't have an exact answer.
Probably a good opportunity for the TB to show itself :)
My input would be - to justify new sample for dpdk+third-party-lib it has to demonstrate one of:
a) clear performance gain for the existing dpdk application,
i.e under scenario X with library Y dpdk app Z shows N% better performance.
(PQos example).
b) how to integrate dpdk based app with some well-known and widely used technology.
(tap example, using fuse to implement vhost example).
c) How to expand packet processing with the functionality that is not part of dpdk project. 
So yes, if tomorrow someone will come up with example that does packet compression,
or encryption or DPI using some third party library, I think we at least have to consider to
include it inside dpdk.org/examples.

 As a restriction I would put that the example has to be relatively small and simple 
and demonstrate particular feature usage. 
Plus I think that this third-party library has to be freely available and open-sourced. 

Konstantin

> Examples: what about a zip compression in the forwarding plane?
> What about a VM2VM failover synchronization?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-24 11:21, Ananyev, Konstantin:
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Wednesday, February 24, 2016 10:35 AM
> > To: Ananyev, Konstantin
> > Cc: Richardson, Bruce; dev@dpdk.org; Kantecki, Tomasz
> > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > 
> > 2016-02-24 10:22, Ananyev, Konstantin:
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > > Sent: Wednesday, February 24, 2016 10:10 AM
> > > > To: Thomas Monjalon
> > > > Cc: dev@dpdk.org; Kantecki, Tomasz
> > > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > > >
> > > > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > > > DPDK?
> > > > > > The example makes it much easier to use the technology with DPDK.
> > > > > >
> > > > > > > Maybe the example should be better in the library itself.
> > > > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > > > > >
> > > > > > > I suggest to mention the library in
> > > > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > > >
> > > > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > > > So yes, I would vote -1.
> > > > >
> > > > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > > > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > > > can be of great use to packet processing applications for helping to ensure that
> > > > the app gets the resources it needed - especially in a virtualised world - and
> > > > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > > > is well worthwhile having. It's more effective than a simple doc update in
> > > > raising awareness of the existence of the feature, and also provides for DPDK
> > > > users a readily available app for the user to start playing with to evaluate
> > > > PQoS for their own use-cases.
> > >
> > > +1
> > > I also think it is a good thing to have.
> > > Again user don't have to trust the whitepapers - instead he can run the app
> > > and measure performance gain on his particular platform.
> > 
> > I totally agree the example is good to have.
> > Konstantin, are you thinking it must be hosted in the PQoS lib repository?
> 
> Personally I prefer it to be part of dpdk samples.
> DPDK IO code path is a bit different from what the 'classical' user app usually does -
> a lot of polling, avoid system calls, etc.
> Also it would probably have much better visibility here.
> Again, as Bruce already mentioned,  we have QAT & TAP samples, why we can't have PQoS too.

Indeed the DPDK policies are really flexible.
How would you suggest to decide which examples can enter in DPDK?
Examples: what about a zip compression in the forwarding plane?
What about a VM2VM failover synchronization?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Ananyev, Konstantin]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, February 24, 2016 10:35 AM
> To: Ananyev, Konstantin
> Cc: Richardson, Bruce; dev@dpdk.org; Kantecki, Tomasz
> Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> 
> 2016-02-24 10:22, Ananyev, Konstantin:
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > > Sent: Wednesday, February 24, 2016 10:10 AM
> > > To: Thomas Monjalon
> > > Cc: dev@dpdk.org; Kantecki, Tomasz
> > > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > >
> > > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > > DPDK?
> > > > > The example makes it much easier to use the technology with DPDK.
> > > > >
> > > > > > Maybe the example should be better in the library itself.
> > > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > > > >
> > > > > > I suggest to mention the library in
> > > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > >
> > > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > > So yes, I would vote -1.
> > > >
> > > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > > can be of great use to packet processing applications for helping to ensure that
> > > the app gets the resources it needed - especially in a virtualised world - and
> > > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > > is well worthwhile having. It's more effective than a simple doc update in
> > > raising awareness of the existence of the feature, and also provides for DPDK
> > > users a readily available app for the user to start playing with to evaluate
> > > PQoS for their own use-cases.
> >
> > +1
> > I also think it is a good thing to have.
> > Again user don't have to trust the whitepapers - instead he can run the app
> > and measure performance gain on his particular platform.
> 
> I totally agree the example is good to have.
> Konstantin, are you thinking it must be hosted in the PQoS lib repository?

Personally I prefer it to be part of dpdk samples.
DPDK IO code path is a bit different from what the 'classical' user app usually does -
a lot of polling, avoid system calls, etc.
Also it would probably have much better visibility here.
Again, as Bruce already mentioned,  we have QAT & TAP samples, why we can't have PQoS too.
Konstantin

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Bruce Richardson]

On Wed, Feb 24, 2016 at 11:31:47AM +0100, Thomas Monjalon wrote:
> 2016-02-24 10:10, Bruce Richardson:
> > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > DPDK?
> > > > The example makes it much easier to use the technology with DPDK.
> > > > 
> > > > > Maybe the example should be better in the library itself.
> > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > > > 
> > > > > I suggest to mention the library in
> > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > > 
> > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > So yes, I would vote -1.
> > > 
> > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > can be of great use to packet processing applications for helping to ensure that
> > the app gets the resources it needed - especially in a virtualised world - and
> > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > is well worthwhile having. It's more effective than a simple doc update in
> > raising awareness of the existence of the feature, and also provides for DPDK
> > users a readily available app for the user to start playing with to evaluate
> > PQoS for their own use-cases.
> > I also fail to see what the downside of having the sample app is - it won't add
> > significantly to the project maintenance overhead.
> 
> We need to draw a line in the sand to decide what can go in DPDK examples because
> any code using some DPDK functions could require to be integrated.
> Until now, the examples were used to demonstrate some DPDK API.
> Here you are advocating that examples are "marketing tools" to bring awareness on
> a library.
> I have no strong opinion, except that we cannot host and maintain 100 examples.
> Other opinions are welcome.

I think it comes down to whether it contributes something that would be useful
and appreciated by the users. 

[While the majority of sample apps have been for showing off DPDK APIs, that's
not exclusively the case. There is the DPDK-QAT example, and also the
exception_path example showing how to use DPDK with TUN/TAP. I believe both
these add value for DPDK users and these examples, and others like them,
are worth having.]

So, in summary, I honestly think that having an example that shows platform 
QoS functionality being used with DPDK is worth having as it is something that
would be useful to DPDK users.

/Bruce

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-24 10:22, Ananyev, Konstantin:
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> > Sent: Wednesday, February 24, 2016 10:10 AM
> > To: Thomas Monjalon
> > Cc: dev@dpdk.org; Kantecki, Tomasz
> > Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> > 
> > On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > > DPDK?
> > > > The example makes it much easier to use the technology with DPDK.
> > > >
> > > > > Maybe the example should be better in the library itself.
> > > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > > >
> > > > > I suggest to mention the library in
> > > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > >
> > > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > > So yes, I would vote -1.
> > >
> > Well, the intersection of DPDK and PQoS is what the example is really all about,
> > and as such it is relevant to both DPDK and the library itself. Platform QoS
> > can be of great use to packet processing applications for helping to ensure that
> > the app gets the resources it needed - especially in a virtualised world - and
> > so we believe that having an example in DPDK showing how to use PQoS with DPDK
> > is well worthwhile having. It's more effective than a simple doc update in
> > raising awareness of the existence of the feature, and also provides for DPDK
> > users a readily available app for the user to start playing with to evaluate
> > PQoS for their own use-cases.
> 
> +1 
> I also think it is a good thing to have.
> Again user don't have to trust the whitepapers - instead he can run the app 
> and measure performance gain on his particular platform.

I totally agree the example is good to have.
Konstantin, are you thinking it must be hosted in the PQoS lib repository?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-24 10:10, Bruce Richardson:
> On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > DPDK?
> > > The example makes it much easier to use the technology with DPDK.
> > > 
> > > > Maybe the example should be better in the library itself.
> > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > > 
> > > > I suggest to mention the library in
> > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> > 
> > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > So yes, I would vote -1.
> > 
> Well, the intersection of DPDK and PQoS is what the example is really all about,
> and as such it is relevant to both DPDK and the library itself. Platform QoS
> can be of great use to packet processing applications for helping to ensure that
> the app gets the resources it needed - especially in a virtualised world - and
> so we believe that having an example in DPDK showing how to use PQoS with DPDK
> is well worthwhile having. It's more effective than a simple doc update in
> raising awareness of the existence of the feature, and also provides for DPDK
> users a readily available app for the user to start playing with to evaluate
> PQoS for their own use-cases.
> I also fail to see what the downside of having the sample app is - it won't add
> significantly to the project maintenance overhead.

We need to draw a line in the sand to decide what can go in DPDK examples because
any code using some DPDK functions could require to be integrated.
Until now, the examples were used to demonstrate some DPDK API.
Here you are advocating that examples are "marketing tools" to bring awareness on
a library.
I have no strong opinion, except that we cannot host and maintain 100 examples.
Other opinions are welcome.

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Wednesday, February 24, 2016 10:10 AM
> To: Thomas Monjalon
> Cc: dev@dpdk.org; Kantecki, Tomasz
> Subject: Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension
> 
> On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> > 2016-02-23 23:03, Kantecki, Tomasz:
> > > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > > DPDK?
> > > The example makes it much easier to use the technology with DPDK.
> > >
> > > > Maybe the example should be better in the library itself.
> > > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > >
> > > > I suggest to mention the library in
> > > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> >
> > I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> > So yes, I would vote -1.
> >
> Well, the intersection of DPDK and PQoS is what the example is really all about,
> and as such it is relevant to both DPDK and the library itself. Platform QoS
> can be of great use to packet processing applications for helping to ensure that
> the app gets the resources it needed - especially in a virtualised world - and
> so we believe that having an example in DPDK showing how to use PQoS with DPDK
> is well worthwhile having. It's more effective than a simple doc update in
> raising awareness of the existence of the feature, and also provides for DPDK
> users a readily available app for the user to start playing with to evaluate
> PQoS for their own use-cases.

+1 
I also think it is a good thing to have.
Again user don't have to trust the whitepapers - instead he can run the app 
and measure performance gain on his particular platform.
Konstantin   

> I also fail to see what the downside of having the sample app is - it won't add
> significantly to the project maintenance overhead.
> 
> Regards,
> /Bruce

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Bruce Richardson]

On Wed, Feb 24, 2016 at 09:24:33AM +0100, Thomas Monjalon wrote:
> 2016-02-23 23:03, Kantecki, Tomasz:
> > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > If there is nothing specific in DPDK for PQos, why writing an example in
> > > DPDK?
> > The example makes it much easier to use the technology with DPDK.
> > 
> > > Maybe the example should be better in the library itself.
> > The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> > 
> > > I suggest to mention the library in
> > > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> > Ok it can be added to this document. Does it imply -1 for the sample code idea?
> 
> I may be wrong but I have the feeling the example is more about PQoS than DPDK.
> So yes, I would vote -1.
> 
Well, the intersection of DPDK and PQoS is what the example is really all about,
and as such it is relevant to both DPDK and the library itself. Platform QoS
can be of great use to packet processing applications for helping to ensure that
the app gets the resources it needed - especially in a virtualised world - and
so we believe that having an example in DPDK showing how to use PQoS with DPDK
is well worthwhile having. It's more effective than a simple doc update in
raising awareness of the existence of the feature, and also provides for DPDK
users a readily available app for the user to start playing with to evaluate
PQoS for their own use-cases.
I also fail to see what the downside of having the sample app is - it won't add
significantly to the project maintenance overhead.

Regards,
/Bruce

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-23 23:03, Kantecki, Tomasz:
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > If there is nothing specific in DPDK for PQos, why writing an example in
> > DPDK?
> The example makes it much easier to use the technology with DPDK.
> 
> > Maybe the example should be better in the library itself.
> The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.
> 
> > I suggest to mention the library in
> > doc/guides/linux_gsg/nic_perf_intel_platform.rst
> Ok it can be added to this document. Does it imply -1 for the sample code idea?

I may be wrong but I have the feeling the example is more about PQoS than DPDK.
So yes, I would vote -1.

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Kantecki, Tomasz]

> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> If there is nothing specific in DPDK for PQos, why writing an example in
> DPDK?
The example makes it much easier to use the technology with DPDK.

> Maybe the example should be better in the library itself.
The library in question (https://github.com/01org/intel-cmt-cat) has a couple of examples but none of them refers to DPDK.

> I suggest to mention the library in
> doc/guides/linux_gsg/nic_perf_intel_platform.rst
Ok it can be added to this document. Does it imply -1 for the sample code idea?

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-23 10:03, Kantecki, Tomasz:
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > Are you OK with concept behind V2 PQoS patch as described above?
> > 
> > So you mean that PQoS don't really need some DPDK integration?
> > You just want to show how to use it with DPDK?
> 
> You are correct. The technology can be used without tight DPDK integration but will require users to reach out to existing open source code.
> Based on received feedback about EAL PQoS extensions we backed off to sort of a sample code.

If there is nothing specific in DPDK for PQos, why writing an example in DPDK?
Maybe the example should be better in the library itself.
I suggest to mention the library in doc/guides/linux_gsg/nic_perf_intel_platform.rst

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Kantecki, Tomasz]

> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Are you OK with concept behind V2 PQoS patch as described above?
> 
> So you mean that PQoS don't really need some DPDK integration?
> You just want to show how to use it with DPDK?

You are correct. The technology can be used without tight DPDK integration but will require users to reach out to existing open source code.
Based on received feedback about EAL PQoS extensions we backed off to sort of a sample code.

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Thomas Monjalon]

2016-02-22 13:12, Kantecki, Tomasz:
> Off the mail list, we received feedback that extending EAL commands is not an option due to Intel Architecture nature of CAT.
> Consequently, we changed direction when working on V2 PQoS patches in roder to: 
> - reduce DPDK PQoS code print and reuse existing open source code
> - remove IA specific options from EAL
> 
> Current V2 patch, yet to be shared on the mail-list in a few days, implements a sample code (reuse of skeleton) that links against existing 01.org PQoS library. This eliminates need for code included in V1 librte_pqos patch.
> The same sample code implements a C module that parses application specific part of the command line with CAT configuration options (same format as V1 EAL command). The module is easy to re-use in other applications as needed.
> In the future, if there is a need to change this C module into a regular DPDK library then we'll be happy to do so.
> 
> Are you OK with concept behind V2 PQoS patch as described above?

So you mean that PQoS don't really need some DPDK integration?
You just want to show how to use it with DPDK?

Re: [dpdk-dev] [PATCH] eal: Initial implementation of PQoS EAL extension

[Kantecki, Tomasz]

> EAL should not depend on the PQoS library.
> Please could you hook the command line parsing and help to add these new
> options from outside?

Thanks for the feedback - we heard about dependency problem from a few sources.
 
> The command line options may be convenient in some cases but not
> straightforward for some applications.
> >From now, the runtime options must be also configurable with an API.
> Please consider a clean configuration API first in the library.
> Then it can be hooked in the EAL command line.
> A thought: should we move the EAL command line in a separate library with a
> hook API?

Thanks for the suggestion and the feedback.

Off the mail list, we received feedback that extending EAL commands is not an option due to Intel Architecture nature of CAT.
Consequently, we changed direction when working on V2 PQoS patches in roder to: 
- reduce DPDK PQoS code print and reuse existing open source code
- remove IA specific options from EAL

Current V2 patch, yet to be shared on the mail-list in a few days, implements a sample code (reuse of skeleton) that links against existing 01.org PQoS library. This eliminates need for code included in V1 librte_pqos patch.
The same sample code implements a C module that parses application specific part of the command line with CAT configuration options (same format as V1 EAL command). The module is easy to re-use in other applications as needed.
In the future, if there is a need to change this C module into a regular DPDK library then we'll be happy to do so.

Are you OK with concept behind V2 PQoS patch as described above?

Thanks,
Tomasz 
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.