[PATCH v4 0/3] add uncore api to be called through l3fwd-power

[PATCH v4 0/3] add uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to allow uncore frequency
adjustment. This will be called through the l3fwd-power app and gives the ability to set the 
minimum and maximum uncore frequency to both min, max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.

Tadhg Kearney (3):
  power: add uncore frequency control API to the power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_uncore.c                  | 299 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  37 ++
 doc/guides/rel_notes/release_22_11.rst        |   5 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 122 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_uncore.c                  | 447 ++++++++++++++++++
 lib/power/rte_power_uncore.h                  | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1145 insertions(+), 3 deletions(-)
 create mode 100644 app/test/test_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

-- 
2.25.1

[PATCH v4 1/3] power: add uncore frequency control API to the power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 21453 bytes --]

Add API to allow uncore frequency adjustment. This is done through
manipulating related uncore frequency control sysfs entries to
adjust the minimum and maximum uncore frequency values.
Nine API's are being added that are all public and experimental.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  37 ++
 doc/guides/rel_notes/release_22_11.rst |   5 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_uncore.c           | 447 +++++++++++++++++++++++++
 lib/power/rte_power_uncore.h           | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 696 insertions(+)
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..49ff3edef0 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,43 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Uncore API
+----------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Uncore
+~~~~~~~~~~~~~~~~~~~~~~~
+* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
+  original min & max for pkg & die.
+
+* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
+
+* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
+
+* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
+
+* **Uncore Power Max**: Set max uncore freq index for pkg & die.
+
+* **Uncore Power Min**: Set min uncore freq index for pkg & die.
+
+* **Get Num Freqs**: Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
+
+* **Get Num Dies**: Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 8c021cf050..8e184034d8 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -55,6 +55,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added uncore frequency control API to the power library.**
+
+  Add api to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..80cdeb72d4 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
new file mode 100644
index 0000000000..b3004e5bfc
--- /dev/null
+++ b/lib/power/rte_power_uncore.c
@@ -0,0 +1,447 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_uncore.h b/lib/power/rte_power_uncore.h
new file mode 100644
index 0000000000..2be6546f49
--- /dev/null
+++ b/lib/power/rte_power_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_UNCORE_H
+#define _RTE_POWER_UNCORE_H
+
+/**
+ * @file
+ * RTE Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set the new frequency for a specific die on a package by indicating the index of
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Scale up the frequency of a specific die on a package to the highest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Scale down the frequency of a specific die on a package to the lowest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v4 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 +++++
 examples/l3fwd-power/main.c                   | 122 +++++++++++++++++-
 2 files changed, 148 insertions(+), 3 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..d1a32594c0 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -179,6 +180,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1623,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1682,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1838,7 +1916,7 @@ parse_ep_config(const char *q_arg)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, enabled_uncore = 0;
 	char **argvopt;
 	int option_index;
 	uint32_t limit;
@@ -1864,7 +1942,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P:uUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1971,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2463,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2476,23 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_exit(pkg, die);
+			if (ret < 0) {
+				RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deninit successfully for pkg %02u die %02u\n"
+					, pkg, die);
+				return -1;
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v4 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 app/test/meson.build         |   2 +
 app/test/test_power_uncore.c | 299 +++++++++++++++++++++++++++++++++++
 2 files changed, 301 insertions(+)
 create mode 100644 app/test/test_power_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index bf1d81f84a..170401ccdc 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -100,6 +100,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -240,6 +241,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_uncore.c b/app/test/test_power_uncore.c
new file mode 100644
index 0000000000..7bc3ed7260
--- /dev/null
+++ b/app/test/test_power_uncore.c
@@ -0,0 +1,299 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_uncore);
-- 
2.25.1

Re: [PATCH v4 2/3] l3fwd-power: add option to call uncore API

[Hunt, David]

On 20/09/2022 10:48, Tadhg Kearney wrote:
> Add option for setting uncore frequency min/max/index, through uncore API.
> This will be set for each package and die on the SKU. On exit, uncore min
> and max frequency will be reverted back to previous frequencies.
>
> Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
> ---
>   .../sample_app_ug/l3_forward_power_man.rst    |  29 +++++
>   examples/l3fwd-power/main.c                   | 122 +++++++++++++++++-
>   2 files changed, 148 insertions(+), 3 deletions(-)
>
> diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
> index 8f6d906200..08ac8ef369 100644
> --- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
> +++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
> @@ -97,6 +97,12 @@ where,
>   *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
>       Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
>   
> +*   -u: optional, sets uncore min/max frequency to minimum value.
> +
> +*   -U: optional, sets uncore min/max frequency to maximum value.
> +
> +*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
> +
>   *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
>   
>   *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
> @@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
>   .. code-block:: console
>   
>           ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
> +
> +Setting Uncore Values
> +---------------------
> +
> +Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
> +This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
> +Three options are available for setting uncore frequency:
> +
> +``-u``
> +  This will set uncore minimum and maximum frequencies to minimum possible value.
> +
> +``-U``
> +  This will set uncore minimum and maximum frequencies to maximum possible value.
> +
> +``-i``
> +  This will allow you to set the specific uncore frequency index that you want, by setting
> +  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
> +  maximum to minimum.
> +  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
> +
> +.. code-block:: console
> +
> +        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
> diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
> index 887c6eae3f..d1a32594c0 100644
> --- a/examples/l3fwd-power/main.c
> +++ b/examples/l3fwd-power/main.c
> @@ -47,6 +47,7 @@
>   #include <rte_metrics.h>
>   #include <rte_telemetry.h>
>   #include <rte_power_pmd_mgmt.h>
> +#include <rte_power_uncore.h>
>   
>   #include "perf_core.h"
>   #include "main.h"
> @@ -179,6 +180,12 @@ enum busy_rate {
>   	FULL = 100
>   };
>   
> +enum uncore_choice {
> +	UNCORE_MIN = 0,
> +	UNCORE_MAX = 1,
> +	UNCORE_IDX = 2
> +};
> +
>   /* reference poll count to measure core busyness */
>   #define DEFAULT_COUNT 10000
>   /*
> @@ -1616,6 +1623,9 @@ print_usage(const char *prgname)
>   		"  [--max-pkt-len PKTLEN]\n"
>   		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
>   		"  -P: enable promiscuous mode\n"
> +		"  -u: set min/max frequency for uncore to minimum value\n"
> +		"  -U: set min/max frequency for uncore to maximum value\n"
> +		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
>   		"  --config (port,queue,lcore): rx queues configuration\n"
>   		"  --high-perf-cores CORELIST: list of high performance cores\n"
>   		"  --perf-config: similar as config, cores specified as indices"
> @@ -1672,6 +1682,74 @@ static int parse_max_pkt_len(const char *pktlen)
>   	return len;
>   }
>   
> +static int
> +parse_uncore_options(enum uncore_choice choice, const char *argument)
> +{
> +	unsigned int die, pkg, max_pkg, max_die;
> +	int ret = 0;
> +	max_pkg = rte_power_uncore_get_num_pkgs();
> +	if (max_pkg == 0)
> +		return -1;
> +
> +	for (pkg = 0; pkg < max_pkg; pkg++) {
> +		max_die = rte_power_uncore_get_num_dies(pkg);
> +		if (max_die == 0)
> +			return -1;
> +		for (die = 0; die < max_die; die++) {
> +			ret = rte_power_uncore_init(pkg, die);
> +			if (ret == -1) {
> +				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
> +				, pkg, die);
> +				return ret;
> +			}
> +			if (choice == UNCORE_MIN) {
> +				ret = rte_power_uncore_freq_min(pkg, die);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
> +					, pkg, die);
> +					return ret;
> +				}
> +			} else if (choice == UNCORE_MAX) {
> +				ret = rte_power_uncore_freq_max(pkg, die);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
> +					, pkg, die);
> +					return ret;
> +				}
> +			} else if (choice == UNCORE_IDX) {
> +				char *ptr = NULL;
> +				int frequency_index = strtol(argument, &ptr, 10);
> +				if (argument == ptr) {
> +					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
> +					return -1;
> +				}
> +				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
> +				if (frequency_index > freq_array_len - 1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Frequency index given out of range, please choose a value from 0 to %d.\n",
> +					freq_array_len);
> +					return -1;
> +				}
> +				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
> +					pkg, die);
> +					return ret;
> +				}
> +			} else {
> +				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
> +				return -1;
> +			}
> +		}
> +	}
> +
> +	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
> +	return ret;
> +}
> +
>   static int
>   parse_portmask(const char *portmask)
>   {
> @@ -1838,7 +1916,7 @@ parse_ep_config(const char *q_arg)
>   static int
>   parse_args(int argc, char **argv)
>   {
> -	int opt, ret;
> +	int opt, ret, enabled_uncore = 0;
>   	char **argvopt;
>   	int option_index;
>   	uint32_t limit;
> @@ -1864,7 +1942,7 @@ parse_args(int argc, char **argv)
>   
>   	argvopt = argv;
>   
> -	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
> +	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P:uUi:",
>   				lgopts, &option_index)) != EOF) {
>   
>   		switch (opt) {
> @@ -1893,6 +1971,27 @@ parse_args(int argc, char **argv)
>   			limit = parse_max_pkt_len(optarg);
>   			freq_tlb[HGH] = limit;
>   			break;
> +		case 'u':
> +			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
> +		case 'U':
> +			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
> +		case 'i':
> +			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
>   		/* long options */
>   		case 0:
>   			if (!strncmp(lgopts[option_index].name, "config", 6)) {
> @@ -2364,7 +2463,7 @@ init_power_library(void)
>   static int
>   deinit_power_library(void)
>   {
> -	unsigned int lcore_id;
> +	unsigned int lcore_id, max_pkg, max_die, die, pkg;
>   	int ret = 0;
>   
>   	RTE_LCORE_FOREACH(lcore_id) {
> @@ -2377,6 +2476,23 @@ deinit_power_library(void)
>   			return ret;
>   		}
>   	}
> +
> +	max_pkg = rte_power_uncore_get_num_pkgs();
> +	if (max_pkg == 0)
> +		return -1;
> +	for (pkg = 0; pkg < max_pkg; pkg++) {
> +		max_die = rte_power_uncore_get_num_dies(pkg);
> +		if (max_die == 0)
> +			return -1;
> +		for (die = 0; die < max_die; die++) {
> +			ret = rte_power_uncore_exit(pkg, die);
> +			if (ret < 0) {
> +				RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deninit successfully for pkg %02u die %02u\n"
> +					, pkg, die);
> +				return -1;
> +			}
> +		}
> +	}
>   	return ret;
>   }
>   

Thanks Tadhg. I've also tested this and it sets the uncore frequencies 
as expected with the -U, -u and -i options.

Reviewed-by: David Hunt <david.hunt@intel.com>

Re: [PATCH v4 1/3] power: add uncore frequency control API to the power library

[Hunt, David]

On 20/09/2022 10:48, Tadhg Kearney wrote:
> Add API to allow uncore frequency adjustment. This is done through
> manipulating related uncore frequency control sysfs entries to
> adjust the minimum and maximum uncore frequency values.
> Nine API's are being added that are all public and experimental.
>
> Signed-off-by: Tadhg Kearney<tadhg.kearney@intel.com>
> ---
>   doc/guides/prog_guide/power_man.rst    |  37 ++
>   doc/guides/rel_notes/release_22_11.rst |   5 +
>   lib/power/meson.build                  |   2 +
>   lib/power/rte_power_uncore.c           | 447 +++++++++++++++++++++++++
>   lib/power/rte_power_uncore.h           | 194 +++++++++++
>   lib/power/version.map                  |  11 +
>   6 files changed, 696 insertions(+)
>   create mode 100644 lib/power/rte_power_uncore.c
>   create mode 100644 lib/power/rte_power_uncore.h
>
> diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
> index 98cfd3c1f3..49ff3edef0 100644
> --- a/doc/guides/prog_guide/power_man.rst
> +++ b/doc/guides/prog_guide/power_man.rst
> @@ -276,6 +276,43 @@ API Overview for Ethernet PMD Power Management
>   * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
>     Scaling mode.
>   
> +Uncore API
> +----------
> +
> +Abstract
> +~~~~~~~~
> +
> +Uncore is a term used by Intel to describe the functions of a microprocessor that are
> +not in the core, but which must be closely connected to the core to achieve high performance;
> +L3 cache, on-die memory controller, etc.
> +Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
> +
> +The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
> +for x86 platform. The driver is available from kernel version 5.6 and above.
> +This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.

Need to mention that the uncore frequency control is made available in 
the kernel by enabling the CONFIG_INTEL_UNCORE_FREQ_CONTROL kernel 
option which was added in 5.6


> +
> +
> +API Overview for Uncore
> +~~~~~~~~~~~~~~~~~~~~~~~
> +* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
> +  original min & max for pkg & die.
> +
> +* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
> +
> +* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
> +
> +* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
> +
> +* **Uncore Power Max**: Set max uncore freq index for pkg & die.
> +
> +* **Uncore Power Min**: Set min uncore freq index for pkg & die.
> +
> +* **Get Num Freqs**: Get the number of frequencies in the index array.
> +
> +* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
> +
> +* **Get Num Dies**: Get the number of die's on a given package.
> +
>   References
>   ----------
>   
> diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
> index 8c021cf050..8e184034d8 100644
> --- a/doc/guides/rel_notes/release_22_11.rst
> +++ b/doc/guides/rel_notes/release_22_11.rst
> @@ -55,6 +55,11 @@ New Features
>        Also, make sure to start the actual text at the margin.
>        =======================================================
>   
> +* **Added uncore frequency control API to the power library.**
> +
> +  Add api to allow uncore frequency adjustment. This is done through
> +  manipulating related uncore frequency control sysfs entries to
> +  adjust the minimum and maximum uncore frequency values.
>   
>   Removed Items
>   -------------
> diff --git a/lib/power/meson.build b/lib/power/meson.build
> index ba8d66074b..80cdeb72d4 100644
> --- a/lib/power/meson.build
> +++ b/lib/power/meson.build
> @@ -21,12 +21,14 @@ sources = files(
>           'rte_power.c',
>           'rte_power_empty_poll.c',
>           'rte_power_pmd_mgmt.c',
> +        'rte_power_uncore.c',
>   )
>   headers = files(
>           'rte_power.h',
>           'rte_power_empty_poll.h',
>           'rte_power_pmd_mgmt.h',
>           'rte_power_guest_channel.h',
> +        'rte_power_uncore.h',
>   )
>   if cc.has_argument('-Wno-cast-qual')
>       cflags += '-Wno-cast-qual'
> diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
> new file mode 100644
> index 0000000000..b3004e5bfc
> --- /dev/null
> +++ b/lib/power/rte_power_uncore.c
> @@ -0,0 +1,447 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2022 Intel Corporation
> + */
> +
> +#include <errno.h>
> +#include <dirent.h>
> +#include <fnmatch.h>
> +
> +#include <rte_memcpy.h>
> +
> +#include "rte_power_uncore.h"
> +#include "power_common.h"
> +
> +#define MAX_UNCORE_FREQS 32
> +#define MAX_NUMA_DIE 8
> +#define BUS_FREQ     100000
> +#define FILTER_LENGTH 18
> +#define PACKAGE_FILTER "package_%02u_die_*"
> +#define DIE_FILTER "package_%02u_die_%02u"
> +#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
> +#define POWER_GOVERNOR_PERF "performance"
> +#define POWER_UNCORE_SYSFILE_MAX_FREQ \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
> +#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
> +#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
> +#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
> +
> +
> +struct uncore_power_info {
> +	unsigned int die;                    /** Core die id */
> +	unsigned int pkg;                    /** Package id */
> +	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
> +	uint32_t nb_freqs;                   /** Number of available freqs */
> +	FILE *f_cur_min;                     /** FD of scaling_min */
> +	FILE *f_cur_max;                     /** FD of scaling_max */
> +	uint32_t curr_idx;                   /** Freq index in freqs array */
> +	uint32_t org_min_freq;               /** Original min freq of uncore */
> +	uint32_t org_max_freq;               /** Original max freq of uncore */
> +	uint32_t init_max_freq;              /** System max uncore freq */
> +	uint32_t init_min_freq;              /** System min uncore freq */
> +} __rte_cache_aligned;
> +
> +static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
> +
> +static int
> +set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
> +{
> +	uint32_t target_uncore_freq, curr_max_freq;
> +	int ret;
> +
> +	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
> +		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
> +				"should be less than %u\n", idx, ui->nb_freqs);
> +		return -1;
> +	}
> +
> +	target_uncore_freq = ui->freqs[idx];
> +
> +	/* check current max freq, so that the value to be flushed first
> +	 * can be accurately recorded
> +	 */
> +	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
> +			ui->pkg, ui->die);
> +	if (ui->f_cur_max == NULL) {
> +		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
> +				POWER_UNCORE_SYSFILE_MAX_FREQ);
> +		return -1;
> +	}
> +	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
> +	if (ret < 0) {
> +		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
> +				POWER_UNCORE_SYSFILE_MAX_FREQ);
> +		fclose(ui->f_cur_max);
> +		return -1;
> +	}
> +
> +	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
> +	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
> +		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
> +				"pkg %02u die %02u\n", ui->pkg, ui->die);
> +		return -1;
> +	}
> +
> +	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
> +		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
> +				"pkg %02u die %02u\n", ui->pkg, ui->die);
> +		return -1;
> +	}
> +
> +	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
> +				target_uncore_freq, ui->pkg, ui->die);
> +
> +
> +	/* write the minimum value first if the target freq is less than current max */
> +	if (target_uncore_freq <= curr_max_freq) {
> +		fflush(ui->f_cur_min);
> +		fflush(ui->f_cur_max);
> +	} else {
> +		fflush(ui->f_cur_max);
> +		fflush(ui->f_cur_min);
> +	}
> +	ui->curr_idx = idx;
> +
> +	return 0;
> +}
> +
> +/**
> + * Fopen the sys file for the future setting of the uncore die frequency.
> + */
> +static int
> +power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
> +{
> +	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
> +	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
> +	int ret;
> +
> +	/* open and read all uncore sys files */
> +	/* Base max */
> +	open_core_sysfs_file(&f_base_max, "r", POWER_UNCORE_SYSFILE_BASE_MAX_FREQ,
> +			ui->pkg, ui->die);
> +	if (f_base_max == NULL) {
> +		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
> +				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
> +		goto err;
> +	}
> +	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
> +	if (ret < 0) {
> +		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
> +				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
> +		goto err;
> +	}
> +
> +	/* Base min */
> +	open_core_sysfs_file(&f_base_min, "r", POWER_UNCORE_SYSFILE_BASE_MIN_FREQ,
> +		ui->pkg, ui->die);
> +	if (f_base_min == NULL) {
> +		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
> +				POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
> +		goto err;
> +	}
> +	if (f_base_min != NULL) {
> +		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
> +		if (ret < 0) {
> +			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
> +					POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
> +			goto err;
> +		}
> +	}
> +
> +	/* Curr min */
> +	open_core_sysfs_file(&f_min, "rw+", POWER_UNCORE_SYSFILE_MIN_FREQ,
> +			ui->pkg, ui->die);
> +	if (f_min == NULL) {
> +		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
> +				POWER_UNCORE_SYSFILE_MIN_FREQ);
> +		goto err;
> +	}
> +	if (f_min != NULL) {
> +		ret = read_core_sysfs_u32(f_min, &min_freq);
> +		if (ret < 0) {
> +			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
> +					POWER_UNCORE_SYSFILE_MIN_FREQ);
> +			goto err;
> +		}
> +	}
> +
> +	/* Curr max */
> +	open_core_sysfs_file(&f_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
> +			ui->pkg, ui->die);
> +	if (f_max == NULL) {
> +		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
> +				POWER_UNCORE_SYSFILE_MAX_FREQ);
> +		goto err;
> +	}
> +	if (f_max != NULL) {
> +		ret = read_core_sysfs_u32(f_max, &max_freq);
> +		if (ret < 0) {
> +			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
> +					POWER_UNCORE_SYSFILE_MAX_FREQ);
> +			goto err;
> +		}
> +	}
> +
> +	/* assign file handles */
> +	ui->f_cur_min = f_min;
> +	ui->f_cur_max = f_max;
> +	/* save current min + max freq's so that they can be restored on exit */
> +	ui->org_min_freq = min_freq;
> +	ui->org_max_freq = max_freq;
> +	ui->init_max_freq = base_max_freq;
> +	ui->init_min_freq = base_min_freq;
> +
> +	return 0;
> +
> +err:
> +	if (f_base_min != NULL)
> +		fclose(f_base_min);
> +	if (f_base_max != NULL)
> +		fclose(f_base_max);
> +	if (f_min != NULL)
> +		fclose(f_min);
> +	if (f_max != NULL)
> +		fclose(f_max);
> +	return -1;
> +}
> +
> +/**
> + * Get the available uncore frequencies of the specific die by reading the
> + * sys file.
> + */
> +static int
> +power_get_available_uncore_freqs(struct uncore_power_info *ui)
> +{
> +	int ret = -1;
> +	uint32_t i, num_uncore_freqs = 0;
> +
> +	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
> +	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
> +		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
> +				num_uncore_freqs);
> +		goto out;
> +	}
> +
> +	/* Generate the uncore freq bucket array. */
> +	for (i = 0; i < num_uncore_freqs; i++)
> +		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
> +
> +	ui->nb_freqs = num_uncore_freqs;
> +
> +	ret = 0;
> +
> +	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
> +			num_uncore_freqs, ui->pkg, ui->die);
> +
> +out:
> +	return ret;
> +}
> +
> +static int
> +check_pkg_die_values(unsigned int pkg, unsigned int die)
> +{
> +	unsigned int max_pkgs, max_dies;
> +	max_pkgs = rte_power_uncore_get_num_pkgs();
> +	if (max_pkgs == 0)
> +		return -1;
> +	if (pkg >= max_pkgs) {
> +		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
> +				pkg, max_pkgs);
> +		return -1;
> +	}
> +
> +	max_dies = rte_power_uncore_get_num_dies(pkg);
> +	if (max_dies == 0)
> +		return -1;
> +	if (die >= max_dies) {
> +		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
> +				die, max_dies);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_power_uncore_init(unsigned int pkg, unsigned int die)
> +{
> +	struct uncore_power_info *ui;
> +
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	ui = &uncore_info[pkg][die];
> +	ui->die = die;
> +	ui->pkg = pkg;
> +
> +	/* Init for setting uncore die frequency */
> +	if (power_init_for_setting_uncore_freq(ui) < 0) {
> +		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
> +				"pkg %02u die %02u\n", pkg, die);
> +		return -1;
> +	}
> +
> +	/* Get the available frequencies */
> +	if (power_get_available_uncore_freqs(ui) < 0) {
> +		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
> +				"pkg %02u die %02u\n", pkg, die);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_power_uncore_exit(unsigned int pkg, unsigned int die)
> +{
> +	struct uncore_power_info *ui;
> +
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	ui = &uncore_info[pkg][die];
> +
> +	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
> +		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
> +				"pkg %02u die %02u\n", ui->pkg, ui->die);
> +		return -1;
> +	}
> +
> +	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
> +		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
> +				"pkg %02u die %02u\n", ui->pkg, ui->die);
> +		return -1;
> +	}
> +
> +	fflush(ui->f_cur_min);
> +	fflush(ui->f_cur_max);
> +
> +	/* Close FD of setting freq */
> +	fclose(ui->f_cur_min);
> +	fclose(ui->f_cur_max);
> +	ui->f_cur_min = NULL;
> +	ui->f_cur_max = NULL;
> +
> +	return 0;
> +}
> +
> +uint32_t
> +rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
> +{
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	return uncore_info[pkg][die].curr_idx;
> +}
> +
> +int
> +rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
> +{
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
> +}
> +
> +int
> +rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
> +{
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
> +}
> +
> +
> +int
> +rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
> +{
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	struct uncore_power_info *ui = &uncore_info[pkg][die];
> +
> +	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
> +}
> +
> +int
> +rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
> +{
> +	int ret = check_pkg_die_values(pkg, die);
> +	if (ret < 0)
> +		return -1;
> +
> +	return uncore_info[pkg][die].nb_freqs;
> +}
> +
> +unsigned int
> +rte_power_uncore_get_num_pkgs(void)
> +{
> +	DIR *d;
> +	struct dirent *dir;
> +	unsigned int count = 0;
> +	char filter[FILTER_LENGTH];
> +
> +	d = opendir(UNCORE_FREQUENCY_DIR);
> +	if (d == NULL) {
> +		RTE_LOG(ERR, POWER,
> +		"Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
> +		return 0;
> +	}


Suggest re-wording this message. I'm running a 5.17 kernel, and still 
see this message because CONFIG_INTEL_UNCORE_FREQ_CONTROL is not set.

Maybe "Uncore frequency management not supported/enabled on this kernel. 
Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL."

Also, the '\n' is missing from the end of the line.


> +
> +	/* search by incrementing file name for max pkg file value */
> +	while ((dir = readdir(d)) != NULL) {
> +		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
> +		/* make sure filter string is in file name (don't include hidden files) */
> +		if (fnmatch(filter, dir->d_name, 0) == 0)
> +			count++;
> +	}
> +
> +	closedir(d);
> +
> +	return count;
> +}
> +
> +unsigned int
> +rte_power_uncore_get_num_dies(unsigned int pkg)
> +{
> +	DIR *d;
> +	struct dirent *dir;
> +	unsigned int count = 0, max_pkgs;
> +	char filter[FILTER_LENGTH];
> +
> +	max_pkgs = rte_power_uncore_get_num_pkgs();
> +	if (max_pkgs == 0)
> +		return 0;
> +	if (pkg >= max_pkgs) {
> +		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
> +		return 0;
> +	}
> +
> +	d = opendir(UNCORE_FREQUENCY_DIR);
> +	if (d == NULL) {
> +		RTE_LOG(ERR, POWER,
> +		"Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
> +		return 0;
> +	}


Suggest re-wording this message. I'm running a 5.17 kernel, and still 
see this message because CONFIG_INTEL_UNCORE_FREQ_CONTROL is not set.

Maybe "Uncore frequency management not supported/enabled on this kernel. 
Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL."

Also, the '\n' is missing from the end of the line.


> +
> +	/* search by incrementing file name for max die file value */
> +	while ((dir = readdir(d)) != NULL) {
> +		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
> +		/* make sure filter string is in file name (don't include hidden files) */
> +		if (fnmatch(filter, dir->d_name, 0) == 0)
> +			count++;
> +	}
> +
> +	closedir(d);
> +
> +	return count;
> +}
> diff --git a/lib/power/rte_power_uncore.h b/lib/power/rte_power_uncore.h
> new file mode 100644
> index 0000000000..2be6546f49
> --- /dev/null
> +++ b/lib/power/rte_power_uncore.h
> @@ -0,0 +1,194 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2022 Intel Corporation
> + */
> +
> +#ifndef _RTE_POWER_UNCORE_H
> +#define _RTE_POWER_UNCORE_H
> +
> +/**
> + * @file
> + * RTE Uncore Frequency Management
> + */
> +
> +#include "rte_power.h"
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * Initialize uncore frequency management for specific die on a package. It will get the available
> + * frequencies and prepare to set new die frequencies.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  - 0 on success.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_uncore_init(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
> + * max values to previous values before initialization of API.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  - 0 on success.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_uncore_exit(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Return the current index of available frequencies of a specific die on a package.
> + * It should be protected outside of this function for threadsafe.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  The current index of available frequencies.
> + *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
> + */
> +__rte_experimental
> +uint32_t
> +rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Set the new frequency for a specific die on a package by indicating the index of
> + * available frequencies.
> + * It should be protected outside of this function for threadsafe.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + * @param index
> + *  The index of available frequencies.
> + *
> + * @return
> + *  - 1 on success with frequency changed.
> + *  - 0 on success without frequency changed.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
> +
> +/**
> + * Scale up the frequency of a specific die on a package to the highest according to the
> + * available frequencies.
> + * It should be protected outside of this function for threadsafe.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  - 1 on success with frequency changed.
> + *  - 0 on success without frequency changed.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Scale down the frequency of a specific die on a package to the lowest according to the
> + * available frequencies.
> + * It should be protected outside of this function for threadsafe.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  - 1 on success with frequency changed.
> + *  - 0 on success without frequency changed.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Return the list length of available frequencies in the index array.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.
> + *
> + * @return
> + *  - The number of available index's in frequency array.
> + *  - Negative on error.
> + */
> +__rte_experimental
> +int
> +rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
> +
> +/**
> + * Return the number of packages (CPUs) on a system by parsing the uncore
> + * sysfs directory.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @return
> + *  - Zero on error.
> + *  - Number of package on system on success.
> + */
> +__rte_experimental
> +unsigned int
> +rte_power_uncore_get_num_pkgs(void);
> +
> +/**
> + * Return the number of dies for pakckages (CPUs) specified from parsing
> + * the uncore sysfs directory.
> + *
> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + *
> + * @return
> + *  - Zero on error.
> + *  - Number of dies for package on sucecss.
> + */
> +__rte_experimental
> +unsigned int
> +rte_power_uncore_get_num_dies(unsigned int pkg);
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
> diff --git a/lib/power/version.map b/lib/power/version.map
> index f9b2947adf..8fccbf20f7 100644
> --- a/lib/power/version.map
> +++ b/lib/power/version.map
> @@ -48,4 +48,15 @@ EXPERIMENTAL {
>   	rte_power_pmd_mgmt_set_pause_duration;
>   	rte_power_pmd_mgmt_set_scaling_freq_max;
>   	rte_power_pmd_mgmt_set_scaling_freq_min;
> +
> +	# added in 22.11
> +	rte_power_get_uncore_freq;
> +	rte_power_set_uncore_freq;
> +	rte_power_uncore_exit;
> +	rte_power_uncore_freq_max;
> +	rte_power_uncore_freq_min;
> +	rte_power_uncore_get_num_dies;
> +	rte_power_uncore_get_num_freqs;
> +	rte_power_uncore_get_num_pkgs;
> +	rte_power_uncore_init;
>   };


Once the above requested changes are addressed:

Reviewed-by: David Hunt <david.hunt@intel.com>

Re: [PATCH v4 3/3] test/power: add unit tests for uncore API

[Hunt, David]

Hi Tadhg,

On 20/09/2022 10:48, Tadhg Kearney wrote:
> Add basic unit tests covering all nine uncore API's.
>
> Signed-off-by: Tadhg Kearney<tadhg.kearney@intel.com>
> ---
>   app/test/meson.build         |   2 +
>   app/test/test_power_uncore.c | 299 +++++++++++++++++++++++++++++++++++
>   2 files changed, 301 insertions(+)
>   create mode 100644 app/test/test_power_uncore.c
>
> diff --git a/app/test/meson.build b/app/test/meson.build
> index bf1d81f84a..170401ccdc 100644
> --- a/app/test/meson.build
> +++ b/app/test/meson.build
> @@ -100,6 +100,7 @@ test_sources = files(
>           'test_power.c',
>           'test_power_cpufreq.c',
>           'test_power_kvm_vm.c',
> +        'test_power_uncore.c',
>           'test_prefetch.c',
>           'test_rand_perf.c',
>           'test_rawdev.c',
> @@ -240,6 +241,7 @@ fast_tests = [
>           ['power_cpufreq_autotest', false, true],
>           ['power_autotest', true, true],
>           ['power_kvm_vm_autotest', false, true],
> +        ['power_uncore_autotest', true, true],
>           ['reorder_autotest', true, true],
>           ['service_autotest', true, true],
>           ['thash_autotest', true, true],
> diff --git a/app/test/test_power_uncore.c b/app/test/test_power_uncore.c
> new file mode 100644
> index 0000000000..7bc3ed7260
> --- /dev/null
> +++ b/app/test/test_power_uncore.c
> @@ -0,0 +1,299 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2022 Intel Corporation
> + */
> +
> +#include "test.h"
> +
> +#ifndef RTE_LIB_POWER
> +
> +static int
> +test_power_uncore(void)
> +{
> +	printf("Power management library not supported, skipping test\n");
> +	return TEST_SKIPPED;
> +}
> +
> +#else
> +#include <rte_power_uncore.h>
> +#include <power_common.h>
> +
> +#define MAX_UNCORE_FREQS 32
> +
> +#define VALID_PKG 0
> +#define VALID_DIE 0
> +#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
> +#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
> +#define VALID_INDEX 1
> +#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
> +
> +static int check_power_uncore_init(void)
> +{
> +	int ret;
> +
> +	/* Test initialisation of uncore configuration*/
> +	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Cannot initialise uncore power management for pkg %u die %u, this "
> +			"may occur if environment is not configured "
> +			"correctly(APCI cpufreq) or operating in another valid "
> +			"Power management environment\n", VALID_PKG, VALID_DIE);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
> +	if (ret == 0) {
> +		printf("Unexpectedly was able to initialise uncore power management "
> +			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_get_uncore_freq(void)
> +{
> +	int ret;
> +
> +	/* Successfully get uncore freq */
> +	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Failed to get uncore frequency for pkg %u die %u\n",
> +							VALID_PKG, VALID_DIE);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
> +	if (ret >= 0) {
> +		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
> +							INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_set_uncore_freq(void)
> +{
> +	int ret;
> +
> +	/* Successfully set uncore freq */
> +	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
> +	if (ret < 0) {
> +		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
> +							VALID_PKG, VALID_DIE, VALID_INDEX);
> +		return -1;
> +	}
> +
> +	/* Try to unsuccessfully set invalid uncore freq index */
> +	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
> +	if (ret == 0) {
> +		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
> +							VALID_PKG, VALID_DIE, INVALID_INDEX);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
> +	if (ret == 0) {
> +		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
> +							INVALID_PKG, INVALID_DIE, VALID_INDEX);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_freq_max(void)
> +{
> +	int ret;
> +
> +	/* Successfully get max uncore freq */
> +	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Failed to set max uncore frequency for pkg %u die %u\n",
> +							VALID_PKG, VALID_DIE);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
> +	if (ret == 0) {
> +		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
> +							INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_freq_min(void)
> +{
> +	int ret;
> +
> +	/* Successfully get min uncore freq */
> +	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Failed to set min uncore frequency for pkg %u die %u\n",
> +							VALID_PKG, VALID_DIE);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
> +	if (ret == 0) {
> +		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
> +							INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_get_num_freqs(void)
> +{
> +	int ret;
> +
> +	/* Successfully get number of uncore freq */
> +	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
> +							VALID_PKG, VALID_DIE);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
> +	if (ret >= 0) {
> +		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
> +							INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_get_num_pkgs(void)
> +{
> +	int ret;
> +
> +	/* Successfully get number of uncore pkgs */
> +	ret = rte_power_uncore_get_num_pkgs();
> +	if (ret == 0) {
> +		printf("Failed to get number of uncore pkgs\n");
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_get_num_dies(void)
> +{
> +	int ret;
> +
> +	/* Successfully get number of uncore dies */
> +	ret = rte_power_uncore_get_num_dies(VALID_PKG);
> +	if (ret == 0) {
> +		printf("Failed to get number of uncore dies for pkg %u\n",
> +							VALID_PKG);
> +		return -1;
> +	}
> +
> +	/* Unsuccessful test */
> +	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
> +	if (ret > 0) {
> +		printf("Unexpectedly got number of invalid dies for pkg %u\n",
> +							INVALID_PKG);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +check_power_uncore_exit(void)
> +{
> +	int ret;
> +
> +	/* Successfully exit uncore power management */
> +	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
> +	if (ret < 0) {
> +		printf("Failed to exit uncore power management for pkg %u die %u\n",
> +							VALID_PKG, VALID_DIE);
> +	}
> +
> +	/* Unsuccessful Test */
> +	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
> +	if (ret == 0) {
> +		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
> +							INVALID_PKG, INVALID_DIE);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +test_power_uncore(void)
> +{
> +	int ret;
> +
> +	ret = rte_power_uncore_get_num_pkgs();
> +	if (ret == 0) {
> +		printf("Uncore frequency management only supported on x86 with linux kernel >= 5.6.");
> +		return TEST_SKIPPED;
> +	}
> +


Suggest re-wording this message. I'm running a 5.17 kernel, and still 
see this message because CONFIG_INTEL_UNCORE_FREQ_CONTROL is not set.

Maybe "Uncore frequency management not supported/enabled on this kernel. 
Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL."

Also, the '\n' is missing from the end of the line.


> +	ret = check_power_uncore_init();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_get_uncore_freq();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_set_uncore_freq();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_freq_max();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_freq_min();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_get_num_freqs();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_get_num_pkgs();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_get_num_dies();
> +	if (ret < 0)
> +		goto fail_all;
> +
> +	ret = check_power_uncore_exit();
> +	if (ret < 0)
> +		return -1;
> +
> +	return 0;
> +
> +fail_all:
> +	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
> +	return -1;
> +}
> +#endif
> +
> +REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_uncore);


The rest looks good to me, and once the above requested changes are 
addressed:

Reviewed-by: David Hunt <david.hunt@intel.com>

[PATCH v5 0/3] add uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to allow uncore
frequency adjustment. This will be called through the l3fwd-power app and gives the
ability to set the minimum and maximum uncore frequency to both min,
max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.
v5:
Improve error message for uncore access not working.


Tadhg Kearney (3):
  power: add uncore frequency control API to the power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_uncore.c                  | 301 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  38 ++
 doc/guides/rel_notes/release_22_11.rst        |   5 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 122 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_uncore.c                  | 451 ++++++++++++++++++
 lib/power/rte_power_uncore.h                  | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1152 insertions(+), 3 deletions(-)
 create mode 100644 app/test/test_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

-- 
2.25.1

[PATCH v5 1/3] power: add uncore frequency control API to the power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 21651 bytes --]

Add API to allow uncore frequency adjustment. This is done through
manipulating related uncore frequency control sysfs entries to
adjust the minimum and maximum uncore frequency values.
Nine API's are being added that are all public and experimental.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  38 +++
 doc/guides/rel_notes/release_22_11.rst |   5 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_uncore.c           | 451 +++++++++++++++++++++++++
 lib/power/rte_power_uncore.h           | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 701 insertions(+)
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..cf2f5d8b69 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,44 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Uncore API
+----------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Uncore
+~~~~~~~~~~~~~~~~~~~~~~~
+* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
+  original min & max for pkg & die.
+
+* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
+
+* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
+
+* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
+
+* **Uncore Power Max**: Set max uncore freq index for pkg & die.
+
+* **Uncore Power Min**: Set min uncore freq index for pkg & die.
+
+* **Get Num Freqs**: Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
+
+* **Get Num Dies**: Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 684bf74596..cc104c084a 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -67,6 +67,11 @@ New Features
 
   * Added support to set device link down/up.
 
+* **Added uncore frequency control API to the power library.**
+
+  Add api to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..80cdeb72d4 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
new file mode 100644
index 0000000000..463522a137
--- /dev/null
+++ b/lib/power/rte_power_uncore.c
@@ -0,0 +1,451 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel."
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel."
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_uncore.h b/lib/power/rte_power_uncore.h
new file mode 100644
index 0000000000..2be6546f49
--- /dev/null
+++ b/lib/power/rte_power_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_UNCORE_H
+#define _RTE_POWER_UNCORE_H
+
+/**
+ * @file
+ * RTE Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set the new frequency for a specific die on a package by indicating the index of
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Scale up the frequency of a specific die on a package to the highest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Scale down the frequency of a specific die on a package to the lowest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v5 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 +++++
 examples/l3fwd-power/main.c                   | 122 +++++++++++++++++-
 2 files changed, 148 insertions(+), 3 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..d1a32594c0 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -179,6 +180,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1623,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1682,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1838,7 +1916,7 @@ parse_ep_config(const char *q_arg)
 static int
 parse_args(int argc, char **argv)
 {
-	int opt, ret;
+	int opt, ret, enabled_uncore = 0;
 	char **argvopt;
 	int option_index;
 	uint32_t limit;
@@ -1864,7 +1942,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P:uUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1971,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2463,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2476,23 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_exit(pkg, die);
+			if (ret < 0) {
+				RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deninit successfully for pkg %02u die %02u\n"
+					, pkg, die);
+				return -1;
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v5 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
---
 app/test/meson.build         |   2 +
 app/test/test_power_uncore.c | 301 +++++++++++++++++++++++++++++++++++
 2 files changed, 303 insertions(+)
 create mode 100644 app/test/test_power_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index d5cad72116..8a864be957 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -241,6 +242,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_uncore.c b/app/test/test_power_uncore.c
new file mode 100644
index 0000000000..5d64524e20
--- /dev/null
+++ b/app/test/test_power_uncore.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management not supported/enabled on this kernel."
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_uncore);
-- 
2.25.1

[PATCH v6 0/3] add uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to allow
uncore frequency adjustment. This will be called through the l3fwd-power app
and gives the ability to set the minimum and maximum uncore frequency to both min, max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.
v5:
Improve error message for uncore access not working.
v6:
Fix uncore exit being called if uncore option not selected.


Tadhg Kearney (3):
  power: add uncore frequency control API to the power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_uncore.c                  | 301 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  38 ++
 doc/guides/rel_notes/release_22_11.rst        |   5 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 126 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_uncore.c                  | 451 ++++++++++++++++++
 lib/power/rte_power_uncore.h                  | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1157 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

-- 
2.25.1

[PATCH v6 1/3] power: add uncore frequency control API to the power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 21792 bytes --]

Add API to allow uncore frequency adjustment. This is done through
manipulating related uncore frequency control sysfs entries to
adjust the minimum and maximum uncore frequency values.
Nine API's are being added that are all public and experimental.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  38 +++
 doc/guides/rel_notes/release_22_11.rst |   5 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_uncore.c           | 451 +++++++++++++++++++++++++
 lib/power/rte_power_uncore.h           | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 701 insertions(+)
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..cf2f5d8b69 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,44 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Uncore API
+----------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Uncore
+~~~~~~~~~~~~~~~~~~~~~~~
+* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
+  original min & max for pkg & die.
+
+* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
+
+* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
+
+* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
+
+* **Uncore Power Max**: Set max uncore freq index for pkg & die.
+
+* **Uncore Power Min**: Set min uncore freq index for pkg & die.
+
+* **Get Num Freqs**: Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
+
+* **Get Num Dies**: Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index cb7677fd3c..5d3f815b54 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -75,6 +75,11 @@ New Features
   * Added ``rte_event_eth_tx_adapter_instance_get`` to get Tx adapter
     instance ID for specified ethernet device ID and Tx queue index.
 
+* **Added uncore frequency control API to the power library.**
+
+  Add api to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..80cdeb72d4 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
new file mode 100644
index 0000000000..c9049e2b6c
--- /dev/null
+++ b/lib/power/rte_power_uncore.c
@@ -0,0 +1,451 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_uncore.h b/lib/power/rte_power_uncore.h
new file mode 100644
index 0000000000..2be6546f49
--- /dev/null
+++ b/lib/power/rte_power_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_UNCORE_H
+#define _RTE_POWER_UNCORE_H
+
+/**
+ * @file
+ * RTE Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set the new frequency for a specific die on a package by indicating the index of
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Scale up the frequency of a specific die on a package to the highest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Scale down the frequency of a specific die on a package to the lowest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v6 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++++
 examples/l3fwd-power/main.c                   | 126 +++++++++++++++++-
 2 files changed, 153 insertions(+), 2 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..0b6acc8e58 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -161,6 +162,9 @@ static struct rte_timer telemetry_timer;
 /* stats index returned by metrics lib */
 int telstats_index;
 
+/* flag to check if uncore option enabled */
+int enabled_uncore = -1;
+
 struct telstats_name {
 	char name[RTE_ETH_XSTATS_NAME_SIZE];
 };
@@ -179,6 +183,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1626,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1685,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1864,7 +1945,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P:uUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1974,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2466,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2479,26 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	/* if uncore option was set */
+	if (enabled_uncore == 0) {
+		max_pkg = rte_power_uncore_get_num_pkgs();
+		if (max_pkg == 0)
+			return -1;
+		for (pkg = 0; pkg < max_pkg; pkg++) {
+			max_die = rte_power_uncore_get_num_dies(pkg);
+			if (max_die == 0)
+				return -1;
+			for (die = 0; die < max_die; die++) {
+				ret = rte_power_uncore_exit(pkg, die);
+				if (ret < 0) {
+					RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deinit successfully for pkg %02u die %02u\n"
+						, pkg, die);
+					return -1;
+				}
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v6 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 app/test/meson.build         |   2 +
 app/test/test_power_uncore.c | 301 +++++++++++++++++++++++++++++++++++
 2 files changed, 303 insertions(+)
 create mode 100644 app/test/test_power_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index d5cad72116..8a864be957 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -241,6 +242,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_uncore.c b/app/test/test_power_uncore.c
new file mode 100644
index 0000000000..da64987ce3
--- /dev/null
+++ b/app/test/test_power_uncore.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_uncore);
-- 
2.25.1

Re: [PATCH v6 2/3] l3fwd-power: add option to call uncore API

[Hunt, David]

Hi Tadhg,

On 28/09/2022 10:06, Tadhg Kearney wrote:
> Add option for setting uncore frequency min/max/index, through uncore API.
> This will be set for each package and die on the SKU. On exit, uncore min
> and max frequency will be reverted back to previous frequencies.
>
> Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
> Reviewed-by: David Hunt <david.hunt@intel.com>
> ---
>   .../sample_app_ug/l3_forward_power_man.rst    |  29 ++++
>   examples/l3fwd-power/main.c                   | 126 +++++++++++++++++-
>   2 files changed, 153 insertions(+), 2 deletions(-)
>
> diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
> index 8f6d906200..08ac8ef369 100644
> --- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
> +++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
> @@ -97,6 +97,12 @@ where,
>   *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
>       Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
>   
> +*   -u: optional, sets uncore min/max frequency to minimum value.
> +
> +*   -U: optional, sets uncore min/max frequency to maximum value.
> +
> +*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
> +
>   *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
>   
>   *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
> @@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
>   .. code-block:: console
>   
>           ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
> +
> +Setting Uncore Values
> +---------------------
> +
> +Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
> +This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
> +Three options are available for setting uncore frequency:
> +
> +``-u``
> +  This will set uncore minimum and maximum frequencies to minimum possible value.
> +
> +``-U``
> +  This will set uncore minimum and maximum frequencies to maximum possible value.
> +
> +``-i``
> +  This will allow you to set the specific uncore frequency index that you want, by setting
> +  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
> +  maximum to minimum.
> +  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
> +
> +.. code-block:: console
> +
> +        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
> diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
> index 887c6eae3f..0b6acc8e58 100644
> --- a/examples/l3fwd-power/main.c
> +++ b/examples/l3fwd-power/main.c
> @@ -47,6 +47,7 @@
>   #include <rte_metrics.h>
>   #include <rte_telemetry.h>
>   #include <rte_power_pmd_mgmt.h>
> +#include <rte_power_uncore.h>
>   
>   #include "perf_core.h"
>   #include "main.h"
> @@ -161,6 +162,9 @@ static struct rte_timer telemetry_timer;
>   /* stats index returned by metrics lib */
>   int telstats_index;
>   
> +/* flag to check if uncore option enabled */
> +int enabled_uncore = -1;
> +
>   struct telstats_name {
>   	char name[RTE_ETH_XSTATS_NAME_SIZE];
>   };
> @@ -179,6 +183,12 @@ enum busy_rate {
>   	FULL = 100
>   };
>   
> +enum uncore_choice {
> +	UNCORE_MIN = 0,
> +	UNCORE_MAX = 1,
> +	UNCORE_IDX = 2
> +};
> +
>   /* reference poll count to measure core busyness */
>   #define DEFAULT_COUNT 10000
>   /*
> @@ -1616,6 +1626,9 @@ print_usage(const char *prgname)
>   		"  [--max-pkt-len PKTLEN]\n"
>   		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
>   		"  -P: enable promiscuous mode\n"
> +		"  -u: set min/max frequency for uncore to minimum value\n"
> +		"  -U: set min/max frequency for uncore to maximum value\n"
> +		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
>   		"  --config (port,queue,lcore): rx queues configuration\n"
>   		"  --high-perf-cores CORELIST: list of high performance cores\n"
>   		"  --perf-config: similar as config, cores specified as indices"
> @@ -1672,6 +1685,74 @@ static int parse_max_pkt_len(const char *pktlen)
>   	return len;
>   }
>   
> +static int
> +parse_uncore_options(enum uncore_choice choice, const char *argument)
> +{
> +	unsigned int die, pkg, max_pkg, max_die;
> +	int ret = 0;
> +	max_pkg = rte_power_uncore_get_num_pkgs();
> +	if (max_pkg == 0)
> +		return -1;
> +
> +	for (pkg = 0; pkg < max_pkg; pkg++) {
> +		max_die = rte_power_uncore_get_num_dies(pkg);
> +		if (max_die == 0)
> +			return -1;
> +		for (die = 0; die < max_die; die++) {
> +			ret = rte_power_uncore_init(pkg, die);
> +			if (ret == -1) {
> +				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
> +				, pkg, die);
> +				return ret;
> +			}
> +			if (choice == UNCORE_MIN) {
> +				ret = rte_power_uncore_freq_min(pkg, die);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
> +					, pkg, die);
> +					return ret;
> +				}
> +			} else if (choice == UNCORE_MAX) {
> +				ret = rte_power_uncore_freq_max(pkg, die);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
> +					, pkg, die);
> +					return ret;
> +				}
> +			} else if (choice == UNCORE_IDX) {
> +				char *ptr = NULL;
> +				int frequency_index = strtol(argument, &ptr, 10);
> +				if (argument == ptr) {
> +					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
> +					return -1;
> +				}
> +				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
> +				if (frequency_index > freq_array_len - 1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Frequency index given out of range, please choose a value from 0 to %d.\n",
> +					freq_array_len);
> +					return -1;
> +				}
> +				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
> +				if (ret == -1) {
> +					RTE_LOG(INFO, L3FWD_POWER,
> +					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
> +					pkg, die);
> +					return ret;
> +				}
> +			} else {
> +				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
> +				return -1;
> +			}
> +		}
> +	}
> +
> +	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
> +	return ret;
> +}
> +
>   static int
>   parse_portmask(const char *portmask)
>   {
> @@ -1864,7 +1945,7 @@ parse_args(int argc, char **argv)
>   
>   	argvopt = argv;
>   
> -	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
> +	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P:uUi:",

There's been a spurious ':' added after the 'P' for promiscuous mode, 
thereby breaking it.


>   				lgopts, &option_index)) != EOF) {
>   
>   		switch (opt) {
> @@ -1893,6 +1974,27 @@ parse_args(int argc, char **argv)
>   			limit = parse_max_pkt_len(optarg);
>   			freq_tlb[HGH] = limit;
>   			break;
> +		case 'u':
> +			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
> +		case 'U':
> +			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
> +		case 'i':
> +			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
> +			if (enabled_uncore < 0) {
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
>   		/* long options */
>   		case 0:
>   			if (!strncmp(lgopts[option_index].name, "config", 6)) {
> @@ -2364,7 +2466,7 @@ init_power_library(void)
>   static int
>   deinit_power_library(void)
>   {
> -	unsigned int lcore_id;
> +	unsigned int lcore_id, max_pkg, max_die, die, pkg;
>   	int ret = 0;
>   
>   	RTE_LCORE_FOREACH(lcore_id) {
> @@ -2377,6 +2479,26 @@ deinit_power_library(void)
>   			return ret;
>   		}
>   	}
> +
> +	/* if uncore option was set */
> +	if (enabled_uncore == 0) {
> +		max_pkg = rte_power_uncore_get_num_pkgs();
> +		if (max_pkg == 0)
> +			return -1;
> +		for (pkg = 0; pkg < max_pkg; pkg++) {
> +			max_die = rte_power_uncore_get_num_dies(pkg);
> +			if (max_die == 0)
> +				return -1;
> +			for (die = 0; die < max_die; die++) {
> +				ret = rte_power_uncore_exit(pkg, die);
> +				if (ret < 0) {
> +					RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deinit successfully for pkg %02u die %02u\n"
> +						, pkg, die);
> +					return -1;
> +				}
> +			}
> +		}
> +	}
>   	return ret;
>   }
>   

[PATCH v7 0/3] add uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to allow
uncore frequency adjustment. This will be called through the l3fwd-power app
and gives the ability to set the minimum and maximum uncore frequency to both min,
max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.
v5:
Improve error message for uncore access not working.
v6:
Fix uncore exit being called if uncore option not selected.
v7:
Address ml comments.


Tadhg Kearney (3):
  power: add uncore frequency control API to the power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_uncore.c                  | 301 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  38 ++
 doc/guides/rel_notes/release_22_11.rst        |   5 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 126 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_uncore.c                  | 451 ++++++++++++++++++
 lib/power/rte_power_uncore.h                  | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1157 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

-- 
2.25.1

[PATCH v7 1/3] power: add uncore frequency control API to the power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 21792 bytes --]

Add API to allow uncore frequency adjustment. This is done through
manipulating related uncore frequency control sysfs entries to
adjust the minimum and maximum uncore frequency values.
Nine API's are being added that are all public and experimental.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  38 +++
 doc/guides/rel_notes/release_22_11.rst |   5 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_uncore.c           | 451 +++++++++++++++++++++++++
 lib/power/rte_power_uncore.h           | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 701 insertions(+)
 create mode 100644 lib/power/rte_power_uncore.c
 create mode 100644 lib/power/rte_power_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..cf2f5d8b69 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,44 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Uncore API
+----------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Uncore
+~~~~~~~~~~~~~~~~~~~~~~~
+* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
+  original min & max for pkg & die.
+
+* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
+
+* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
+
+* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
+
+* **Uncore Power Max**: Set max uncore freq index for pkg & die.
+
+* **Uncore Power Min**: Set min uncore freq index for pkg & die.
+
+* **Get Num Freqs**: Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
+
+* **Get Num Dies**: Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index cb7677fd3c..5d3f815b54 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -75,6 +75,11 @@ New Features
   * Added ``rte_event_eth_tx_adapter_instance_get`` to get Tx adapter
     instance ID for specified ethernet device ID and Tx queue index.
 
+* **Added uncore frequency control API to the power library.**
+
+  Add api to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..80cdeb72d4 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_uncore.c b/lib/power/rte_power_uncore.c
new file mode 100644
index 0000000000..c9049e2b6c
--- /dev/null
+++ b/lib/power/rte_power_uncore.c
@@ -0,0 +1,451 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_uncore.h b/lib/power/rte_power_uncore.h
new file mode 100644
index 0000000000..2be6546f49
--- /dev/null
+++ b/lib/power/rte_power_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_UNCORE_H
+#define _RTE_POWER_UNCORE_H
+
+/**
+ * @file
+ * RTE Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set the new frequency for a specific die on a package by indicating the index of
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Scale up the frequency of a specific die on a package to the highest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Scale down the frequency of a specific die on a package to the lowest according to the
+ * available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ * @param die
+ *  Die number.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v7 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++++
 examples/l3fwd-power/main.c                   | 126 +++++++++++++++++-
 2 files changed, 153 insertions(+), 2 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..0704e06290 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -161,6 +162,9 @@ static struct rte_timer telemetry_timer;
 /* stats index returned by metrics lib */
 int telstats_index;
 
+/* flag to check if uncore option enabled */
+int enabled_uncore = -1;
+
 struct telstats_name {
 	char name[RTE_ETH_XSTATS_NAME_SIZE];
 };
@@ -179,6 +183,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1626,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1685,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1864,7 +1945,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:PuUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1974,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2466,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2479,26 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	/* if uncore option was set */
+	if (enabled_uncore == 0) {
+		max_pkg = rte_power_uncore_get_num_pkgs();
+		if (max_pkg == 0)
+			return -1;
+		for (pkg = 0; pkg < max_pkg; pkg++) {
+			max_die = rte_power_uncore_get_num_dies(pkg);
+			if (max_die == 0)
+				return -1;
+			for (die = 0; die < max_die; die++) {
+				ret = rte_power_uncore_exit(pkg, die);
+				if (ret < 0) {
+					RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deinit successfully for pkg %02u die %02u\n"
+						, pkg, die);
+					return -1;
+				}
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v7 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
---
 app/test/meson.build         |   2 +
 app/test/test_power_uncore.c | 301 +++++++++++++++++++++++++++++++++++
 2 files changed, 303 insertions(+)
 create mode 100644 app/test/test_power_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index d5cad72116..8a864be957 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -241,6 +242,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_uncore.c b/app/test/test_power_uncore.c
new file mode 100644
index 0000000000..da64987ce3
--- /dev/null
+++ b/app/test/test_power_uncore.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_uncore);
-- 
2.25.1

Re: [PATCH v7 0/3] add uncore api to be called through l3fwd-power

[Hunt, David]

Hi Tadhg,

On 28/09/2022 14:30, Tadhg Kearney wrote:
> This is targeting 22.11 and aims to add an API to DPDK power library to allow
> uncore frequency adjustment. This will be called through the l3fwd-power app
> and gives the ability to set the minimum and maximum uncore frequency to both min,
> max or specific frequency index.
>
> Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
> Reviewed-by: David Hunt <david.hunt@intel.com>
> ---
>
> v2:
> Fix compilation warnings and errors.
> v3:
> Remove addition of x86 global macros.
> Add 2 new API's for getting package and die numbers from system.
> Address comments from mailing list.
> Improve efficiency of code and code quality.
> v4:
> Fix compilation warnings and errors.
> v5:
> Improve error message for uncore access not working.
> v6:
> Fix uncore exit being called if uncore option not selected.
> v7:
> Address ml comments.
>
>
> Tadhg Kearney (3):
>    power: add uncore frequency control API to the power library
>    l3fwd-power: add option to call uncore API
>    test/power: add unit tests for uncore API
>
>   app/test/meson.build                          |   2 +
>   app/test/test_power_uncore.c                  | 301 ++++++++++++
>   doc/guides/prog_guide/power_man.rst           |  38 ++
>   doc/guides/rel_notes/release_22_11.rst        |   5 +
>   .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
>   examples/l3fwd-power/main.c                   | 126 ++++-
>   lib/power/meson.build                         |   2 +
>   lib/power/rte_power_uncore.c                  | 451 ++++++++++++++++++
>   lib/power/rte_power_uncore.h                  | 194 ++++++++
>   lib/power/version.map                         |  11 +
>   10 files changed, 1157 insertions(+), 2 deletions(-)
>   create mode 100644 app/test/test_power_uncore.c
>   create mode 100644 lib/power/rte_power_uncore.c
>   create mode 100644 lib/power/rte_power_uncore.h


Latest revision looks good to me. Consider the series:

Acked-by: David Hunt <david.hunt@intel.com>

Re: [PATCH v7 1/3] power: add uncore frequency control API to the power library

[Thomas Monjalon]

28/09/2022 15:30, Tadhg Kearney:
> Add API to allow uncore frequency adjustment. This is done through
> manipulating related uncore frequency control sysfs entries to
> adjust the minimum and maximum uncore frequency values.
> Nine API's are being added that are all public and experimental.

You cannot introduce an API without explaining what it is about.
Maybe I'm an idiot but I don't know what is "uncore".
I see it is explained in the documentation,
but few words in the commit message would not be too much.
At least you must say it for Linux on Intel,
and which feature it is controlling.

> +Uncore API
> +----------
> +
> +Abstract
> +~~~~~~~~
> +
> +Uncore is a term used by Intel to describe the functions of a microprocessor that are
> +not in the core, but which must be closely connected to the core to achieve high performance;
> +L3 cache, on-die memory controller, etc.
> +Significant power savings can be achieved by reducing the uncore frequency to its lowest value.

So this is an Intel thing.

> +
> +The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
> +for x86 platform. The driver is available from kernel version 5.6 and above.
> +Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
> +This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.

It is correctly named "intel-uncore" in the Linux kernel.
Why not having "Intel" in the DPDK feature name?

> +
> +
> +API Overview for Uncore
> +~~~~~~~~~~~~~~~~~~~~~~~

A blank line is missing here.

> +* **Uncore Power Init**: Initialise uncore power, populate frequency array and record
> +  original min & max for pkg & die.
> +
> +* **Uncore Power Exit**: Exit uncore power, restoring original min & max for pkg & die.
> +
> +* **Get Uncore Power Freq**: Get current uncore freq index for pkg & die.
> +
> +* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg & die (min and max will be the same).
> +
> +* **Uncore Power Max**: Set max uncore freq index for pkg & die.
> +
> +* **Uncore Power Min**: Set min uncore freq index for pkg & die.
> +
> +* **Get Num Freqs**: Get the number of frequencies in the index array.
> +
> +* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
> +
> +* **Get Num Dies**: Get the number of die's on a given package.

Not sure what you are listing here. Are they functions?
If you really want to keep a list, I suggest using a definition list
available in RST syntax.
If you want to provide an explanation easy to read,
full sentences connecting things together would be better.

> +
>  References
>  ----------
>  
> diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
> index cb7677fd3c..5d3f815b54 100644
> --- a/doc/guides/rel_notes/release_22_11.rst
> +++ b/doc/guides/rel_notes/release_22_11.rst
> @@ -75,6 +75,11 @@ New Features
>    * Added ``rte_event_eth_tx_adapter_instance_get`` to get Tx adapter
>      instance ID for specified ethernet device ID and Tx queue index.
>  
> +* **Added uncore frequency control API to the power library.**
> +
> +  Add api to allow uncore frequency adjustment. This is done through

s/api/API/

> +  manipulating related uncore frequency control sysfs entries to
> +  adjust the minimum and maximum uncore frequency values.

It is Linux-only for Intel hardware only.

> --- /dev/null
> +++ b/lib/power/rte_power_uncore.c

I would add "intel" in the filename.

[...]
> +#define UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
> +#define POWER_GOVERNOR_PERF "performance"
> +#define POWER_UNCORE_SYSFILE_MAX_FREQ \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
> +#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
> +#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
> +#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
> +		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"

It is for Intel CPU only, right?

> + * This function should NOT be called in the fast path.
> + *
> + * @param pkg
> + *  Package number.
> + * @param die
> + *  Die number.

To me it is not clear what they are.
Is it possible to better explain "pkg" and "die" somewhere?
Is it related to NUMA nodes?

RE: [PATCH v7 1/3] power: add uncore frequency control API to the power library

[Kearney, Tadhg]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday 4 October 2022 18:09
> To: Kearney, Tadhg <tadhg.kearney@intel.com>
> Cc: dev@dpdk.org; Hunt, David <david.hunt@intel.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>; Pattan, Reshma <reshma.pattan@intel.com>
> Subject: Re: [PATCH v7 1/3] power: add uncore frequency control API to the
> power library
> 
> 28/09/2022 15:30, Tadhg Kearney:
> > Add API to allow uncore frequency adjustment. This is done through
> > manipulating related uncore frequency control sysfs entries to adjust
> > the minimum and maximum uncore frequency values.
> > Nine API's are being added that are all public and experimental.
> 
> You cannot introduce an API without explaining what it is about.
> Maybe I'm an idiot but I don't know what is "uncore".
> I see it is explained in the documentation, but few words in the commit
> message would not be too much.
> At least you must say it for Linux on Intel, and which feature it is controlling.
> 
> > +Uncore API
> > +----------
> > +
> > +Abstract
> > +~~~~~~~~
> > +
> > +Uncore is a term used by Intel to describe the functions of a
> > +microprocessor that are not in the core, but which must be closely
> > +connected to the core to achieve high performance;
> > +L3 cache, on-die memory controller, etc.
> > +Significant power savings can be achieved by reducing the uncore
> frequency to its lowest value.
> 
> So this is an Intel thing.

Yes, so far the uncore kernel implementation covers Intel hardware.

> 
> > +
> > +The Linux kernel provides the driver “intel-uncore-frequency" to
> > +control the uncore frequency limits for x86 platform. The driver is
> available from kernel version 5.6 and above.
> > +Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in
> the kernel, which was added in 5.6.
> > +This manipulates the contest of MSR 0x620, which sets min/max of the
> uncore for the SKU.
> 
> It is correctly named "intel-uncore" in the Linux kernel.
> Why not having "Intel" in the DPDK feature name?
> 
> > +
> > +
> > +API Overview for Uncore
> > +~~~~~~~~~~~~~~~~~~~~~~~
> 
> A blank line is missing here.
> 
> > +* **Uncore Power Init**: Initialise uncore power, populate frequency
> > +array and record
> > +  original min & max for pkg & die.
> > +
> > +* **Uncore Power Exit**: Exit uncore power, restoring original min & max
> for pkg & die.
> > +
> > +* **Get Uncore Power Freq**: Get current uncore freq index for pkg &
> die.
> > +
> > +* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg &
> die (min and max will be the same).
> > +
> > +* **Uncore Power Max**: Set max uncore freq index for pkg & die.
> > +
> > +* **Uncore Power Min**: Set min uncore freq index for pkg & die.
> > +
> > +* **Get Num Freqs**: Get the number of frequencies in the index array.
> > +
> > +* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
> > +
> > +* **Get Num Dies**: Get the number of die's on a given package.
> 
> Not sure what you are listing here. Are they functions?
> If you really want to keep a list, I suggest using a definition list available in RST
> syntax.
> If you want to provide an explanation easy to read, full sentences connecting
> things together would be better.

Agreed.

> 
> > +
> >  References
> >  ----------
> >
> > diff --git a/doc/guides/rel_notes/release_22_11.rst
> > b/doc/guides/rel_notes/release_22_11.rst
> > index cb7677fd3c..5d3f815b54 100644
> > --- a/doc/guides/rel_notes/release_22_11.rst
> > +++ b/doc/guides/rel_notes/release_22_11.rst
> > @@ -75,6 +75,11 @@ New Features
> >    * Added ``rte_event_eth_tx_adapter_instance_get`` to get Tx adapter
> >      instance ID for specified ethernet device ID and Tx queue index.
> >
> > +* **Added uncore frequency control API to the power library.**
> > +
> > +  Add api to allow uncore frequency adjustment. This is done through
> 
> s/api/API/
> 
> > +  manipulating related uncore frequency control sysfs entries to
> > + adjust the minimum and maximum uncore frequency values.
> 
> It is Linux-only for Intel hardware only.
> 
> > --- /dev/null
> > +++ b/lib/power/rte_power_uncore.c
> 
> I would add "intel" in the filename.
> 
> [...]
> > +#define UNCORE_FREQUENCY_DIR
> "/sys/devices/system/cpu/intel_uncore_frequency"
> > +#define POWER_GOVERNOR_PERF "performance"
> > +#define POWER_UNCORE_SYSFILE_MAX_FREQ \
> > +
> 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> die_%02u/max_freq_khz"
> > +#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
> > +
> 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> die_%02u/min_freq_khz"
> > +#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
> > +
> 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> die_%02u/initial_max_freq_khz"
> > +#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
> > +
> 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> die_%02u/initial_min_freq_khz"
> 
> It is for Intel CPU only, right?

Currently only Intel CPUs are covered by these sysfs entries, but it is possible that other platforms will be included in the future.

> 
> > + * This function should NOT be called in the fast path.
> > + *
> > + * @param pkg
> > + *  Package number.
> > + * @param die
> > + *  Die number.
> 
> To me it is not clear what they are.
> Is it possible to better explain "pkg" and "die" somewhere?
> Is it related to NUMA nodes?

Each NUMA node is a package, which can contain 1 or more dies. These dies are connected in a package together via the UNCORE mesh.
Dies may appear as separate NUMA nodes, or a group of dies on a packages may appear as a single NUMA node, depending on the BIOS configuration.
Header descriptions will be changed to:
    * Package number. Each physical CPU in a system is referred to as a package.
    * Die number. Each package can have several dies connected together via the uncore mesh.

> 

Regards, Tadhg

Re: [PATCH v7 1/3] power: add uncore frequency control API to the power library

[Thomas Monjalon]

05/10/2022 12:50, Kearney, Tadhg:
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Tuesday 4 October 2022 18:09
> > To: Kearney, Tadhg <tadhg.kearney@intel.com>
> > Cc: dev@dpdk.org; Hunt, David <david.hunt@intel.com>; Burakov, Anatoly
> > <anatoly.burakov@intel.com>; Pattan, Reshma <reshma.pattan@intel.com>
> > Subject: Re: [PATCH v7 1/3] power: add uncore frequency control API to the
> > power library
> > 
> > 28/09/2022 15:30, Tadhg Kearney:
> > > Add API to allow uncore frequency adjustment. This is done through
> > > manipulating related uncore frequency control sysfs entries to adjust
> > > the minimum and maximum uncore frequency values.
> > > Nine API's are being added that are all public and experimental.
> > 
> > You cannot introduce an API without explaining what it is about.
> > Maybe I'm an idiot but I don't know what is "uncore".
> > I see it is explained in the documentation, but few words in the commit
> > message would not be too much.
> > At least you must say it for Linux on Intel, and which feature it is controlling.
> > 
> > > +Uncore API
> > > +----------
> > > +
> > > +Abstract
> > > +~~~~~~~~
> > > +
> > > +Uncore is a term used by Intel to describe the functions of a
> > > +microprocessor that are not in the core, but which must be closely
> > > +connected to the core to achieve high performance;
> > > +L3 cache, on-die memory controller, etc.
> > > +Significant power savings can be achieved by reducing the uncore
> > frequency to its lowest value.
> > 
> > So this is an Intel thing.
> 
> Yes, so far the uncore kernel implementation covers Intel hardware.
> 
> > 
> > > +
> > > +The Linux kernel provides the driver “intel-uncore-frequency" to
> > > +control the uncore frequency limits for x86 platform. The driver is
> > available from kernel version 5.6 and above.
> > > +Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in
> > the kernel, which was added in 5.6.
> > > +This manipulates the contest of MSR 0x620, which sets min/max of the
> > uncore for the SKU.
> > 
> > It is correctly named "intel-uncore" in the Linux kernel.
> > Why not having "Intel" in the DPDK feature name?
> > 
> > > +
> > > +
> > > +API Overview for Uncore
> > > +~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > A blank line is missing here.
> > 
> > > +* **Uncore Power Init**: Initialise uncore power, populate frequency
> > > +array and record
> > > +  original min & max for pkg & die.
> > > +
> > > +* **Uncore Power Exit**: Exit uncore power, restoring original min & max
> > for pkg & die.
> > > +
> > > +* **Get Uncore Power Freq**: Get current uncore freq index for pkg &
> > die.
> > > +
> > > +* **Set Uncore Power Freq**: Set min & max uncore freq index for pkg &
> > die (min and max will be the same).
> > > +
> > > +* **Uncore Power Max**: Set max uncore freq index for pkg & die.
> > > +
> > > +* **Uncore Power Min**: Set min uncore freq index for pkg & die.
> > > +
> > > +* **Get Num Freqs**: Get the number of frequencies in the index array.
> > > +
> > > +* **Get Num Pkgs**: Get the number of packages (CPUs) on the system.
> > > +
> > > +* **Get Num Dies**: Get the number of die's on a given package.
> > 
> > Not sure what you are listing here. Are they functions?
> > If you really want to keep a list, I suggest using a definition list available in RST
> > syntax.
> > If you want to provide an explanation easy to read, full sentences connecting
> > things together would be better.
> 
> Agreed.
> 
> > 
> > > +
> > >  References
> > >  ----------
> > >
> > > diff --git a/doc/guides/rel_notes/release_22_11.rst
> > > b/doc/guides/rel_notes/release_22_11.rst
> > > index cb7677fd3c..5d3f815b54 100644
> > > --- a/doc/guides/rel_notes/release_22_11.rst
> > > +++ b/doc/guides/rel_notes/release_22_11.rst
> > > @@ -75,6 +75,11 @@ New Features
> > >    * Added ``rte_event_eth_tx_adapter_instance_get`` to get Tx adapter
> > >      instance ID for specified ethernet device ID and Tx queue index.
> > >
> > > +* **Added uncore frequency control API to the power library.**
> > > +
> > > +  Add api to allow uncore frequency adjustment. This is done through
> > 
> > s/api/API/
> > 
> > > +  manipulating related uncore frequency control sysfs entries to
> > > + adjust the minimum and maximum uncore frequency values.
> > 
> > It is Linux-only for Intel hardware only.
> > 
> > > --- /dev/null
> > > +++ b/lib/power/rte_power_uncore.c
> > 
> > I would add "intel" in the filename.
> > 
> > [...]
> > > +#define UNCORE_FREQUENCY_DIR
> > "/sys/devices/system/cpu/intel_uncore_frequency"
> > > +#define POWER_GOVERNOR_PERF "performance"
> > > +#define POWER_UNCORE_SYSFILE_MAX_FREQ \
> > > +
> > 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> > die_%02u/max_freq_khz"
> > > +#define POWER_UNCORE_SYSFILE_MIN_FREQ  \
> > > +
> > 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> > die_%02u/min_freq_khz"
> > > +#define POWER_UNCORE_SYSFILE_BASE_MAX_FREQ \
> > > +
> > 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> > die_%02u/initial_max_freq_khz"
> > > +#define POWER_UNCORE_SYSFILE_BASE_MIN_FREQ  \
> > > +
> > 	"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_
> > die_%02u/initial_min_freq_khz"
> > 
> > It is for Intel CPU only, right?
> 
> Currently only Intel CPUs are covered by these sysfs entries, but it is possible that other platforms will be included in the future.

No, these sysfs have "intel" in their names.
So it will never become something not Intel.
At the very minimum, you should name the defines with _INTEL_

> > > + * This function should NOT be called in the fast path.
> > > + *
> > > + * @param pkg
> > > + *  Package number.
> > > + * @param die
> > > + *  Die number.
> > 
> > To me it is not clear what they are.
> > Is it possible to better explain "pkg" and "die" somewhere?
> > Is it related to NUMA nodes?
> 
> Each NUMA node is a package, which can contain 1 or more dies. These dies are connected in a package together via the UNCORE mesh.
> Dies may appear as separate NUMA nodes, or a group of dies on a packages may appear as a single NUMA node, depending on the BIOS configuration.
> Header descriptions will be changed to:
>     * Package number. Each physical CPU in a system is referred to as a package.
>     * Die number. Each package can have several dies connected together via the uncore mesh.

OK better, thanks.

[PATCH v8 0/3] add Intel uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to
allow uncore frequency adjustment. This will be called through the l3fwd-power
app and gives the ability to set the minimum and maximum uncore frequency to
both min, max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.
v5:
Improve error message for uncore access not working.
v6:
Fix uncore exit being called if uncore option not selected.
v7:
Address mailing list comments.
v8:
Address mailing list comments.

Tadhg Kearney (3):
  power: add Intel uncore frequency control API to power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_intel_uncore.c            | 301 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  54 +++
 doc/guides/rel_notes/release_22_11.rst        |   6 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 126 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_intel_uncore.c            | 451 ++++++++++++++++++
 lib/power/rte_power_intel_uncore.h            | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1174 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.h

-- 
2.25.1

[PATCH v8 1/3] power: add Intel uncore frequency control API to power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 23594 bytes --]

Add API to allow uncore frequency adjustment. Uncore is a
term used by Intel to describe function of a microprocessor
that are closely connected to the core to achieve high
performance. This is done through manipulating related
uncore frequency control sysfs entries to adjust the
minimum and maximum uncore frequency values and works
on Linux for Intel hardware.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  54 +++
 doc/guides/rel_notes/release_22_11.rst |   6 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_intel_uncore.c     | 451 +++++++++++++++++++++++++
 lib/power/rte_power_intel_uncore.h     | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 718 insertions(+)
 create mode 100644 lib/power/rte_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..89bc23aa9d 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,60 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Intel Uncore API
+----------------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Intel Uncore
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Overview of each function in the Intel Uncore API, with explanation of what
+they do. Each function should not be called in the fast path.
+
+* **Uncore Power Init**
+    Initialize uncore power, populate frequency array and record
+    original min & max for die on pkg.
+
+* **Uncore Power Exit**
+    Exit uncore power, restoring original min & max for die on pkg.
+
+* **Get Uncore Power Freq**
+    Get current uncore freq index for die on pkg.
+
+* **Set Uncore Power Freq**
+    Set min & max uncore freq index for die on pkg to specified index value
+    (min and max will be the same).
+
+* **Uncore Power Max**
+    Set min & max uncore freq to maximum frequency index for die on pkg
+    (min and max will be the same).
+
+* **Uncore Power Min**
+    Set min & max uncore freq to minimum frequency index for die on pkg
+    (min and max will be the same).
+
+* **Get Num Freqs**
+    Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**
+    Get the number of packages (CPU's) on the system.
+
+* **Get Num Dies**
+    Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index ac67e7e710..3f21a6126b 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -123,6 +123,12 @@ New Features
   into single event containing ``rte_event_vector``
   whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added Intel uncore frequency control API to the power library.**
+
+  Add API to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values, which works on
+  Linux with Intel hardware only.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..c8de545b6c 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_intel_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_intel_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_intel_uncore.c b/lib/power/rte_power_intel_uncore.c
new file mode 100644
index 0000000000..4ba1ed7b4e
--- /dev/null
+++ b/lib/power/rte_power_intel_uncore.c
@@ -0,0 +1,451 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_intel_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define INTEL_UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(INTEL_UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(INTEL_UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_intel_uncore.h b/lib/power/rte_power_intel_uncore.h
new file mode 100644
index 0000000000..3e891f4001
--- /dev/null
+++ b/lib/power/rte_power_intel_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_INTEL_UNCORE_H
+#define _RTE_POWER_INTEL_UNCORE_H
+
+/**
+ * @file
+ * RTE Intel Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to specified
+ * index value.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to maximum
+ * value according to the available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to minimum
+ * value according to the available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v8 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++++
 examples/l3fwd-power/main.c                   | 126 +++++++++++++++++-
 2 files changed, 153 insertions(+), 2 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..d6d71719e5 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_intel_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -161,6 +162,9 @@ static struct rte_timer telemetry_timer;
 /* stats index returned by metrics lib */
 int telstats_index;
 
+/* flag to check if uncore option enabled */
+int enabled_uncore = -1;
+
 struct telstats_name {
 	char name[RTE_ETH_XSTATS_NAME_SIZE];
 };
@@ -179,6 +183,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1626,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1685,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1864,7 +1945,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:PuUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1974,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2466,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2479,26 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	/* if uncore option was set */
+	if (enabled_uncore == 0) {
+		max_pkg = rte_power_uncore_get_num_pkgs();
+		if (max_pkg == 0)
+			return -1;
+		for (pkg = 0; pkg < max_pkg; pkg++) {
+			max_die = rte_power_uncore_get_num_dies(pkg);
+			if (max_die == 0)
+				return -1;
+			for (die = 0; die < max_die; die++) {
+				ret = rte_power_uncore_exit(pkg, die);
+				if (ret < 0) {
+					RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deinit successfully for pkg %02u die %02u\n"
+						, pkg, die);
+					return -1;
+				}
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v8 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 app/test/meson.build               |   2 +
 app/test/test_power_intel_uncore.c | 301 +++++++++++++++++++++++++++++
 2 files changed, 303 insertions(+)
 create mode 100644 app/test/test_power_intel_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index d5cad72116..2ce923a6b3 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_intel_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -241,6 +242,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_intel_uncore.c b/app/test/test_power_intel_uncore.c
new file mode 100644
index 0000000000..4615ce5d19
--- /dev/null
+++ b/app/test/test_power_intel_uncore.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_intel_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_intel_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_uncore_autotest, test_power_intel_uncore);
-- 
2.25.1

[PATCH v9 0/3] add Intel uncore api to be called through l3fwd-power

[Tadhg Kearney]

This is targeting 22.11 and aims to add an API to DPDK power library to
allow uncore frequency adjustment. This will be called through the l3fwd-power
app and gives the ability to set the minimum and maximum uncore frequency to
both min, max or specific frequency index.

Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---

v2:
Fix compilation warnings and errors.
v3:
Remove addition of x86 global macros.
Add 2 new API's for getting package and die numbers from system.
Address comments from mailing list.
Improve efficiency of code and code quality.
v4:
Fix compilation warnings and errors.
v5:
Improve error message for uncore access not working.
v6:
Fix uncore exit being called if uncore option not selected.
v7:
Address mailing list comments.
v8:
Address mailing list comments.
v9:
Fix build errors.

Tadhg Kearney (3):
  power: add Intel uncore frequency control API to power library
  l3fwd-power: add option to call uncore API
  test/power: add unit tests for uncore API

 app/test/meson.build                          |   2 +
 app/test/test_power_intel_uncore.c            | 301 ++++++++++++
 doc/guides/prog_guide/power_man.rst           |  54 +++
 doc/guides/rel_notes/release_22_11.rst        |   6 +
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++
 examples/l3fwd-power/main.c                   | 126 ++++-
 lib/power/meson.build                         |   2 +
 lib/power/rte_power_intel_uncore.c            | 451 ++++++++++++++++++
 lib/power/rte_power_intel_uncore.h            | 194 ++++++++
 lib/power/version.map                         |  11 +
 10 files changed, 1174 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.h

-- 
2.25.1

[PATCH v9 1/3] power: add Intel uncore frequency control API to power library

[Tadhg Kearney]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 23594 bytes --]

Add API to allow uncore frequency adjustment. Uncore is a
term used by Intel to describe function of a microprocessor
that are closely connected to the core to achieve high
performance. This is done through manipulating related
uncore frequency control sysfs entries to adjust the
minimum and maximum uncore frequency values and works
on Linux for Intel hardware.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 doc/guides/prog_guide/power_man.rst    |  54 +++
 doc/guides/rel_notes/release_22_11.rst |   6 +
 lib/power/meson.build                  |   2 +
 lib/power/rte_power_intel_uncore.c     | 451 +++++++++++++++++++++++++
 lib/power/rte_power_intel_uncore.h     | 194 +++++++++++
 lib/power/version.map                  |  11 +
 6 files changed, 718 insertions(+)
 create mode 100644 lib/power/rte_power_intel_uncore.c
 create mode 100644 lib/power/rte_power_intel_uncore.h

diff --git a/doc/guides/prog_guide/power_man.rst b/doc/guides/prog_guide/power_man.rst
index 98cfd3c1f3..89bc23aa9d 100644
--- a/doc/guides/prog_guide/power_man.rst
+++ b/doc/guides/prog_guide/power_man.rst
@@ -276,6 +276,60 @@ API Overview for Ethernet PMD Power Management
 * **Set Scaling Max Freq**: Set the maximum frequency (kHz) to be used in Frequency
   Scaling mode.
 
+Intel Uncore API
+----------------
+
+Abstract
+~~~~~~~~
+
+Uncore is a term used by Intel to describe the functions of a microprocessor that are
+not in the core, but which must be closely connected to the core to achieve high performance;
+L3 cache, on-die memory controller, etc.
+Significant power savings can be achieved by reducing the uncore frequency to its lowest value.
+
+The Linux kernel provides the driver “intel-uncore-frequency" to control the uncore frequency limits
+for x86 platform. The driver is available from kernel version 5.6 and above.
+Also CONFIG_INTEL_UNCORE_FREQ_CONTROL will need to be enabled in the kernel, which was added in 5.6.
+This manipulates the contest of MSR 0x620, which sets min/max of the uncore for the SKU.
+
+
+API Overview for Intel Uncore
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Overview of each function in the Intel Uncore API, with explanation of what
+they do. Each function should not be called in the fast path.
+
+* **Uncore Power Init**
+    Initialize uncore power, populate frequency array and record
+    original min & max for die on pkg.
+
+* **Uncore Power Exit**
+    Exit uncore power, restoring original min & max for die on pkg.
+
+* **Get Uncore Power Freq**
+    Get current uncore freq index for die on pkg.
+
+* **Set Uncore Power Freq**
+    Set min & max uncore freq index for die on pkg to specified index value
+    (min and max will be the same).
+
+* **Uncore Power Max**
+    Set min & max uncore freq to maximum frequency index for die on pkg
+    (min and max will be the same).
+
+* **Uncore Power Min**
+    Set min & max uncore freq to minimum frequency index for die on pkg
+    (min and max will be the same).
+
+* **Get Num Freqs**
+    Get the number of frequencies in the index array.
+
+* **Get Num Pkgs**
+    Get the number of packages (CPU's) on the system.
+
+* **Get Num Dies**
+    Get the number of die's on a given package.
+
 References
 ----------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index ac67e7e710..3f21a6126b 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -123,6 +123,12 @@ New Features
   into single event containing ``rte_event_vector``
   whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
+* **Added Intel uncore frequency control API to the power library.**
+
+  Add API to allow uncore frequency adjustment. This is done through
+  manipulating related uncore frequency control sysfs entries to
+  adjust the minimum and maximum uncore frequency values, which works on
+  Linux with Intel hardware only.
 
 Removed Items
 -------------
diff --git a/lib/power/meson.build b/lib/power/meson.build
index ba8d66074b..c8de545b6c 100644
--- a/lib/power/meson.build
+++ b/lib/power/meson.build
@@ -21,12 +21,14 @@ sources = files(
         'rte_power.c',
         'rte_power_empty_poll.c',
         'rte_power_pmd_mgmt.c',
+        'rte_power_intel_uncore.c',
 )
 headers = files(
         'rte_power.h',
         'rte_power_empty_poll.h',
         'rte_power_pmd_mgmt.h',
         'rte_power_guest_channel.h',
+        'rte_power_intel_uncore.h',
 )
 if cc.has_argument('-Wno-cast-qual')
     cflags += '-Wno-cast-qual'
diff --git a/lib/power/rte_power_intel_uncore.c b/lib/power/rte_power_intel_uncore.c
new file mode 100644
index 0000000000..4ba1ed7b4e
--- /dev/null
+++ b/lib/power/rte_power_intel_uncore.c
@@ -0,0 +1,451 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#include <errno.h>
+#include <dirent.h>
+#include <fnmatch.h>
+
+#include <rte_memcpy.h>
+
+#include "rte_power_intel_uncore.h"
+#include "power_common.h"
+
+#define MAX_UNCORE_FREQS 32
+#define MAX_NUMA_DIE 8
+#define BUS_FREQ     100000
+#define FILTER_LENGTH 18
+#define PACKAGE_FILTER "package_%02u_die_*"
+#define DIE_FILTER "package_%02u_die_%02u"
+#define INTEL_UNCORE_FREQUENCY_DIR "/sys/devices/system/cpu/intel_uncore_frequency"
+#define POWER_GOVERNOR_PERF "performance"
+#define POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/max_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/min_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_max_freq_khz"
+#define POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ  \
+		"/sys/devices/system/cpu/intel_uncore_frequency/package_%02u_die_%02u/initial_min_freq_khz"
+
+
+struct uncore_power_info {
+	unsigned int die;                    /** Core die id */
+	unsigned int pkg;                    /** Package id */
+	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
+	uint32_t nb_freqs;                   /** Number of available freqs */
+	FILE *f_cur_min;                     /** FD of scaling_min */
+	FILE *f_cur_max;                     /** FD of scaling_max */
+	uint32_t curr_idx;                   /** Freq index in freqs array */
+	uint32_t org_min_freq;               /** Original min freq of uncore */
+	uint32_t org_max_freq;               /** Original max freq of uncore */
+	uint32_t init_max_freq;              /** System max uncore freq */
+	uint32_t init_min_freq;              /** System min uncore freq */
+} __rte_cache_aligned;
+
+static struct uncore_power_info uncore_info[RTE_MAX_NUMA_NODES][MAX_NUMA_DIE];
+
+static int
+set_uncore_freq_internal(struct uncore_power_info *ui, uint32_t idx)
+{
+	uint32_t target_uncore_freq, curr_max_freq;
+	int ret;
+
+	if (idx >= MAX_UNCORE_FREQS || idx >= ui->nb_freqs) {
+		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
+				"should be less than %u\n", idx, ui->nb_freqs);
+		return -1;
+	}
+
+	target_uncore_freq = ui->freqs[idx];
+
+	/* check current max freq, so that the value to be flushed first
+	 * can be accurately recorded
+	 */
+	open_core_sysfs_file(&ui->f_cur_max, "rw+", POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (ui->f_cur_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		return -1;
+	}
+	ret = read_core_sysfs_u32(ui->f_cur_max, &curr_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		fclose(ui->f_cur_max);
+		return -1;
+	}
+
+	/* check this value first before fprintf value to f_cur_max, so value isn't overwritten */
+	if (fprintf(ui->f_cur_min, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", target_uncore_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write new uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	POWER_DEBUG_TRACE("Uncore frequency '%u' to be set for pkg %02u die %02u\n",
+				target_uncore_freq, ui->pkg, ui->die);
+
+
+	/* write the minimum value first if the target freq is less than current max */
+	if (target_uncore_freq <= curr_max_freq) {
+		fflush(ui->f_cur_min);
+		fflush(ui->f_cur_max);
+	} else {
+		fflush(ui->f_cur_max);
+		fflush(ui->f_cur_min);
+	}
+	ui->curr_idx = idx;
+
+	return 0;
+}
+
+/**
+ * Fopen the sys file for the future setting of the uncore die frequency.
+ */
+static int
+power_init_for_setting_uncore_freq(struct uncore_power_info *ui)
+{
+	FILE *f_base_min = NULL, *f_base_max = NULL, *f_min = NULL, *f_max = NULL;
+	uint32_t base_min_freq = 0, base_max_freq = 0, min_freq = 0, max_freq = 0;
+	int ret;
+
+	/* open and read all uncore sys files */
+	/* Base max */
+	open_core_sysfs_file(&f_base_max, "r", POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_base_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+	ret = read_core_sysfs_u32(f_base_max, &base_max_freq);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MAX_FREQ);
+		goto err;
+	}
+
+	/* Base min */
+	open_core_sysfs_file(&f_base_min, "r", POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ,
+		ui->pkg, ui->die);
+	if (f_base_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ);
+		goto err;
+	}
+	if (f_base_min != NULL) {
+		ret = read_core_sysfs_u32(f_base_min, &base_min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_BASE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr min */
+	open_core_sysfs_file(&f_min, "rw+", POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ,
+			ui->pkg, ui->die);
+	if (f_min == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ);
+		goto err;
+	}
+	if (f_min != NULL) {
+		ret = read_core_sysfs_u32(f_min, &min_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_MIN_FREQ);
+			goto err;
+		}
+	}
+
+	/* Curr max */
+	open_core_sysfs_file(&f_max, "rw+", POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ,
+			ui->pkg, ui->die);
+	if (f_max == NULL) {
+		RTE_LOG(DEBUG, POWER, "failed to open %s\n",
+				POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+		goto err;
+	}
+	if (f_max != NULL) {
+		ret = read_core_sysfs_u32(f_max, &max_freq);
+		if (ret < 0) {
+			RTE_LOG(DEBUG, POWER, "Failed to read %s\n",
+					POWER_INTEL_UNCORE_SYSFILE_MAX_FREQ);
+			goto err;
+		}
+	}
+
+	/* assign file handles */
+	ui->f_cur_min = f_min;
+	ui->f_cur_max = f_max;
+	/* save current min + max freq's so that they can be restored on exit */
+	ui->org_min_freq = min_freq;
+	ui->org_max_freq = max_freq;
+	ui->init_max_freq = base_max_freq;
+	ui->init_min_freq = base_min_freq;
+
+	return 0;
+
+err:
+	if (f_base_min != NULL)
+		fclose(f_base_min);
+	if (f_base_max != NULL)
+		fclose(f_base_max);
+	if (f_min != NULL)
+		fclose(f_min);
+	if (f_max != NULL)
+		fclose(f_max);
+	return -1;
+}
+
+/**
+ * Get the available uncore frequencies of the specific die by reading the
+ * sys file.
+ */
+static int
+power_get_available_uncore_freqs(struct uncore_power_info *ui)
+{
+	int ret = -1;
+	uint32_t i, num_uncore_freqs = 0;
+
+	num_uncore_freqs = (ui->init_max_freq - ui->init_min_freq) / BUS_FREQ + 1;
+	if (num_uncore_freqs >= MAX_UNCORE_FREQS) {
+		RTE_LOG(ERR, POWER, "Too many available uncore frequencies: %d\n",
+				num_uncore_freqs);
+		goto out;
+	}
+
+	/* Generate the uncore freq bucket array. */
+	for (i = 0; i < num_uncore_freqs; i++)
+		ui->freqs[i] = ui->init_max_freq - (i) * BUS_FREQ;
+
+	ui->nb_freqs = num_uncore_freqs;
+
+	ret = 0;
+
+	POWER_DEBUG_TRACE("%d frequency(s) of pkg %02u die %02u are available\n",
+			num_uncore_freqs, ui->pkg, ui->die);
+
+out:
+	return ret;
+}
+
+static int
+check_pkg_die_values(unsigned int pkg, unsigned int die)
+{
+	unsigned int max_pkgs, max_dies;
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return -1;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Package number %02u can not exceed %u\n",
+				pkg, max_pkgs);
+		return -1;
+	}
+
+	max_dies = rte_power_uncore_get_num_dies(pkg);
+	if (max_dies == 0)
+		return -1;
+	if (die >= max_dies) {
+		RTE_LOG(DEBUG, POWER, "Die number %02u can not exceed %u\n",
+				die, max_dies);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+	ui->die = die;
+	ui->pkg = pkg;
+
+	/* Init for setting uncore die frequency */
+	if (power_init_for_setting_uncore_freq(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot init for setting uncore frequency for "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	/* Get the available frequencies */
+	if (power_get_available_uncore_freqs(ui) < 0) {
+		RTE_LOG(DEBUG, POWER, "Cannot get available uncore frequencies of "
+				"pkg %02u die %02u\n", pkg, die);
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die)
+{
+	struct uncore_power_info *ui;
+
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	ui = &uncore_info[pkg][die];
+
+	if (fprintf(ui->f_cur_min, "%u", ui->org_min_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	if (fprintf(ui->f_cur_max, "%u", ui->org_max_freq) < 0) {
+		RTE_LOG(ERR, POWER, "Fail to write original uncore frequency for "
+				"pkg %02u die %02u\n", ui->pkg, ui->die);
+		return -1;
+	}
+
+	fflush(ui->f_cur_min);
+	fflush(ui->f_cur_max);
+
+	/* Close FD of setting freq */
+	fclose(ui->f_cur_min);
+	fclose(ui->f_cur_max);
+	ui->f_cur_min = NULL;
+	ui->f_cur_max = NULL;
+
+	return 0;
+}
+
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].curr_idx;
+}
+
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), index);
+}
+
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), 0);
+}
+
+
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	struct uncore_power_info *ui = &uncore_info[pkg][die];
+
+	return set_uncore_freq_internal(&(uncore_info[pkg][die]), ui->nb_freqs - 1);
+}
+
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die)
+{
+	int ret = check_pkg_die_values(pkg, die);
+	if (ret < 0)
+		return -1;
+
+	return uncore_info[pkg][die].nb_freqs;
+}
+
+unsigned int
+rte_power_uncore_get_num_pkgs(void)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0;
+	char filter[FILTER_LENGTH];
+
+	d = opendir(INTEL_UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max pkg file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, PACKAGE_FILTER, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
+
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg)
+{
+	DIR *d;
+	struct dirent *dir;
+	unsigned int count = 0, max_pkgs;
+	char filter[FILTER_LENGTH];
+
+	max_pkgs = rte_power_uncore_get_num_pkgs();
+	if (max_pkgs == 0)
+		return 0;
+	if (pkg >= max_pkgs) {
+		RTE_LOG(DEBUG, POWER, "Invalid package number\n");
+		return 0;
+	}
+
+	d = opendir(INTEL_UNCORE_FREQUENCY_DIR);
+	if (d == NULL) {
+		RTE_LOG(ERR, POWER,
+		"Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return 0;
+	}
+
+	/* search by incrementing file name for max die file value */
+	while ((dir = readdir(d)) != NULL) {
+		snprintf(filter, FILTER_LENGTH, DIE_FILTER, pkg, count);
+		/* make sure filter string is in file name (don't include hidden files) */
+		if (fnmatch(filter, dir->d_name, 0) == 0)
+			count++;
+	}
+
+	closedir(d);
+
+	return count;
+}
diff --git a/lib/power/rte_power_intel_uncore.h b/lib/power/rte_power_intel_uncore.h
new file mode 100644
index 0000000000..3e891f4001
--- /dev/null
+++ b/lib/power/rte_power_intel_uncore.h
@@ -0,0 +1,194 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Intel Corporation
+ */
+
+#ifndef _RTE_POWER_INTEL_UNCORE_H
+#define _RTE_POWER_INTEL_UNCORE_H
+
+/**
+ * @file
+ * RTE Intel Uncore Frequency Management
+ */
+
+#include "rte_power.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * Initialize uncore frequency management for specific die on a package. It will get the available
+ * frequencies and prepare to set new die frequencies.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_init(unsigned int pkg, unsigned int die);
+
+/**
+ * Exit uncore frequency management on a specific die on a package. It will restore uncore min and
+ * max values to previous values before initialization of API.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 0 on success.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_exit(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the current index of available frequencies of a specific die on a package.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  The current index of available frequencies.
+ *  If error, it will return 'RTE_POWER_INVALID_FREQ_INDEX = (~0)'.
+ */
+__rte_experimental
+uint32_t
+rte_power_get_uncore_freq(unsigned int pkg, unsigned int die);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to specified
+ * index value.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ * @param index
+ *  The index of available frequencies.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_set_uncore_freq(unsigned int pkg, unsigned int die, uint32_t index);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to maximum
+ * value according to the available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_max(unsigned int pkg, unsigned int die);
+
+/**
+ * Set minimum and maximum uncore frequency for specified die on a package to minimum
+ * value according to the available frequencies.
+ * It should be protected outside of this function for threadsafe.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - 1 on success with frequency changed.
+ *  - 0 on success without frequency changed.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_freq_min(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the list length of available frequencies in the index array.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ * @param die
+ *  Die number. Each package can have several dies connected together via the uncore mesh.
+ *
+ * @return
+ *  - The number of available index's in frequency array.
+ *  - Negative on error.
+ */
+__rte_experimental
+int
+rte_power_uncore_get_num_freqs(unsigned int pkg, unsigned int die);
+
+/**
+ * Return the number of packages (CPUs) on a system by parsing the uncore
+ * sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of package on system on success.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_pkgs(void);
+
+/**
+ * Return the number of dies for pakckages (CPUs) specified from parsing
+ * the uncore sysfs directory.
+ *
+ * This function should NOT be called in the fast path.
+ *
+ * @param pkg
+ *  Package number. Each physical CPU in a system is referred to as a package.
+ *
+ * @return
+ *  - Zero on error.
+ *  - Number of dies for package on sucecss.
+ */
+__rte_experimental
+unsigned int
+rte_power_uncore_get_num_dies(unsigned int pkg);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/lib/power/version.map b/lib/power/version.map
index f9b2947adf..8fccbf20f7 100644
--- a/lib/power/version.map
+++ b/lib/power/version.map
@@ -48,4 +48,15 @@ EXPERIMENTAL {
 	rte_power_pmd_mgmt_set_pause_duration;
 	rte_power_pmd_mgmt_set_scaling_freq_max;
 	rte_power_pmd_mgmt_set_scaling_freq_min;
+
+	# added in 22.11
+	rte_power_get_uncore_freq;
+	rte_power_set_uncore_freq;
+	rte_power_uncore_exit;
+	rte_power_uncore_freq_max;
+	rte_power_uncore_freq_min;
+	rte_power_uncore_get_num_dies;
+	rte_power_uncore_get_num_freqs;
+	rte_power_uncore_get_num_pkgs;
+	rte_power_uncore_init;
 };
-- 
2.25.1

[PATCH v9 2/3] l3fwd-power: add option to call uncore API

[Tadhg Kearney]

Add option for setting uncore frequency min/max/index, through uncore API.
This will be set for each package and die on the SKU. On exit, uncore min
and max frequency will be reverted back to previous frequencies.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 .../sample_app_ug/l3_forward_power_man.rst    |  29 ++++
 examples/l3fwd-power/main.c                   | 126 +++++++++++++++++-
 2 files changed, 153 insertions(+), 2 deletions(-)

diff --git a/doc/guides/sample_app_ug/l3_forward_power_man.rst b/doc/guides/sample_app_ug/l3_forward_power_man.rst
index 8f6d906200..08ac8ef369 100644
--- a/doc/guides/sample_app_ug/l3_forward_power_man.rst
+++ b/doc/guides/sample_app_ug/l3_forward_power_man.rst
@@ -97,6 +97,12 @@ where,
 *   -P: Sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address.
     Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted.
 
+*   -u: optional, sets uncore min/max frequency to minimum value.
+
+*   -U: optional, sets uncore min/max frequency to maximum value.
+
+*   -i (frequency index): optional, sets uncore frequency to frequency index value, by setting min and max values to be the same.
+
 *   --config (port,queue,lcore)[,(port,queue,lcore)]: determines which queues from which ports are mapped to which cores.
 
 *   --max-pkt-len: optional, maximum packet length in decimal (64-9600)
@@ -364,3 +370,26 @@ in the DPDK Programmer's Guide for more details on PMD power management.
 .. code-block:: console
 
         ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" --pmd-mgmt=scale
+
+Setting Uncore Values
+---------------------
+
+Uncore frequency can be adjusted through manipulating related sysfs entries to adjust the minimum and maximum uncore values.
+This will be set for each package and die on the SKU. The driver for enabling this is available from kernel version 5.6 and above.
+Three options are available for setting uncore frequency:
+
+``-u``
+  This will set uncore minimum and maximum frequencies to minimum possible value.
+
+``-U``
+  This will set uncore minimum and maximum frequencies to maximum possible value.
+
+``-i``
+  This will allow you to set the specific uncore frequency index that you want, by setting
+  the uncore frequency to a frequency pointed by index. Frequency index's are set 100MHz apart from
+  maximum to minimum.
+  Frequency index values are in descending order, ie, index 0 is maximum frequency index.
+
+.. code-block:: console
+
+        ./<build_dir>/examples/dpdk-l3fwd-power -l 1-3 -- -p 0x0f --config="(0,0,2),(0,1,3)" -i 1
diff --git a/examples/l3fwd-power/main.c b/examples/l3fwd-power/main.c
index 887c6eae3f..d6d71719e5 100644
--- a/examples/l3fwd-power/main.c
+++ b/examples/l3fwd-power/main.c
@@ -47,6 +47,7 @@
 #include <rte_metrics.h>
 #include <rte_telemetry.h>
 #include <rte_power_pmd_mgmt.h>
+#include <rte_power_intel_uncore.h>
 
 #include "perf_core.h"
 #include "main.h"
@@ -161,6 +162,9 @@ static struct rte_timer telemetry_timer;
 /* stats index returned by metrics lib */
 int telstats_index;
 
+/* flag to check if uncore option enabled */
+int enabled_uncore = -1;
+
 struct telstats_name {
 	char name[RTE_ETH_XSTATS_NAME_SIZE];
 };
@@ -179,6 +183,12 @@ enum busy_rate {
 	FULL = 100
 };
 
+enum uncore_choice {
+	UNCORE_MIN = 0,
+	UNCORE_MAX = 1,
+	UNCORE_IDX = 2
+};
+
 /* reference poll count to measure core busyness */
 #define DEFAULT_COUNT 10000
 /*
@@ -1616,6 +1626,9 @@ print_usage(const char *prgname)
 		"  [--max-pkt-len PKTLEN]\n"
 		"  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 		"  -P: enable promiscuous mode\n"
+		"  -u: set min/max frequency for uncore to minimum value\n"
+		"  -U: set min/max frequency for uncore to maximum value\n"
+		"  -i (frequency index): set min/max frequency for uncore to specified frequency index\n"
 		"  --config (port,queue,lcore): rx queues configuration\n"
 		"  --high-perf-cores CORELIST: list of high performance cores\n"
 		"  --perf-config: similar as config, cores specified as indices"
@@ -1672,6 +1685,74 @@ static int parse_max_pkt_len(const char *pktlen)
 	return len;
 }
 
+static int
+parse_uncore_options(enum uncore_choice choice, const char *argument)
+{
+	unsigned int die, pkg, max_pkg, max_die;
+	int ret = 0;
+	max_pkg = rte_power_uncore_get_num_pkgs();
+	if (max_pkg == 0)
+		return -1;
+
+	for (pkg = 0; pkg < max_pkg; pkg++) {
+		max_die = rte_power_uncore_get_num_dies(pkg);
+		if (max_die == 0)
+			return -1;
+		for (die = 0; die < max_die; die++) {
+			ret = rte_power_uncore_init(pkg, die);
+			if (ret == -1) {
+				RTE_LOG(INFO, L3FWD_POWER, "Unable to initialize uncore for pkg %02u die %02u\n"
+				, pkg, die);
+				return ret;
+			}
+			if (choice == UNCORE_MIN) {
+				ret = rte_power_uncore_freq_min(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set the uncore min/max to minimum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_MAX) {
+				ret = rte_power_uncore_freq_max(pkg, die);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set uncore min/max to maximum uncore frequency value for pkg %02u die %02u\n"
+					, pkg, die);
+					return ret;
+				}
+			} else if (choice == UNCORE_IDX) {
+				char *ptr = NULL;
+				int frequency_index = strtol(argument, &ptr, 10);
+				if (argument == ptr) {
+					RTE_LOG(INFO, L3FWD_POWER, "Index given is not a valid number.");
+					return -1;
+				}
+				int freq_array_len = rte_power_uncore_get_num_freqs(pkg, die);
+				if (frequency_index > freq_array_len - 1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Frequency index given out of range, please choose a value from 0 to %d.\n",
+					freq_array_len);
+					return -1;
+				}
+				ret = rte_power_set_uncore_freq(pkg, die, frequency_index);
+				if (ret == -1) {
+					RTE_LOG(INFO, L3FWD_POWER,
+					"Unable to set min/max uncore index value for pkg %02u die %02u\n",
+					pkg, die);
+					return ret;
+				}
+			} else {
+				RTE_LOG(INFO, L3FWD_POWER, "Uncore choice provided invalid\n");
+				return -1;
+			}
+		}
+	}
+
+	RTE_LOG(INFO, L3FWD_POWER, "Successfully set max/min/index uncore frequency.\n");
+	return ret;
+}
+
 static int
 parse_portmask(const char *portmask)
 {
@@ -1864,7 +1945,7 @@ parse_args(int argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:P",
+	while ((opt = getopt_long(argc, argvopt, "p:l:m:h:PuUi:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1893,6 +1974,27 @@ parse_args(int argc, char **argv)
 			limit = parse_max_pkt_len(optarg);
 			freq_tlb[HGH] = limit;
 			break;
+		case 'u':
+			enabled_uncore = parse_uncore_options(UNCORE_MIN, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'U':
+			enabled_uncore = parse_uncore_options(UNCORE_MAX, NULL);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case 'i':
+			enabled_uncore = parse_uncore_options(UNCORE_IDX, optarg);
+			if (enabled_uncore < 0) {
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		/* long options */
 		case 0:
 			if (!strncmp(lgopts[option_index].name, "config", 6)) {
@@ -2364,7 +2466,7 @@ init_power_library(void)
 static int
 deinit_power_library(void)
 {
-	unsigned int lcore_id;
+	unsigned int lcore_id, max_pkg, max_die, die, pkg;
 	int ret = 0;
 
 	RTE_LCORE_FOREACH(lcore_id) {
@@ -2377,6 +2479,26 @@ deinit_power_library(void)
 			return ret;
 		}
 	}
+
+	/* if uncore option was set */
+	if (enabled_uncore == 0) {
+		max_pkg = rte_power_uncore_get_num_pkgs();
+		if (max_pkg == 0)
+			return -1;
+		for (pkg = 0; pkg < max_pkg; pkg++) {
+			max_die = rte_power_uncore_get_num_dies(pkg);
+			if (max_die == 0)
+				return -1;
+			for (die = 0; die < max_die; die++) {
+				ret = rte_power_uncore_exit(pkg, die);
+				if (ret < 0) {
+					RTE_LOG(ERR, L3FWD_POWER, "Failed to exit uncore deinit successfully for pkg %02u die %02u\n"
+						, pkg, die);
+					return -1;
+				}
+			}
+		}
+	}
 	return ret;
 }
 
-- 
2.25.1

[PATCH v9 3/3] test/power: add unit tests for uncore API

[Tadhg Kearney]

Add basic unit tests covering all nine uncore API's.

Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
Reviewed-by: David Hunt <david.hunt@intel.com>
Acked-by: David Hunt <david.hunt@intel.com>
---
 app/test/meson.build               |   2 +
 app/test/test_power_intel_uncore.c | 301 +++++++++++++++++++++++++++++
 2 files changed, 303 insertions(+)
 create mode 100644 app/test/test_power_intel_uncore.c

diff --git a/app/test/meson.build b/app/test/meson.build
index d5cad72116..396b133959 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -101,6 +101,7 @@ test_sources = files(
         'test_power.c',
         'test_power_cpufreq.c',
         'test_power_kvm_vm.c',
+        'test_power_intel_uncore.c',
         'test_prefetch.c',
         'test_rand_perf.c',
         'test_rawdev.c',
@@ -241,6 +242,7 @@ fast_tests = [
         ['power_cpufreq_autotest', false, true],
         ['power_autotest', true, true],
         ['power_kvm_vm_autotest', false, true],
+        ['power_intel_uncore_autotest', true, true],
         ['reorder_autotest', true, true],
         ['service_autotest', true, true],
         ['thash_autotest', true, true],
diff --git a/app/test/test_power_intel_uncore.c b/app/test/test_power_intel_uncore.c
new file mode 100644
index 0000000000..31163af84e
--- /dev/null
+++ b/app/test/test_power_intel_uncore.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2022 Intel Corporation
+ */
+
+#include "test.h"
+
+#ifndef RTE_LIB_POWER
+
+static int
+test_power_intel_uncore(void)
+{
+	printf("Power management library not supported, skipping test\n");
+	return TEST_SKIPPED;
+}
+
+#else
+#include <rte_power_intel_uncore.h>
+#include <power_common.h>
+
+#define MAX_UNCORE_FREQS 32
+
+#define VALID_PKG 0
+#define VALID_DIE 0
+#define INVALID_PKG (rte_power_uncore_get_num_pkgs() + 1)
+#define INVALID_DIE (rte_power_uncore_get_num_dies(VALID_PKG) + 1)
+#define VALID_INDEX 1
+#define INVALID_INDEX (MAX_UNCORE_FREQS + 1)
+
+static int check_power_uncore_init(void)
+{
+	int ret;
+
+	/* Test initialisation of uncore configuration*/
+	ret = rte_power_uncore_init(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Cannot initialise uncore power management for pkg %u die %u, this "
+			"may occur if environment is not configured "
+			"correctly(APCI cpufreq) or operating in another valid "
+			"Power management environment\n", VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_init(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to initialise uncore power management "
+			"for pkg %u die %u\n", INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_get_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully get uncore freq */
+	ret = rte_power_get_uncore_freq(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_get_uncore_freq(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got invalid uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_set_uncore_freq(void)
+{
+	int ret;
+
+	/* Successfully set uncore freq */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, VALID_INDEX);
+	if (ret < 0) {
+		printf("Failed to set uncore frequency for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	/* Try to unsuccessfully set invalid uncore freq index */
+	ret = rte_power_set_uncore_freq(VALID_PKG, VALID_DIE, INVALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore index for pkg %u die %u index %u\n",
+							VALID_PKG, VALID_DIE, INVALID_INDEX);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_set_uncore_freq(INVALID_PKG, INVALID_DIE, VALID_INDEX);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid uncore frequency for pkg %u die %u index %u\n",
+							INVALID_PKG, INVALID_DIE, VALID_INDEX);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_max(void)
+{
+	int ret;
+
+	/* Successfully get max uncore freq */
+	ret = rte_power_uncore_freq_max(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set max uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_max(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid max uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_freq_min(void)
+{
+	int ret;
+
+	/* Successfully get min uncore freq */
+	ret = rte_power_uncore_freq_min(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to set min uncore frequency for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_freq_min(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly set invalid min uncore frequency for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_freqs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore freq */
+	ret = rte_power_uncore_get_num_freqs(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to get number of uncore frequencies for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+		return -1;
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_get_num_freqs(INVALID_PKG, INVALID_DIE);
+	if (ret >= 0) {
+		printf("Unexpectedly got number of invalid frequencies for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_pkgs(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore pkgs */
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Failed to get number of uncore pkgs\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_get_num_dies(void)
+{
+	int ret;
+
+	/* Successfully get number of uncore dies */
+	ret = rte_power_uncore_get_num_dies(VALID_PKG);
+	if (ret == 0) {
+		printf("Failed to get number of uncore dies for pkg %u\n",
+							VALID_PKG);
+		return -1;
+	}
+
+	/* Unsuccessful test */
+	ret = rte_power_uncore_get_num_dies(INVALID_PKG);
+	if (ret > 0) {
+		printf("Unexpectedly got number of invalid dies for pkg %u\n",
+							INVALID_PKG);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+check_power_uncore_exit(void)
+{
+	int ret;
+
+	/* Successfully exit uncore power management */
+	ret = rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	if (ret < 0) {
+		printf("Failed to exit uncore power management for pkg %u die %u\n",
+							VALID_PKG, VALID_DIE);
+	}
+
+	/* Unsuccessful Test */
+	ret = rte_power_uncore_exit(INVALID_PKG, INVALID_DIE);
+	if (ret == 0) {
+		printf("Unexpectedly was able to exit uncore power management for pkg %u die %u\n",
+							INVALID_PKG, INVALID_DIE);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+test_power_intel_uncore(void)
+{
+	int ret;
+
+	ret = rte_power_uncore_get_num_pkgs();
+	if (ret == 0) {
+		printf("Uncore frequency management not supported/enabled on this kernel. "
+		"Please enable CONFIG_INTEL_UNCORE_FREQ_CONTROL if on x86 with linux kernel"
+		" >= 5.6\n");
+		return TEST_SKIPPED;
+	}
+
+	ret = check_power_uncore_init();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_get_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_set_uncore_freq();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_max();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_freq_min();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_freqs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_pkgs();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_get_num_dies();
+	if (ret < 0)
+		goto fail_all;
+
+	ret = check_power_uncore_exit();
+	if (ret < 0)
+		return -1;
+
+	return 0;
+
+fail_all:
+	rte_power_uncore_exit(VALID_PKG, VALID_DIE);
+	return -1;
+}
+#endif
+
+REGISTER_TEST_COMMAND(power_intel_uncore_autotest, test_power_intel_uncore);
-- 
2.25.1

Re: [PATCH v9 1/3] power: add Intel uncore frequency control API to power library

[Stephen Hemminger]

On Thu,  6 Oct 2022 09:38:01 +0000
Tadhg Kearney <tadhg.kearney@intel.com> wrote:

> Add API to allow uncore frequency adjustment. Uncore is a
> term used by Intel to describe function of a microprocessor
> that are closely connected to the core to achieve high
> performance. This is done through manipulating related
> uncore frequency control sysfs entries to adjust the
> minimum and maximum uncore frequency values and works
> on Linux for Intel hardware.
> 
> Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
> Reviewed-by: David Hunt <david.hunt@intel.com>
> Acked-by: David Hunt <david.hunt@intel.com>

Looks like this is missing an opportunity for a more general
long term solution in DPDK.

Shouldn't this be a general thing like the Linux kernel scheduler.
Uncore is Intel specific, but there is already big/little cores
on many ARM platforms.

Re: [PATCH v9 1/3] power: add Intel uncore frequency control API to power library

[Hunt, David]

On 06/10/2022 18:32, Stephen Hemminger wrote:
> On Thu,  6 Oct 2022 09:38:01 +0000
> Tadhg Kearney <tadhg.kearney@intel.com> wrote:
>
>> Add API to allow uncore frequency adjustment. Uncore is a
>> term used by Intel to describe function of a microprocessor
>> that are closely connected to the core to achieve high
>> performance. This is done through manipulating related
>> uncore frequency control sysfs entries to adjust the
>> minimum and maximum uncore frequency values and works
>> on Linux for Intel hardware.
>>
>> Signed-off-by: Tadhg Kearney <tadhg.kearney@intel.com>
>> Reviewed-by: David Hunt <david.hunt@intel.com>
>> Acked-by: David Hunt <david.hunt@intel.com>


Hi Stephen,

> Looks like this is missing an opportunity for a more general
> long term solution in DPDK.


We're hoping that this is the first step along the path to that 
long-term solution. It's like the power library frequency control for 
cores, which was initially Intel only, and then more architectures were 
added over time. The API's are experimental, so can be adapted if needed.


> Shouldn't this be a general thing like the Linux kernel scheduler.

I don't think the kernel scheduler has any concept of uncore busyness, 
the uncore frequency is typically controlled by hardware, and if there's 
enough polling going on, the frequency of the uncore will remain high 
(if uncore frequency scaling is enabled). We're addressing that in this 
patch in that if the application realises that it's not processing a lot 
of packets even though most of it's cores are polling, it can tell the 
hardware to scale down the uncore to save power.


> Uncore is Intel specific, but there is already big/little cores
> on many ARM platforms.

I don't think big/little is related to uncore frequency scaling. The 
big/little cores still need to communicate via that architecture's 
communications bus (called uncore in Intel's case, though I've seen that 
term used on other architectures also). Where an architecture can scale 
the frequency of this communications bus, this patch set's functionality 
can be extended in the future to cover this.

Regards,
Dave

Re: [PATCH v9 1/3] power: add Intel uncore frequency control API to power library

[Thomas Monjalon]

06/10/2022 11:38, Tadhg Kearney:
> +API Overview for Intel Uncore
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Overview of each function in the Intel Uncore API, with explanation of what
> +they do. Each function should not be called in the fast path.
> +
> +* **Uncore Power Init**
> +    Initialize uncore power, populate frequency array and record
> +    original min & max for die on pkg.
> +
> +* **Uncore Power Exit**
> +    Exit uncore power, restoring original min & max for die on pkg.
> +
> +* **Get Uncore Power Freq**
> +    Get current uncore freq index for die on pkg.
> +
> +* **Set Uncore Power Freq**
> +    Set min & max uncore freq index for die on pkg to specified index value
> +    (min and max will be the same).
> +
> +* **Uncore Power Max**
> +    Set min & max uncore freq to maximum frequency index for die on pkg
> +    (min and max will be the same).
> +
> +* **Uncore Power Min**
> +    Set min & max uncore freq to minimum frequency index for die on pkg
> +    (min and max will be the same).
> +
> +* **Get Num Freqs**
> +    Get the number of frequencies in the index array.
> +
> +* **Get Num Pkgs**
> +    Get the number of packages (CPU's) on the system.
> +
> +* **Get Num Dies**
> +    Get the number of die's on a given package.

This is the role of doxygen documentation to explain API.
I don't understand why it is there.
Anyway I've converted it into a definition list,
which the proper RST syntax for what you do.

>          'rte_power.c',
>          'rte_power_empty_poll.c',
>          'rte_power_pmd_mgmt.c',
> +        'rte_power_intel_uncore.c',

In general, we keep such list in alphabetical order.

[...]
> +struct uncore_power_info {
> +	unsigned int die;                    /** Core die id */
> +	unsigned int pkg;                    /** Package id */
> +	uint32_t freqs[MAX_UNCORE_FREQS];/** Frequency array */
> +	uint32_t nb_freqs;                   /** Number of available freqs */
> +	FILE *f_cur_min;                     /** FD of scaling_min */
> +	FILE *f_cur_max;                     /** FD of scaling_max */
> +	uint32_t curr_idx;                   /** Freq index in freqs array */
> +	uint32_t org_min_freq;               /** Original min freq of uncore */
> +	uint32_t org_max_freq;               /** Original max freq of uncore */
> +	uint32_t init_max_freq;              /** System max uncore freq */
> +	uint32_t init_min_freq;              /** System min uncore freq */
> +} __rte_cache_aligned;

No need of doxygen syntax in a .c file.
And an alignment is wrong.

[...]
> +		RTE_LOG(DEBUG, POWER, "Invalid uncore frequency index %u, which "
> +				"should be less than %u\n", idx, ui->nb_freqs);

When you have time, it would be good to switch to dynamic logging.
See RTE_LOG_REGISTER_DEFAULT


[...]
> +#ifndef _RTE_POWER_INTEL_UNCORE_H
> +#define _RTE_POWER_INTEL_UNCORE_H

underscore prefix is reserved for reserved keywords

[...]
> +/**
> + * Exit uncore frequency management on a specific die on a package. It will restore uncore min and

Screen width is limited, while scrolling bar is infinite,
so don't hesitate to break your lines shorter when possible,
like at the end of a sentence.

Re: [PATCH v9 0/3] add Intel uncore api to be called through l3fwd-power

[Thomas Monjalon]

06/10/2022 11:38, Tadhg Kearney:
> This is targeting 22.11 and aims to add an API to DPDK power library to
> allow uncore frequency adjustment. This will be called through the l3fwd-power
> app and gives the ability to set the minimum and maximum uncore frequency to
> both min, max or specific frequency index.
> 
> Signed-off-by: tadhgkearney <tadhg.kearney@intel.com>
> Reviewed-by: David Hunt <david.hunt@intel.com>
> Acked-by: David Hunt <david.hunt@intel.com>

I did some edits to make it more comfortable to my eyes.
Applied, thanks.

I really would like 2 follow-ups please:
- some general clean-up in power library code and doc.
- some discussion about how we can make this library more generic.