From: Anatoly Burakov <anatoly.burakov@intel.com>
To: dev@dpdk.org
Cc: john.mcnamara@intel.com, bruce.richardson@intel.com,
pablo.de.lara.guarch@intel.com, david.hunt@intel.com,
mohammad.abdul.awal@intel.com, thomas@monjalon.net,
ferruh.yigit@intel.com
Subject: [dpdk-dev] [RFC v2 2/9] usertools/lib: add platform info library
Date: Thu, 15 Nov 2018 15:47:14 +0000 [thread overview]
Message-ID: <3b48e558478fdad1e9e6a8e58f723734be79ae21.1542291869.git.anatoly.burakov@intel.com> (raw)
In-Reply-To: <cover.1542291869.git.anatoly.burakov@intel.com>
In-Reply-To: <cover.1542291869.git.anatoly.burakov@intel.com>
Add a library that will parse system information:
* NUMA nodes
* Cores and threads
* Mapping from NUMA node and core to thread id's
* Hyperthreading support status
* RAM size
* Default hugepage size as reported by kernel
This can be used by scripts.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
usertools/DPDKConfigLib/PlatformInfo.py | 130 ++++++++++++++++++++++++
usertools/DPDKConfigLib/Util.py | 16 +++
2 files changed, 146 insertions(+)
create mode 100755 usertools/DPDKConfigLib/PlatformInfo.py
create mode 100755 usertools/DPDKConfigLib/Util.py
diff --git a/usertools/DPDKConfigLib/PlatformInfo.py b/usertools/DPDKConfigLib/PlatformInfo.py
new file mode 100755
index 000000000..734d22026
--- /dev/null
+++ b/usertools/DPDKConfigLib/PlatformInfo.py
@@ -0,0 +1,130 @@
+#!/usr/bin/python
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+
+from .Util import *
+import re
+import glob
+
+__SYSFS_CPU_INFO_PATH = "/sys/devices/system/cpu"
+__SYSFS_IOMMU_CLASS_PATH = "/sys/class/iommu"
+__KERNEL_HPSZ_PATH = "/sys/kernel/mm/hugepages/hugepages-*"
+
+try:
+ xrange # python 2
+except NameError:
+ xrange = range # python 3
+
+
+__CPU_FLAGS_TO_HP_SIZE = {
+ # x86 supports 2M and 1G pages
+ "pse": 2048,
+ "pdpe1gb": 1048576
+}
+__IOMMU_CPU_FLAGS = ["vmx", "vms"]
+
+__HT_CPU_FLAGS = ["ht"]
+
+
+def _parse_cpuinfo(pinfo):
+ core_info_list = []
+ with open("/proc/cpuinfo") as f:
+ cur_core = {}
+ for line in f:
+ line = line.strip()
+ # if we've reached end of current core info, store it and clear it
+ if line == "":
+ core_info_list.append(cur_core)
+ cur_core = {}
+ continue
+ key, value = kv_split(line, ":")
+ cur_core[key] = value
+ # parse flags - they're the same for all CPU's so only parse the first one
+ flags = set(core_info_list[0]["flags"].split())
+ for flag in flags:
+ if flag in __CPU_FLAGS_TO_HP_SIZE:
+ pinfo.hugepage_sizes_supported.append(__CPU_FLAGS_TO_HP_SIZE[flag])
+ elif flag in __IOMMU_CPU_FLAGS:
+ pinfo.iommu_supported = True
+ elif flag in __HT_CPU_FLAGS:
+ pinfo.hyperthreading_supported = True
+
+ # parse cores and sockets
+ numa_nodes = set()
+ core_map = {}
+ for core_dict in core_info_list:
+ thread_id = int(core_dict["processor"])
+ core_id = int(core_dict["core id"])
+ numa_node = int(core_dict["physical id"])
+
+ core_map.setdefault((numa_node, core_id), []).append(thread_id)
+ numa_nodes.add(numa_node)
+
+ # now, populate PlatformInfo with our, well, info - convert to lists
+ pinfo.numa_nodes = list(numa_nodes)
+ pinfo.core_map = core_map
+
+
+def _parse_meminfo(pinfo):
+ meminfo_data = {}
+ with open("/proc/meminfo") as f:
+ for line in f:
+ key, value = kv_split(line, ":")
+ meminfo_data[key] = value
+
+ # regex used to capture kilobytes
+ r = re.compile("(\d+) kB")
+
+ # total ram size
+ m = r.match(meminfo_data["MemTotal"])
+ if not m:
+ raise RuntimeError("BUG: Bad regular expression")
+ pinfo.ram_size = int(m.group(1))
+
+ # hugepages may not be supported
+ if "Hugepagesize" in meminfo_data:
+ m = r.match(meminfo_data["Hugepagesize"])
+ if not m:
+ raise RuntimeError("BUG: Bad regular expression")
+ pinfo.default_hugepage_size = int(m.group(1))
+
+
+def _find_enabled_hugepage_sizes():
+ paths = glob.glob(__KERNEL_HPSZ_PATH)
+ r = re.compile("hugepages-(\d+)kB")
+ sizes = []
+ for p in paths:
+ p = os.path.basename(p)
+ m = r.search(p)
+ if not m:
+ raise RuntimeError("BUG: Bad regular expression")
+ sizes.append(int(m.group(1)))
+ return sizes
+
+
+def _iommu_is_enabled():
+ pass
+
+
+class PlatformInfo:
+ def __init__(self):
+ self.update()
+
+ def reset(self):
+ self.numa_nodes = []
+ self.hyperthreading_supported = False
+ self.core_map = {} # numa_node, core_id: [thread_id]
+ self.iommu_supported = False
+ self.iommu_mode = ""
+ self.bootloader_iommu_mode = ""
+ self.hugepage_sizes_supported = []
+ self.hugepage_sizes_enabled = []
+ self.default_hugepage_size = 0
+ self.ram_size = 0
+
+ def update(self):
+ self.reset()
+ _parse_cpuinfo(self)
+ _parse_meminfo(self)
+ self.hugepage_sizes_enabled = _find_enabled_hugepage_sizes()
diff --git a/usertools/DPDKConfigLib/Util.py b/usertools/DPDKConfigLib/Util.py
new file mode 100755
index 000000000..42434e728
--- /dev/null
+++ b/usertools/DPDKConfigLib/Util.py
@@ -0,0 +1,16 @@
+#!/usr/bin/python
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# split line into key-value pair, cleaning up the values in the process
+def kv_split(line, separator):
+ # just in case
+ line = line.strip()
+
+ tokens = line.split(separator, 1)
+ key, value = None, None
+ if len(tokens) > 0:
+ key = tokens[0].strip()
+ if len(tokens) > 1:
+ value = tokens[1].strip()
+ return key, value
--
2.17.1
next prev parent reply other threads:[~2018-11-15 15:47 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-25 15:59 [dpdk-dev] [RFC 0/9] Modularize and enhance DPDK Python scripts Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 1/9] usertools: add DPDK config lib python library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 2/9] usertools/lib: add platform info library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 3/9] usertools/cpu_layout: rewrite to use DPDKConfigLib Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 4/9] usertools/lib: support FreeBSD for platform info Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 5/9] usertools/lib: add device information library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 6/9] usertools/devbind: switch to using DPDKConfigLib Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 7/9] usertools/lib: add hugepage information library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 8/9] usertools: add hugepage info script Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 9/9] usertools/lib: add GRUB utility library for hugepage config Anatoly Burakov
2018-06-26 1:09 ` Kevin Wilson
2018-06-26 9:05 ` Burakov, Anatoly
2018-08-14 10:11 ` [dpdk-dev] [RFC 0/9] Modularize and enhance DPDK Python scripts Burakov, Anatoly
2018-08-28 8:16 ` Burakov, Anatoly
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 " Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 1/9] usertools: add DPDK config lib python library Anatoly Burakov
2018-11-16 0:45 ` Stephen Hemminger
2018-11-16 11:49 ` Burakov, Anatoly
2018-11-16 14:09 ` Wiles, Keith
2018-11-16 14:13 ` Richardson, Bruce
2018-11-16 14:37 ` Burakov, Anatoly
2018-11-16 14:55 ` Thomas Monjalon
2018-11-16 15:41 ` Wiles, Keith
2018-11-16 15:43 ` Burakov, Anatoly
2018-11-16 15:58 ` Thomas Monjalon
2018-11-16 16:10 ` Bruce Richardson
2018-11-16 16:08 ` Bruce Richardson
2018-11-16 15:38 ` Wiles, Keith
2018-11-15 15:47 ` Anatoly Burakov [this message]
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 3/9] usertools/cpu_layout: rewrite to use DPDKConfigLib Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 4/9] usertools/lib: support FreeBSD for platform info Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 5/9] usertools/lib: add device information library Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 6/9] usertools/devbind: switch to using DPDKConfigLib Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 7/9] usertools/lib: add hugepage information library Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 8/9] usertools: add hugepage info script Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 9/9] usertools/lib: add GRUB utility library for hugepage config Anatoly Burakov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3b48e558478fdad1e9e6a8e58f723734be79ae21.1542291869.git.anatoly.burakov@intel.com \
--to=anatoly.burakov@intel.com \
--cc=bruce.richardson@intel.com \
--cc=david.hunt@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=john.mcnamara@intel.com \
--cc=mohammad.abdul.awal@intel.com \
--cc=pablo.de.lara.guarch@intel.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).