DPDK patches and discussions
 help / color / mirror / Atom feed
From: Anatoly Burakov <anatoly.burakov@intel.com>
To: dev@dpdk.org
Cc: john.mcnamara@intel.com, bruce.richardson@intel.com,
	pablo.de.lara.guarch@intel.com, david.hunt@intel.com,
	mohammad.abdul.awal@intel.com
Subject: [dpdk-dev] [RFC 2/9] usertools/lib: add platform info library
Date: Mon, 25 Jun 2018 16:59:39 +0100
Message-ID: <0ed69167149eca2965ad2e6fb222e74e0ea90a2a.1529940601.git.anatoly.burakov@intel.com> (raw)
In-Reply-To: <cover.1529940601.git.anatoly.burakov@intel.com>
In-Reply-To: <cover.1529940601.git.anatoly.burakov@intel.com>

Add a library that will parse system information:

* NUMA nodes
* Cores and threads
* Mapping from NUMA node and core to thread id's
* Hyperthreading support status
* RAM size
* Default hugepage size as reported by kernel

This can be used by scripts.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 usertools/DPDKConfigLib/PlatformInfo.py | 130 ++++++++++++++++++++++++
 usertools/DPDKConfigLib/Util.py         |  16 +++
 2 files changed, 146 insertions(+)
 create mode 100755 usertools/DPDKConfigLib/PlatformInfo.py
 create mode 100755 usertools/DPDKConfigLib/Util.py

diff --git a/usertools/DPDKConfigLib/PlatformInfo.py b/usertools/DPDKConfigLib/PlatformInfo.py
new file mode 100755
index 000000000..734d22026
--- /dev/null
+++ b/usertools/DPDKConfigLib/PlatformInfo.py
@@ -0,0 +1,130 @@
+#!/usr/bin/python
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+
+from .Util import *
+import re
+import glob
+
+__SYSFS_CPU_INFO_PATH = "/sys/devices/system/cpu"
+__SYSFS_IOMMU_CLASS_PATH = "/sys/class/iommu"
+__KERNEL_HPSZ_PATH = "/sys/kernel/mm/hugepages/hugepages-*"
+
+try:
+    xrange  # python 2
+except NameError:
+    xrange = range  # python 3
+
+
+__CPU_FLAGS_TO_HP_SIZE = {
+    # x86 supports 2M and 1G pages
+    "pse": 2048,
+    "pdpe1gb": 1048576
+}
+__IOMMU_CPU_FLAGS = ["vmx", "vms"]
+
+__HT_CPU_FLAGS = ["ht"]
+
+
+def _parse_cpuinfo(pinfo):
+    core_info_list = []
+    with open("/proc/cpuinfo") as f:
+        cur_core = {}
+        for line in f:
+            line = line.strip()
+            # if we've reached end of current core info, store it and clear it
+            if line == "":
+                core_info_list.append(cur_core)
+                cur_core = {}
+                continue
+            key, value = kv_split(line, ":")
+            cur_core[key] = value
+    # parse flags - they're the same for all CPU's so only parse the first one
+    flags = set(core_info_list[0]["flags"].split())
+    for flag in flags:
+        if flag in __CPU_FLAGS_TO_HP_SIZE:
+            pinfo.hugepage_sizes_supported.append(__CPU_FLAGS_TO_HP_SIZE[flag])
+        elif flag in __IOMMU_CPU_FLAGS:
+            pinfo.iommu_supported = True
+        elif flag in __HT_CPU_FLAGS:
+            pinfo.hyperthreading_supported = True
+
+    # parse cores and sockets
+    numa_nodes = set()
+    core_map = {}
+    for core_dict in core_info_list:
+        thread_id = int(core_dict["processor"])
+        core_id = int(core_dict["core id"])
+        numa_node = int(core_dict["physical id"])
+
+        core_map.setdefault((numa_node, core_id), []).append(thread_id)
+        numa_nodes.add(numa_node)
+
+    # now, populate PlatformInfo with our, well, info - convert to lists
+    pinfo.numa_nodes = list(numa_nodes)
+    pinfo.core_map = core_map
+
+
+def _parse_meminfo(pinfo):
+    meminfo_data = {}
+    with open("/proc/meminfo") as f:
+        for line in f:
+            key, value = kv_split(line, ":")
+            meminfo_data[key] = value
+
+    # regex used to capture kilobytes
+    r = re.compile("(\d+) kB")
+
+    # total ram size
+    m = r.match(meminfo_data["MemTotal"])
+    if not m:
+        raise RuntimeError("BUG: Bad regular expression")
+    pinfo.ram_size = int(m.group(1))
+
+    # hugepages may not be supported
+    if "Hugepagesize" in meminfo_data:
+        m = r.match(meminfo_data["Hugepagesize"])
+        if not m:
+            raise RuntimeError("BUG: Bad regular expression")
+        pinfo.default_hugepage_size = int(m.group(1))
+
+
+def _find_enabled_hugepage_sizes():
+    paths = glob.glob(__KERNEL_HPSZ_PATH)
+    r = re.compile("hugepages-(\d+)kB")
+    sizes = []
+    for p in paths:
+        p = os.path.basename(p)
+        m = r.search(p)
+        if not m:
+            raise RuntimeError("BUG: Bad regular expression")
+        sizes.append(int(m.group(1)))
+    return sizes
+
+
+def _iommu_is_enabled():
+    pass
+
+
+class PlatformInfo:
+    def __init__(self):
+        self.update()
+
+    def reset(self):
+        self.numa_nodes = []
+        self.hyperthreading_supported = False
+        self.core_map = {}  # numa_node, core_id: [thread_id]
+        self.iommu_supported = False
+        self.iommu_mode = ""
+        self.bootloader_iommu_mode = ""
+        self.hugepage_sizes_supported = []
+        self.hugepage_sizes_enabled = []
+        self.default_hugepage_size = 0
+        self.ram_size = 0
+
+    def update(self):
+        self.reset()
+        _parse_cpuinfo(self)
+        _parse_meminfo(self)
+        self.hugepage_sizes_enabled = _find_enabled_hugepage_sizes()
diff --git a/usertools/DPDKConfigLib/Util.py b/usertools/DPDKConfigLib/Util.py
new file mode 100755
index 000000000..42434e728
--- /dev/null
+++ b/usertools/DPDKConfigLib/Util.py
@@ -0,0 +1,16 @@
+#!/usr/bin/python
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# split line into key-value pair, cleaning up the values in the process
+def kv_split(line, separator):
+    # just in case
+    line = line.strip()
+
+    tokens = line.split(separator, 1)
+    key, value = None, None
+    if len(tokens) > 0:
+        key = tokens[0].strip()
+    if len(tokens) > 1:
+        value = tokens[1].strip()
+    return key, value
-- 
2.17.1

  parent reply	other threads:[~2018-06-25 15:59 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-06-25 15:59 [dpdk-dev] [RFC 0/9] Modularize and enhance DPDK Python scripts Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 1/9] usertools: add DPDK config lib python library Anatoly Burakov
2018-06-25 15:59 ` Anatoly Burakov [this message]
2018-06-25 15:59 ` [dpdk-dev] [RFC 3/9] usertools/cpu_layout: rewrite to use DPDKConfigLib Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 4/9] usertools/lib: support FreeBSD for platform info Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 5/9] usertools/lib: add device information library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 6/9] usertools/devbind: switch to using DPDKConfigLib Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 7/9] usertools/lib: add hugepage information library Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 8/9] usertools: add hugepage info script Anatoly Burakov
2018-06-25 15:59 ` [dpdk-dev] [RFC 9/9] usertools/lib: add GRUB utility library for hugepage config Anatoly Burakov
2018-06-26  1:09   ` Kevin Wilson
2018-06-26  9:05     ` Burakov, Anatoly
2018-08-14 10:11 ` [dpdk-dev] [RFC 0/9] Modularize and enhance DPDK Python scripts Burakov, Anatoly
2018-08-28  8:16   ` Burakov, Anatoly
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 " Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 1/9] usertools: add DPDK config lib python library Anatoly Burakov
2018-11-16  0:45   ` Stephen Hemminger
2018-11-16 11:49     ` Burakov, Anatoly
2018-11-16 14:09       ` Wiles, Keith
2018-11-16 14:13         ` Richardson, Bruce
2018-11-16 14:37           ` Burakov, Anatoly
2018-11-16 14:55             ` Thomas Monjalon
2018-11-16 15:41               ` Wiles, Keith
2018-11-16 15:43               ` Burakov, Anatoly
2018-11-16 15:58                 ` Thomas Monjalon
2018-11-16 16:10                   ` Bruce Richardson
2018-11-16 16:08                 ` Bruce Richardson
2018-11-16 15:38             ` Wiles, Keith
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 2/9] usertools/lib: add platform info library Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 3/9] usertools/cpu_layout: rewrite to use DPDKConfigLib Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 4/9] usertools/lib: support FreeBSD for platform info Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 5/9] usertools/lib: add device information library Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 6/9] usertools/devbind: switch to using DPDKConfigLib Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 7/9] usertools/lib: add hugepage information library Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 8/9] usertools: add hugepage info script Anatoly Burakov
2018-11-15 15:47 ` [dpdk-dev] [RFC v2 9/9] usertools/lib: add GRUB utility library for hugepage config Anatoly Burakov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0ed69167149eca2965ad2e6fb222e74e0ea90a2a.1529940601.git.anatoly.burakov@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=john.mcnamara@intel.com \
    --cc=mohammad.abdul.awal@intel.com \
    --cc=pablo.de.lara.guarch@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git