From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0530EA04B5; Wed, 9 Sep 2020 20:51:12 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 930C01BE0C; Wed, 9 Sep 2020 20:51:12 +0200 (CEST) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by dpdk.org (Postfix) with ESMTP id C03611DB8 for ; Wed, 9 Sep 2020 20:51:10 +0200 (CEST) Received: by mail-pf1-f176.google.com with SMTP id d6so2947979pfn.9 for ; Wed, 09 Sep 2020 11:51:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JNZhlqA0HL6xtg7cmc56XBH9ktObKwREVwYmKJpkUuQ=; b=Akh9ul08rIab90sFQ70PyFCBjDk27L1U5Q0jjoSV/zhNlojhXEE8YJZTSpb3+9Tc2Q opsIER0s/x0th31PchQdK/3nIspUqfBRhp/w57RUyF3ho+cZVJLFicEaMnFtfGue1mdL jX/G8e33He5r7CasSvLJ98LoSSz5fAT3RVs6xEWZDy9ekE85E3QuLhmcVR30DZltx6WP V3JuFGr44TlwzVvy8zz3UiLGvNRe92I/JDNKh26HaMtSA4tWIjRoBng8T+elNawQjKXj B0BPxK59v7aTPWtAzXihxCyQdxohBuF5DtU8dBFO8PlbSvBnjT0a4Mr3DDUj9XH9AcRi JLkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JNZhlqA0HL6xtg7cmc56XBH9ktObKwREVwYmKJpkUuQ=; b=LUlKfM/J3B2HDUhBfgRUqBtK1wA1JPU4AJvDCo1TXdDQO+WNrWLo6c9+snTm2HkWqr TxMMEgEwDrFLbuIVvNWjQt/PHjNZJnf7+9ND2wCEwyDljtzQhAD/EgMv5zxDX+rDWeqy NwyuWcN5IjYFB9ykJuXBhvSipLJVpTxp0HMwyXImUVHMxphI2QOzyPvX9qhwFWVKp1g1 IW+lAqM9pcEjEH47meaOqj+6U+XxiteFq0w6LAsIXcifhUNrt4NEKD4LDLRomxGjFDz5 C6MvT8OxXJSA/xiiHJMrIdqz74lC3jlR13Rl3rzp2j8k1nhLiS2NoBF4/xCW8A3xbSEB jDAQ== X-Gm-Message-State: AOAM531COa7vWP0s6yORqi9VBv4QyGtymfBwB/lMFPIqL+l1EqZ4vC7g CzjvfUnYAeL3Q3JaCIoZ5Ydp9pz8aY7B+w== X-Google-Smtp-Source: ABdhPJzazOMFqr5FqWt3ejZI5sld6+1pKhOGVcl1tHEXzYOz2pxRtyjJlkdNIh6n+Vo/k0O5esWoBQ== X-Received: by 2002:a63:242:: with SMTP id 63mr1581071pgc.182.1599677469338; Wed, 09 Sep 2020 11:51:09 -0700 (PDT) Received: from hermes.corp.microsoft.com (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id x29sm2837408pga.23.2020.09.09.11.51.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2020 11:51:08 -0700 (PDT) From: Stephen Hemminger To: dev@dpdk.org Cc: Stephen Hemminger Date: Wed, 9 Sep 2020 11:51:01 -0700 Message-Id: <20200909185101.26977-1-stephen@networkplumber.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200901165643.15668-1-stephen@networkplumber.org> References: <20200901165643.15668-1-stephen@networkplumber.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v7] usertools: add a huge page setup script X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This is an improved version of the setup of huge pages bases on earlier DPDK setup. Differences are: * autodetects NUMA vs non NUMA * allows setting different page sizes recent kernels support multiple sizes. * accepts a parameter in bytes (not pages). * can display current hugepage settings. Most users will just use --setup argument but if necessary the steps of clearing old settings and mounting/umounting can be done individually. Signed-off-by: Stephen Hemminger --- v7 - fix issues with show and add Total column cleanup whitespace in hugepages.rst doc/guides/tools/hugepages.rst | 79 ++++++++++ doc/guides/tools/index.rst | 1 + usertools/dpdk-hugepages.py | 270 +++++++++++++++++++++++++++++++++ usertools/meson.build | 7 +- 4 files changed, 356 insertions(+), 1 deletion(-) create mode 100644 doc/guides/tools/hugepages.rst create mode 100755 usertools/dpdk-hugepages.py diff --git a/doc/guides/tools/hugepages.rst b/doc/guides/tools/hugepages.rst new file mode 100644 index 000000000000..40e5387a682c --- /dev/null +++ b/doc/guides/tools/hugepages.rst @@ -0,0 +1,79 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2020 Microsoft Corporation + +dpdk-hugpages Application +========================== + +The ``dpdk-hugpages`` tool is a Data Plane Development Kit (DPDK) utility +that helps in reserving hugepages. +As well as checking for current settings. + + +Running the Application +----------------------- + +The tool has a number of command line options: + +.. code-block:: console + + + dpdk-hugpages [options] + + +OPTIONS +------- + +* ``-h, --help`` + + Display usage information and quit + +* ``-s, --show`` + + Print the current huge page configuration + +* ``-c driver, --clear`` + + Clear existing huge page reservation + +* ``-m, --mount`` + + Mount the huge page filesystem + +* ``-u, --unmount`` + + Unmount the huge page filesystem + +* ``-n NODE, --node=NODE`` + + Set NUMA node to reserve pages on + +* ``-p SIZE, --pagesize=SIZE`` + + Select hugepage size to use. + If not specified the default system huge page size is used. + +* ``-r SIZE, --reserve=SIZE`` + + Reserve huge pages. + Size is in bytes with K, M or G suffix. + +* ``--setup SIZE`` + + Short cut to clear, unmount, reserve and mount. + +.. warning:: + + While any user can run the ``dpdk-hugpages.py`` script to view the + status of huge pages, modifying the setup requires root privileges. + + +Examples +-------- + +To display current huge page settings:: + + dpdk-hugpages.py -s + +To a complete setup of with 2 Gigabyte of 1G huge pages:: + + dpdk-hugpages.py -p 1G --setup 2G diff --git a/doc/guides/tools/index.rst b/doc/guides/tools/index.rst index c721943606f9..93dde4148e90 100644 --- a/doc/guides/tools/index.rst +++ b/doc/guides/tools/index.rst @@ -11,6 +11,7 @@ DPDK Tools User Guides proc_info pdump pmdinfo + hugepages devbind flow-perf testbbdev diff --git a/usertools/dpdk-hugepages.py b/usertools/dpdk-hugepages.py new file mode 100755 index 000000000000..a78e9b94567a --- /dev/null +++ b/usertools/dpdk-hugepages.py @@ -0,0 +1,270 @@ +#! /usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2020 Microsoft Corporation +"""Script to query and setup huge pages for DPDK applications.""" + +import argparse +import glob +import os +import re +import sys +from math import log2 + +# Standard binary prefix +BINARY_PREFIX = "KMG" + +# systemd mount point for huge pages +HUGE_MOUNT = "/dev/hugepages" + + +def fmt_memsize(kb): + '''Format memory size in kB into conventional format''' + logk = int(log2(kb) / 10) + suffix = BINARY_PREFIX[logk] + unit = 2**(logk * 10) + return '{}{}b'.format(int(kb / unit), suffix) + + +def get_memsize(arg): + '''Convert memory size with suffix to kB''' + match = re.match(r'(\d+)([' + BINARY_PREFIX + r']?)$', arg.upper()) + if match is None: + sys.exit('{} is not a valid page size'.format(arg)) + num = float(match.group(1)) + suffix = match.group(2) + if suffix == "": + return int(num / 1024) + idx = BINARY_PREFIX.find(suffix) + return int(num * (2**(idx * 10))) + + +def is_numa(): + '''Test if NUMA is necessary on this system''' + return os.path.exists('/sys/devices/system/node') + + +def get_hugepages(path): + '''Read number of reserved pages''' + with open(path + '/nr_hugepages') as nr_hugpages: + return int(nr_hugpages.read()) + return 0 + + +def set_hugepages(path, pages): + '''Write the number of reserved huge pages''' + filename = path + '/nr_hugepages' + try: + with open(filename, 'w') as nr_hugpages: + nr_hugpages.write('{}\n'.format(pages)) + except PermissionError: + sys.exit('Permission denied: need to be root!') + except FileNotFoundError: + filename = os.path.basename(path) + size = filename[10:] + sys.exit('{} is not a valid system huge page size'.format(size)) + + +def show_numa_pages(): + '''Show huge page reservations on Numa system''' + print('Node Pages Size Total') + for numa_path in glob.glob('/sys/devices/system/node/node*'): + node = numa_path[29:] # slice after /sys/devices/system/node/node + path = numa_path + '/hugepages' + for hdir in os.listdir(path): + pages = get_hugepages(path + '/' + hdir) + if pages > 0: + kb = int(hdir[10:-2]) # slice out of hugepages-NNNkB + print('{:<4} {:<5} {:<6} {}'.format(node, pages, + fmt_memsize(kb), + fmt_memsize(pages * kb))) + + +def show_non_numa_pages(): + '''Show huge page reservations on non Numa system''' + print('Pages Size Total') + path = '/sys/kernel/mm/hugepages' + for hdir in os.listdir(path): + pages = get_hugepages(path + '/' + hdir) + if pages > 0: + kb = int(hdir[10:-2]) + print('{:<5} {:<6} {}'.format(pages, fmt_memsize(kb), + fmt_memsize(pages * kb))) + + +def show_pages(): + '''Show existing huge page settings''' + if is_numa(): + show_numa_pages() + else: + show_non_numa_pages() + + +def clear_pages(): + '''Clear all existing huge page mappings''' + if is_numa(): + dirs = glob.glob( + '/sys/devices/system/node/node*/hugepages/hugepages-*') + else: + dirs = glob.glob('/sys/kernel/mm/hugepages/hugepages-*') + + for path in dirs: + set_hugepages(path, 0) + + +def default_pagesize(): + '''Get default huge page size from /proc/meminfo''' + with open('/proc/meminfo') as meminfo: + for line in meminfo: + if line.startswith('Hugepagesize:'): + return int(line.split()[1]) + return None + + +def set_numa_pages(pages, hugepgsz, node=None): + '''Set huge page reservation on Numa system''' + if node: + nodes = ['/sys/devices/system/node/node{}/hugepages'.format(node)] + else: + nodes = glob.glob('/sys/devices/system/node/node*/hugepages') + + for node_path in nodes: + huge_path = '{}/hugepages-{}kB'.format(node_path, hugepgsz) + set_hugepages(huge_path, pages) + + +def set_non_numa_pages(pages, hugepgsz): + '''Set huge page reservation on non Numa system''' + path = '/sys/kernel/mm/hugepages/hugepages-{}kB'.format(hugepgsz) + set_hugepages(path, pages) + + +def reserve_pages(pages, hugepgsz, node=None): + '''Sets the number of huge pages to be reserved''' + if node or is_numa(): + set_numa_pages(pages, hugepgsz, node=node) + else: + set_non_numa_pages(pages, hugepgsz) + + +def get_mountpoints(): + '''get list of of where hugepage filesystem is mounted''' + mounted = [] + with open('/proc/mounts') as mounts: + for line in mounts: + fields = line.split() + if fields[2] != 'hugetlbfs': + continue + mounted.append(fields[1]) + return mounted + + +def mount_huge(pagesize, mountpoint): + '''mount the huge tlb file system''' + if mountpoint in get_mountpoints(): + print(mountpoint, "already mounted") + return + cmd = "mount -t hugetlbfs" + if pagesize: + cmd += ' -o pagesize={}'.format(pagesize * 1024) + cmd += ' nodev ' + mountpoint + os.system(cmd) + + +def umount_huge(mountpoint): + '''unmount the huge tlb file system (if mounted)''' + if mountpoint in get_mountpoints(): + os.system("umount " + mountpoint) + + +def show_mount(): + '''Show where huge page filesystem is mounted''' + mounted = get_mountpoints() + if mounted: + print("Hugepages mounted on", *mounted) + else: + print("Hugepages not mounted") + + +def main(): + '''Process the command line arguments and setup huge pages''' + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description="Setup huge pages", + epilog=""" +Examples: + +To display current huge page settings: + %(prog)s -s + +To a complete setup of with 2 Gigabyte of 1G huge pages: + %(prog)s -p 1G --setup 2G +""") + parser.add_argument( + '--show', + '-s', + action='store_true', + help="print the current huge page configuration") + parser.add_argument( + '--clear', '-c', action='store_true', help="clear existing huge pages") + parser.add_argument( + '--mount', + '-m', + action='store_true', + help='mount the huge page filesystem') + parser.add_argument( + '--unmount', + '-u', + action='store_true', + help='unmount the system huge page directory') + parser.add_argument( + '--node', '-n', help='select numa node to reserve pages on') + parser.add_argument( + '--pagesize', + '-p', + metavar='SIZE', + help='choose huge page size to use') + parser.add_argument( + '--reserve', + '-r', + metavar='SIZE', + help='reserve huge pages. Size is in bytes with K, M, or G suffix') + parser.add_argument( + '--setup', + metavar='SIZE', + help='setup huge pages by doing clear, unmount, reserve and mount') + args = parser.parse_args() + + if args.setup: + args.clear = True + args.unmount = True + args.reserve = args.setup + args.mount = True + + if args.pagesize: + pagesize_kb = get_memsize(args.pagesize) + else: + pagesize_kb = default_pagesize() + + if args.clear: + clear_pages() + if args.unmount: + umount_huge(HUGE_MOUNT) + + if args.reserve: + reserve_kb = get_memsize(args.reserve) + if reserve_kb % pagesize_kb != 0: + sys.exit( + 'Huge reservation {}kB is not a multiple of page size {}kB'. + format(reserve_kb, pagesize_kb)) + reserve_pages( + int(reserve_kb / pagesize_kb), pagesize_kb, node=args.node) + if args.mount: + mount_huge(pagesize_kb, HUGE_MOUNT) + if args.show: + show_pages() + print() + show_mount() + + +if __name__ == "__main__": + main() diff --git a/usertools/meson.build b/usertools/meson.build index 64e27238f45b..596eaefb0e23 100644 --- a/usertools/meson.build +++ b/usertools/meson.build @@ -1,4 +1,9 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -install_data(['dpdk-devbind.py', 'dpdk-pmdinfo.py', 'dpdk-telemetry.py'], install_dir: 'bin') +install_data([ + 'dpdk-devbind.py', + 'dpdk-pmdinfo.py', + 'dpdk-telemetry.py', + 'dpdk-hugepages.py' +],install_dir: 'bin') -- 2.27.0