[dpdk-dev] [PATCH] scripts: add script for generating customised build config

[dpdk-dev] [PATCH] scripts: add script for generating customised build config

[Bruce Richardson]

This patch adds in the dpdk_config.py script file. It can be used
to generate custom build-time configurations for DPDK either manually on
the commandline or by calling it from other scripts. It takes as parameters
the base config template to use, and output directory to which the modified
configuration will be written. Other optional parameters are then taken
as y/n values which should be adjusted in the config file, and a special
-m flag is provided to override the default RTE_MACHINE setting in the
config template too.

Example, to create a build configuration with extra non-default PMDs
enabled, and the kernel modules disabled:

  ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
     IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
     NFP_PMD=y BNX2X_PMD=y

See the included help text in the script for more details.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 scripts/dpdk_config.py | 198 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100755 scripts/dpdk_config.py

diff --git a/scripts/dpdk_config.py b/scripts/dpdk_config.py
new file mode 100755
index 0000000..01236d8
--- /dev/null
+++ b/scripts/dpdk_config.py
@@ -0,0 +1,198 @@
+#! /usr/bin/env python
+#
+#   BSD LICENSE
+#
+#   Copyright(c) 2016 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+from __future__ import print_function
+import sys
+import os.path
+import getopt
+import subprocess
+
+out = "build"
+base = None
+machine = None
+
+
+def print_usage(cmd, f):
+    print("""Usage: %s -b <base_config> [-o <output_dir>] [options...] """
+          % os.path.basename(cmd), file=f)
+
+
+def print_help(cmd):
+    print_usage(cmd, sys.stdout)
+    print("""
+Generates a new DPDK build time configuration in the given output directory,
+using as a starting point the provided base configuration. Once completed
+successfully, DPDK can then be built using that configuration by running the
+command "make -C <output_dir>".
+
+-b, --base-config=CONFIG
+    Use the configuration given by CONFIG as the starting point for the
+    build configuration. CONFIG must be a valid DPDK default configuration
+    provided by a file "defconfig_CONFIG" in the top level "config"
+    directory.
+    Example: "-b x86_64-native-linuxapp-gcc"
+
+-o, --output-dir=DIR
+    Use DIR as the resulting output directory. If unspecified the default
+    value of "build" is used.
+
+-m, --machine-type=TYPE
+    Override the machine-type value given in the default configuration by
+    setting it to TYPE. Using regular options, regular y/n values can be
+    changed, but not string options, so this explicit override for machine
+    type exists to allow changing it.
+
+-h, --help
+    Print this help text and then exit
+
+Other options passed after these on the commandline should be in the format of
+"name=y|n" and are overrides to be applied to the configuration to toggle the
+y|n settings in the file.
+
+Matches are applied on partial values, so for example "DEBUG=y" will turn on
+all options ending in "DEBUG". This means that a full option need not always be
+specified, for example, "CONFIG_RTE_LIBRTE_" prefixes can generally be omitted.
+
+Examples:
+
+To generate a configuration for DPDK
+- based off x86_64-native-linuxapp-gcc,
+- changing the machine type to "default",
+- turning on the pcap PMD,
+- disabling the KNI and igb_uio kernel modules, and
+- placing the output in a suitably named folder
+use:
+
+    %s -o x86_64-default-linuxapp-gcc -b x86_64-native-linuxapp-gcc \\
+        -m default PMD_PCAP=y IGB_UIO=n KNI_KMOD=n
+
+""" % os.path.basename(cmd))
+
+
+def parse_opts(argv):
+    global out, base, machine
+    long_opts = ["output-dir", "base-config", "machine-type", "help"]
+    try:
+        opts, args = getopt.getopt(argv[1:], "o:b:m:h", long_opts)
+    except getopt.GetoptError as err:
+        print(str(err))
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+
+    for o, a in opts:
+        if o in ["-o", "--output-dir"]:
+            out = a
+        elif o in ["-b", "--base-config"]:
+            base = a
+        elif o in ["-m", "--machine-type"]:
+            machine = a
+        elif o in ["-h", "--help"]:
+            print_help(argv[0])
+            sys.exit()
+        else:
+            print("Error: Unhandled option '%s'\n" % o,
+                  file=sys.stderr)
+            sys.exit(1)
+
+    if base is None:
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+    if not os.path.exists("config/defconfig_%s" % base):
+        print("Error, invalid base configuration specified: %s" % base)
+        sys.exit(1)
+
+    enable_args = [a[:-2] for a in args if a[-2:] == "=y"]
+    disable_args = [a[:-2] for a in args if a[-2:] == "=n"]
+    invalid = [a for a in args if a[-2:] not in ("=y", "=n")]
+    if len(invalid) > 0:
+        print("Error, options not ending in '=y' or '=n': %s"
+              % str(invalid))
+        sys.exit(1)
+    return (enable_args, disable_args)
+
+
+def run_make_config():
+    try:
+        # the flush parameter is not always implemented so provide fallback
+        print("Running 'make config'...", end=' ', flush=True)
+    except Exception:
+        print("Running 'make config'...")
+
+    cmd = "make config T=%s O=%s" % (base, out)
+    if sys.platform.startswith("freebsd"):
+        cmd = cmd.replace("make", "gmake", 1)
+    if subprocess.call(cmd.split()) != 0:
+        print("make config called failed")
+        sys.exit(1)
+
+
+def load_file(fn):
+    fd = open(fn)
+    contents = [line.strip() for line in fd.readlines()]
+    fd.close()
+    return contents
+
+
+def write_file(fn, config):
+    fd = open(fn, "w")
+    fd.writelines(["%s\n" % line for line in config])
+    fd.close()
+
+
+def change_settings(enable, disable):
+    config = load_file("%s/.config" % out)
+    print("Enabling settings %s" % str(enable))
+    for entry in enable:
+        config = [line.replace("%s=n" % entry, "%s=y" % entry)
+                  for line in config]
+    print("Disabling settings %s" % str(disable))
+    for entry in disable:
+        config = [line.replace("%s=y" % entry, "%s=n" % entry)
+                  for line in config]
+
+    if machine is not None:
+        print("Setting machine to '%s'" % machine)
+        config.append("CONFIG_RTE_MACHINE=\"%s\"" % machine)
+    write_file("%s/.config" % out, config)
+
+
+def main():
+    enable, disable = parse_opts(sys.argv)
+    run_make_config()
+    change_settings(enable, disable)
+    print("Done")
+
+
+if __name__ == "__main__":
+    main()
-- 
2.5.5

Re: [dpdk-dev] [PATCH] scripts: add script for generating customised build config

[Thomas Monjalon]

Hi Bruce,

Thanks for pushing this idea.

2016-04-19 11:27, Bruce Richardson:
> This patch adds in the dpdk_config.py script file. It can be used
> to generate custom build-time configurations for DPDK either manually on
> the commandline or by calling it from other scripts. It takes as parameters
> the base config template to use, and output directory to which the modified
> configuration will be written. Other optional parameters are then taken
> as y/n values which should be adjusted in the config file, and a special
> -m flag is provided to override the default RTE_MACHINE setting in the
> config template too.
> 
> Example, to create a build configuration with extra non-default PMDs
> enabled, and the kernel modules disabled:
> 
>   ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
>      IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
>      NFP_PMD=y BNX2X_PMD=y

Would it be possible to use it without -b option to update a configuration?

Why not name it scripts/configure.py with a symlink ./configure in the
top directory?

Should we be able to list every options for a "-b defconfig"?

Would it be a good idea to manage dependencies checks in this script?

Re: [dpdk-dev] [PATCH] scripts: add script for generating customised build config

[Bruce Richardson]

On Tue, Apr 19, 2016 at 02:30:08PM +0200, Thomas Monjalon wrote:
> Hi Bruce,
> 
> Thanks for pushing this idea.
> 
> 2016-04-19 11:27, Bruce Richardson:
> > This patch adds in the dpdk_config.py script file. It can be used
> > to generate custom build-time configurations for DPDK either manually on
> > the commandline or by calling it from other scripts. It takes as parameters
> > the base config template to use, and output directory to which the modified
> > configuration will be written. Other optional parameters are then taken
> > as y/n values which should be adjusted in the config file, and a special
> > -m flag is provided to override the default RTE_MACHINE setting in the
> > config template too.
> > 
> > Example, to create a build configuration with extra non-default PMDs
> > enabled, and the kernel modules disabled:
> > 
> >   ./scripts/dpdk_config.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
> >      IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
> >      NFP_PMD=y BNX2X_PMD=y
> 
> Would it be possible to use it without -b option to update a configuration?
> 
Interesting idea, but what would that really give us over manual editing? If
calling from a script, the user can just rebuild the config from scratch.

> Why not name it scripts/configure.py with a symlink ./configure in the
> top directory?

No objections here. It's just not really a "normal" configure script, instead
it's one designed to make it easy for me to generate directories with lots of
different build settings in them.
If we do want to make it to be "the" way to build DPDK, we can perhaps look at
additional enhancements to speed it up by working with the defconfigs directly,
as I'm not happy right now with the speed of the "make config" step.

On the other hand, I'm very happy with having it short and of limited scope.
Actual code only takes up about 50% of the file. :-)

> 
> Should we be able to list every options for a "-b defconfig"?
> 
Good idea, I think.
Under what conditions? Only if an invalid config is provided, or as part of
regular help text or otherwise?

> Would it be a good idea to manage dependencies checks in this script?
> 
I'd rather not do so here.
a) I don't think there are a huge number of dependencies to manage
b) I'd like to keep it from getting too long and complicated.

For 95% of use cases, being able to set a few y/n values on and off, and change
the machine type should suffice. I wasn't going for a 100% solution.

/Bruce

[dpdk-dev] [PATCH v2] scripts: add script for generating customised build config

[Bruce Richardson]

This patch adds in the configure.py script file. It can be used to
generate custom build-time configurations for DPDK either manually on
the commandline or by calling it from other scripts. It takes as parameters
the base config template to use, and output directory to which the modified
configuration will be written. Other optional parameters are then taken
as y/n values which should be adjusted in the config file, and a special
-m flag is provided to override the default RTE_MACHINE setting in the
config template too.

Example, to create a build configuration with extra non-default PMDs
enabled, and the kernel modules disabled:

  ./scripts/configure.py -b $RTE_TARGET -o $RTE_TARGET PMD_PCAP=y \
     IGB_UIO=n KNI_KMOD=n MLX4_PMD=y MLX5_PMD=y SZEDATA2=y \
     NFP_PMD=y BNX2X_PMD=y

See the included help text in the script for more details.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---

V2: Add listing configs in help text and on error with invalid cfg.
    Renamed from dpdk_config.py to configure.py
---
 scripts/configure.py | 205 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 205 insertions(+)
 create mode 100755 scripts/configure.py

diff --git a/scripts/configure.py b/scripts/configure.py
new file mode 100755
index 0000000..58b8734
--- /dev/null
+++ b/scripts/configure.py
@@ -0,0 +1,205 @@
+#! /usr/bin/env python
+#
+#   BSD LICENSE
+#
+#   Copyright(c) 2016 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+from __future__ import print_function
+import sys
+import os.path
+import getopt
+import subprocess
+
+out = "build"
+base = None
+machine = None
+
+
+def print_usage(cmd, f):
+    print("""Usage: %s -b <base_config> [-o <output_dir>] [options...] """
+          % os.path.basename(cmd), file=f)
+
+
+def get_valid_configs():
+    c = [f[10:] for f in os.listdir("config") if f.startswith("defconfig_")]
+    c.sort()
+    return c
+
+
+def print_help(cmd):
+    print_usage(cmd, sys.stdout)
+    print("""
+Generates a new DPDK build time configuration in the given output directory,
+using as a starting point the provided base configuration. Once completed
+successfully, DPDK can then be built using that configuration by running the
+command "make -C <output_dir>".
+
+-b, --base-config=CONFIG
+    Use the configuration given by CONFIG as the starting point for the
+    build configuration. CONFIG must be a valid DPDK default configuration
+    provided by a file "defconfig_CONFIG" in the top level "config"
+    directory.
+    Valid --base-config values:\n\t\t%s
+
+-o, --output-dir=DIR
+    Use DIR as the resulting output directory. If unspecified the default
+    value of "build" is used.
+
+-m, --machine-type=TYPE
+    Override the machine-type value given in the default configuration by
+    setting it to TYPE. Using regular options, regular y/n values can be
+    changed, but not string options, so this explicit override for machine
+    type exists to allow changing it.
+
+-h, --help
+    Print this help text and then exit
+
+Other options passed after these on the commandline should be in the format of
+"name=y|n" and are overrides to be applied to the configuration to toggle the
+y|n settings in the file.
+
+Matches are applied on partial values, so for example "DEBUG=y" will turn on
+all options ending in "DEBUG". This means that a full option need not always be
+specified, for example, "CONFIG_RTE_LIBRTE_" prefixes can generally be omitted.
+
+Examples:
+
+To generate a configuration for DPDK
+- based off x86_64-native-linuxapp-gcc,
+- changing the machine type to "default",
+- turning on the pcap PMD,
+- disabling the KNI and igb_uio kernel modules, and
+- placing the output in a suitably named folder
+use:
+
+    %s -o x86_64-default-linuxapp-gcc -b x86_64-native-linuxapp-gcc \\
+        -m default PMD_PCAP=y IGB_UIO=n KNI_KMOD=n
+
+""" % (",\n\t\t".join(get_valid_configs()), os.path.basename(cmd)))
+
+
+def parse_opts(argv):
+    global out, base, machine
+    long_opts = ["output-dir", "base-config", "machine-type", "help"]
+    try:
+        opts, args = getopt.getopt(argv[1:], "o:b:m:h", long_opts)
+    except getopt.GetoptError as err:
+        print(str(err))
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+
+    for o, a in opts:
+        if o in ["-o", "--output-dir"]:
+            out = a
+        elif o in ["-b", "--base-config"]:
+            base = a
+        elif o in ["-m", "--machine-type"]:
+            machine = a
+        elif o in ["-h", "--help"]:
+            print_help(argv[0])
+            sys.exit()
+        else:
+            print("Error: Unhandled option '%s'\n" % o,
+                  file=sys.stderr)
+            sys.exit(1)
+
+    if base is None:
+        print_usage(argv[0], sys.stderr)
+        sys.exit(1)
+    if not os.path.exists("config/defconfig_%s" % base):
+        print("Error, invalid base configuration specified: %s" % base)
+        print("Valid cfgs: %s" % ",\n\t".join(get_valid_configs()))
+        sys.exit(1)
+
+    enable_args = [a[:-2] for a in args if a[-2:] == "=y"]
+    disable_args = [a[:-2] for a in args if a[-2:] == "=n"]
+    invalid = [a for a in args if a[-2:] not in ("=y", "=n")]
+    if len(invalid) > 0:
+        print("Error, options not ending in '=y' or '=n': %s"
+              % str(invalid))
+        sys.exit(1)
+    return (enable_args, disable_args)
+
+
+def run_make_config():
+    try:
+        # the flush parameter is not always implemented so provide fallback
+        print("Running 'make config'...", end=' ', flush=True)
+    except Exception:
+        print("Running 'make config'...")
+
+    cmd = "make config T=%s O=%s" % (base, out)
+    if sys.platform.startswith("freebsd"):
+        cmd = cmd.replace("make", "gmake", 1)
+    if subprocess.call(cmd.split()) != 0:
+        print("make config called failed")
+        sys.exit(1)
+
+
+def load_file(fn):
+    fd = open(fn)
+    contents = [line.strip() for line in fd.readlines()]
+    fd.close()
+    return contents
+
+
+def write_file(fn, config):
+    fd = open(fn, "w")
+    fd.writelines(["%s\n" % line for line in config])
+    fd.close()
+
+
+def change_settings(enable, disable):
+    config = load_file("%s/.config" % out)
+    print("Enabling settings %s" % str(enable))
+    for entry in enable:
+        config = [line.replace("%s=n" % entry, "%s=y" % entry)
+                  for line in config]
+    print("Disabling settings %s" % str(disable))
+    for entry in disable:
+        config = [line.replace("%s=y" % entry, "%s=n" % entry)
+                  for line in config]
+
+    if machine is not None:
+        print("Setting machine to '%s'" % machine)
+        config.append("CONFIG_RTE_MACHINE=\"%s\"" % machine)
+    write_file("%s/.config" % out, config)
+
+
+def main():
+    enable, disable = parse_opts(sys.argv)
+    run_make_config()
+    change_settings(enable, disable)
+    print("Done")
+
+
+if __name__ == "__main__":
+    main()
-- 
2.5.5

Re: [dpdk-dev] [PATCH v2] scripts: add script for generating customised build config

[Thomas Monjalon]

Hi Bruce

2016-04-29 16:48, Bruce Richardson:
> This patch adds in the configure.py script file. It can be used to
> generate custom build-time configurations for DPDK either manually on
> the commandline or by calling it from other scripts. It takes as parameters
> the base config template to use, and output directory to which the modified
> configuration will be written. Other optional parameters are then taken
> as y/n values which should be adjusted in the config file, and a special
> -m flag is provided to override the default RTE_MACHINE setting in the
> config template too.

You were planning to use this script but not make it the official way of
building DPDK. I think it should be the official way.
I would suggest to integrate it quietly and improve it to make it more
official later.
What do you think is missing in the current design to take it as a base
for later work?
I really would like to see the dependencies (internal and external)
managed in this script.

[...]
> +-m, --machine-type=TYPE
> +    Override the machine-type value given in the default configuration by
> +    setting it to TYPE. Using regular options, regular y/n values can be
> +    changed, but not string options, so this explicit override for machine
> +    type exists to allow changing it.

I think we should make possible to change string values as boolean ones in
a generic way.

Detail: the MAINTAINERS file must be updated for this new file.

Re: [dpdk-dev] [PATCH v2] scripts: add script for generating customised build config

[Bruce Richardson]

On Tue, Jun 07, 2016 at 09:36:37AM +0200, Thomas Monjalon wrote:
> Hi Bruce
> 
> 2016-04-29 16:48, Bruce Richardson:
> > This patch adds in the configure.py script file. It can be used to
> > generate custom build-time configurations for DPDK either manually on
> > the commandline or by calling it from other scripts. It takes as parameters
> > the base config template to use, and output directory to which the modified
> > configuration will be written. Other optional parameters are then taken
> > as y/n values which should be adjusted in the config file, and a special
> > -m flag is provided to override the default RTE_MACHINE setting in the
> > config template too.
> 
> You were planning to use this script but not make it the official way of
> building DPDK. I think it should be the official way.
> I would suggest to integrate it quietly and improve it to make it more
> official later.
> What do you think is missing in the current design to take it as a base
> for later work?
> I really would like to see the dependencies (internal and external)
> managed in this script.
> 

I agree, we can look to add that in future.

> [...]
> > +-m, --machine-type=TYPE
> > +    Override the machine-type value given in the default configuration by
> > +    setting it to TYPE. Using regular options, regular y/n values can be
> > +    changed, but not string options, so this explicit override for machine
> > +    type exists to allow changing it.
> 
> I think we should make possible to change string values as boolean ones in
> a generic way.
> 

Yes. I just didn't implement this before since there are so few string values 
in our config, and I didn't think it was that useful apart from machine type. If
in future, we will use this script as the official configuration method, then
it does need to be added.

> Detail: the MAINTAINERS file must be updated for this new file.

Yep.

/Bruce