Re: [PATCH] build: output a dependency log in build directory

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 7/30/2024 3:55 PM, Bruce Richardson wrote:
> As meson processes our DPDK source tree it manages dependencies
> specified by each individual driver. To enable future analysis of the
> dependency links between components, log the dependencies of each DPDK
> component as it gets processed. This could potentially allow other tools
> to automatically enable or disable components based on the desired end
> components to be built, e.g. if the user requests net/ice, ensure that
> common/iavf is also enabled in the drivers.
> 
> The output file produced is in "dot" or "graphviz" format, which allows
> producing a graphical representation of the DPDK dependency tree if so
> desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> image file "build/deps.dot.png".
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>

I tested it quickly, good to have this dependency list, at least it
makes some duplicated dependencies obvious from .dot file.

But generated dependency graph is too large to be useful, does it make
sense to have a new meson option that control this dependency generation:
-Dgenerate_deps=apps
-Dgenerate_deps=libs
-Dgenerate_deps=drivers
-Dgenerate_deps=all

Not sure about what should be the default option,
as this is not always required/used, disabling this option by default
saves unnecessary work,
but disabling it by default may cause not noticing when it is broken,
perhaps this can be addressed by adding this option to
'devtools/test-meson-builds.sh'.

btw, it generates following warning:

WARNING: You should add the boolean check kwarg to the run_command call.


         It currently defaults to false,


         but it will default to true in future releases of meson.


         See also: https://github.com/mesonbuild/meson/issues/9300


> ---
>  app/meson.build        |  1 +
>  buildtools/log-deps.py | 43 ++++++++++++++++++++++++++++++++++++++++++
>  buildtools/meson.build |  2 ++
>  drivers/meson.build    |  1 +
>  lib/meson.build        |  1 +
>  5 files changed, 48 insertions(+)
>  create mode 100644 buildtools/log-deps.py
> 
> diff --git a/app/meson.build b/app/meson.build
> index 5b2c80c7a1..6afa457f4c 100644
> --- a/app/meson.build
> +++ b/app/meson.build
> @@ -76,6 +76,7 @@ foreach app:apps
>  
>      if build
>          subdir(name)
> +        run_command([log_deps_cmd, name, deps])
>          if not build and require_apps
>              error('Cannot build explicitly requested app "@0@".\n'.format(name)
>                    + '\tReason: ' + reason)
> diff --git a/buildtools/log-deps.py b/buildtools/log-deps.py
> new file mode 100644
> index 0000000000..a4331fa15b
> --- /dev/null
> +++ b/buildtools/log-deps.py
> @@ -0,0 +1,43 @@
> +#! /usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Intel Corporation
> +
> +"""Utility script to build up a list of dependencies from meson."""
> +
> +import os
> +import sys
> +
> +
> +def file_to_list(filename):
> +    """Read file into a list of strings."""
> +    with open(filename) as f:
> +        return f.readlines()
> +
> +
> +def list_to_file(filename, lines):
> +    """Write a list of strings out to a file."""
> +    with open(filename, 'w') as f:
> +        f.writelines(lines)
> +
> +
> +depsfile = f'{os.environ["MESON_BUILD_ROOT"]}/deps.dot'
> +
> +# to reset the deps file on each build, the script is called without any params
> +if len(sys.argv) == 1:
> +    os.remove(depsfile)
> +    sys.exit(0)
> +
> +try:
> +    contents = file_to_list(depsfile)
> +except FileNotFoundError:
> +    contents = ['digraph {\n', '}\n']
> +
> +component = sys.argv[1]
> +if len(sys.argv) > 2:
> +    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'
> +else:
> +    contents[-1] = f'"{component}"\n'
> +
> +contents.append('}\n')
> +
> +list_to_file(depsfile, contents)
> diff --git a/buildtools/meson.build b/buildtools/meson.build
> index 3adf34e1a8..332f0f3d38 100644
> --- a/buildtools/meson.build
> +++ b/buildtools/meson.build
> @@ -24,6 +24,8 @@ get_numa_count_cmd = py3 + files('get-numa-count.py')
>  get_test_suites_cmd = py3 + files('get-test-suites.py')
>  has_hugepages_cmd = py3 + files('has-hugepages.py')
>  cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
> +log_deps_cmd = py3 + files('log-deps.py')
> +run_command(log_deps_cmd)  # call with no parameters to reset the file
>  
>  # install any build tools that end-users might want also
>  install_data([
> diff --git a/drivers/meson.build b/drivers/meson.build
> index 66931d4241..44935e067c 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -154,6 +154,7 @@ foreach subpath:subdirs
>          if build
>              # pull in driver directory which should update all the local variables
>              subdir(drv_path)
> +            run_command([log_deps_cmd, drv_path.underscorify(), deps])
>  
>              if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 and require_iova_in_mbuf
>                  build = false
> diff --git a/lib/meson.build b/lib/meson.build
> index 162287753f..da2815465f 100644
> --- a/lib/meson.build
> +++ b/lib/meson.build
> @@ -160,6 +160,7 @@ foreach l:libraries
>  
>      if build
>          subdir(l)
> +        run_command([log_deps_cmd, l, deps])
>          if not build and require_libs
>              error('Cannot build explicitly requested lib "@0@".\n'.format(name)
>                      +'\tReason: ' + reason)