[dpdk-dev] [RFC PATCH 0/3] add custom logging to DPDK meson runs

[dpdk-dev] [RFC PATCH 0/3] add custom logging to DPDK meson runs

[Bruce Richardson]

While the meson.log file in each build directory contains lots of
information, much of it is not of interest to developers, and it can be
hard to see the explicitly messages from the dpdk build files among all
the rest of the content. Futhermore, while at the end of the a
configuration run we print out the list of enable/disabled components,
this is never recorded anywhere for anyone who wants to check that build
configuration later.

This patchset attempts to solve these issues by supporting a
dpdk-meson.log file in each DPDK build folder containing the output from
the DPDK meson.build files - including the final summary information for
later checking.

Bruce Richardson (3):
  build: add a dpdk-specific meson log file
  build: shorten top-level build file
  build: write messages to dpdk build log file

 app/test/meson.build          |  2 +-
 buildtools/meson.build        |  1 +
 buildtools/write-log-entry.py | 23 +++++++++++++++++++++++
 config/arm/meson.build        |  2 ++
 config/meson.build            | 18 ++++++++++++++++++
 config/x86/meson.build        |  2 ++
 drivers/meson.build           |  8 +++++---
 examples/meson.build          |  1 +
 lib/meson.build               |  8 +++++---
 meson.build                   | 31 ++++++++++++-------------------
 10 files changed, 70 insertions(+), 26 deletions(-)
 create mode 100644 buildtools/write-log-entry.py

-- 
2.25.1

[dpdk-dev] [RFC PATCH 1/3] build: add a dpdk-specific meson log file

[Bruce Richardson]

While meson itself writes a log file of all the tests it runs and the
output of those, to help debug any build problems and provide more
developer information it can be helpful to have a logfile created and
controlled by the dpdk meson.build files themselves.

Since meson doesn't directly support this we need to use run_command to
implement it manually. Using a python script to do the writing of the log
makes things cross-platform, and as initial log entries we can add in the
messages about the defined internal dependencies, since these are of
developer interest only.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/meson.build        |  1 +
 buildtools/write-log-entry.py | 23 +++++++++++++++++++++++
 drivers/meson.build           |  2 +-
 lib/meson.build               |  2 +-
 meson.build                   |  8 ++++++--
 5 files changed, 32 insertions(+), 4 deletions(-)
 create mode 100644 buildtools/write-log-entry.py

diff --git a/buildtools/meson.build b/buildtools/meson.build
index 04808dabc..35a8f4290 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+log = py3 + [files('write-log-entry.py'), 'dpdk-meson.log']
diff --git a/buildtools/write-log-entry.py b/buildtools/write-log-entry.py
new file mode 100644
index 000000000..6ec748a4f
--- /dev/null
+++ b/buildtools/write-log-entry.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+from sys import argv, exit
+from time import strftime
+from os import environ
+from os.path import join as path_join
+
+if len(argv) < 2: # error if no filename
+    exit(1)
+if len(argv) < 3: # do nothing if no contents to log
+    exit(0)
+
+build_dir = environ.get('MESON_BUILD_ROOT', '.')
+filename = path_join(build_dir, argv[1])
+
+# append dates to headings
+if argv[2].startswith('===='):
+    argv.append(strftime('@@ %Y-%m-%d %X'))
+
+with open(filename, 'a') as log:
+    print(" ".join(argv[2:]), file=log)
diff --git a/drivers/meson.build b/drivers/meson.build
index a5a6fed06..c008abd4a 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -228,7 +228,7 @@ foreach subpath:subdirs
 			set_variable('shared_@0@'.format(lib_name), shared_dep)
 			set_variable('static_@0@'.format(lib_name), static_dep)
 			dependency_name = ''.join(lib_name.split('rte_'))
-			message('drivers/@0@: Defining dependency "@1@"'.format(
+			run_command(log, 'drivers/@0@: Defining dependency "@1@"'.format(
 					drv_path, dependency_name))
 		endif # build
 	endforeach
diff --git a/lib/meson.build b/lib/meson.build
index dd55b5cb5..2715c0f66 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -200,7 +200,7 @@ foreach l:libraries
 
 		set_variable('shared_rte_' + name, shared_dep)
 		set_variable('static_rte_' + name, static_dep)
-		message('lib/@0@: Defining dependency "@1@"'.format(
+		run_command(log, 'lib/@0@: Defining dependency "@1@"'.format(
 				dir_name, name))
 	endif # if build
 endforeach
diff --git a/meson.build b/meson.build
index 61d9a4f5f..ab950a2ce 100644
--- a/meson.build
+++ b/meson.build
@@ -11,6 +11,11 @@ project('DPDK', 'C',
 	meson_version: '>= 0.47.1'
 )
 
+# get tool paths and scripts
+subdir('buildtools')
+
+run_command(log, '======= Starting meson =======')
+
 # set up some global vars for compiler, platform, configuration, etc.
 cc = meson.get_compiler('c')
 dpdk_conf = configuration_data()
@@ -40,8 +45,7 @@ global_inc = include_directories('.', 'config',
 	'lib/librte_eal/@0@/include'.format(arch_subdir),
 )
 
-# do configuration and get tool paths
-subdir('buildtools')
+# do main build config
 subdir('config')
 
 # build libs and drivers
-- 
2.25.1

[dpdk-dev] [RFC PATCH 2/3] build: shorten top-level build file

[Bruce Richardson]

Move some basic configuration settings out of the top-level meson.build
file to the "config/meson.build" file, so as to keep the top-level file as
clear and as short as possible.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 config/meson.build | 18 ++++++++++++++++++
 meson.build        | 17 -----------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/config/meson.build b/config/meson.build
index 258b01d06..10643fdeb 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -14,6 +14,24 @@ foreach env:supported_exec_envs
 	set_variable('is_' + env, exec_env == env)
 endforeach
 
+# set the basic ISA
+if host_machine.cpu_family().startswith('x86')
+	arch_subdir = 'x86'
+elif host_machine.cpu_family().startswith('arm') or host_machine.cpu_family().startswith('aarch')
+	arch_subdir = 'arm'
+elif host_machine.cpu_family().startswith('ppc')
+	arch_subdir = 'ppc'
+endif
+
+# configure the build, and make sure configs here and in root folder are
+# able to be included in any file. We also store a global array of include dirs
+# for passing to pmdinfogen scripts
+global_inc = include_directories('.', '..',
+	'../lib/librte_eal/include',
+	'../lib/librte_eal/@0@/include'.format(host_machine.system()),
+	'../lib/librte_eal/@0@/include'.format(arch_subdir),
+)
+
 # MS linker requires special treatment.
 # TODO: use cc.get_linker_id() with Meson >= 0.54
 is_ms_linker = is_windows and (cc.get_id() == 'clang')
diff --git a/meson.build b/meson.build
index ab950a2ce..8333d264f 100644
--- a/meson.build
+++ b/meson.build
@@ -28,23 +28,6 @@ dpdk_libs_disabled = []
 dpdk_drvs_disabled = []
 abi_version_file = files('ABI_VERSION')
 
-if host_machine.cpu_family().startswith('x86')
-	arch_subdir = 'x86'
-elif host_machine.cpu_family().startswith('arm') or host_machine.cpu_family().startswith('aarch')
-	arch_subdir = 'arm'
-elif host_machine.cpu_family().startswith('ppc')
-	arch_subdir = 'ppc'
-endif
-
-# configure the build, and make sure configs here and in config folder are
-# able to be included in any file. We also store a global array of include dirs
-# for passing to pmdinfogen scripts
-global_inc = include_directories('.', 'config',
-	'lib/librte_eal/include',
-	'lib/librte_eal/@0@/include'.format(host_machine.system()),
-	'lib/librte_eal/@0@/include'.format(arch_subdir),
-)
-
 # do main build config
 subdir('config')
 
-- 
2.25.1

[dpdk-dev] [RFC PATCH 3/3] build: write messages to dpdk build log file

[Bruce Richardson]

Add additional info into the DPDK-specific logs by copying or replacing all
"message()" calls with writes to the log file, making the build
configuration, including lists of enabled/disabled drivers etc. available
after the meson run has completed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/meson.build   | 2 +-
 config/arm/meson.build | 2 ++
 config/x86/meson.build | 2 ++
 drivers/meson.build    | 6 ++++--
 examples/meson.build   | 1 +
 lib/meson.build        | 5 +++--
 meson.build            | 6 ++++++
 7 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/app/test/meson.build b/app/test/meson.build
index 8bfb02890..b2d90cfd8 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -439,7 +439,7 @@ dpdk_test = executable('dpdk-test',
 	install: true)
 
 has_hugepage = run_command('has-hugepage.sh').stdout().strip() != '0'
-message('hugepage availability: @0@'.format(has_hugepage))
+run_command(log, 'hugepage availability: @0@'.format(has_hugepage))
 
 # some perf tests (eg: memcpy perf autotest)take very long
 # to complete, so timeout to 10 minutes
diff --git a/config/arm/meson.build b/config/arm/meson.build
index b49203fa8..31ac79480 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -182,6 +182,7 @@ else
 	endforeach
 
 	message('Implementer : ' + machine[0])
+	run_command(log, 'config/arm/meson.build: Implementer = ' + machine[0])
 	foreach flag: machine[1]
 		if flag.length() > 0
 			dpdk_conf.set(flag[0], flag[1])
@@ -205,6 +206,7 @@ else
 	endforeach
 endif
 message(machine_args)
+run_command(log, 'config/arm/meson.build: machine_args = ', machine_args)
 
 if (cc.get_define('__ARM_NEON', args: machine_args) != '' or
     cc.get_define('__aarch64__', args: machine_args) != '')
diff --git a/config/x86/meson.build b/config/x86/meson.build
index 31bfa63b1..0008717b8 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -13,6 +13,8 @@ endif
 # we require SSE4.2 for DPDK
 if cc.get_define('__SSE4_2__', args: machine_args) == ''
 	message('SSE 4.2 not enabled by default, explicitly enabling')
+	run_command(log, 'config/x86/meson.build:',
+			'SSE 4.2 not enabled by default, explicitly enabling')
 	machine_args += '-msse4'
 endif
 
diff --git a/drivers/meson.build b/drivers/meson.build
index c008abd4a..fd925114e 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -90,8 +90,10 @@ foreach subpath:subdirs
 				if not is_variable('shared_rte_' + d)
 					build = false
 					reason = 'Missing internal dependency, "@0@"'.format(d)
-					message('Disabling @1@ [@2@]: missing internal dependency "@0@"'
-							.format(d, name, 'drivers/' + drv_path))
+					log_msg = 'Disabling @1@ [@2@]: missing internal dependency "@0@"'.format(
+						d, name, 'drivers/' + drv_path)
+					message(log_msg)
+					run_command(log, log_msg)
 				else
 					shared_deps += [get_variable('shared_rte_' + d)]
 					static_deps += [get_variable('static_rte_' + d)]
diff --git a/examples/meson.build b/examples/meson.build
index 414ec55cc..8f9922cad 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -105,5 +105,6 @@ foreach example: examples
 		error('Cannot build requested example "' + name + '"')
 	else
 		message('Skipping example "' + name + '"')
+		run_command(log, 'Skipping example "' + name + '"')
 	endif
 endforeach
diff --git a/lib/meson.build b/lib/meson.build
index 2715c0f66..aac07eb50 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -112,8 +112,9 @@ foreach l:libraries
 			static_dep = shared_dep
 		else
 			if is_windows and use_function_versioning
-				message('@0@: Function versioning is not supported by Windows.'
-				.format(name))
+				log_msg = '@0@: Function versioning is not supported by Windows.'.format(name)
+				message(log_msg)
+				run_command(log, log_msg)
 			endif
 
 			if use_function_versioning
diff --git a/meson.build b/meson.build
index 8333d264f..a587579f3 100644
--- a/meson.build
+++ b/meson.build
@@ -15,6 +15,8 @@ project('DPDK', 'C',
 subdir('buildtools')
 
 run_command(log, '======= Starting meson =======')
+run_command(log, 'Meson version', meson.version())
+run_command(log, 'DPDK version', meson.project_version())
 
 # set up some global vars for compiler, platform, configuration, etc.
 cc = meson.get_compiler('c')
@@ -76,6 +78,7 @@ foreach lib:enabled_libs
 	endif
 endforeach
 message(output_message + '\n')
+run_command(log, output_message)
 
 output_message = '\n===============\nDrivers Enabled\n===============\n'
 foreach class:dpdk_driver_classes
@@ -92,6 +95,7 @@ foreach class:dpdk_driver_classes
 	endforeach
 endforeach
 message(output_message + '\n')
+run_command(log, output_message)
 
 output_message = '\n=================\nContent Skipped\n=================\n'
 output_message += '\nlibs:\n\t'
@@ -105,3 +109,5 @@ foreach drv:dpdk_drvs_disabled
 	output_message += drv + ':\t' + reason + '\n\t'
 endforeach
 message(output_message + '\n')
+run_command(log, output_message)
+run_command(log, 'Meson done\n')
-- 
2.25.1

Re: [dpdk-dev] [RFC PATCH 0/3] add custom logging to DPDK meson runs

[Bruce Richardson]

On Thu, Oct 22, 2020 at 03:59:41PM +0100, Bruce Richardson wrote:
> While the meson.log file in each build directory contains lots of
> information, much of it is not of interest to developers, and it can be
> hard to see the explicitly messages from the dpdk build files among all
> the rest of the content. Futhermore, while at the end of the a
> configuration run we print out the list of enable/disabled components,
> this is never recorded anywhere for anyone who wants to check that build
> configuration later.
> 
> This patchset attempts to solve these issues by supporting a
> dpdk-meson.log file in each DPDK build folder containing the output from
> the DPDK meson.build files - including the final summary information for
> later checking.
> 
> Bruce Richardson (3):
>   build: add a dpdk-specific meson log file
>   build: shorten top-level build file
>   build: write messages to dpdk build log file
>
Ping on this old RFC set.

From lack of interest, I'd suggest we drop this set from patchwork as
"Rejected", but if others do thing it's worth pursuing I can investigate
doing a v2, to update to latest codebase.

/Bruce 

Re: [dpdk-dev] [RFC PATCH 0/3] add custom logging to DPDK meson runs

[Bruce Richardson]

On Wed, Sep 15, 2021 at 05:23:24PM +0100, Bruce Richardson wrote:
> On Thu, Oct 22, 2020 at 03:59:41PM +0100, Bruce Richardson wrote:
> > While the meson.log file in each build directory contains lots of
> > information, much of it is not of interest to developers, and it can be
> > hard to see the explicitly messages from the dpdk build files among all
> > the rest of the content. Futhermore, while at the end of the a
> > configuration run we print out the list of enable/disabled components,
> > this is never recorded anywhere for anyone who wants to check that build
> > configuration later.
> > 
> > This patchset attempts to solve these issues by supporting a
> > dpdk-meson.log file in each DPDK build folder containing the output from
> > the DPDK meson.build files - including the final summary information for
> > later checking.
> > 
> > Bruce Richardson (3):
> >   build: add a dpdk-specific meson log file
> >   build: shorten top-level build file
> >   build: write messages to dpdk build log file
> >
> Ping on this old RFC set.
> 
> From lack of interest, I'd suggest we drop this set from patchwork as
> "Rejected", but if others do thing it's worth pursuing I can investigate
> doing a v2, to update to latest codebase.
> 
No further feedback received. Marking as rejected to avoid filling up
patchwork.