[dpdk-dev] [PATCH] devtools: add test script for meson builds

[dpdk-dev] [PATCH] devtools: add test script for meson builds

[Bruce Richardson]

To simplify testing with the meson and ninja builds, we can add a script
to set up and do multiple builds. Currently this script sets up:

* clang and gcc builds
* builds using static and shared linkage for binaries (libs are always
   built as both)
* a build using the lowest instruction-set level for x86 (-march=nehalem)
* cross-builds for each cross-file listed in config/arm

Each build is configured in a directory ending in *-build, and then for
the build stage, we just call ninja in each directory in turn. [i.e. we
assume every directory ending in "-build" is a meson build, which is
probably an ok assumption].

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 devtools/test-meson-builds.sh | 55 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100755 devtools/test-meson-builds.sh

diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
new file mode 100755
index 000000000..07a0f4e3c
--- /dev/null
+++ b/devtools/test-meson-builds.sh
@@ -0,0 +1,55 @@
+#! /bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# run meson to auto-configure the various builds
+# * all builds get put in a directory whose name ends in "-build"
+# * if a build-directory already exists we assume it was properly configured
+mesonbuilds ()
+{
+	if [ "$MESON" == "" ]; then
+		MESON=meson;
+	fi;
+	_MESON="$MESON --werror -Dexamples=all";
+
+	# configure shared and static linked builds with gcc and clang
+	for c in gcc clang; do
+		for s in static shared; do
+			builddir=$c-$s-build;
+			if [ -d $builddir ]; then
+				continue;
+			fi
+			echo CC="ccache $c" $_MESON --default-library=$s -Ddefault_library=$s $builddir;
+			CC="ccache $c" $_MESON --default-library=$s -Ddefault_library=$s $builddir || break;
+		done;
+	done;
+
+	# test compilation with minimal x86 instruction set
+	if [ ! -d default-build ] ; then
+		cmd="$_MESON -Dmachine=nehalem default-build";
+		echo $cmd && $cmd || break
+	fi
+
+	# enable cross compilation if gcc cross-compiler is found
+	if command -v aarch64-linux-gnu-gcc >/dev/null 2>&1 ;  then
+		for f in config/arm/arm*gcc; do
+			builddir=`basename $f | awk -F'_' '{print $2}'`;
+			builddir=$builddir-build;
+			if [ -d $builddir ]; then
+				continue;
+			fi
+			cmd="$_MESON --cross-file $f $builddir";
+			echo $cmd && $cmd || break;
+		done;
+	fi;
+}
+
+run_ninja ()
+{
+	for d in *-build ; do
+		ninja -C $d || break
+	done
+}
+
+mesonbuilds
+run_ninja
-- 
2.14.3

Re: [dpdk-dev] [PATCH] devtools: add test script for meson builds

[Bruce Richardson]

On Tue, Apr 24, 2018 at 01:32:55PM +0100, Bruce Richardson wrote:
> To simplify testing with the meson and ninja builds, we can add a script
> to set up and do multiple builds. Currently this script sets up:
> 
> * clang and gcc builds
> * builds using static and shared linkage for binaries (libs are always
>    built as both)
> * a build using the lowest instruction-set level for x86 (-march=nehalem)
> * cross-builds for each cross-file listed in config/arm
> 
> Each build is configured in a directory ending in *-build, and then for
> the build stage, we just call ninja in each directory in turn. [i.e. we
> assume every directory ending in "-build" is a meson build, which is
> probably an ok assumption].
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

By way of update, below is the list I have of open items for the meson
build of DPDK. While it may seems quite a list, it also shows the amount of
material we have in DPDK itself!

While I plan do a number of these, e.g. doing up the documentation on
meson, it would be great if people, particularly driver maintainers, could
look to add support for the missing items in the next release of DPDK. The
biggest gap to close is getting more of the NIC drivers supported.

Please send me on any corrections, too.

Regards,
/Bruce

Gaps for meson build:

- Build of html/pdf docs
- Documentation updates
	- Programmers guide update about the build system itself and
	    it's way of working
	- GSG updates on using meson
- Improved support for unit tests (e.g. benchmark vs regular tests)
- Vector driver support on ARM for ixgbe and i40e
- Map file checking for experimental tags.
- ICC support
- 32-bit compilation

- Library compilation completeness:
	- app - Done
	- drivers/baseband
		 null
		 turbo_sw
	- drivers/bus - Done
	- drivers/crypto
		- aesni_gcm
		- aesni_mb
		- armv8
		- kasumi
		- mrvl
		- scheduler
		- snow3g
		- zuc
	- drivers/event
		- opdl
	- drivers/mempool - Done
	- drivers/net
		- ark
		- avf
		- avp
		- bnx2x
		- bnxt
		- cxgbe
		- ena
		- failsafe
		- kni
		- liquidio
		- mlx4
		- mlx5
		- mvpp2
		- nfp
		- qede
		- softnic
		- szedata2
		- tap
		- vdev_netvsc
		- vhost
		- vmxnet3
	- drivers/raw - [some patches submitted]
	- examples/
		- multi_process
		- netmap_compat
		- performance-thread
		- quota_watermark
		- server_node_efd
		- vm_power_manager
	- kernel/
		 kni
	- lib/ - Done
	- test/
		- cmdline_test
		- test-acl
		- test-pipeline

Re: [dpdk-dev] [PATCH] devtools: add test script for meson builds

[Bruce Richardson]

On Tue, Apr 24, 2018 at 01:32:55PM +0100, Bruce Richardson wrote:
> To simplify testing with the meson and ninja builds, we can add a script
> to set up and do multiple builds. Currently this script sets up:
> 
> * clang and gcc builds
> * builds using static and shared linkage for binaries (libs are always
>    built as both)
> * a build using the lowest instruction-set level for x86 (-march=nehalem)
> * cross-builds for each cross-file listed in config/arm
> 
> Each build is configured in a directory ending in *-build, and then for
> the build stage, we just call ninja in each directory in turn. [i.e. we
> assume every directory ending in "-build" is a meson build, which is
> probably an ok assumption].
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  devtools/test-meson-builds.sh | 55 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 55 insertions(+)
>  create mode 100755 devtools/test-meson-builds.sh
> 
For anyone using this script, I'd recommend updating your copy of meson to
version 0.46. There are some performance optimizations in it that
significantly reduce the time it takes to have meson configure a build.
Actual build time with ninja is as fast as ever, however! :-)

/Bruce

[dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Thomas Monjalon]

From: Bruce Richardson <bruce.richardson@intel.com>

To simplify testing with the meson and ninja builds, we can add a script
to set up and do multiple builds. Currently this script sets up:

* clang and gcc builds
* builds using static and shared linkage for binaries (libs are always
   built as both)
* a build using the lowest instruction-set level for x86 (-march=nehalem)
* cross-builds for each cross-file listed in config/arm

Each build is configured in a directory ending in *-build, and then for
the build stage, we just call ninja in each directory in turn. [i.e. we
assume every directory starting with "build-" is a meson build, which is
probably an ok assumption].

It can use the same configuration file as for the legacy test-build.sh.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2: it is a rework with 3 major changes
   - automatically stop on error thanks to -e
   - directory name starts with "build-"
   - optionally load a config file to get some environment variables
---
 MAINTAINERS                   |  1 +
 devtools/test-meson-builds.sh | 75 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100755 devtools/test-meson-builds.sh

diff --git a/MAINTAINERS b/MAINTAINERS
index e56c72687..4d015fe53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -86,6 +86,7 @@ F: devtools/get-maintainer.sh
 F: devtools/git-log-fixes.sh
 F: devtools/load-devel-config
 F: devtools/test-build.sh
+F: devtools/test-meson-builds.sh
 F: license/
 
 
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
new file mode 100755
index 000000000..8ce0a1d31
--- /dev/null
+++ b/devtools/test-meson-builds.sh
@@ -0,0 +1,75 @@
+#! /bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# Run meson to auto-configure the various builds.
+# * all builds get put in a directory whose name starts with "build-"
+# * if a build-directory already exists we assume it was properly configured
+# Run ninja after configuration is done.
+
+default_path=$PATH
+
+# Load config options
+. $(dirname $(readlink -e $0))/load-devel-config
+
+reset_env ()
+{
+	export PATH=$default_path
+	unset CROSS
+	unset ARMV8_CRYPTO_LIB_PATH
+	unset FLEXRAN_SDK
+	unset LIBMUSDK_PATH
+	unset LIBSSO_SNOW3G_PATH
+	unset LIBSSO_KASUMI_PATH
+	unset LIBSSO_ZUC_PATH
+	unset PQOS_INSTALL_PATH
+}
+
+cd $(dirname $(readlink -m $0))/..
+
+load_config ()
+{
+	reset_env
+	. $(dirname $(readlink -e $0))/load-devel-config
+	MESON=${MESON:-meson}
+}
+
+build () # <directory> <meson options>
+{
+	dir=$1
+	shift
+	if [ ! -d "$dir" ] ; then
+		options="--werror -Dexamples=all $*"
+		# TODO: the configuration variables $DPDK_DEP_CFLAGS
+		# and $DPDK_DEP_LDFLAGS are not handled in this script
+		echo "$MESON $options $dir"
+		$MESON $options $dir
+		unset CC
+	fi
+	echo "ninja -C $dir"
+	ninja -C $dir
+}
+
+# shared and static linked builds with gcc and clang
+for c in gcc clang ; do
+	for s in static shared ; do
+		load_config
+		export CC="ccache $c"
+		build build-$c-$s --default-library=$s
+	done
+done
+
+# test compilation with minimal x86 instruction set
+load_config
+build build-x86-default -Dmachine=nehalem
+
+# enable cross compilation if gcc cross-compiler is found
+for f in config/arm/arm*gcc ; do
+	DPDK_TARGET=$(basename $f | tr '_' '-')
+	load_config
+	CROSS=${CROSS:-aarch64-linux-gnu-}
+	if ! command -v ${CROSS}gcc >/dev/null 2>&1 ; then
+		continue
+	fi
+	build build-$(echo $DPDK_TARGET | cut -d'-' -f-2) --cross-file $f
+done
-- 
2.16.2

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Bruce Richardson]

On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> From: Bruce Richardson <bruce.richardson@intel.com>
> 
> To simplify testing with the meson and ninja builds, we can add a script
> to set up and do multiple builds. Currently this script sets up:
> 
> * clang and gcc builds
> * builds using static and shared linkage for binaries (libs are always
>    built as both)
> * a build using the lowest instruction-set level for x86 (-march=nehalem)
> * cross-builds for each cross-file listed in config/arm
> 
> Each build is configured in a directory ending in *-build, and then for
> the build stage, we just call ninja in each directory in turn. [i.e. we
> assume every directory starting with "build-" is a meson build, which is
> probably an ok assumption].
> 
> It can use the same configuration file as for the legacy test-build.sh.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> v2: it is a rework with 3 major changes
>    - automatically stop on error thanks to -e
>    - directory name starts with "build-"
>    - optionally load a config file to get some environment variables
> ---
>  MAINTAINERS                   |  1 +
>  devtools/test-meson-builds.sh | 75 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 76 insertions(+)
>  create mode 100755 devtools/test-meson-builds.sh
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e56c72687..4d015fe53 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -86,6 +86,7 @@ F: devtools/get-maintainer.sh
>  F: devtools/git-log-fixes.sh
>  F: devtools/load-devel-config
>  F: devtools/test-build.sh
> +F: devtools/test-meson-builds.sh
>  F: license/
>  
>  
> diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
> new file mode 100755
> index 000000000..8ce0a1d31
> --- /dev/null
> +++ b/devtools/test-meson-builds.sh
> @@ -0,0 +1,75 @@
> +#! /bin/sh -e
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +# Run meson to auto-configure the various builds.
> +# * all builds get put in a directory whose name starts with "build-"
> +# * if a build-directory already exists we assume it was properly configured
> +# Run ninja after configuration is done.
> +
> +default_path=$PATH
> +
> +# Load config options
> +. $(dirname $(readlink -e $0))/load-devel-config
> +

Why is this needed here, it seems to be called before each individual
config anyway.

> +reset_env ()
> +{
> +	export PATH=$default_path
> +	unset CROSS
> +	unset ARMV8_CRYPTO_LIB_PATH
> +	unset FLEXRAN_SDK
> +	unset LIBMUSDK_PATH
> +	unset LIBSSO_SNOW3G_PATH
> +	unset LIBSSO_KASUMI_PATH
> +	unset LIBSSO_ZUC_PATH
> +	unset PQOS_INSTALL_PATH

These variables bar PATH are unused by meson build, so should be removed
here to avoid giving the impression they are use. $CROSS is used by the
script so perhaps it can be kept. However, the whole point of the
cross-files is that you include all the needed details of your compiler
there. I think we should move away from using the CROSS value completely,
and use the cross-files properly.

> +}
> +
> +cd $(dirname $(readlink -m $0))/..
> +
I don't think we should force the builds to be always put into the base
directory. Instead of using cd, I think we should instead capture the base
directory path and pass that to meson.

> +load_config ()
> +{
> +	reset_env
> +	. $(dirname $(readlink -e $0))/load-devel-config
> +	MESON=${MESON:-meson}
> +}
Why does this need to be done each time?

> +
> +build () # <directory> <meson options>
> +{
> +	dir=$1
> +	shift
> +	if [ ! -d "$dir" ] ; then
> +		options="--werror -Dexamples=all $*"
> +		# TODO: the configuration variables $DPDK_DEP_CFLAGS
> +		# and $DPDK_DEP_LDFLAGS are not handled in this script
> +		echo "$MESON $options $dir"
> +		$MESON $options $dir
> +		unset CC
> +	fi
> +	echo "ninja -C $dir"
> +	ninja -C $dir
> +}
> +
> +# shared and static linked builds with gcc and clang
> +for c in gcc clang ; do
> +	for s in static shared ; do
> +		load_config
> +		export CC="ccache $c"
> +		build build-$c-$s --default-library=$s
> +	done
> +done
> +
> +# test compilation with minimal x86 instruction set
> +load_config
> +build build-x86-default -Dmachine=nehalem
> +
> +# enable cross compilation if gcc cross-compiler is found
> +for f in config/arm/arm*gcc ; do
> +	DPDK_TARGET=$(basename $f | tr '_' '-')
> +	load_config
> +	CROSS=${CROSS:-aarch64-linux-gnu-}
> +	if ! command -v ${CROSS}gcc >/dev/null 2>&1 ; then
> +		continue
> +	fi
> +	build build-$(echo $DPDK_TARGET | cut -d'-' -f-2) --cross-file $f
> +done
> -- 
> 2.16.2
> 

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Thomas Monjalon]

25/05/2018 17:18, Bruce Richardson:
> On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > +default_path=$PATH
> > +
> > +# Load config options
> > +. $(dirname $(readlink -e $0))/load-devel-config
> > +
> 
> Why is this needed here, it seems to be called before each individual
> config anyway.

Right, it can be removed from here.

> > +reset_env ()
> > +{
> > +	export PATH=$default_path
> > +	unset CROSS
> > +	unset ARMV8_CRYPTO_LIB_PATH
> > +	unset FLEXRAN_SDK
> > +	unset LIBMUSDK_PATH
> > +	unset LIBSSO_SNOW3G_PATH
> > +	unset LIBSSO_KASUMI_PATH
> > +	unset LIBSSO_ZUC_PATH
> > +	unset PQOS_INSTALL_PATH
> 
> These variables bar PATH are unused by meson build, so should be removed
> here to avoid giving the impression they are use.

Actually they should be used when compiling.
PATH can be used to allow a toolchain which is not in the standard path.
And dependencies _PATH variables can be specified only for some builds.
Example: I have libsso only for x86 64-bit, so I do not set it
for 32-bit or ARM builds. The config file reads DPDK_TARGET to know.
Note: DPDK_TARGET is not yet set correctly for every builds in this version.

> $CROSS is used by the
> script so perhaps it can be kept. However, the whole point of the
> cross-files is that you include all the needed details of your compiler
> there. I think we should move away from using the CROSS value completely,
> and use the cross-files properly.

Yes we can remove CROSS if it is redundant with config files in config/arm/.
But I do not understand why these files cannot be agnostic regarding the
name (CROSS) of the toolchain.
To me it is very strange to have the binary names hard-linked in the configs.
Anyway, this discussion is out of the scope of this script.
So I am for removing the CROSS variable and use aarch64-linux-gnu-gcc
as it is hard written in every ARM configs for now.

> > +cd $(dirname $(readlink -m $0))/..
> > +
> I don't think we should force the builds to be always put into the base
> directory. Instead of using cd, I think we should instead capture the base
> directory path and pass that to meson.

OK to not force the directory.
You want to build in the current directory?
If yes, we can just remove this "cd" and no need to pass a base directory
to meson.

> > +load_config ()
> > +{
> > +	reset_env
> > +	. $(dirname $(readlink -e $0))/load-devel-config
> > +	MESON=${MESON:-meson}
> > +}
> Why does this need to be done each time?

Because the config could be different for each build (see above).

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Bruce Richardson]

On Sat, May 26, 2018 at 11:32:53AM +0200, Thomas Monjalon wrote:
> 25/05/2018 17:18, Bruce Richardson:
> > On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > > +default_path=$PATH
> > > +
> > > +# Load config options
> > > +. $(dirname $(readlink -e $0))/load-devel-config
> > > +
> > 
> > Why is this needed here, it seems to be called before each individual
> > config anyway.
> 
> Right, it can be removed from here.
> 
> > > +reset_env ()
> > > +{
> > > +	export PATH=$default_path
> > > +	unset CROSS
> > > +	unset ARMV8_CRYPTO_LIB_PATH
> > > +	unset FLEXRAN_SDK
> > > +	unset LIBMUSDK_PATH
> > > +	unset LIBSSO_SNOW3G_PATH
> > > +	unset LIBSSO_KASUMI_PATH
> > > +	unset LIBSSO_ZUC_PATH
> > > +	unset PQOS_INSTALL_PATH
> > 
> > These variables bar PATH are unused by meson build, so should be removed
> > here to avoid giving the impression they are use.
> 
> Actually they should be used when compiling.
> PATH can be used to allow a toolchain which is not in the standard path.
> And dependencies _PATH variables can be specified only for some builds.
> Example: I have libsso only for x86 64-bit, so I do not set it
> for 32-bit or ARM builds. The config file reads DPDK_TARGET to know.
> Note: DPDK_TARGET is not yet set correctly for every builds in this version.
> 

We don't need DPDK_TARGET variable any more, however, environmental vars
can be used by this script for setting up PATH and CFLAGS/LDFLAGS etc. as
needed. However, environmental vars bar those standard ones are ignored by
a meson build, so they would be only for script use. Therefore I'm
suggesting removing only the unused ones.

> > $CROSS is used by the
> > script so perhaps it can be kept. However, the whole point of the
> > cross-files is that you include all the needed details of your compiler
> > there. I think we should move away from using the CROSS value completely,
> > and use the cross-files properly.
> 
> Yes we can remove CROSS if it is redundant with config files in config/arm/.
> But I do not understand why these files cannot be agnostic regarding the
> name (CROSS) of the toolchain.
> To me it is very strange to have the binary names hard-linked in the configs.
> Anyway, this discussion is out of the scope of this script.
> So I am for removing the CROSS variable and use aarch64-linux-gnu-gcc
> as it is hard written in every ARM configs for now.

You can use CROSS in this script to set PATH for each build. Again, though
it would only be for script, not meson use.

> 
> > > +cd $(dirname $(readlink -m $0))/..
> > > +
> > I don't think we should force the builds to be always put into the base
> > directory. Instead of using cd, I think we should instead capture the base
> > directory path and pass that to meson.
> 
> OK to not force the directory.
> You want to build in the current directory?
> If yes, we can just remove this "cd" and no need to pass a base directory
> to meson.

We would need to pass the base to meson, because otherwise meson assumes
the top-level meson.build file is in the current directory, i.e. calling
"meson build-dir" is the same as "meson . build-dir". If we want to allow
using this script from other places on filesystem, we need to either "cd"
to the base dir as you do now, or else explicitly pass in the basedir. The
latter I think is a better option as it would allow building from any
location on the filesystem.

> 
> > > +load_config ()
> > > +{
> > > +	reset_env
> > > +	. $(dirname $(readlink -e $0))/load-devel-config
> > > +	MESON=${MESON:-meson}
> > > +}
> > Why does this need to be done each time?
> 
> Because the config could be different for each build (see above).
> 
How would it be different, it's the same command called with the same
environment each time?

/Bruce

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Thomas Monjalon]

28/05/2018 11:33, Bruce Richardson:
> On Sat, May 26, 2018 at 11:32:53AM +0200, Thomas Monjalon wrote:
> > 25/05/2018 17:18, Bruce Richardson:
> > > On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > > > +cd $(dirname $(readlink -m $0))/..
> > > > +
> > > I don't think we should force the builds to be always put into the base
> > > directory. Instead of using cd, I think we should instead capture the base
> > > directory path and pass that to meson.
> > 
> > OK to not force the directory.
> > You want to build in the current directory?
> > If yes, we can just remove this "cd" and no need to pass a base directory
> > to meson.
> 
> We would need to pass the base to meson, because otherwise meson assumes
> the top-level meson.build file is in the current directory, i.e. calling
> "meson build-dir" is the same as "meson . build-dir". If we want to allow
> using this script from other places on filesystem, we need to either "cd"
> to the base dir as you do now, or else explicitly pass in the basedir. The
> latter I think is a better option as it would allow building from any
> location on the filesystem.

I agree.

I don't understand the meson syntax:
	meson [ options ] [ source directory ] [ build directory ]
If there is only one argument, it is the build directory?

I could send a v5 to add the source directory in the meson command.
It would be:
	srcdir=$(dirname $(readlink -m $0))/..
	$MESON $options $srcdir $builddir


> > > > +load_config ()
> > > > +{
> > > > +	reset_env
> > > > +	. $(dirname $(readlink -e $0))/load-devel-config
> > > > +	MESON=${MESON:-meson}
> > > > +}
> > > Why does this need to be done each time?
> > 
> > Because the config could be different for each build (see above).
> > 
> How would it be different, it's the same command called with the same
> environment each time?

No, the idea is to adapt the environment to the build target.
As an example, the dependencies can be different for 32-bit and 64-bit.

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Bruce Richardson]

On Mon, May 28, 2018 at 12:26:24PM +0200, Thomas Monjalon wrote:
> 28/05/2018 11:33, Bruce Richardson:
> > On Sat, May 26, 2018 at 11:32:53AM +0200, Thomas Monjalon wrote:
> > > 25/05/2018 17:18, Bruce Richardson:
> > > > On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > > > > +cd $(dirname $(readlink -m $0))/..
> > > > > +
> > > > I don't think we should force the builds to be always put into the base
> > > > directory. Instead of using cd, I think we should instead capture the base
> > > > directory path and pass that to meson.
> > > 
> > > OK to not force the directory.
> > > You want to build in the current directory?
> > > If yes, we can just remove this "cd" and no need to pass a base directory
> > > to meson.
> > 
> > We would need to pass the base to meson, because otherwise meson assumes
> > the top-level meson.build file is in the current directory, i.e. calling
> > "meson build-dir" is the same as "meson . build-dir". If we want to allow
> > using this script from other places on filesystem, we need to either "cd"
> > to the base dir as you do now, or else explicitly pass in the basedir. The
> > latter I think is a better option as it would allow building from any
> > location on the filesystem.
> 
> I agree.
> 
> I don't understand the meson syntax:
> 	meson [ options ] [ source directory ] [ build directory ]
> If there is only one argument, it is the build directory?
> 
> I could send a v5 to add the source directory in the meson command.
> It would be:
> 	srcdir=$(dirname $(readlink -m $0))/..
> 	$MESON $options $srcdir $builddir
> 

Yes, exactly.

> 
> > > > > +load_config () +{ +	reset_env +	. $(dirname $(readlink -e
> > > > > $0))/load-devel-config +	MESON=${MESON:-meson} +}
> > > > Why does this need to be done each time?
> > > 
> > > Because the config could be different for each build (see above).
> > > 
> > How would it be different, it's the same command called with the same
> > environment each time?
> 
> No, the idea is to adapt the environment to the build target.  As an
> example, the dependencies can be different for 32-bit and 64-bit.
> 
I would hope that dependency detection should solve that, but since you
already have support for that in existing build script via environment
vars, I have no objection to leveraging that in the meson scripts. Overall,
though, I'd prefer to ensure that the detection works so that everyone only
needs one environment setup in order to get all builds working
simultaneously.

/Bruce

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Thomas Monjalon]

28/05/2018 15:20, Bruce Richardson:
> On Mon, May 28, 2018 at 12:26:24PM +0200, Thomas Monjalon wrote:
> > 28/05/2018 11:33, Bruce Richardson:
> > > On Sat, May 26, 2018 at 11:32:53AM +0200, Thomas Monjalon wrote:
> > > > 25/05/2018 17:18, Bruce Richardson:
> > > > > On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > > > > > +load_config () +{ +	reset_env +	. $(dirname $(readlink -e
> > > > > > $0))/load-devel-config +	MESON=${MESON:-meson} +}
> > > > > Why does this need to be done each time?
> > > > 
> > > > Because the config could be different for each build (see above).
> > > > 
> > > How would it be different, it's the same command called with the same
> > > environment each time?
> > 
> > No, the idea is to adapt the environment to the build target.  As an
> > example, the dependencies can be different for 32-bit and 64-bit.
> > 
> I would hope that dependency detection should solve that, but since you
> already have support for that in existing build script via environment
> vars, I have no objection to leveraging that in the meson scripts. Overall,
> though, I'd prefer to ensure that the detection works so that everyone only
> needs one environment setup in order to get all builds working
> simultaneously.

The dependency detection cannot work if I have dependencies in uncommon
directories.
I think it is important to allow testing compilation with dependencies
which are available but not installed, by providing paths.

Re: [dpdk-dev] [PATCH v2] devtools: add test script for meson builds

[Bruce Richardson]

On Tue, May 29, 2018 at 12:38:14PM +0200, Thomas Monjalon wrote:
> 28/05/2018 15:20, Bruce Richardson:
> > On Mon, May 28, 2018 at 12:26:24PM +0200, Thomas Monjalon wrote:
> > > 28/05/2018 11:33, Bruce Richardson:
> > > > On Sat, May 26, 2018 at 11:32:53AM +0200, Thomas Monjalon wrote:
> > > > > 25/05/2018 17:18, Bruce Richardson:
> > > > > > On Fri, May 25, 2018 at 04:51:58PM +0200, Thomas Monjalon wrote:
> > > > > > > +load_config () +{ +	reset_env +	. $(dirname $(readlink -e
> > > > > > > $0))/load-devel-config +	MESON=${MESON:-meson} +}
> > > > > > Why does this need to be done each time?
> > > > > 
> > > > > Because the config could be different for each build (see above).
> > > > > 
> > > > How would it be different, it's the same command called with the same
> > > > environment each time?
> > > 
> > > No, the idea is to adapt the environment to the build target.  As an
> > > example, the dependencies can be different for 32-bit and 64-bit.
> > > 
> > I would hope that dependency detection should solve that, but since you
> > already have support for that in existing build script via environment
> > vars, I have no objection to leveraging that in the meson scripts. Overall,
> > though, I'd prefer to ensure that the detection works so that everyone only
> > needs one environment setup in order to get all builds working
> > simultaneously.
> 
> The dependency detection cannot work if I have dependencies in uncommon
> directories.
> I think it is important to allow testing compilation with dependencies
> which are available but not installed, by providing paths.
> 
Yes, but that should be done via meson configure options to specify the
paths, rather than via environmental variables. AFAIK there is no support in
meson for querying the environment*, every variable that could affect the
build should be explicitly specified 

/Bruce

* Ok, no direct way. There are probably ways in which it could be done,
but it's not the meson way to do things.

[dpdk-dev] [PATCH v3] devtools: add test script for meson builds

[Thomas Monjalon]

From: Bruce Richardson <bruce.richardson@intel.com>

To simplify testing with the meson and ninja builds, we can add a script
to set up and do multiple builds. Currently this script sets up:

* clang and gcc builds
* builds using static and shared linkage for binaries (libs are always
   built as both)
* a build using the lowest instruction-set level for x86 (-march=nehalem)
* cross-builds for each cross-file listed in config/arm

Each build is configured in a directory ending in *-build, and then for
the build stage, we just call ninja in each directory in turn. [i.e. we
assume every directory starting with "build-" is a meson build, which is
probably an ok assumption].

It can use the same configuration file as for the legacy test-build.sh.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2: it is a rework with 3 major changes
   - automatically stop on error thanks to -e
   - directory name starts with "build-"
   - optionally load a config file to get some environment variables
v3:
   - remove forcing "cd": use current directory
   - remove CROSS: use hard-written aarch64-linux-gnu-gcc
   - remove config load in script start
---
 MAINTAINERS                   |  1 +
 devtools/test-meson-builds.sh | 68 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
 create mode 100755 devtools/test-meson-builds.sh

diff --git a/MAINTAINERS b/MAINTAINERS
index e56c72687..4d015fe53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -86,6 +86,7 @@ F: devtools/get-maintainer.sh
 F: devtools/git-log-fixes.sh
 F: devtools/load-devel-config
 F: devtools/test-build.sh
+F: devtools/test-meson-builds.sh
 F: license/
 
 
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
new file mode 100755
index 000000000..3283dbb7f
--- /dev/null
+++ b/devtools/test-meson-builds.sh
@@ -0,0 +1,68 @@
+#! /bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# Run meson to auto-configure the various builds.
+# * all builds get put in a directory whose name starts with "build-"
+# * if a build-directory already exists we assume it was properly configured
+# Run ninja after configuration is done.
+
+default_path=$PATH
+
+reset_env ()
+{
+	export PATH=$default_path
+	unset ARMV8_CRYPTO_LIB_PATH
+	unset FLEXRAN_SDK
+	unset LIBMUSDK_PATH
+	unset LIBSSO_SNOW3G_PATH
+	unset LIBSSO_KASUMI_PATH
+	unset LIBSSO_ZUC_PATH
+	unset PQOS_INSTALL_PATH
+}
+
+load_config ()
+{
+	reset_env
+	. $(dirname $(readlink -e $0))/load-devel-config
+	MESON=${MESON:-meson}
+}
+
+build () # <directory> <meson options>
+{
+	dir=$1
+	shift
+	if [ ! -d "$dir" ] ; then
+		options="--werror -Dexamples=all $*"
+		# TODO: the configuration variables $DPDK_DEP_CFLAGS
+		# and $DPDK_DEP_LDFLAGS are not handled in this script
+		echo "$MESON $options $dir"
+		$MESON $options $dir
+		unset CC
+	fi
+	echo "ninja -C $dir"
+	ninja -C $dir
+}
+
+# shared and static linked builds with gcc and clang
+for c in gcc clang ; do
+	for s in static shared ; do
+		load_config
+		export CC="ccache $c"
+		build build-$c-$s --default-library=$s
+	done
+done
+
+# test compilation with minimal x86 instruction set
+load_config
+build build-x86-default -Dmachine=nehalem
+
+# enable cross compilation if gcc cross-compiler is found
+for f in config/arm/arm*gcc ; do
+	DPDK_TARGET=$(basename $f | tr '_' '-')
+	load_config
+	if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1 ; then
+		continue
+	fi
+	build build-$(echo $DPDK_TARGET | cut -d'-' -f-2) --cross-file $f
+done
-- 
2.16.2

[dpdk-dev] [PATCH v4] devtools: add test script for meson builds

[Thomas Monjalon]

From: Bruce Richardson <bruce.richardson@intel.com>

To simplify testing with the meson and ninja builds, we can add a script
to set up and do multiple builds. Currently this script sets up:

* clang and gcc builds
* builds using static and shared linkage for binaries (libs are always
   built as both)
* a build using the lowest instruction-set level for x86 (-march=nehalem)
* cross-builds for each cross-file listed in config/arm

Each build is configured in a directory ending in *-build, and then for
the build stage, we just call ninja in each directory in turn. [i.e. we
assume every directory starting with "build-" is a meson build, which is
probably an ok assumption].

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2: it is a rework with 3 major changes
   - automatically stop on error thanks to -e
   - directory name starts with "build-"
   - optionally load a config file to get some environment variables
v3:
   - remove forcing "cd": use current directory
   - remove CROSS: use hard-written aarch64-linux-gnu-gcc
   - remove config load in script start
v4:
   - remove config file loading (will be improved and sent later)
   - the v4 is closer to what Bruce sent as v1
---
 MAINTAINERS                   |  1 +
 devtools/test-meson-builds.sh | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
 create mode 100755 devtools/test-meson-builds.sh

diff --git a/MAINTAINERS b/MAINTAINERS
index e56c72687..4d015fe53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -86,6 +86,7 @@ F: devtools/get-maintainer.sh
 F: devtools/git-log-fixes.sh
 F: devtools/load-devel-config
 F: devtools/test-build.sh
+F: devtools/test-meson-builds.sh
 F: license/
 
 
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
new file mode 100755
index 000000000..78d712aea
--- /dev/null
+++ b/devtools/test-meson-builds.sh
@@ -0,0 +1,43 @@
+#! /bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# Run meson to auto-configure the various builds.
+# * all builds get put in a directory whose name starts with "build-"
+# * if a build-directory already exists we assume it was properly configured
+# Run ninja after configuration is done.
+
+MESON=${MESON:-meson}
+
+build () # <directory> <meson options>
+{
+	dir=$1
+	shift
+	if [ ! -d "$dir" ] ; then
+		options="--werror -Dexamples=all $*"
+		echo "$MESON $options $dir"
+		$MESON $options $dir
+		unset CC
+	fi
+	echo "ninja -C $dir"
+	ninja -C $dir
+}
+
+# shared and static linked builds with gcc and clang
+for c in gcc clang ; do
+	for s in static shared ; do
+		export CC="ccache $c"
+		build build-$c-$s --default-library=$s
+	done
+done
+
+# test compilation with minimal x86 instruction set
+build build-x86-default -Dmachine=nehalem
+
+# enable cross compilation if gcc cross-compiler is found
+for f in config/arm/arm*gcc ; do
+	if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1 ; then
+		continue
+	fi
+	build build-$(basename $f | tr '_' '-' | cut -d'-' -f-2) --cross-file $f
+done
-- 
2.16.2

Re: [dpdk-dev] [PATCH v4] devtools: add test script for meson builds

[Bruce Richardson]

On Sat, May 26, 2018 at 01:21:42PM +0200, Thomas Monjalon wrote:
> From: Bruce Richardson <bruce.richardson@intel.com>
> 
> To simplify testing with the meson and ninja builds, we can add a script
> to set up and do multiple builds. Currently this script sets up:
> 
> * clang and gcc builds
> * builds using static and shared linkage for binaries (libs are always
>    built as both)
> * a build using the lowest instruction-set level for x86 (-march=nehalem)
> * cross-builds for each cross-file listed in config/arm
> 
> Each build is configured in a directory ending in *-build, and then for
> the build stage, we just call ninja in each directory in turn. [i.e. we
> assume every directory starting with "build-" is a meson build, which is
> probably an ok assumption].
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> v2: it is a rework with 3 major changes
>    - automatically stop on error thanks to -e
>    - directory name starts with "build-"
>    - optionally load a config file to get some environment variables
> v3:
>    - remove forcing "cd": use current directory
>    - remove CROSS: use hard-written aarch64-linux-gnu-gcc
>    - remove config load in script start
> v4:
>    - remove config file loading (will be improved and sent later)
>    - the v4 is closer to what Bruce sent as v1
> ---

Yes, it's closer, but there is also some good material in your v2 that we
probably want to rework into this later on. I suggest in the name of
progress we merge this version and iterate on improving it later.

/Bruce

[dpdk-dev] [PATCH v5] devtools: add test script for meson builds

[Thomas Monjalon]

From: Bruce Richardson <bruce.richardson@intel.com>

To simplify testing with the meson and ninja builds, we can add a script
to set up and do multiple builds. Currently this script sets up:

* clang and gcc builds
* builds using static and shared linkage for binaries (libs are always
   built as both)
* a build using the lowest instruction-set level for x86 (-march=nehalem)
* cross-builds for each cross-file listed in config/arm

Each build is configured in a directory ending in *-build, and then for
the build stage, we just call ninja in each directory in turn. [i.e. we
assume every directory starting with "build-" is a meson build, which is
probably an ok assumption].

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2: it is a rework with 3 major changes
   - automatically stop on error thanks to -e
   - directory name starts with "build-"
   - optionally load a config file to get some environment variables
v3:
   - remove forcing "cd": use current directory
   - remove CROSS: use hard-written aarch64-linux-gnu-gcc
   - remove config load in script start
v4:
   - remove config file loading (will be improved and sent later)
   - the v4 is closer to what Bruce sent as v1
v5:
   - specify srcdir to build from anywhere
   - use ccache for ARM
---
 MAINTAINERS                   |  1 +
 devtools/test-meson-builds.sh | 46 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
 create mode 100755 devtools/test-meson-builds.sh

diff --git a/MAINTAINERS b/MAINTAINERS
index e56c72687..4d015fe53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -86,6 +86,7 @@ F: devtools/get-maintainer.sh
 F: devtools/git-log-fixes.sh
 F: devtools/load-devel-config
 F: devtools/test-build.sh
+F: devtools/test-meson-builds.sh
 F: license/
 
 
diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
new file mode 100755
index 000000000..9868c325b
--- /dev/null
+++ b/devtools/test-meson-builds.sh
@@ -0,0 +1,46 @@
+#! /bin/sh -e
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# Run meson to auto-configure the various builds.
+# * all builds get put in a directory whose name starts with "build-"
+# * if a build-directory already exists we assume it was properly configured
+# Run ninja after configuration is done.
+
+srcdir=$(dirname $(readlink -m $0))/..
+MESON=${MESON:-meson}
+
+build () # <directory> <meson options>
+{
+	builddir=$1
+	shift
+	if [ ! -d "$builddir" ] ; then
+		options="--werror -Dexamples=all $*"
+		echo "$MESON $options $srcdir $builddir"
+		$MESON $options $srcdir $builddir
+		unset CC
+	fi
+	echo "ninja -C $builddir"
+	ninja -C $builddir
+}
+
+# shared and static linked builds with gcc and clang
+for c in gcc clang ; do
+	for s in static shared ; do
+		export CC="ccache $c"
+		build build-$c-$s --default-library=$s
+	done
+done
+
+# test compilation with minimal x86 instruction set
+build build-x86-default -Dmachine=nehalem
+
+# enable cross compilation if gcc cross-compiler is found
+for f in config/arm/arm*gcc ; do
+	c=aarch64-linux-gnu-gcc
+	if ! command -v $c >/dev/null 2>&1 ; then
+		continue
+	fi
+	export CC="ccache $c"
+	build build-$(basename $f | tr '_' '-' | cut -d'-' -f-2) --cross-file $f
+done
-- 
2.16.2

Re: [dpdk-dev] [PATCH v5] devtools: add test script for meson builds

[Richardson, Bruce]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, May 29, 2018 3:43 PM
> To: dev@dpdk.org
> Cc: Richardson, Bruce <bruce.richardson@intel.com>
> Subject: [PATCH v5] devtools: add test script for meson builds
> 
> From: Bruce Richardson <bruce.richardson@intel.com>
> 
> To simplify testing with the meson and ninja builds, we can add a script
> to set up and do multiple builds. Currently this script sets up:
> 
> * clang and gcc builds
> * builds using static and shared linkage for binaries (libs are always
>    built as both)
> * a build using the lowest instruction-set level for x86 (-march=nehalem)
> * cross-builds for each cross-file listed in config/arm
> 
> Each build is configured in a directory ending in *-build, and then for
> the build stage, we just call ninja in each directory in turn. [i.e. we
> assume every directory starting with "build-" is a meson build, which is
> probably an ok assumption].
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> v2: it is a rework with 3 major changes
>    - automatically stop on error thanks to -e
>    - directory name starts with "build-"
>    - optionally load a config file to get some environment variables
> v3:
>    - remove forcing "cd": use current directory
>    - remove CROSS: use hard-written aarch64-linux-gnu-gcc
>    - remove config load in script start
> v4:
>    - remove config file loading (will be improved and sent later)
>    - the v4 is closer to what Bruce sent as v1
> v5:
>    - specify srcdir to build from anywhere
>    - use ccache for ARM
> ---
>  MAINTAINERS                   |  1 +
>  devtools/test-meson-builds.sh | 46
> +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 47 insertions(+)
>  create mode 100755 devtools/test-meson-builds.sh

LGTM

Re: [dpdk-dev] [PATCH v5] devtools: add test script for meson builds

[Thomas Monjalon]

29/05/2018 16:58, Richardson, Bruce:
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Tuesday, May 29, 2018 3:43 PM
> > To: dev@dpdk.org
> > Cc: Richardson, Bruce <bruce.richardson@intel.com>
> > Subject: [PATCH v5] devtools: add test script for meson builds
> > 
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > 
> > To simplify testing with the meson and ninja builds, we can add a script
> > to set up and do multiple builds. Currently this script sets up:
> > 
> > * clang and gcc builds
> > * builds using static and shared linkage for binaries (libs are always
> >    built as both)
> > * a build using the lowest instruction-set level for x86 (-march=nehalem)
> > * cross-builds for each cross-file listed in config/arm
> > 
> > Each build is configured in a directory ending in *-build, and then for
> > the build stage, we just call ninja in each directory in turn. [i.e. we
> > assume every directory starting with "build-" is a meson build, which is
> > probably an ok assumption].
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > v2: it is a rework with 3 major changes
> >    - automatically stop on error thanks to -e
> >    - directory name starts with "build-"
> >    - optionally load a config file to get some environment variables
> > v3:
> >    - remove forcing "cd": use current directory
> >    - remove CROSS: use hard-written aarch64-linux-gnu-gcc
> >    - remove config load in script start
> > v4:
> >    - remove config file loading (will be improved and sent later)
> >    - the v4 is closer to what Bruce sent as v1
> > v5:
> >    - specify srcdir to build from anywhere
> >    - use ccache for ARM
> > ---
> >  MAINTAINERS                   |  1 +
> >  devtools/test-meson-builds.sh | 46
> > +++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 47 insertions(+)
> >  create mode 100755 devtools/test-meson-builds.sh
> 
> LGTM

Applied, thanks