[dpdk-dev] [PATCH 0/8] simpler and better build test

[dpdk-dev] [PATCH 0/8] simpler and better build test

[Thomas Monjalon]

These are some improvements to the script test-build.sh which
makes compilation testing easier in various environments with
different dependencies.

Thomas Monjalon (8):
  scripts: fix run in any directory
  scripts: remove legacy build method test
  scripts: test build of performance-thread example
  scripts: stop build test after first error
  scripts: allow tuning build options per test target
  scripts: allow tuning any test build option
  scripts: hook build test config
  scripts: add verbose test build option

 scripts/checkpatches.sh |   2 +-
 scripts/test-build.sh   | 124 ++++++++++++++++++++++++++++++++++--------------
 2 files changed, 90 insertions(+), 36 deletions(-)

-- 
2.7.0

[dpdk-dev] [PATCH 1/8] scripts: fix run in any directory

[Thomas Monjalon]

The path to load-devel-config.sh must be relative to the script.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/checkpatches.sh | 2 +-
 scripts/test-build.sh   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/checkpatches.sh b/scripts/checkpatches.sh
index fcb24fd..5c58a20 100755
--- a/scripts/checkpatches.sh
+++ b/scripts/checkpatches.sh
@@ -33,7 +33,7 @@
 # Load config options:
 # - DPDK_CHECKPATCH_PATH
 # - DPDK_CHECKPATCH_LINE_LENGTH
-. scripts/load-devel-config.sh
+. $(dirname $(readlink -e $0))/load-devel-config.sh
 
 length=${DPDK_CHECKPATCH_LINE_LENGTH:-80}
 
diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index 5f3cab5..be8275e 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -38,7 +38,7 @@
 # - DPDK_DEP_MOFED (y/[n])
 # - DPDK_DEP_PCAP (y/[n])
 # - DPDK_NOTIFY (notify-send)
-. scripts/load-devel-config.sh
+. $(dirname $(readlink -e $0))/load-devel-config.sh
 
 print_usage () {
 	echo "usage: $(basename $0) [-h] [-jX] [-s] [config1 [config2] ...]]"
-- 
2.7.0

[dpdk-dev] [PATCH 2/8] scripts: remove legacy build method test

[Thomas Monjalon]

Building with "make install T=" is now deprecated.
The script will test only the standard "make" command.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index be8275e..b23d6c8 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -96,7 +96,7 @@ cd $(dirname $(readlink -m $0))/..
 config () # <directory> <target> <options>
 {
 	if [ ! -e $1/.config ] ; then
-		echo Custom configuration
+		echo "================== Configure $1"
 		make T=$2 O=$1 config
 		echo $3 | grep -q next || \
 		sed -ri           's,(NEXT_ABI=)y,\1n,' $1/.config
@@ -140,21 +140,23 @@ for conf in $configs ; do
 	options=$(echo $conf | cut -d'+' -sf2- --output-delimiter='-')
 	if [ -z "$options" ] ; then
 		dir=$target
-		config $dir $target
-		# Use install rule
-		make -j$J T=$target install EXTRA_CFLAGS="$DPDK_DEP_CFLAGS" EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS"
-		$short || make -j$J T=$target examples O=$dir/examples EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS"
 	else
 		dir=$target-$options
-		config $dir $target $options
-		echo "================== Build $dir"
-		# Use O variable without install
-		make -j$J O=$dir EXTRA_CFLAGS="$DPDK_DEP_CFLAGS" EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS"
-		echo "================== Build examples for $dir"
-		make -j$J -sC examples RTE_SDK=$(pwd) RTE_TARGET=$dir O=$(readlink -m $dir/examples) EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS"
 	fi
-	echo "################## $dir done."
+	config $dir $target $options
+
+	echo "================== Build $dir"
+	make -j$J EXTRA_CFLAGS="$DPDK_DEP_CFLAGS" \
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" O=$dir
 	! $short || break
+	echo "================== Build examples for $dir"
+	export RTE_SDK=$(pwd)
+	export RTE_TARGET=$dir
+	make -j$J -sC examples \
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" \
+		O=$(readlink -m $dir/examples)
+	unset RTE_TARGET
+	echo "################## $dir done."
 done
 
 if ! $short ; then
-- 
2.7.0

[dpdk-dev] [PATCH 3/8] scripts: test build of performance-thread example

[Thomas Monjalon]

This example is not part of the baseline because of its
experimental state. That's why it must be tested separately.

Fixes: b700090c8cec ("examples/performance-thread: mark as experimental")

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index b23d6c8..ffad7ee 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -155,6 +155,10 @@ for conf in $configs ; do
 	make -j$J -sC examples \
 		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" \
 		O=$(readlink -m $dir/examples)
+	! echo $target | grep -q '^x86_64' || \
+	make -j$J -sC examples/performance-thread \
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" \
+		O=$(readlink -m $dir/examples/performance-thread)
 	unset RTE_TARGET
 	echo "################## $dir done."
 done
-- 
2.7.0

[dpdk-dev] [PATCH 4/8] scripts: stop build test after first error

[Thomas Monjalon]

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index ffad7ee..ffa7bea 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -64,6 +64,7 @@ print_help () {
 
 J=$DPDK_MAKE_JOBS
 short=false
+maxerr=-Wfatal-errors
 while getopts hj:s ARG ; do
 	case $ARG in
 		j ) J=$OPTARG ;;
@@ -146,7 +147,7 @@ for conf in $configs ; do
 	config $dir $target $options
 
 	echo "================== Build $dir"
-	make -j$J EXTRA_CFLAGS="$DPDK_DEP_CFLAGS" \
+	make -j$J EXTRA_CFLAGS="$maxerr $DPDK_DEP_CFLAGS" \
 		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" O=$dir
 	! $short || break
 	echo "================== Build examples for $dir"
-- 
2.7.0

[dpdk-dev] [PATCH 5/8] scripts: allow tuning build options per test target

[Thomas Monjalon]

The global variables are reloaded between each build to allow
having different config options based on DPDK_TARGET.

Some checks can now be removed from the script as they can
be done in the devel config file.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index ffa7bea..f28c289 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -30,6 +30,8 @@
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+default_path=$PATH
+
 # Load config options:
 # - AESNI_MULTI_BUFFER_LIB_PATH
 # - DPDK_BUILD_TEST_CONFIGS (target1+option1+option2 target2)
@@ -37,6 +39,10 @@
 # - DPDK_DEP_LDFLAGS
 # - DPDK_DEP_MOFED (y/[n])
 # - DPDK_DEP_PCAP (y/[n])
+# - DPDK_DEP_SSL (y/[n])
+# - DPDK_DEP_SZE (y/[n])
+# - DPDK_DEP_ZLIB (y/[n])
+# - DPDK_MAKE_JOBS (int)
 # - DPDK_NOTIFY (notify-send)
 . $(dirname $(readlink -e $0))/load-devel-config.sh
 
@@ -94,6 +100,21 @@ trap on_exit EXIT
 
 cd $(dirname $(readlink -m $0))/..
 
+reset_env ()
+{
+	export PATH=$default_path
+	unset CROSS
+	unset DPDK_DEP_CFLAGS
+	unset DPDK_DEP_LDFLAGS
+	unset DPDK_DEP_MOFED
+	unset DPDK_DEP_PCAP
+	unset DPDK_DEP_SSL
+	unset DPDK_DEP_SZE
+	unset DPDK_DEP_ZLIB
+	unset AESNI_MULTI_BUFFER_LIB_PATH
+	unset PQOS_INSTALL_PATH
+}
+
 config () # <directory> <target> <options>
 {
 	if [ ! -e $1/.config ] ; then
@@ -103,16 +124,14 @@ config () # <directory> <target> <options>
 		sed -ri           's,(NEXT_ABI=)y,\1n,' $1/.config
 		! echo $3 | grep -q shared || \
 		sed -ri         's,(SHARED_LIB=)n,\1y,' $1/.config
-		echo $2 | grep -q '^i686' || \
+		! echo $2 | grep -q '^x86_64' || \
 		sed -ri               's,(NUMA=)n,\1y,' $1/.config
 		sed -ri         's,(PCI_CONFIG=)n,\1y,' $1/.config
 		sed -ri    's,(LIBRTE_IEEE1588=)n,\1y,' $1/.config
 		sed -ri             's,(BYPASS=)n,\1y,' $1/.config
 		test "$DPDK_DEP_MOFED" != y || \
-		echo $2 | grep -q '^clang$' || \
 		sed -ri           's,(MLX._PMD=)n,\1y,' $1/.config
 		test "$DPDK_DEP_SZE" != y || \
-		echo $2 | grep -q '^i686' || \
 		sed -ri       's,(PMD_SZEDATA2=)n,\1y,' $1/.config
 		test "$DPDK_DEP_ZLIB" != y || \
 		sed -ri          's,(BNX2X_PMD=)n,\1y,' $1/.config
@@ -120,17 +139,13 @@ config () # <directory> <target> <options>
 		test "$DPDK_DEP_PCAP" != y || \
 		sed -ri               's,(PCAP=)n,\1y,' $1/.config
 		test -z "$AESNI_MULTI_BUFFER_LIB_PATH" || \
-		! echo $2 | grep -q '^x86_64' || \
 		sed -ri       's,(PMD_AESNI_MB=)n,\1y,' $1/.config
 		test -z "$AESNI_MULTI_BUFFER_LIB_PATH" || \
-		! echo $2 | grep -q '^x86_64' || \
 		sed -ri      's,(PMD_AESNI_GCM=)n,\1y,' $1/.config
 		test "$DPDK_DEP_SSL" != y || \
 		sed -ri            's,(PMD_QAT=)n,\1y,' $1/.config
 		sed -ri        's,(KNI_VHOST.*=)n,\1y,' $1/.config
 		sed -ri           's,(SCHED_.*=)n,\1y,' $1/.config
-		! echo $2 | grep -q '^i686' || \
-		sed -ri              's,(POWER=)y,\1n,' $1/.config
 		sed -ri 's,(TEST_PMD_RECORD_.*=)n,\1y,' $1/.config
 		sed -ri            's,(DEBUG.*=)n,\1y,' $1/.config
 	fi
@@ -138,6 +153,11 @@ config () # <directory> <target> <options>
 
 for conf in $configs ; do
 	target=$(echo $conf | cut -d'+' -f1)
+	# reload config with DPDK_TARGET set
+	DPDK_TARGET=$target
+	reset_env
+	. $(dirname $(readlink -e $0))/load-devel-config.sh
+
 	options=$(echo $conf | cut -d'+' -sf2- --output-delimiter='-')
 	if [ -z "$options" ] ; then
 		dir=$target
-- 
2.7.0

[dpdk-dev] [PATCH 6/8] scripts: allow tuning any test build option

[Thomas Monjalon]

Any build option can be enabled or disabled by appending the end
of its name to the config name.
Examples:
	+INTRINSICS to enable CONFIG_RTE_FORCE_INTRINSICS
	~RXTX_CALLBACKS to disable CONFIG_RTE_ETHDEV_RXTX_CALLBACKS

These builtin (lowercase) options are also added for convenience:
	+debug to enable every debug options
	+default to set target machine as default

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 46 ++++++++++++++++++++++++++++++++--------------
 1 file changed, 32 insertions(+), 14 deletions(-)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index f28c289..bd4b501 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -34,7 +34,7 @@ default_path=$PATH
 
 # Load config options:
 # - AESNI_MULTI_BUFFER_LIB_PATH
-# - DPDK_BUILD_TEST_CONFIGS (target1+option1+option2 target2)
+# - DPDK_BUILD_TEST_CONFIGS (defconfig1+option1+option2 defconfig2)
 # - DPDK_DEP_CFLAGS
 # - DPDK_DEP_LDFLAGS
 # - DPDK_DEP_MOFED (y/[n])
@@ -61,10 +61,14 @@ print_help () {
 	        -jX   use X parallel jobs in "make"
 	        -s    short test with only first config without examples/doc
 
-	config: defconfig name followed by switches delimited with "+" sign
-	        Example: x86_64-native-linuxapp-gcc+next+shared
-	        Default is to enable most of the options.
+	config: defconfig[[~][+]option1[[~][+]option2...]]
+	        Example: x86_64-native-linuxapp-gcc+debug~RXTX_CALLBACKS
+	        The lowercase options are defined inside $(basename $0).
+	        The uppercase options can be the end of a defconfig option
+	        to enable if prefixed with '+' or to disable if prefixed with '~'.
+	        Default is to automatically enable most of the options.
 	        The external dependencies are setup with DPDK_DEP_* variables.
+	        If no config on command line, DPDK_BUILD_TEST_CONFIGS is used.
 	END_OF_HELP
 }
 
@@ -120,10 +124,19 @@ config () # <directory> <target> <options>
 	if [ ! -e $1/.config ] ; then
 		echo "================== Configure $1"
 		make T=$2 O=$1 config
-		echo $3 | grep -q next || \
+
+		echo 'Customize configuration'
+		# Built-in options (lowercase)
+		! echo $3 | grep -q '+default' || \
+		sed -ri 's,(RTE_MACHINE=")native,\1default,' $1/.config
+		echo $3 | grep -q '+next' || \
 		sed -ri           's,(NEXT_ABI=)y,\1n,' $1/.config
-		! echo $3 | grep -q shared || \
+		! echo $3 | grep -q '+shared' || \
 		sed -ri         's,(SHARED_LIB=)n,\1y,' $1/.config
+		! echo $3 | grep -q '+debug' || \
+		sed -ri            's,(DEBUG.*=)n,\1y,' $1/.config
+
+		# Automatic configuration
 		! echo $2 | grep -q '^x86_64' || \
 		sed -ri               's,(NUMA=)n,\1y,' $1/.config
 		sed -ri         's,(PCI_CONFIG=)n,\1y,' $1/.config
@@ -147,23 +160,28 @@ config () # <directory> <target> <options>
 		sed -ri        's,(KNI_VHOST.*=)n,\1y,' $1/.config
 		sed -ri           's,(SCHED_.*=)n,\1y,' $1/.config
 		sed -ri 's,(TEST_PMD_RECORD_.*=)n,\1y,' $1/.config
-		sed -ri            's,(DEBUG.*=)n,\1y,' $1/.config
+
+		# Explicit enabler/disabler (uppercase)
+		for option in $(echo $3 | sed 's,[~+], &,g') ; do
+			pattern=$(echo $option | cut -c2-)
+			if echo $option | grep -q '^~' ; then
+				sed -ri "s,($pattern=)y,\1n," $1/.config
+			elif echo $option | grep -q '^+' ; then
+				sed -ri "s,($pattern=)n,\1y," $1/.config
+			fi
+		done
 	fi
 }
 
 for conf in $configs ; do
-	target=$(echo $conf | cut -d'+' -f1)
+	target=$(echo $conf | sed 's,[~+].*,,')
 	# reload config with DPDK_TARGET set
 	DPDK_TARGET=$target
 	reset_env
 	. $(dirname $(readlink -e $0))/load-devel-config.sh
 
-	options=$(echo $conf | cut -d'+' -sf2- --output-delimiter='-')
-	if [ -z "$options" ] ; then
-		dir=$target
-	else
-		dir=$target-$options
-	fi
+	options=$(echo $conf | sed 's,[^~+]*,,')
+	dir=$conf
 	config $dir $target $options
 
 	echo "================== Build $dir"
-- 
2.7.0

[dpdk-dev] [PATCH 7/8] scripts: hook build test config

[Thomas Monjalon]

Insert a hook at the end of the config procedure, after having
adapted the configuration to the environment variables and the options
passed to the script.
It allows to better tune the automatic configuration of the build tests
in a function located in the devel config file.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index bd4b501..808e8e4 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -160,6 +160,7 @@ config () # <directory> <target> <options>
 		sed -ri        's,(KNI_VHOST.*=)n,\1y,' $1/.config
 		sed -ri           's,(SCHED_.*=)n,\1y,' $1/.config
 		sed -ri 's,(TEST_PMD_RECORD_.*=)n,\1y,' $1/.config
+		build_config_hook $1 $2 $3
 
 		# Explicit enabler/disabler (uppercase)
 		for option in $(echo $3 | sed 's,[~+], &,g') ; do
@@ -173,6 +174,12 @@ config () # <directory> <target> <options>
 	fi
 }
 
+# default empty hook to override in devel config
+build_config_hook () # <directory> <target> <options>
+{
+	:
+}
+
 for conf in $configs ; do
 	target=$(echo $conf | sed 's,[~+].*,,')
 	# reload config with DPDK_TARGET set
-- 
2.7.0

[dpdk-dev] [PATCH 8/8] scripts: add verbose test build option

[Thomas Monjalon]

The option -v enables the verbose mode when testing a build.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 scripts/test-build.sh | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index 808e8e4..a486244 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -74,11 +74,13 @@ print_help () {
 
 J=$DPDK_MAKE_JOBS
 short=false
+unset verbose
 maxerr=-Wfatal-errors
-while getopts hj:s ARG ; do
+while getopts hj:sv ARG ; do
 	case $ARG in
 		j ) J=$OPTARG ;;
 		s ) short=true ;;
+		v ) verbose='V=1' ;;
 		h ) print_help ; exit 0 ;;
 		? ) print_usage ; exit 1 ;;
 	esac
@@ -193,17 +195,17 @@ for conf in $configs ; do
 
 	echo "================== Build $dir"
 	make -j$J EXTRA_CFLAGS="$maxerr $DPDK_DEP_CFLAGS" \
-		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" O=$dir
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose O=$dir
 	! $short || break
 	echo "================== Build examples for $dir"
 	export RTE_SDK=$(pwd)
 	export RTE_TARGET=$dir
 	make -j$J -sC examples \
-		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" \
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
 		O=$(readlink -m $dir/examples)
 	! echo $target | grep -q '^x86_64' || \
 	make -j$J -sC examples/performance-thread \
-		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" \
+		EXTRA_LDFLAGS="$DPDK_DEP_LDFLAGS" $verbose \
 		O=$(readlink -m $dir/examples/performance-thread)
 	unset RTE_TARGET
 	echo "################## $dir done."
-- 
2.7.0

Re: [dpdk-dev] [PATCH 0/8] simpler and better build test

[Thomas Monjalon]

2016-03-29 18:15, Thomas Monjalon:
> These are some improvements to the script test-build.sh which
> makes compilation testing easier in various environments with
> different dependencies.
> 
> Thomas Monjalon (8):
>   scripts: fix run in any directory
>   scripts: remove legacy build method test
>   scripts: test build of performance-thread example
>   scripts: stop build test after first error
>   scripts: allow tuning build options per test target
>   scripts: allow tuning any test build option
>   scripts: hook build test config
>   scripts: add verbose test build option

Applied