[dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

When testing compilation and checking ABI compatibility,
there is no real need of static binaries eating disks.
The static linkage of applications are tested with GCC and Clang,
plus some examples are statically linked.
The after-installation build test is limited to "helloworld" example.
Note the meson static build test was already limited to "l3fwd" example.

The ABI compatibility is checked on shared libraries, so no need
running this test a second time on builds intended for static linking.
However, limiting ABI check to "shared builds" means all test cases
must have a "shared build" occurence.
As a consequence the 32-bit build test is switched to shared linking.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 devtools/test-meson-builds.sh | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index 7280b7a93d..ed44d4ffb1 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -166,6 +166,9 @@ build () # <directory> <target compiler | cross file> <meson options>
 	config $srcdir $builds_dir/$targetdir $cross --werror $*
 	compile $builds_dir/$targetdir
 	if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+		if echo $* | grep -qw -- '--default-library=static' ; then
+			return # skip ABI check for static build
+		fi
 		abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION
 		if [ ! -d $abirefdir/$targetdir ]; then
 			# clone current sources
@@ -230,7 +233,7 @@ if check_cc_flags '-m32' ; then
 		export PKG_CONFIG_LIBDIR='/usr/lib/pkgconfig'
 	fi
 	target_override='i386-pc-linux-gnu'
-	build build-32b cc -Dc_args='-m32' -Dc_link_args='-m32'
+	build build-32b cc -Dc_args='-m32' -Dc_link_args='-m32' $use_shared
 	target_override=
 	unset PKG_CONFIG_LIBDIR
 fi
@@ -274,7 +277,8 @@ if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
 	export PKGCONF="pkg-config --define-prefix"
 	for example in $examples; do
 		echo "## Building $example"
+		[ $example = helloworld ] && static=static || static= # save disk space
 		$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example \
-			clean shared static >&$veryverbose
+			clean shared $static >&$veryverbose
 	done
 fi
-- 
2.29.2

Re: [dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[Bruce Richardson]

On Mon, Dec 07, 2020 at 06:33:19PM +0100, Thomas Monjalon wrote:
> When testing compilation and checking ABI compatibility,
> there is no real need of static binaries eating disks.
> The static linkage of applications are tested with GCC and Clang,
> plus some examples are statically linked.
> The after-installation build test is limited to "helloworld" example.
> Note the meson static build test was already limited to "l3fwd" example.
> 
> The ABI compatibility is checked on shared libraries, so no need
> running this test a second time on builds intended for static linking.
> However, limiting ABI check to "shared builds" means all test cases
> must have a "shared build" occurence.
> As a consequence the 32-bit build test is switched to shared linking.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>  devtools/test-meson-builds.sh | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)

I think this might be better as two patches - one for the ABI check changes
and a second for the example build changes with installed DPDK.

> 
> diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
> index 7280b7a93d..ed44d4ffb1 100755
> --- a/devtools/test-meson-builds.sh
> +++ b/devtools/test-meson-builds.sh
> @@ -166,6 +166,9 @@ build () # <directory> <target compiler | cross file> <meson options>
>  	config $srcdir $builds_dir/$targetdir $cross --werror $*
>  	compile $builds_dir/$targetdir
>  	if [ -n "$DPDK_ABI_REF_VERSION" ]; then
> +		if echo $* | grep -qw -- '--default-library=static' ; then
> +			return # skip ABI check for static build
> +		fi
>  		abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION
>  		if [ ! -d $abirefdir/$targetdir ]; then
>  			# clone current sources
> @@ -230,7 +233,7 @@ if check_cc_flags '-m32' ; then
>  		export PKG_CONFIG_LIBDIR='/usr/lib/pkgconfig'
>  	fi
>  	target_override='i386-pc-linux-gnu'
> -	build build-32b cc -Dc_args='-m32' -Dc_link_args='-m32'
> +	build build-32b cc -Dc_args='-m32' -Dc_link_args='-m32' $use_shared
>  	target_override=
>  	unset PKG_CONFIG_LIBDIR
>  fi
> @@ -274,7 +277,8 @@ if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
>  	export PKGCONF="pkg-config --define-prefix"
>  	for example in $examples; do
>  		echo "## Building $example"
> +		[ $example = helloworld ] && static=static || static= # save disk space
>  		$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example \
> -			clean shared static >&$veryverbose
> +			clean shared $static >&$veryverbose
>  	done
>  fi

Just wonder are we likely to miss things with this change? Would changing
the order to do a clean at the end to free back up the disk space not
achieve much the same result while still saving disk space?

Re: [dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

07/12/2020 18:47, Bruce Richardson:
> On Mon, Dec 07, 2020 at 06:33:19PM +0100, Thomas Monjalon wrote:
> > When testing compilation and checking ABI compatibility,
> > there is no real need of static binaries eating disks.
> > The static linkage of applications are tested with GCC and Clang,
> > plus some examples are statically linked.
> > The after-installation build test is limited to "helloworld" example.
> > Note the meson static build test was already limited to "l3fwd" example.
> > 
> > The ABI compatibility is checked on shared libraries, so no need
> > running this test a second time on builds intended for static linking.
> > However, limiting ABI check to "shared builds" means all test cases
> > must have a "shared build" occurence.
> > As a consequence the 32-bit build test is switched to shared linking.
> > 
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> >  devtools/test-meson-builds.sh | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> I think this might be better as two patches - one for the ABI check changes
> and a second for the example build changes with installed DPDK.

Yes could be 2 patches.


> >  	for example in $examples; do
> >  		echo "## Building $example"
> > +		[ $example = helloworld ] && static=static || static= # save disk space
> >  		$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example \
> > -			clean shared static >&$veryverbose
> > +			clean shared $static >&$veryverbose
> >  	done
> >  fi
> 
> Just wonder are we likely to miss things with this change? Would changing
> the order to do a clean at the end to free back up the disk space not
> achieve much the same result while still saving disk space?

Not building static flavour of most examples is also faster.
Ideally we should not rebuild an example if the libs did not change.

To the question "will we miss something", the difference between static
and shared examples is just the pkg-config call in the Makefile.
I think the risk is small.

Re: [dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[Bruce Richardson]

On Mon, Dec 07, 2020 at 07:12:32PM +0100, Thomas Monjalon wrote:
> 07/12/2020 18:47, Bruce Richardson:
> > On Mon, Dec 07, 2020 at 06:33:19PM +0100, Thomas Monjalon wrote:
> > > When testing compilation and checking ABI compatibility,
> > > there is no real need of static binaries eating disks.
> > > The static linkage of applications are tested with GCC and Clang,
> > > plus some examples are statically linked.
> > > The after-installation build test is limited to "helloworld" example.
> > > Note the meson static build test was already limited to "l3fwd" example.
> > > 
> > > The ABI compatibility is checked on shared libraries, so no need
> > > running this test a second time on builds intended for static linking.
> > > However, limiting ABI check to "shared builds" means all test cases
> > > must have a "shared build" occurence.
> > > As a consequence the 32-bit build test is switched to shared linking.
> > > 
> > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > ---
> > >  devtools/test-meson-builds.sh | 8 ++++++--
> > >  1 file changed, 6 insertions(+), 2 deletions(-)
> > 
> > I think this might be better as two patches - one for the ABI check changes
> > and a second for the example build changes with installed DPDK.
> 
> Yes could be 2 patches.
> 
> 
> > >  	for example in $examples; do
> > >  		echo "## Building $example"
> > > +		[ $example = helloworld ] && static=static || static= # save disk space
> > >  		$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example \
> > > -			clean shared static >&$veryverbose
> > > +			clean shared $static >&$veryverbose
> > >  	done
> > >  fi
> > 
> > Just wonder are we likely to miss things with this change? Would changing
> > the order to do a clean at the end to free back up the disk space not
> > achieve much the same result while still saving disk space?
> 
> Not building static flavour of most examples is also faster.
> Ideally we should not rebuild an example if the libs did not change.
> 
> To the question "will we miss something", the difference between static
> and shared examples is just the pkg-config call in the Makefile.
> I think the risk is small.
> 
Yes, for the majority of the apps that is the case. However, the only
concern I have is for a number of the apps which link directly against a
driver or two. Looking at vm_power_manager example, which links against 3
drivers, I see that the extra flags are only added for shared builds so we
should be ok for that one anyway.

Therefore ok with this change exactly as you suggest.

/Bruce

Re: [dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[David Marchand]

On Mon, Dec 7, 2020 at 6:33 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> When testing compilation and checking ABI compatibility,
> there is no real need of static binaries eating disks.
> The static linkage of applications are tested with GCC and Clang,
> plus some examples are statically linked.
> The after-installation build test is limited to "helloworld" example.
> Note the meson static build test was already limited to "l3fwd" example.
>
> The ABI compatibility is checked on shared libraries, so no need
> running this test a second time on builds intended for static linking.
> However, limiting ABI check to "shared builds" means all test cases
> must have a "shared build" occurence.
> As a consequence the 32-bit build test is switched to shared linking.

I see no reason to tie the ABI check to default-library.

What about the mingw target?


What you want is to avoid doing duplicate ABI checks.
This happens for the gcc/clang x86 builds, so I'd rather control the
ABI checks out of the build() function (passing a new parameter?).

-- 
David Marchand

Re: [dpdk-dev] [PATCH 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

08/12/2020 16:37, David Marchand:
> On Mon, Dec 7, 2020 at 6:33 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > When testing compilation and checking ABI compatibility,
> > there is no real need of static binaries eating disks.
> > The static linkage of applications are tested with GCC and Clang,
> > plus some examples are statically linked.
> > The after-installation build test is limited to "helloworld" example.
> > Note the meson static build test was already limited to "l3fwd" example.
> >
> > The ABI compatibility is checked on shared libraries, so no need
> > running this test a second time on builds intended for static linking.
> > However, limiting ABI check to "shared builds" means all test cases
> > must have a "shared build" occurence.
> > As a consequence the 32-bit build test is switched to shared linking.
> 
> I see no reason to tie the ABI check to default-library.

The only reason is that ABI check triggers binary installation,
which is big when statically linked.

> What about the mingw target?

ABI check is not required for Windows.
BTW there are issues with DLL support.

> What you want is to avoid doing duplicate ABI checks.
> This happens for the gcc/clang x86 builds, so I'd rather control the
> ABI checks out of the build() function (passing a new parameter?).

Yes, it would be cleaner to separate ABI check requirement
and static linking.
In v2, ABI check will be enabled explicitly when calling "build" function
for shared builds.

[dpdk-dev] [PATCH v2 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

When testing compilation and checking ABI compatibility,
there is no real need of static binaries eating disks.

The static linkage of applications was already well tested,
though the static examples tested with meson were limited to "l3fwd" only.
The static build test with make is limited to "helloworld" example.

The ABI compatibility is checked on shared libraries,
and there is no need to test again on similar builds.
A new parameter is added to the function "build",
so the ABI check is enabled only for native gcc and clang shared builds,
32-bit, generic armv8 and ppc cross compilations.
In other words, it is disabled for some static builds and some Arm ones.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
v2:
- separate ABI check enablement from default library
- disable ABI check in specific Arm builds
---
 devtools/test-meson-builds.sh | 32 ++++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
index 00e3d0b443..0e79e1b2bd 100755
--- a/devtools/test-meson-builds.sh
+++ b/devtools/test-meson-builds.sh
@@ -146,13 +146,15 @@ install_target () # <builddir> <installdir>
 	DESTDIR=$2 $ninja_cmd -C $1 install >&$veryverbose
 }
 
-build () # <directory> <target compiler | cross file> <meson options>
+build () # <directory> <target cc | cross file> <ABI check> [meson options]
 {
 	targetdir=$1
 	shift
 	crossfile=
 	[ -r $1 ] && crossfile=$1 || targetcc=$1
 	shift
+	abicheck=$1
+	shift
 	# skip build if compiler not available
 	command -v ${CC##* } >/dev/null 2>&1 || return 0
 	if [ -n "$crossfile" ] ; then
@@ -165,7 +167,7 @@ build () # <directory> <target compiler | cross file> <meson options>
 	load_env $targetcc || return 0
 	config $srcdir $builds_dir/$targetdir $cross --werror $*
 	compile $builds_dir/$targetdir
-	if [ -n "$DPDK_ABI_REF_VERSION" ]; then
+	if [ -n "$DPDK_ABI_REF_VERSION" -a "$abicheck" = ABI ] ; then
 		abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION
 		if [ ! -d $abirefdir/$targetdir ]; then
 			# clone current sources
@@ -207,8 +209,13 @@ build () # <directory> <target compiler | cross file> <meson options>
 for c in gcc clang ; do
 	command -v $c >/dev/null 2>&1 || continue
 	for s in static shared ; do
+		if [ $s = shared ] ; then
+			abicheck=ABI
+		else
+			abicheck=skipABI # save time and disk space
+		fi
 		export CC="$CCACHE $c"
-		build build-$c-$s $c --default-library=$s
+		build build-$c-$s $c $abicheck --default-library=$s
 		unset CC
 	done
 done
@@ -220,7 +227,8 @@ default_machine='nehalem'
 if ! check_cc_flags "-march=$default_machine" ; then
 	default_machine='corei7'
 fi
-build build-x86-default cc -Dlibdir=lib -Dmachine=$default_machine $use_shared
+build build-x86-default cc ABI \
+	-Dlibdir=lib -Dmachine=$default_machine $use_shared
 
 # 32-bit with default compiler
 if check_cc_flags '-m32' ; then
@@ -235,29 +243,32 @@ if check_cc_flags '-m32' ; then
 		export PKG_CONFIG_LIBDIR='/usr/lib/pkgconfig'
 	fi
 	target_override='i386-pc-linux-gnu'
-	build build-32b cc -Dc_args='-m32' -Dc_link_args='-m32'
+	build build-32b cc ABI -Dc_args='-m32' -Dc_link_args='-m32'
 	target_override=
 	unset PKG_CONFIG_LIBDIR
 fi
 
 # x86 MinGW
-build build-x86-mingw $srcdir/config/x86/cross-mingw -Dexamples=helloworld
+build build-x86-mingw $srcdir/config/x86/cross-mingw skipABI \
+	-Dexamples=helloworld
 
 # generic armv8a with clang as host compiler
 f=$srcdir/config/arm/arm64_armv8_linux_gcc
 export CC="clang"
-build build-arm64-host-clang $f $use_shared
+build build-arm64-host-clang $f ABI $use_shared
 unset CC
 # some gcc/arm configurations
 for f in $srcdir/config/arm/arm64_[bdo]*gcc ; do
 	export CC="$CCACHE gcc"
-	build build-$(basename $f | tr '_' '-' | cut -d'-' -f-2) $f $use_shared
+	targetdir=build-$(basename $f | tr '_' '-' | cut -d'-' -f-2)
+	build $targetdir $f skipABI $use_shared
 	unset CC
 done
 
 # ppc configurations
 for f in $srcdir/config/ppc/ppc* ; do
-	build build-$(basename $f | cut -d'-' -f-2) $f $use_shared
+	targetdir=build-$(basename $f | cut -d'-' -f-2)
+	build $targetdir $f ABI $use_shared
 done
 
 # Test installation of the x86-default target, to be used for checking
@@ -279,7 +290,8 @@ if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
 	export PKGCONF="pkg-config --define-prefix"
 	for example in $examples; do
 		echo "## Building $example"
+		[ $example = helloworld ] && static=static || static= # save disk space
 		$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example \
-			clean shared static >&$veryverbose
+			clean shared $static >&$veryverbose
 	done
 fi
-- 
2.29.2

Re: [dpdk-dev] [PATCH v2 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

13/01/2021 20:05, Thomas Monjalon:
> When testing compilation and checking ABI compatibility,
> there is no real need of static binaries eating disks.
> 
> The static linkage of applications was already well tested,
> though the static examples tested with meson were limited to "l3fwd" only.
> The static build test with make is limited to "helloworld" example.
> 
> The ABI compatibility is checked on shared libraries,
> and there is no need to test again on similar builds.
> A new parameter is added to the function "build",
> so the ABI check is enabled only for native gcc and clang shared builds,
> 32-bit, generic armv8 and ppc cross compilations.
> In other words, it is disabled for some static builds and some Arm ones.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> v2:
> - separate ABI check enablement from default library
> - disable ABI check in specific Arm builds
> ---
[...]
> -build build-x86-default cc -Dlibdir=lib -Dmachine=$default_machine $use_shared
> +build build-x86-default cc ABI \
> +	-Dlibdir=lib -Dmachine=$default_machine $use_shared

After a second thought, I think this one should be "skipABI".

Re: [dpdk-dev] [PATCH v2 1/1] devtools: avoid installing static binaries

[David Marchand]

On Wed, Jan 13, 2021 at 11:01 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 13/01/2021 20:05, Thomas Monjalon:
> > When testing compilation and checking ABI compatibility,
> > there is no real need of static binaries eating disks.
> >
> > The static linkage of applications was already well tested,
> > though the static examples tested with meson were limited to "l3fwd" only.
> > The static build test with make is limited to "helloworld" example.
> >
> > The ABI compatibility is checked on shared libraries,
> > and there is no need to test again on similar builds.
> > A new parameter is added to the function "build",
> > so the ABI check is enabled only for native gcc and clang shared builds,
> > 32-bit, generic armv8 and ppc cross compilations.
> > In other words, it is disabled for some static builds and some Arm ones.
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > v2:
> > - separate ABI check enablement from default library
> > - disable ABI check in specific Arm builds
> > ---
> [...]
> > -build build-x86-default cc -Dlibdir=lib -Dmachine=$default_machine $use_shared
> > +build build-x86-default cc ABI \
> > +     -Dlibdir=lib -Dmachine=$default_machine $use_shared
>
> After a second thought, I think this one should be "skipABI".

No opinion on this one.

The title might need some tweak, since you also disabled the ABI check
on some ARM targets.
The rest lgtm.

Acked-by: David Marchand <david.marchand@redhat.com>

-- 
David Marchand

Re: [dpdk-dev] [PATCH v2 1/1] devtools: avoid installing static binaries

[Thomas Monjalon]

15/01/2021 16:24, David Marchand:
> On Wed, Jan 13, 2021 at 11:01 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > 13/01/2021 20:05, Thomas Monjalon:
> > > When testing compilation and checking ABI compatibility,
> > > there is no real need of static binaries eating disks.
> > >
> > > The static linkage of applications was already well tested,
> > > though the static examples tested with meson were limited to "l3fwd" only.
> > > The static build test with make is limited to "helloworld" example.
> > >
> > > The ABI compatibility is checked on shared libraries,
> > > and there is no need to test again on similar builds.
> > > A new parameter is added to the function "build",
> > > so the ABI check is enabled only for native gcc and clang shared builds,
> > > 32-bit, generic armv8 and ppc cross compilations.
> > > In other words, it is disabled for some static builds and some Arm ones.
> > >
> > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > ---
> > > v2:
> > > - separate ABI check enablement from default library
> > > - disable ABI check in specific Arm builds
> > > ---
> > [...]
> > > -build build-x86-default cc -Dlibdir=lib -Dmachine=$default_machine $use_shared
> > > +build build-x86-default cc ABI \
> > > +     -Dlibdir=lib -Dmachine=$default_machine $use_shared
> >
> > After a second thought, I think this one should be "skipABI".
> 
> No opinion on this one.
> 
> The title might need some tweak, since you also disabled the ABI check
> on some ARM targets.

Yes, you're right.
Disabling some ABI checks is a way to reduce the number
of static binaries, but it should be visible in the title.

> The rest lgtm.
> 
> Acked-by: David Marchand <david.marchand@redhat.com>

Applied with title "devtools: reduce ABI checks and static binaries"