From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4AEE8A04B7; Wed, 14 Oct 2020 11:43:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DF22D1DDB8; Wed, 14 Oct 2020 11:43:46 +0200 (CEST) Received: from mail-108-mta187.mxroute.com (mail-108-mta187.mxroute.com [136.175.108.187]) by dpdk.org (Postfix) with ESMTP id 492901DDB6 for ; Wed, 14 Oct 2020 11:43:45 +0200 (CEST) Received: from filter004.mxroute.com ([149.28.56.236] 149.28.56.236.vultr.com) (Authenticated sender: mN4UYu2MZsgR) by mail-108-mta187.mxroute.com (ZoneMTA) with ESMTPSA id 175267e667d00027ef.001 for (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256); Wed, 14 Oct 2020 09:43:39 +0000 X-Zone-Loop: 20ee4aa812631ad1260446d464321d9dcec081d2fe7a X-Originating-IP: [149.28.56.236] Received: from echo.mxrouting.net (echo.mxrouting.net [116.202.222.109]) by filter004.mxroute.com (Postfix) with ESMTPS id 387363EAE0; Wed, 14 Oct 2020 09:43:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=UjtD8mseLjvH/Gw54Ja5AYCY4u9rw2N+Lamg8T3y3uQ=; b=LYBD7d21wztXBK/KL0sNgPuiot FF2+eg6oyX5W3scjLl0shjq//wngUzyPxD6Q/Y5EzzHuHR94YnYbNrBmO6wrxs9WloFkq2UZm/NOL Hxp9fjInhXl0QNKVIrcv52y1rRGlIvuna+JhSFVQAgXe+KjvpXtHpp/H0xyFcc1uXw5nvkHlSvw/9 r6nTG+tpekvc603vrYIf10vtGrK4oz1fVHDd4Hyi7rUqhayvhpio9oOcG1SgkCoxIF+UepmgrEYEy bJE8NC1T9uFVTT7H9NKd8AkY3iXqZ0a7LcOGOY2ejZWp9bqRwT46m8eRkMDu8pBwJqDbTUDMGxScM Ph7PAbLg==; To: Conor Walsh , nhorman@tuxdriver.com, bruce.richardson@intel.com, thomas@monjalon.net, david.marchand@redhat.com Cc: dev@dpdk.org References: <20201012080829.3446-1-conor.walsh@intel.com> <20201012130348.3212-1-conor.walsh@intel.com> <20201012130348.3212-3-conor.walsh@intel.com> From: "Kinsella, Ray" Autocrypt: addr=mdr@ashroe.eu; keydata= mQINBFv8B3wBEAC+5ImcgbIvadt3axrTnt7Sxch3FsmWTTomXfB8YiuHT8KL8L/bFRQSL1f6 ASCHu3M89EjYazlY+vJUWLr0BhK5t/YI7bQzrOuYrl9K94vlLwzD19s/zB/g5YGGR5plJr0s JtJsFGEvF9LL3e+FKMRXveQxBB8A51nAHfwG0WSyx53d61DYz7lp4/Y4RagxaJoHp9lakn8j HV2N6rrnF+qt5ukj5SbbKWSzGg5HQF2t0QQ5tzWhCAKTfcPlnP0GymTBfNMGOReWivi3Qqzr S51Xo7hoGujUgNAM41sxpxmhx8xSwcQ5WzmxgAhJ/StNV9cb3HWIoE5StCwQ4uXOLplZNGnS uxNdegvKB95NHZjRVRChg/uMTGpg9PqYbTIFoPXjuk27sxZLRJRrueg4tLbb3HM39CJwSB++ YICcqf2N+GVD48STfcIlpp12/HI+EcDSThzfWFhaHDC0hyirHxJyHXjnZ8bUexI/5zATn/ux TpMbc/vicJxeN+qfaVqPkCbkS71cHKuPluM3jE8aNCIBNQY1/j87k5ELzg3qaesLo2n1krBH bKvFfAmQuUuJT84/IqfdVtrSCTabvDuNBDpYBV0dGbTwaRfE7i+LiJJclUr8lOvHUpJ4Y6a5 0cxEPxm498G12Z3NoY/mP5soItPIPtLR0rA0fage44zSPwp6cQARAQABtBxSYXkgS2luc2Vs bGEgPG1kckBhc2hyb2UuZXU+iQJUBBMBCAA+FiEEcDUDlKDJaDuJlfZfdJdaH/sCCpsFAlv8 B3wCGyMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQdJdaH/sCCptdtRAAl0oE msa+djBVYLIsax+0f8acidtWg2l9f7kc2hEjp9h9aZCpPchQvhhemtew/nKavik3RSnLTAyn B3C/0GNlmvI1l5PFROOgPZwz4xhJKGN7jOsRrbkJa23a8ly5UXwF3Vqnlny7D3z+7cu1qq/f VRK8qFyWkAb+xgqeZ/hTcbJUWtW+l5Zb+68WGEp8hB7TuJLEWb4+VKgHTpQ4vElYj8H3Z94a 04s2PJMbLIZSgmKDASnyrKY0CzTpPXx5rSJ1q+B1FCsfepHLqt3vKSALa3ld6bJ8fSJtDUJ7 JLiU8dFZrywgDIVme01jPbjJtUScW6jONLvhI8Z2sheR71UoKqGomMHNQpZ03ViVWBEALzEt TcjWgJFn8yAmxqM4nBnZ+hE3LbMo34KCHJD4eg18ojDt3s9VrDLa+V9fNxUHPSib9FD9UX/1 +nGfU/ZABmiTuUDM7WZdXri7HaMpzDRJUKI6b+/uunF8xH/h/MHW16VuMzgI5dkOKKv1LejD dT5mA4R+2zBS+GsM0oa2hUeX9E5WwjaDzXtVDg6kYq8YvEd+m0z3M4e6diFeLS77/sAOgaYL 92UcoKD+Beym/fVuC6/55a0e12ksTmgk5/ZoEdoNQLlVgd2INtvnO+0k5BJcn66ZjKn3GbEC VqFbrnv1GnA58nEInRCTzR1k26h9nmS5Ag0EW/wHfAEQAMth1vHr3fOZkVOPfod3M6DkQir5 xJvUW5EHgYUjYCPIa2qzgIVVuLDqZgSCCinyooG5dUJONVHj3nCbITCpJp4eB3PI84RPfDcC hf/V34N/Gx5mTeoymSZDBmXT8YtvV/uJvn+LvHLO4ZJdvq5ZxmDyxfXFmkm3/lLw0+rrNdK5 pt6OnVlCqEU9tcDBezjUwDtOahyV20XqxtUttN4kQWbDRkhT+HrA9WN9l2HX91yEYC+zmF1S OhBqRoTPLrR6g4sCWgFywqztpvZWhyIicJipnjac7qL/wRS+wrWfsYy6qWLIV80beN7yoa6v ccnuy4pu2uiuhk9/edtlmFE4dNdoRf7843CV9k1yRASTlmPkU59n0TJbw+okTa9fbbQgbIb1 pWsAuicRHyLUIUz4f6kPgdgty2FgTKuPuIzJd1s8s6p2aC1qo+Obm2gnBTduB+/n1Jw+vKpt 07d+CKEKu4CWwvZZ8ktJJLeofi4hMupTYiq+oMzqH+V1k6QgNm0Da489gXllU+3EFC6W1qKj tkvQzg2rYoWeYD1Qn8iXcO4Fpk6wzylclvatBMddVlQ6qrYeTmSbCsk+m2KVrz5vIyja0o5Y yfeN29s9emXnikmNfv/dA5fpi8XCANNnz3zOfA93DOB9DBf0TQ2/OrSPGjB3op7RCfoPBZ7u AjJ9dM7VABEBAAGJAjwEGAEIACYWIQRwNQOUoMloO4mV9l90l1of+wIKmwUCW/wHfAIbDAUJ CWYBgAAKCRB0l1of+wIKm3KlD/9w/LOG5rtgtCUWPl4B3pZvGpNym6XdK8cop9saOnE85zWf u+sKWCrxNgYkYP7aZrYMPwqDvilxhbTsIJl5HhPgpTO1b0i+c0n1Tij3EElj5UCg3q8mEc17 c+5jRrY3oz77g7E3oPftAjaq1ybbXjY4K32o3JHFR6I8wX3m9wJZJe1+Y+UVrrjY65gZFxcA thNVnWKErarVQGjeNgHV4N1uF3pIx3kT1N4GSnxhoz4Bki91kvkbBhUgYfNflGURfZT3wIKK +d50jd7kqRouXUCzTdzmDh7jnYrcEFM4nvyaYu0JjSS5R672d9SK5LVIfWmoUGzqD4AVmUW8 pcv461+PXchuS8+zpltR9zajl72Q3ymlT4BTAQOlCWkD0snBoKNUB5d2EXPNV13nA0qlm4U2 GpROfJMQXjV6fyYRvttKYfM5xYKgRgtP0z5lTAbsjg9WFKq0Fndh7kUlmHjuAIwKIV4Tzo75 QO2zC0/NTaTjmrtiXhP+vkC4pcrOGNsbHuaqvsc/ZZ0siXyYsqbctj/sCd8ka2r94u+c7o4l BGaAm+FtwAfEAkXHu4y5Phuv2IRR+x1wTey1U1RaEPgN8xq0LQ1OitX4t2mQwjdPihZQBCnZ wzOrkbzlJMNrMKJpEgulmxAHmYJKgvZHXZXtLJSejFjR0GdHJcL5rwVOMWB8cg== Message-ID: <219edcae-a8cd-1b53-54a9-bcef61359094@ashroe.eu> Date: Wed, 14 Oct 2020 10:43:35 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: <20201012130348.3212-3-conor.walsh@intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH v6 2/4] devtools: abi and UX changes for test-meson-builds.sh X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 12/10/2020 14:03, Conor Walsh wrote: > This patch adds new features to test-meson-builds.sh that help to make > the process of using the script easier, the patch also includes > changes to make the abi breakage checks more performant. Avoid commentary such as the above. I reduce the following list of bullets to a single paragraph describing the change. The core of this change is to improve build times. So describe reducing the number of build to 2 and using the pre-build references, and thats it. > Changes/Additions: > - Command line arguments added, the changes are fully backwards > compatible and all previous environmental variables are still supported > - All paths supplied by user are converted to absolute paths if they > are relative as meson has a bug that can sometimes error if a > relative path is supplied to it. > - abi check/generation code moved to function to improve readability > - Only 2 abi checks will now be completed: > - 1 x86_64 gcc or clang check > - 1 ARM gcc or clang check > It is not necessary to check abi breakages in every build > - abi checks can now make use of prebuilt abi references from a http > or local source, it is hoped these would be hosted on dpdk.org in > the future. > Invoke using "./test-meson-builds.sh [-b ] > [-a ] [-u ] > [-d ]" > - : directory to store builds (relative or absolute) > - : dpdk tag e.g. "v20.11" or "latest" > - : http location or directory to get prebuilt > abi references from > - : directory to store abi references > (relative or absolute) > e.g. "./test-meson-builds.sh -a latest" > If no flags are specified test-meson-builds.sh will run the standard > meson tests with default options unless environmental variables are > specified. > > Signed-off-by: Conor Walsh > > --- > devtools/test-meson-builds.sh | 170 +++++++++++++++++++++++++++------- > 1 file changed, 138 insertions(+), 32 deletions(-) > > diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh > index a87de635a..b45506fb0 100755 > --- a/devtools/test-meson-builds.sh > +++ b/devtools/test-meson-builds.sh > @@ -1,12 +1,73 @@ > #! /bin/sh -e > # SPDX-License-Identifier: BSD-3-Clause > -# Copyright(c) 2018 Intel Corporation > +# Copyright(c) 2018-2020 Intel Corporation > > # Run meson to auto-configure the various builds. > # * all builds get put in a directory whose name starts with "build-" > # * if a build-directory already exists we assume it was properly configured > # Run ninja after configuration is done. > > +# Get arguments > +usage() > +{ > + echo "Usage: $0 > + [-b ] > + [-a ] > + [-u ] > + [-d ]" 1>&2; exit 1; > +} > + > +DPDK_ABI_DEFAULT_URI="http://dpdk.org/abi-refs" > + > +while getopts "a:u:d:b:h" arg; do > + case $arg in > + a) > + if [ -n "$DPDK_ABI_REF_VERSION" ]; then > + echo "DPDK_ABI_REF_VERSION and -a cannot both be set" > + exit 1 > + fi > + DPDK_ABI_REF_VERSION=${OPTARG} ;; > + u) > + if [ -n "$DPDK_ABI_TAR_URI" ]; then > + echo "DPDK_ABI_TAR_URI and -u cannot both be set" > + exit 1 > + fi > + DPDK_ABI_TAR_URI=${OPTARG} ;; > + d) > + if [ -n "$DPDK_ABI_REF_DIR" ]; then > + echo "DPDK_ABI_REF_DIR and -d cannot both be set" > + exit 1 > + fi > + DPDK_ABI_REF_DIR=${OPTARG} ;; > + b) > + if [ -n "$DPDK_BUILD_TEST_DIR" ]; then > + echo "DPDK_BUILD_TEST_DIR and -a cannot both be set" > + exit 1 > + fi > + DPDK_BUILD_TEST_DIR=${OPTARG} ;; > + h) > + usage ;; > + *) > + usage ;; > + esac > +done > + > +if [ -n "$DPDK_ABI_REF_VERSION" ] ; then > + if [ "$DPDK_ABI_REF_VERSION" = "latest" ] ; then > + DPDK_ABI_REF_VERSION=$(git ls-remote --tags http://dpdk.org/git/dpdk | > + sed "s/.*\///" | grep -v "r\|{}" | > + grep '^[^.]*.[^.]*$' | tail -n 1) > + elif [ -z "$(git ls-remote http://dpdk.org/git/dpdk refs/tags/$DPDK_ABI_REF_VERSION)" ] ; then > + echo "$DPDK_ABI_REF_VERSION is not a valid DPDK tag" > + exit 1 > + fi > +fi > +if [ -z $DPDK_ABI_TAR_URI ] ; then > + DPDK_ABI_TAR_URI=$DPDK_ABI_DEFAULT_URI > +fi > +# allow the generation script to override value with env var > +abi_checks_done=${DPDK_ABI_GEN_REF:-0} > + > # set pipefail option if possible > PIPEFAIL="" > set -o | grep -q pipefail && set -o pipefail && PIPEFAIL=1 > @@ -16,7 +77,11 @@ srcdir=$(dirname $(readlink -f $0))/.. > > MESON=${MESON:-meson} > use_shared="--default-library=shared" > -builds_dir=${DPDK_BUILD_TEST_DIR:-.} > +builds_dir=${DPDK_BUILD_TEST_DIR:-$srcdir/builds} > +# ensure path is absolute meson returns error when some paths are relative > +if echo "$builds_dir" | grep -qv '^/'; then > + builds_dir=$srcdir/$builds_dir > +fi > > if command -v gmake >/dev/null 2>&1 ; then > MAKE=gmake > @@ -123,39 +188,49 @@ install_target () # > fi > } > > -build () # > +abi_gen_check () # no options > { > - targetdir=$1 > - shift > - crossfile= > - [ -r $1 ] && crossfile=$1 || targetcc=$1 > - shift > - # skip build if compiler not available > - command -v ${CC##* } >/dev/null 2>&1 || return 0 > - if [ -n "$crossfile" ] ; then > - cross="--cross-file $crossfile" > - targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ > - $crossfile | tr -d "'" | tr -d '"') > - else > - cross= > + abirefdir=${DPDK_ABI_REF_DIR:-$builds_dir/__reference}/$DPDK_ABI_REF_VERSION > + mkdir -p $abirefdir > + # ensure path is absolute meson returns error when some are relative > + if echo "$abirefdir" | grep -qv '^/'; then > + abirefdir=$srcdir/$abirefdir > fi > - load_env $targetcc || return 0 > - config $srcdir $builds_dir/$targetdir $cross --werror $* > - compile $builds_dir/$targetdir > - if [ -n "$DPDK_ABI_REF_VERSION" ]; then > - abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION > - if [ ! -d $abirefdir/$targetdir ]; then > + if [ ! -d $abirefdir/$targetdir ]; then > + > + # try to get abi reference > + if echo "$DPDK_ABI_TAR_URI" | grep -q '^http'; then > + if [ $abi_checks_done -gt -1 ]; then > + if curl --head --fail --silent \ > + "$DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz" \ > + >/dev/null; then > + curl -o $abirefdir/$targetdir.tar.gz \ > + $DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz > + fi > + fi > + elif [ $abi_checks_done -gt -1 ]; then > + if [ -f "$DPDK_ABI_TAR_URI/$targetdir.tar.gz" ]; then > + cp $DPDK_ABI_TAR_URI/$targetdir.tar.gz \ > + $abirefdir/ > + fi > + fi > + if [ -f "$abirefdir/$targetdir.tar.gz" ]; then > + tar -xf $abirefdir/$targetdir.tar.gz \ > + -C $abirefdir >/dev/null > + rm -rf $abirefdir/$targetdir.tar.gz > + # if no reference can be found then generate one > + else > # clone current sources > if [ ! -d $abirefdir/src ]; then > git clone --local --no-hardlinks \ > - --single-branch \ > - -b $DPDK_ABI_REF_VERSION \ > - $srcdir $abirefdir/src > + --single-branch \ > + -b $DPDK_ABI_REF_VERSION \ > + $srcdir $abirefdir/src > fi > > rm -rf $abirefdir/build > config $abirefdir/src $abirefdir/build $cross \ > - -Dexamples= $* > + -Dexamples= $* > compile $abirefdir/build > install_target $abirefdir/build $abirefdir/$targetdir > $srcdir/devtools/gen-abi.sh $abirefdir/$targetdir > @@ -164,17 +239,46 @@ build () # > find $abirefdir/$targetdir/usr/local -name '*.a' -delete > rm -rf $abirefdir/$targetdir/usr/local/bin > rm -rf $abirefdir/$targetdir/usr/local/share > + rm -rf $abirefdir/$targetdir/usr/local/lib > fi > + fi > > - install_target $builds_dir/$targetdir \ > - $(readlink -f $builds_dir/$targetdir/install) > - $srcdir/devtools/gen-abi.sh \ > - $(readlink -f $builds_dir/$targetdir/install) > + install_target $builds_dir/$targetdir \ > + $(readlink -f $builds_dir/$targetdir/install) > + $srcdir/devtools/gen-abi.sh \ > + $(readlink -f $builds_dir/$targetdir/install) > + # check abi if not generating references > + if [ -z $DPDK_ABI_GEN_REF ] ; then > $srcdir/devtools/check-abi.sh $abirefdir/$targetdir \ > $(readlink -f $builds_dir/$targetdir/install) > fi > } > > +build () # > +{ > + targetdir=$1 > + shift > + crossfile= > + [ -r $1 ] && crossfile=$1 || targetcc=$1 > + shift > + # skip build if compiler not available > + command -v ${CC##* } >/dev/null 2>&1 || return 0 > + if [ -n "$crossfile" ] ; then > + cross="--cross-file $crossfile" > + targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ > + $crossfile | tr -d "'" | tr -d '"') > + else > + cross= > + fi > + load_env $targetcc || return 0 > + config $srcdir $builds_dir/$targetdir $cross --werror $* > + compile $builds_dir/$targetdir > + if [ -n "$DPDK_ABI_REF_VERSION" ] && [ $abi_checks_done -lt 1 ] ; then > + abi_gen_check > + abi_checks_done=$((abi_checks_done+1)) > + fi > +} > + > if [ "$1" = "-vv" ] ; then > TEST_MESON_BUILD_VERY_VERBOSE=1 > elif [ "$1" = "-v" ] ; then > @@ -189,7 +293,7 @@ fi > # shared and static linked builds with gcc and clang > for c in gcc clang ; do > command -v $c >/dev/null 2>&1 || continue > - for s in static shared ; do > + for s in shared static ; do > export CC="$CCACHE $c" > build build-$c-$s $c --default-library=$s > unset CC > @@ -211,6 +315,8 @@ build build-x86-mingw $srcdir/config/x86/cross-mingw -Dexamples=helloworld > > # generic armv8a with clang as host compiler > f=$srcdir/config/arm/arm64_armv8_linux_gcc > +# run abi checks with 1 arm build > +abi_checks_done=$((abi_checks_done-1)) > export CC="clang" > build build-arm64-host-clang $f $use_shared > unset CC > @@ -231,7 +337,7 @@ done > build_path=$(readlink -f $builds_dir/build-x86-default) > export DESTDIR=$build_path/install > # No need to reinstall if ABI checks are enabled > -if [ -z "$DPDK_ABI_REF_VERSION" ]; then > +if [ -z "$DPDK_ABI_REF_VERSION" ] ; then > install_target $build_path $DESTDIR > fi > >