From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2626BA04B7; Wed, 14 Oct 2020 12:42:55 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F051B1DE4C; Wed, 14 Oct 2020 12:41:44 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id AC7361DE3F for ; Wed, 14 Oct 2020 12:41:39 +0200 (CEST) IronPort-SDR: cBFbFvI8br4KUBOzZsw9bfdTKkgODn0/2JNhkvCs82zkmasf9Pi4AqcR5CrmMCQRF+fBCDq4P0 CFs7CzCQTJjw== X-IronPort-AV: E=McAfee;i="6000,8403,9773"; a="250781284" X-IronPort-AV: E=Sophos;i="5.77,374,1596524400"; d="scan'208";a="250781284" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Oct 2020 03:41:39 -0700 IronPort-SDR: 6aUb50ok5Y5ELN/wdMqq1hHk4TcWuRd1yFw72oGBWksmg3HXMRZt5y3ngOeCdDeMzS8dqt/WIu jrfTc85led5Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,374,1596524400"; d="scan'208";a="530779994" Received: from unknown (HELO silpixa00400466.ir.intel.com) ([10.237.213.98]) by orsmga005.jf.intel.com with ESMTP; 14 Oct 2020 03:41:37 -0700 From: Conor Walsh To: mdr@ashroe.eu, nhorman@tuxdriver.com, bruce.richardson@intel.com, thomas@monjalon.net, david.marchand@redhat.com Cc: dev@dpdk.org, Conor Walsh Date: Wed, 14 Oct 2020 10:41:24 +0000 Message-Id: <20201014104126.469517-3-conor.walsh@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201014104126.469517-1-conor.walsh@intel.com> References: <20201012130348.3212-1-conor.walsh@intel.com> <20201014104126.469517-1-conor.walsh@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v7 2/4] devtools: abi and UX changes for test-meson-builds.sh X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The core reason for this patch is to reduce the amount of time needed to run abi checks. The number of abi checks being run has been reduced to only 2 (1 x86_64 and 1 arm). The script can now also take adavtage of prebuilt abi references. Invoke using "./test-meson-builds.sh [-b ] [-a ] [-u ] [-d ]" - : directory to store builds (relative or absolute) - : dpdk tag e.g. "v20.11" or "latest" - : http location or directory to get prebuilt abi references from - : directory to store abi references (relative or absolute) e.g. "./test-meson-builds.sh -a latest" If no flags are specified test-meson-builds.sh will run the standard meson tests with default options unless environmental variables are specified. Signed-off-by: Conor Walsh --- devtools/test-meson-builds.sh | 171 +++++++++++++++++++++++++++------- 1 file changed, 139 insertions(+), 32 deletions(-) diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh index a87de635a..6b959eb63 100755 --- a/devtools/test-meson-builds.sh +++ b/devtools/test-meson-builds.sh @@ -1,12 +1,74 @@ #! /bin/sh -e # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Intel Corporation +# Copyright(c) 2018-2020 Intel Corporation # Run meson to auto-configure the various builds. # * all builds get put in a directory whose name starts with "build-" # * if a build-directory already exists we assume it was properly configured # Run ninja after configuration is done. +# Get arguments +usage() +{ + echo "Usage: $0 + [-b ] + [-a ] + [-u ] + [-d ]" 1>&2; exit 1; +} + +# Placeholder default uri +DPDK_ABI_DEFAULT_URI="http://abi-ref.dpdk.org" + +while getopts "a:u:d:b:h" arg; do + case $arg in + a) + if [ -n "$DPDK_ABI_REF_VERSION" ]; then + echo "DPDK_ABI_REF_VERSION and -a cannot both be set" + exit 1 + fi + DPDK_ABI_REF_VERSION=${OPTARG} ;; + u) + if [ -n "$DPDK_ABI_TAR_URI" ]; then + echo "DPDK_ABI_TAR_URI and -u cannot both be set" + exit 1 + fi + DPDK_ABI_TAR_URI=${OPTARG} ;; + d) + if [ -n "$DPDK_ABI_REF_DIR" ]; then + echo "DPDK_ABI_REF_DIR and -d cannot both be set" + exit 1 + fi + DPDK_ABI_REF_DIR=${OPTARG} ;; + b) + if [ -n "$DPDK_BUILD_TEST_DIR" ]; then + echo "DPDK_BUILD_TEST_DIR and -a cannot both be set" + exit 1 + fi + DPDK_BUILD_TEST_DIR=${OPTARG} ;; + h) + usage ;; + *) + usage ;; + esac +done + +if [ -n "$DPDK_ABI_REF_VERSION" ] ; then + if [ "$DPDK_ABI_REF_VERSION" = "latest" ] ; then + DPDK_ABI_REF_VERSION=$(git ls-remote --tags http://dpdk.org/git/dpdk | + sed "s/.*\///" | grep -v "r\|{}" | + grep '^[^.]*.[^.]*$' | tail -n 1) + elif [ -z "$(git ls-remote http://dpdk.org/git/dpdk refs/tags/$DPDK_ABI_REF_VERSION)" ] ; then + echo "$DPDK_ABI_REF_VERSION is not a valid DPDK tag" + exit 1 + fi +fi +if [ -z $DPDK_ABI_TAR_URI ] ; then + DPDK_ABI_TAR_URI=$DPDK_ABI_DEFAULT_URI +fi +# allow the generation script to override value with env var +abi_checks_done=${DPDK_ABI_GEN_REF:-0} + # set pipefail option if possible PIPEFAIL="" set -o | grep -q pipefail && set -o pipefail && PIPEFAIL=1 @@ -16,7 +78,11 @@ srcdir=$(dirname $(readlink -f $0))/.. MESON=${MESON:-meson} use_shared="--default-library=shared" -builds_dir=${DPDK_BUILD_TEST_DIR:-.} +builds_dir=${DPDK_BUILD_TEST_DIR:-$srcdir/builds} +# ensure path is absolute meson returns error when some paths are relative +if echo "$builds_dir" | grep -qv '^/'; then + builds_dir=$srcdir/$builds_dir +fi if command -v gmake >/dev/null 2>&1 ; then MAKE=gmake @@ -123,39 +189,49 @@ install_target () # fi } -build () # +abi_gen_check () # no options { - targetdir=$1 - shift - crossfile= - [ -r $1 ] && crossfile=$1 || targetcc=$1 - shift - # skip build if compiler not available - command -v ${CC##* } >/dev/null 2>&1 || return 0 - if [ -n "$crossfile" ] ; then - cross="--cross-file $crossfile" - targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ - $crossfile | tr -d "'" | tr -d '"') - else - cross= + abirefdir=${DPDK_ABI_REF_DIR:-$builds_dir/__reference}/$DPDK_ABI_REF_VERSION + mkdir -p $abirefdir + # ensure path is absolute meson returns error when some are relative + if echo "$abirefdir" | grep -qv '^/'; then + abirefdir=$srcdir/$abirefdir fi - load_env $targetcc || return 0 - config $srcdir $builds_dir/$targetdir $cross --werror $* - compile $builds_dir/$targetdir - if [ -n "$DPDK_ABI_REF_VERSION" ]; then - abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION - if [ ! -d $abirefdir/$targetdir ]; then + if [ ! -d $abirefdir/$targetdir ]; then + + # try to get abi reference + if echo "$DPDK_ABI_TAR_URI" | grep -q '^http'; then + if [ $abi_checks_done -gt -1 ]; then + if curl --head --fail --silent \ + "$DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz" \ + >/dev/null; then + curl -o $abirefdir/$targetdir.tar.gz \ + $DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz + fi + fi + elif [ $abi_checks_done -gt -1 ]; then + if [ -f "$DPDK_ABI_TAR_URI/$targetdir.tar.gz" ]; then + cp $DPDK_ABI_TAR_URI/$targetdir.tar.gz \ + $abirefdir/ + fi + fi + if [ -f "$abirefdir/$targetdir.tar.gz" ]; then + tar -xf $abirefdir/$targetdir.tar.gz \ + -C $abirefdir >/dev/null + rm -rf $abirefdir/$targetdir.tar.gz + # if no reference can be found then generate one + else # clone current sources if [ ! -d $abirefdir/src ]; then git clone --local --no-hardlinks \ - --single-branch \ - -b $DPDK_ABI_REF_VERSION \ - $srcdir $abirefdir/src + --single-branch \ + -b $DPDK_ABI_REF_VERSION \ + $srcdir $abirefdir/src fi rm -rf $abirefdir/build config $abirefdir/src $abirefdir/build $cross \ - -Dexamples= $* + -Dexamples= $* compile $abirefdir/build install_target $abirefdir/build $abirefdir/$targetdir $srcdir/devtools/gen-abi.sh $abirefdir/$targetdir @@ -164,17 +240,46 @@ build () # find $abirefdir/$targetdir/usr/local -name '*.a' -delete rm -rf $abirefdir/$targetdir/usr/local/bin rm -rf $abirefdir/$targetdir/usr/local/share + rm -rf $abirefdir/$targetdir/usr/local/lib fi + fi - install_target $builds_dir/$targetdir \ - $(readlink -f $builds_dir/$targetdir/install) - $srcdir/devtools/gen-abi.sh \ - $(readlink -f $builds_dir/$targetdir/install) + install_target $builds_dir/$targetdir \ + $(readlink -f $builds_dir/$targetdir/install) + $srcdir/devtools/gen-abi.sh \ + $(readlink -f $builds_dir/$targetdir/install) + # check abi if not generating references + if [ -z $DPDK_ABI_GEN_REF ] ; then $srcdir/devtools/check-abi.sh $abirefdir/$targetdir \ $(readlink -f $builds_dir/$targetdir/install) fi } +build () # +{ + targetdir=$1 + shift + crossfile= + [ -r $1 ] && crossfile=$1 || targetcc=$1 + shift + # skip build if compiler not available + command -v ${CC##* } >/dev/null 2>&1 || return 0 + if [ -n "$crossfile" ] ; then + cross="--cross-file $crossfile" + targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ + $crossfile | tr -d "'" | tr -d '"') + else + cross= + fi + load_env $targetcc || return 0 + config $srcdir $builds_dir/$targetdir $cross --werror $* + compile $builds_dir/$targetdir + if [ -n "$DPDK_ABI_REF_VERSION" ] && [ $abi_checks_done -lt 1 ] ; then + abi_gen_check + abi_checks_done=$((abi_checks_done+1)) + fi +} + if [ "$1" = "-vv" ] ; then TEST_MESON_BUILD_VERY_VERBOSE=1 elif [ "$1" = "-v" ] ; then @@ -189,7 +294,7 @@ fi # shared and static linked builds with gcc and clang for c in gcc clang ; do command -v $c >/dev/null 2>&1 || continue - for s in static shared ; do + for s in shared static ; do export CC="$CCACHE $c" build build-$c-$s $c --default-library=$s unset CC @@ -211,6 +316,8 @@ build build-x86-mingw $srcdir/config/x86/cross-mingw -Dexamples=helloworld # generic armv8a with clang as host compiler f=$srcdir/config/arm/arm64_armv8_linux_gcc +# run abi checks with 1 arm build +abi_checks_done=$((abi_checks_done-1)) export CC="clang" build build-arm64-host-clang $f $use_shared unset CC @@ -231,7 +338,7 @@ done build_path=$(readlink -f $builds_dir/build-x86-default) export DESTDIR=$build_path/install # No need to reinstall if ABI checks are enabled -if [ -z "$DPDK_ABI_REF_VERSION" ]; then +if [ -z "$DPDK_ABI_REF_VERSION" ] ; then install_target $build_path $DESTDIR fi -- 2.25.1