From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D8A17A04B6; Mon, 12 Oct 2020 10:09:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 056781D640; Mon, 12 Oct 2020 10:08:59 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id BAC881D633 for ; Mon, 12 Oct 2020 10:08:56 +0200 (CEST) IronPort-SDR: OKznXwGbuoBkk8nWuPjdcqoy0KymYX6nnFlUwrCxpSMpU6R5sDjL8Ct7vwMcEDdl4lta5BSTXf a6LTWKkF8eXg== X-IronPort-AV: E=McAfee;i="6000,8403,9771"; a="162239018" X-IronPort-AV: E=Sophos;i="5.77,366,1596524400"; d="scan'208";a="162239018" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Oct 2020 01:08:54 -0700 IronPort-SDR: xhO8XwcYvQW/CBGCNVW7oWrDPj+7u5fBHWjeE9k5rUFYNhdIRlqVmSvO0oX5w8R9GcV03I1/Zx MCMWkIRrR8Bw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,366,1596524400"; d="scan'208";a="317841937" Received: from unknown (HELO silpixa00400466.ir.intel.com) ([10.237.213.195]) by orsmga006.jf.intel.com with ESMTP; 12 Oct 2020 01:08:52 -0700 From: Conor Walsh To: mdr@ashroe.eu, nhorman@tuxdriver.com, bruce.richardson@intel.com, thomas@monjalon.net, david.marchand@redhat.com Cc: dev@dpdk.org, Conor Walsh Date: Mon, 12 Oct 2020 08:08:27 +0000 Message-Id: <20201012080829.3446-3-conor.walsh@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201012080829.3446-1-conor.walsh@intel.com> References: <20200918121137.1370883-1-conor.walsh@intel.com> <20201012080829.3446-1-conor.walsh@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v5 2/4] devtools: abi and UX changes for test-meson-builds.sh X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds new features to test-meson-builds.sh that help to make the process of using the script easier, the patch also includes changes to make the abi breakage checks more performant. Changes/Additions: - Command line arguments added, the changes are fully backwards compatible and all previous environmental variables are still supported - All paths supplied by user are converted to absolute paths if they are relative as meson has a bug that can sometimes error if a relative path is supplied to it. - abi check/generation code moved to function to improve readability - Only 2 abi checks will now be completed: - 1 x86_64 gcc or clang check - 1 ARM gcc or clang check It is not necessary to check abi breakages in every build - abi checks can now make use of prebuilt abi references from a http or local source, it is hoped these would be hosted on dpdk.org in the future. Invoke using "./test-meson-builds.sh [-b ] [-a ] [-u ] [-d ]" - : directory to store builds (relative or absolute) - : dpdk tag e.g. "v20.11" or "latest" - : http location or directory to get prebuilt abi references from - : directory to store abi references (relative or absolute) e.g. "./test-meson-builds.sh -a latest" If no flags are specified test-meson-builds.sh will run the standard meson tests with default options unless environmental variables are specified. Signed-off-by: Conor Walsh --- devtools/test-meson-builds.sh | 170 +++++++++++++++++++++++++++------- 1 file changed, 138 insertions(+), 32 deletions(-) diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh index a87de635a..b45506fb0 100755 --- a/devtools/test-meson-builds.sh +++ b/devtools/test-meson-builds.sh @@ -1,12 +1,73 @@ #! /bin/sh -e # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Intel Corporation +# Copyright(c) 2018-2020 Intel Corporation # Run meson to auto-configure the various builds. # * all builds get put in a directory whose name starts with "build-" # * if a build-directory already exists we assume it was properly configured # Run ninja after configuration is done. +# Get arguments +usage() +{ + echo "Usage: $0 + [-b ] + [-a ] + [-u ] + [-d ]" 1>&2; exit 1; +} + +DPDK_ABI_DEFAULT_URI="http://dpdk.org/abi-refs" + +while getopts "a:u:d:b:h" arg; do + case $arg in + a) + if [ -n "$DPDK_ABI_REF_VERSION" ]; then + echo "DPDK_ABI_REF_VERSION and -a cannot both be set" + exit 1 + fi + DPDK_ABI_REF_VERSION=${OPTARG} ;; + u) + if [ -n "$DPDK_ABI_TAR_URI" ]; then + echo "DPDK_ABI_TAR_URI and -u cannot both be set" + exit 1 + fi + DPDK_ABI_TAR_URI=${OPTARG} ;; + d) + if [ -n "$DPDK_ABI_REF_DIR" ]; then + echo "DPDK_ABI_REF_DIR and -d cannot both be set" + exit 1 + fi + DPDK_ABI_REF_DIR=${OPTARG} ;; + b) + if [ -n "$DPDK_BUILD_TEST_DIR" ]; then + echo "DPDK_BUILD_TEST_DIR and -a cannot both be set" + exit 1 + fi + DPDK_BUILD_TEST_DIR=${OPTARG} ;; + h) + usage ;; + *) + usage ;; + esac +done + +if [ -n "$DPDK_ABI_REF_VERSION" ] ; then + if [ "$DPDK_ABI_REF_VERSION" = "latest" ] ; then + DPDK_ABI_REF_VERSION=$(git ls-remote --tags http://dpdk.org/git/dpdk | + sed "s/.*\///" | grep -v "r\|{}" | + grep '^[^.]*.[^.]*$' | tail -n 1) + elif [ -z "$(git ls-remote http://dpdk.org/git/dpdk refs/tags/$DPDK_ABI_REF_VERSION)" ] ; then + echo "$DPDK_ABI_REF_VERSION is not a valid DPDK tag" + exit 1 + fi +fi +if [ -z $DPDK_ABI_TAR_URI ] ; then + DPDK_ABI_TAR_URI=$DPDK_ABI_DEFAULT_URI +fi +# allow the generation script to override value with env var +abi_checks_done=${DPDK_ABI_GEN_REF:-0} + # set pipefail option if possible PIPEFAIL="" set -o | grep -q pipefail && set -o pipefail && PIPEFAIL=1 @@ -16,7 +77,11 @@ srcdir=$(dirname $(readlink -f $0))/.. MESON=${MESON:-meson} use_shared="--default-library=shared" -builds_dir=${DPDK_BUILD_TEST_DIR:-.} +builds_dir=${DPDK_BUILD_TEST_DIR:-$srcdir/builds} +# ensure path is absolute meson returns error when some paths are relative +if echo "$builds_dir" | grep -qv '^/'; then + builds_dir=$srcdir/$builds_dir +fi if command -v gmake >/dev/null 2>&1 ; then MAKE=gmake @@ -123,39 +188,49 @@ install_target () # fi } -build () # +abi_gen_check () # no options { - targetdir=$1 - shift - crossfile= - [ -r $1 ] && crossfile=$1 || targetcc=$1 - shift - # skip build if compiler not available - command -v ${CC##* } >/dev/null 2>&1 || return 0 - if [ -n "$crossfile" ] ; then - cross="--cross-file $crossfile" - targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ - $crossfile | tr -d "'" | tr -d '"') - else - cross= + abirefdir=${DPDK_ABI_REF_DIR:-$builds_dir/__reference}/$DPDK_ABI_REF_VERSION + mkdir -p $abirefdir + # ensure path is absolute meson returns error when some are relative + if echo "$abirefdir" | grep -qv '^/'; then + abirefdir=$srcdir/$abirefdir fi - load_env $targetcc || return 0 - config $srcdir $builds_dir/$targetdir $cross --werror $* - compile $builds_dir/$targetdir - if [ -n "$DPDK_ABI_REF_VERSION" ]; then - abirefdir=${DPDK_ABI_REF_DIR:-reference}/$DPDK_ABI_REF_VERSION - if [ ! -d $abirefdir/$targetdir ]; then + if [ ! -d $abirefdir/$targetdir ]; then + + # try to get abi reference + if echo "$DPDK_ABI_TAR_URI" | grep -q '^http'; then + if [ $abi_checks_done -gt -1 ]; then + if curl --head --fail --silent \ + "$DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz" \ + >/dev/null; then + curl -o $abirefdir/$targetdir.tar.gz \ + $DPDK_ABI_TAR_URI/$DPDK_ABI_REF_VERSION/$targetdir.tar.gz + fi + fi + elif [ $abi_checks_done -gt -1 ]; then + if [ -f "$DPDK_ABI_TAR_URI/$targetdir.tar.gz" ]; then + cp $DPDK_ABI_TAR_URI/$targetdir.tar.gz \ + $abirefdir/ + fi + fi + if [ -f "$abirefdir/$targetdir.tar.gz" ]; then + tar -xf $abirefdir/$targetdir.tar.gz \ + -C $abirefdir >/dev/null + rm -rf $abirefdir/$targetdir.tar.gz + # if no reference can be found then generate one + else # clone current sources if [ ! -d $abirefdir/src ]; then git clone --local --no-hardlinks \ - --single-branch \ - -b $DPDK_ABI_REF_VERSION \ - $srcdir $abirefdir/src + --single-branch \ + -b $DPDK_ABI_REF_VERSION \ + $srcdir $abirefdir/src fi rm -rf $abirefdir/build config $abirefdir/src $abirefdir/build $cross \ - -Dexamples= $* + -Dexamples= $* compile $abirefdir/build install_target $abirefdir/build $abirefdir/$targetdir $srcdir/devtools/gen-abi.sh $abirefdir/$targetdir @@ -164,17 +239,46 @@ build () # find $abirefdir/$targetdir/usr/local -name '*.a' -delete rm -rf $abirefdir/$targetdir/usr/local/bin rm -rf $abirefdir/$targetdir/usr/local/share + rm -rf $abirefdir/$targetdir/usr/local/lib fi + fi - install_target $builds_dir/$targetdir \ - $(readlink -f $builds_dir/$targetdir/install) - $srcdir/devtools/gen-abi.sh \ - $(readlink -f $builds_dir/$targetdir/install) + install_target $builds_dir/$targetdir \ + $(readlink -f $builds_dir/$targetdir/install) + $srcdir/devtools/gen-abi.sh \ + $(readlink -f $builds_dir/$targetdir/install) + # check abi if not generating references + if [ -z $DPDK_ABI_GEN_REF ] ; then $srcdir/devtools/check-abi.sh $abirefdir/$targetdir \ $(readlink -f $builds_dir/$targetdir/install) fi } +build () # +{ + targetdir=$1 + shift + crossfile= + [ -r $1 ] && crossfile=$1 || targetcc=$1 + shift + # skip build if compiler not available + command -v ${CC##* } >/dev/null 2>&1 || return 0 + if [ -n "$crossfile" ] ; then + cross="--cross-file $crossfile" + targetcc=$(sed -n 's,^c[[:space:]]*=[[:space:]]*,,p' \ + $crossfile | tr -d "'" | tr -d '"') + else + cross= + fi + load_env $targetcc || return 0 + config $srcdir $builds_dir/$targetdir $cross --werror $* + compile $builds_dir/$targetdir + if [ -n "$DPDK_ABI_REF_VERSION" ] && [ $abi_checks_done -lt 1 ] ; then + abi_gen_check + abi_checks_done=$((abi_checks_done+1)) + fi +} + if [ "$1" = "-vv" ] ; then TEST_MESON_BUILD_VERY_VERBOSE=1 elif [ "$1" = "-v" ] ; then @@ -189,7 +293,7 @@ fi # shared and static linked builds with gcc and clang for c in gcc clang ; do command -v $c >/dev/null 2>&1 || continue - for s in static shared ; do + for s in shared static ; do export CC="$CCACHE $c" build build-$c-$s $c --default-library=$s unset CC @@ -211,6 +315,8 @@ build build-x86-mingw $srcdir/config/x86/cross-mingw -Dexamples=helloworld # generic armv8a with clang as host compiler f=$srcdir/config/arm/arm64_armv8_linux_gcc +# run abi checks with 1 arm build +abi_checks_done=$((abi_checks_done-1)) export CC="clang" build build-arm64-host-clang $f $use_shared unset CC @@ -231,7 +337,7 @@ done build_path=$(readlink -f $builds_dir/build-x86-default) export DESTDIR=$build_path/install # No need to reinstall if ABI checks are enabled -if [ -z "$DPDK_ABI_REF_VERSION" ]; then +if [ -z "$DPDK_ABI_REF_VERSION" ] ; then install_target $build_path $DESTDIR fi -- 2.25.1