From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8480DA04C8; Fri, 18 Sep 2020 14:12:16 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 50DEE1DA22; Fri, 18 Sep 2020 14:12:03 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id EB6C61DA15 for ; Fri, 18 Sep 2020 14:11:55 +0200 (CEST) IronPort-SDR: ZQt0pmg1+u775nrUAqSrWNuloefXfKzbl29gHFsamUXl5Ie0lHRMv8YQyAOPJa5NJoN0TEWZbj atmAdK0W8qtw== X-IronPort-AV: E=McAfee;i="6000,8403,9747"; a="160842573" X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="160842573" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Sep 2020 05:11:55 -0700 IronPort-SDR: f0Wa+/dURiKhwigqAXnyhGRFf7aI+DFCOsvVvknMhHueojZt+vaj2noMh8A+ZR+Gry8+N6NbAK 19Z5jknpYs9A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="410271223" Received: from silpixa00400466.ir.intel.com ([10.237.213.195]) by fmsmga001.fm.intel.com with ESMTP; 18 Sep 2020 05:11:53 -0700 From: Conor Walsh To: dev@dpdk.org Cc: david.marchand@redhat.com, ray.kinsella@intel.com, nhorman@tuxdriver.com, aconole@redhat.com, maicolgabriel@hotmail.com, thomas@monjalon.net, bruce.richardson@intel.com, anatoly.burakov@intel.com, Conor Walsh Date: Fri, 18 Sep 2020 12:11:36 +0000 Message-Id: <20200918121137.1370883-4-conor.walsh@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200918121137.1370883-1-conor.walsh@intel.com> References: <20200911160332.256343-1-conor.walsh@intel.com> <20200918121137.1370883-1-conor.walsh@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 3/4] buildtools: add script to setup abi checks for meson X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds a script that is intended to be invoked by meson to do the required setup for performing ABI breakage checks at build time. The required ABI dump archives can come from several sources including being generated at build time or prebuilt archives can be pulled from a remote http location or local directory. Invoke using "./abi-setup.py [-t ] -d " - : dpdk tag e.g. "v20.11", default: latest - : path to dpdk source directory, required E.g. "./abi-setup.py -t v20.08 -d /root/dpdk" As this script is intended to be run by meson during a build some options can be specified by environmental variables: - DPDK_ABI_DUMPS_PATH: Can be used to specify a custom directory for the systems dump directories. - CC: The required compiler will be determined using this environmental variable - DPDK_ABI_TAR_URI: Can be used to specify a location that the script can pull prebuilt or cached dump archives from. This can be a remote http location or a local directory After the script has setup an appropriate ABI dump directory using one of the multiple methods available to it, it will print the location of this directory to the command line. Signed-off-by: Conor Walsh --- buildtools/abi-setup.py | 141 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100755 buildtools/abi-setup.py diff --git a/buildtools/abi-setup.py b/buildtools/abi-setup.py new file mode 100755 index 000000000..39616d21f --- /dev/null +++ b/buildtools/abi-setup.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Intel Corporation + +""" +This script is intended to be invoked by meson to do the required setup +for performing ABI breakage checks at build time. +The required ABI dump archives can come from several sources including +being generated at build time or prebuilt archives can be pulled from a +remote http location or local directory. +""" + +import sys +import os +from os.path import abspath, join, exists, isfile +import argparse +import platform +import subprocess +import requests +import tarfile +import shutil + +# Get command line options +def args_parse(): + # Get command line arguments + parser = argparse.ArgumentParser( + description='This script is intended to setup ABI dumps for meson to perform ABI checks\n'+ + 'Supported environmental variables\n'+ + '\t- DPDK_ABI_DUMPS_PATH: Can be used to specify a custom directory for the systems dump directories.\n'+ + '\t- CC: The required compiler will be determined using this environmental variable.\n'+ + '\t- DPDK_ABI_TAR_URI: Can be used to specify a location that the script can pull prebuilt or cached dump archives from. This can be a remote http location or a local directory.\n', + formatter_class=argparse.RawTextHelpFormatter) + parser.add_argument( + '-t', '--tag', dest='tag', type=str, + help='DPDK tag e.g. latest or v20.11', default='latest') + parser.add_argument( + '-d', '--dpdk', dest='dpdk', type=str, + help='Path to DPDK source directory', required=True) + args = parser.parse_args() + return args + +# Function to execute git commands +def call_git(args): + args = list(filter(None, args)) + git_call = subprocess.run(['git'] + args, capture_output=True) + if git_call.returncode != 0: + print('ERROR Git returned an error', file=sys.stderr) + exit(1) + return git_call.stdout.decode('utf-8') + +# Function to execute commands +def call_exec(args): + args = list(filter(None, args)) + exec_call = subprocess.run(args, stdout=subprocess.DEVNULL) + if exec_call.returncode != 0: + print('ERROR Script returned an error', file=sys.stderr) + exit(1) + +# Get required git tag +def get_tag(tag): + tags = call_git(['ls-remote', '--tags', 'http://dpdk.org/git/dpdk']).split('\n') + tags = [t.split('/')[-1].strip() for t in tags if 'rc' not in t and not t.endswith('{}') and t != ''] + if tag == 'latest': + tag = tags[-1] + if tag not in tags: + print('ERROR supplied tag does not exist in DPDK repo', file=sys.stderr) + exit(1) + return tag + +def main(): + args = args_parse() + + tag = get_tag(args.tag) + + # Get the specified compiler from system + if 'CC' in os.environ: + comp = os.environ['CC'] + else: + comp = 'gcc' + + # Get the systems architecture + arch = platform.machine() + + # Get devtools path + devtools_path = abspath(join(args.dpdk,'devtools')) + + # Get the abi dumps folder from args or env fail if none supplied + abi_folder = '' + abi_env = 'DPDK_ABI_DUMPS_PATH' + if abi_env in os.environ: + abi_folder = abspath(os.environ[abi_env]) + else: + abi_folder = abspath(join(args.dpdk,'abi_dumps')) + + # If the directory doesn't exist create it and add a README to explain what it does + if not exists(abi_folder): + os.makedirs(abi_folder) + f=open(abi_folder+'/README','w+') + f.write('This directory has been setup to contain the ABI dump folders needed to perform ABI checks\n') + f.write('Directories here must be in the format {DPDK Tag}-{Compiler ID}-{Architecture}-abi_dump\n') + f.write('e.g. v20.11-gcc-x86_64-abi_dump\n') + f.write('Directories that do not use this format will not be picked up by the meson ABI checks\n') + f.write('This directory is managed automatically unless desired by the user\n') + f.close() + + # Move to abi folder + os.chdir(abi_folder) + abi_dump=tag+'-'+comp+'-'+arch+'-abi_dump' + # Download and untar abi dump if not present + if not exists(abi_dump): + # Check DPDK_ABI_TAR_URI for the location of the tarballs local or web + tar_uri_env = 'DPDK_ABI_TAR_URI' + if tar_uri_env in os.environ: + abi_tar_uri = os.environ[tar_uri_env] + if abi_tar_uri.startswith('http'): + # Download the required tarball + tar_loc = '{}.tar.gz'.format(join(abi_tar_uri,abi_dump)) + r = requests.get(tar_loc) + if r.status_code == 200: + with open('{}.tar.gz'.format(abi_dump), 'wb') as f: + f.write(r.content) + else: + abi_tar_uri = abspath(abi_tar_uri) + try: + shutil.copy('{}.tar.gz'.format(join(abi_tar_uri,abi_dump)), '.') + except FileNotFoundError as error: + pass + if not isfile(abi_dump+'.tar.gz'): + call_exec([join(devtools_path,'gen-abi-tarball.py'), '-t', tag, '-a', arch]) + if not isfile(abi_dump+'.tar.gz'): + print('ERROR ABI check generation failed', file=sys.stderr) + exit(1) + f = tarfile.open('{}.tar.gz'.format(abi_dump)) + f.extractall() + os.remove('{}.tar.gz'.format(abi_dump)) + + # Tell user where specified directory is + print(abspath(abi_dump)) + +if __name__ == "__main__": + main() -- 2.25.1