From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE6E9A04C7; Tue, 15 Sep 2020 17:47:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 131C91C0DC; Tue, 15 Sep 2020 17:47:03 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by dpdk.org (Postfix) with ESMTP id 6FD601C0CE for ; Tue, 15 Sep 2020 17:47:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600184821; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=z8YxE37EfWgQr3HafCIqdxmj7mcpOi3iEbHxX1AVH9o=; b=H5BPjD1nSVQ3N5pePeVrFb1vTPRLM2QeQYLq0KFUA4Il6hQQsMtdBUJeeYde2aO8RWj5b7 ya186n+3RJDQ/UW/nYC5JvFTUw0oGQcWsn6BBkBGPVnMmVu2mvHkRgSFkSJdmmCsdVJQbM 6OQ/WXeAvZm0/aCq7bovZ1ggSVZX8RE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-289-qE5NjnGHN7qjT6p69sXYyg-1; Tue, 15 Sep 2020 10:35:18 -0400 X-MC-Unique: qE5NjnGHN7qjT6p69sXYyg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B775B801AEC; Tue, 15 Sep 2020 14:35:16 +0000 (UTC) Received: from dhcp-25.97.bos.redhat.com (ovpn-114-150.rdu2.redhat.com [10.10.114.150]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 74CC87FB89; Tue, 15 Sep 2020 14:35:10 +0000 (UTC) From: Aaron Conole To: "Burakov\, Anatoly" Cc: Conor Walsh , dev@dpdk.org, david.marchand@redhat.com, ray.kinsella@intel.com, nhorman@tuxdriver.com, maicolgabriel@hotmail.com, thomas@monjalon.net, bruce.richardson@intel.com References: <20200910142121.3995680-1-conor.walsh@intel.com> <20200911160332.256343-1-conor.walsh@intel.com> <20200911160332.256343-3-conor.walsh@intel.com> Date: Tue, 15 Sep 2020 10:35:09 -0400 In-Reply-To: (Anatoly Burakov's message of "Mon, 14 Sep 2020 13:50:12 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Content-Type: text/plain Subject: Re: [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" "Burakov, Anatoly" writes: > On 11-Sep-20 5:03 PM, Conor Walsh wrote: >> This patch adds a script that generates a compressed archive >> containing .dump files which can be used to perform ABI >> breakage checking for the build specified in the parameters. >> Invoke using "./gen-abi-tarball.py -t -a [-cf ]" >> - : dpdk tag e.g. "v20.11" >> - : required architecture e.g. "arm" or "x86_64" >> - : configuration file for cross compiling for another >> system, this flag is not required. >> e.g. "config/arm/arm64_armv8_linux_gcc" >> E.g. "./gen-abi-tarball.py -t latest -a x86_64" >> If a compiler is not specified using the CC environmental variable then >> the script will default to using gcc. >> Using these parameters the script will produce a .tar.gz archive >> containing .dump files required to do ABI breakage checking >> >> Signed-off-by: Conor Walsh >> --- > > Just a general comment: this script looks like it's bash translated to > python. If you're going to do that, you might as well just write it in > bash? > >> devtools/gen-abi-tarball.py | 125 ++++++++++++++++++++++++++++++++++++ >> 1 file changed, 125 insertions(+) >> create mode 100755 devtools/gen-abi-tarball.py >> >> diff --git a/devtools/gen-abi-tarball.py b/devtools/gen-abi-tarball.py >> new file mode 100755 >> index 000000000..06761fca6 >> --- /dev/null >> +++ b/devtools/gen-abi-tarball.py >> @@ -0,0 +1,125 @@ >> +#!/usr/bin/env python3 >> +# SPDX-License-Identifier: BSD-3-Clause >> +# Copyright(c) 2020 Intel Corporation >> + >> +import sys >> +import os >> +import argparse >> + > > It is preferred to wrap executable code in "if __name__ == > "__main__"..., helps with importing code while avoiding executing it > on import. I wonder, since DPDK is supporting windows, we should prefer python to bash? I do prefer to use bash because I don't use windows, but maybe that is a good reason? >> +# Get command line arguments >> +parser = argparse.ArgumentParser(usage='\rThis script is intended to generate ABI dump tarballs\n'+ >> + 'Supported environmental variables\n'+ >> + '\t- CC: The required compiler will be determined using this >> environmental variable.\n') >> +parser.add_argument('-t', '--tag', type=str, dest='tag', help='DPDK tag e.g. latest or v20.11') >> +parser.add_argument('-cf', '--cross-file', type=str, >> dest='crosscompile', help='Set the location of a cross compile >> config') >> +parser.add_argument('-a', '--arch', type=str, dest='arch', help='Arch arm or x86_64') >> +args = parser.parse_args() >> + >> +# Get the DPDK tag if not supplied set as latest >> +if args.tag: >> + user_tag = args.tag >> +else: >> + user_tag = 'latest' >> + print('No tag supplied defaulting to latest') > > There's a "default" option for arguments. > >> + >> +# Get the cross-compile option >> +if args.crosscompile: >> + cross_comp = args.crosscompile >> + if not args.arch: >> + sys.stderr.write('ERROR Arch must be set using -a when using cross compile\n') >> + exit(1) >> + cross_comp = os.path.abspath(cross_comp) >> + cross_comp_meson = '--cross-file '+cross_comp >> +else: >> + cross_comp = '' >> + cross_comp_meson = '' >> + >> +# Get the required system architecture if not supplied set as x86_64 >> +if args.arch: >> + arch = args.arch >> +else: >> + arch = os.popen('uname -m').read().strip() > > There's a built-in python library for this, i think it's called "platform". > >> + print('No system architecture supplied defaulting to '+arch) >> + >> +tag = '' >> +# If the user did not supply tag or wants latest then get latest tag >> +if user_tag == 'latest': >> + # Get latest quarterly build tag from git repo >> + tag = os.popen('git ls-remote --tags http://dpdk.org/git/dpdk | >> grep -v "rc\|{}" | tail -n 1 | sed "s/.*\///"').read().strip() >> +else: >> + tag = user_tag >> + # If the user supplied tag is not in the DPDK repo then fail >> + tag_check = int(os.popen('git ls-remote http://dpdk.org/git/dpdk >> refs/tags/'+tag+' | wc -l').read().strip()) >> + if tag_check != 1: >> + sys.stderr.write('ERROR supplied tag does not exist in DPDK repo\n') >> + exit(1) >> + >> +# Get the specified compiler from system >> +comp_env = 'CC' >> +if comp_env in os.environ: >> + comp = os.environ[comp_env] >> + comp_default = '' >> +else: >> + print('No compiler specified, defaulting to gcc') >> + comp = 'gcc' >> + comp_default = 'CC=gcc' >> + >> +# Print the configuration to the user >> +print('\nSelected Build: '+tag+', Compiler: '+comp+', Architecture: >> '+arch+', Cross Compile: '+cross_comp) > > Here and in similar places: please don't use string concatenation, use > string formatting instead. > >> + >> +# Store the base directory script is working from >> +baseDir = os.getcwd() >> +# Store devtools dir >> +devtoolsDir = os.path.abspath(os.path.dirname(os.path.realpath(sys.argv[0]))) >> + >> +# Create directory for DPDK git repo and build >> +try: >> + os.mkdir('dump_dpdk') >> +except OSError as error: >> + sys.stderr.write('ERROR The dump_dpdk directory could not be >> created, ensure it does not exist before start\n') >> + exit(1) >> +os.chdir('dump_dpdk') >> +# Clone DPDK and switch to specified tag >> +print('Cloning '+tag+' from DPDK git') >> +os.popen('git clone --quiet http://dpdk.org/git/dpdk >/dev/null').read() >> +os.chdir('dpdk') >> +os.popen('git checkout --quiet '+tag+' >/dev/null').read() >> + >> +# Create build folder with meson and set debug build and cross compile (if needed) >> +print('Configuring Meson') >> +os.popen(comp_default+' meson dumpbuild '+cross_comp_meson+' >/dev/null').read() > > You do this os.popen(something > /dev/bull).read() quite a lot, maybe > put it into a function? Also, since you're discarding the data anyway > - > why os.popen().read() instead of os.system()? > >> +os.chdir('dumpbuild') >> +os.popen('meson configure -Dbuildtype=debug >/dev/null').read() >> +print('Building DPDK . . .') >> +#Build DPDK with ninja >> +os.popen('ninja >/dev/null').read() >> +gccDir = os.getcwd() >> + >> +# Create directory for abi dump files >> +dumpDir = os.path.join(baseDir,tag+'-'+comp+'-'+arch+'-abi_dump') >> +try: >> + os.mkdir(dumpDir) >> +except OSError as error: >> + sys.stderr.write('ERROR The '+dumpDir+' directory could not be created\n') >> + os.popen('rm -rf '+os.path.join(baseDir,'dump_dpdk')).read() >> + exit(1) >> + >> +# Create dump files and output to dump directory >> +print('Generating ABI dump files') >> +genabiout = os.popen(os.path.join(devtoolsDir,'gen-abi.sh')+' '+gccDir).read() >> +os.popen('cp dump/* '+dumpDir).read() >> + >> +# Compress the dump directory >> +print('Creating Tarball of dump files') >> +os.chdir(baseDir) >> +origSize = os.popen('du -sh '+dumpDir+' | sed "s/\s.*$//"').read() >> +os.popen('tar -czf '+dumpDir.split('/')[-1]+'.tar.gz '+dumpDir.split('/')[-1]+' >/dev/null').read() >> +newSize = os.popen('du -sh '+dumpDir+'.tar.gz | sed "s/\s.*$//"').read() >> + >> +# Remove all temporary directories >> +print('Cleaning up temporary directories') >> +os.popen('rm -rf '+dumpDir).read() >> +os.popen('rm -rf '+os.path.join(baseDir,'dump_dpdk')).read() >> + >> +#Print output of the script to the user >> +print('\nDump of DPDK ABI '+tag+' is available in >> '+dumpDir.split('/')[-1]+'.tar.gz (Original Size: >> '+origSize.strip()+', Compressed Size:'+newSize.strip()+')\n') >>