From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B9A63A04C7; Tue, 15 Sep 2020 17:57:06 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 926B41C0C6; Tue, 15 Sep 2020 17:57:06 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 2D0601C0C2 for ; Tue, 15 Sep 2020 17:57:04 +0200 (CEST) IronPort-SDR: Jn0NJuwu6ZoUdxIFiCEdBjltHNMjvEN9ZmMe8bPFv8pJ/t3DiK8yxXbO4MogReAesQnRFnEM3/ b6QJ8CPHnNBg== X-IronPort-AV: E=McAfee;i="6000,8403,9745"; a="158553201" X-IronPort-AV: E=Sophos;i="5.76,430,1592895600"; d="scan'208";a="158553201" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2020 07:49:49 -0700 IronPort-SDR: cOiYmcC8mxrK/4PmmuuY7fZXInkLGiVxe9wzQDOTdzWu9tpLSIcKw3WLwcf8obG+ADpDd0JFML qmeDX4MZJyBg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,430,1592895600"; d="scan'208";a="319483537" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga002.jf.intel.com with ESMTP; 15 Sep 2020 07:49:48 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 15 Sep 2020 07:49:48 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 15 Sep 2020 07:49:47 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Tue, 15 Sep 2020 07:49:47 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.100) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Tue, 15 Sep 2020 07:49:47 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YhHKda2lH8zTO4J6eSQsbG6YPX9lwSv84ES77BYECTCrQiOp9knoS111PqR7zJ3GKPr+42BtqCP+hLCzcz6bZ+DHbQjDi+6fiJvGF/OJ9YnE8RuHBB/T8dj1MG1Al0rYr4Gj8oQIsNyMtD6zwlpQmNN1yFt4chL/Mnp/onFpm2MYc/rs+UFlEvRPFRwm59f3APVdUtvpncDJfFX75z0JMLRfIZouIF/GEdPGncggleB6xx8KjQxjg8jpzv6TPfWQW7Cl/3tJ8nscf1nf6qywyYOwYU5FC+N68E9Yh0zdxE/DS4PtgxbItwmbG5cyL6J7rqVG4T7KiUy+iz9Zpwg3pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+utdwdEBhJPnvlbjTGnKolcFI5rcfwxOl/tpbLnIBqU=; b=iaXZDYphXR6dpBvsmoSdOEGrlFOq4MR4bcr2Sw7QDdxVBvHbF5Ajh4qQR5UiJTPOWUBisNWNfjSxQN6rzUk7aefOO4AANREEtY/xu45+RIg4dUaSwO+GWKsY6CSZuvkvdbgBT6DA+xSWkrcUVSC/es/pi/zppakRQ8cALmPZIutwBfHQMsFZ0wtxnzeKj8x1IVXiWgogzBrtsCQKKNmGj5q696/KAbdtl/B8UZUGGLXu2zD8gQ0OoHZZ+Me3J6ThN74JL6GGEOV8kL24BlagD4TjnwbRYU9v37sBorzo3yAzUT7+lMDQqJcKUaKC5UzDvw1nNgGOdve4rwl9u/q9Eg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+utdwdEBhJPnvlbjTGnKolcFI5rcfwxOl/tpbLnIBqU=; b=Erjb7tSUw76hhDpBE9h7MQMA8HFkN3TRdVJwnQGyGFEleupvxElJn9xc2GWrbiIjSSyJeqJ3OPZLOdDLrkvZT3boyHjsPm3uB7IsC3UjzGI2y1bsbTRhk3nBB0PqfS4u91YFpY3XA3UoGbIbBRYpjD+reNiR/uFVV/T5/JA4NqE= Received: from BYAPR11MB3799.namprd11.prod.outlook.com (2603:10b6:a03:fb::19) by BYAPR11MB3704.namprd11.prod.outlook.com (2603:10b6:a03:f9::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3391.11; Tue, 15 Sep 2020 14:49:44 +0000 Received: from BYAPR11MB3799.namprd11.prod.outlook.com ([fe80::84c3:7525:84dc:f171]) by BYAPR11MB3799.namprd11.prod.outlook.com ([fe80::84c3:7525:84dc:f171%5]) with mapi id 15.20.3391.011; Tue, 15 Sep 2020 14:49:44 +0000 From: "Walsh, Conor" To: Aaron Conole , "Burakov, Anatoly" CC: "dev@dpdk.org" , "david.marchand@redhat.com" , "Kinsella, Ray" , "nhorman@tuxdriver.com" , "maicolgabriel@hotmail.com" , "thomas@monjalon.net" , "Richardson, Bruce" Thread-Topic: [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives Thread-Index: AQHWiFUpZycajUgZFka67ULmi1wQwqloGoEAgAGv3AWAAAA9cA== Date: Tue, 15 Sep 2020 14:49:43 +0000 Message-ID: References: <20200910142121.3995680-1-conor.walsh@intel.com> <20200911160332.256343-1-conor.walsh@intel.com> <20200911160332.256343-3-conor.walsh@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.5.1.3 dlp-product: dlpe-windows authentication-results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [64.43.134.227] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a6261873-5d1f-4a7f-63a2-08d859869574 x-ms-traffictypediagnostic: BYAPR11MB3704: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2887; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: KrzFYXjIyJfshUz1kNNuDLx0qGC0B1w9aWI/6GzKs5HYN3IZeevUaU1mLRK3jnBAlH4f5ODZkUeRRUvdh8N5SwWKgngZCcJO40S9vqAI0NdgewqZAfw40UJneYE5F8XL7S48NeRlDJ35ccgBds6MSR4CnhtfAprLLW1mcmbrITLnUWahgWt5RHzhW/mNVQbwUz2ubjMSb48Q4gVcNwOZKx5eQomaWBWRq8poZych0Q8ZJqXHYDYL4uyvZiCBEeyMFnHRpyUj3jqvZcm+bq7ypm+C9DItOQRpl0Plel5VGdNk+TZywzuUsg6i6X+HUFmlTHnQp8GrdXKmGjU35jXU1hv7uzWAYE2kLDvpURvqmGrevXDOEHs1eg29xJ4NfMg4P7r+ZG8gvxaiEw2YQ191jusveCCGeyXo6tI7f9bjH1fLwTdp1czHKnEOfBB3BvALwFPmH7Cg5tWOzZV39d9SDg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3799.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(376002)(39860400002)(346002)(136003)(478600001)(107886003)(6636002)(26005)(966005)(4326008)(316002)(54906003)(8936002)(110136005)(9686003)(76116006)(5660300002)(66446008)(66946007)(66476007)(66556008)(186003)(7696005)(64756008)(86362001)(8676002)(2906002)(19273905006)(52536014)(55016002)(33656002)(6506007)(71200400001)(563064011); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: UUmXhbp0H+jjvVflfNF5Xl7J0r6O0kdZCrGTA8/kl860Fx+XNPefFf57VbgRoFCEm++QyCI5SQdeChd8qtSlK9s567TIY805I8+rY9iGU/5tfwHNn9aPxQvMkqt8SlZCZVOg7xIXIK1wXSmu550o1NNb/NwhIj0EytEo8c7UBJdTK9xmb6K000myrmFW5G2hwFEvKEb2khREYju82TEIrnvg98dNarg+EiEZRcncPubNOoXzvVBHqdxNUuPGX8scoVkPdIKqPsuvzCK9BEqJ3+bS0QOrEnrInDVY5CsVGWfek4BPb0XuIwwP4oTf723QZ8Jm1IEKYlc5+KpKLELN/CBgZ2qF6Yo2x8zcDzb6L/srkX5JJstKuWUfiCzZzFSxrQOK1MR6nLO9JMRfSSw4RwOR/RnVybbJ1m1RXuqZCeZttMVyd4KV2ft8Sb1jHdtl8v/8E19piNgB6d4aZ8OofmOTnVz88oZ9PhehA3iNqs0xgzgfWtSK1aic8sxLgsbNkyp+Q7oww4wPuInjT/r6TJj2jBOiKQpK+guXFZkb7+eH6KFJbr2xY/IXyDXfZe0J2rpkxyTW0ko7mV1SuaRw7vvZr5Rq8a1pfAnbXevkvkJz7mu9bF5PSQcEhY6y46qsAc9w5xaqfulVzCmcsGNVpg== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3799.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a6261873-5d1f-4a7f-63a2-08d859869574 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Sep 2020 14:49:43.8039 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: a2A/Xs8iUZKRxaSn7RqUgQUKk3/8BK+3qhVU8s/E5WPsEY3qPlz6YdiHB3qhyYq3OyONSvMxUFJGipNuYrcaQA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3704 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > >> This patch adds a script that generates a compressed archive > >> containing .dump files which can be used to perform ABI breakage > >> checking for the build specified in the parameters. > >> Invoke using "./gen-abi-tarball.py -t -a [-cf ]" > >> - : dpdk tag e.g. "v20.11" > >> - : required architecture e.g. "arm" or "x86_64" > >> - : configuration file for cross compiling for another > >> system, this flag is not required. > >> e.g. "config/arm/arm64_armv8_linux_gcc" > >> E.g. "./gen-abi-tarball.py -t latest -a x86_64" > >> If a compiler is not specified using the CC environmental variable > >> then the script will default to using gcc. > >> Using these parameters the script will produce a .tar.gz archive > >> containing .dump files required to do ABI breakage checking > >> > >> Signed-off-by: Conor Walsh > >> --- > > > > Just a general comment: this script looks like it's bash translated to > > python. If you're going to do that, you might as well just write it in > > bash? I am currently reworking this script to make it less BASH like and use more= Python native functions and modules. Thank you for the feedback Anatoly, it was very helpful. > > > >> devtools/gen-abi-tarball.py | 125 > ++++++++++++++++++++++++++++++++++++ > >> 1 file changed, 125 insertions(+) > >> create mode 100755 devtools/gen-abi-tarball.py > >> > >> diff --git a/devtools/gen-abi-tarball.py > >> b/devtools/gen-abi-tarball.py new file mode 100755 index > >> 000000000..06761fca6 > >> --- /dev/null > >> +++ b/devtools/gen-abi-tarball.py > >> @@ -0,0 +1,125 @@ > >> +#!/usr/bin/env python3 > >> +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Intel > >> +Corporation > >> + > >> +import sys > >> +import os > >> +import argparse > >> + > > > > It is preferred to wrap executable code in "if __name__ =3D=3D > > "__main__"..., helps with importing code while avoiding executing it > > on import. >=20 > I wonder, since DPDK is supporting windows, we should prefer python to > bash? I do prefer to use bash because I don't use windows, but maybe tha= t > is a good reason? This patchset does not support Windows currently but if it did in the futur= e having the scripts written in Python would probably make this easier to i= mplement. >=20 > >> +# Get command line arguments > >> +parser =3D argparse.ArgumentParser(usage=3D'\rThis script is intended= to > generate ABI dump tarballs\n'+ > >> + 'Supported environmental > >> +variables\n'+ > >> + '\t- CC: The required compiler will be determined using this > >> environmental variable.\n') > >> +parser.add_argument('-t', '--tag', type=3Dstr, dest=3D'tag', help=3D'= DPDK > >> +tag e.g. latest or v20.11') parser.add_argument('-cf', > >> +'--cross-file', type=3Dstr, > >> dest=3D'crosscompile', help=3D'Set the location of a cross compile > >> config') > >> +parser.add_argument('-a', '--arch', type=3Dstr, dest=3D'arch', > >> +help=3D'Arch arm or x86_64') args =3D parser.parse_args() > >> + > >> +# Get the DPDK tag if not supplied set as latest if args.tag: > >> + user_tag =3D args.tag > >> +else: > >> + user_tag =3D 'latest' > >> + print('No tag supplied defaulting to latest') > > > > There's a "default" option for arguments. > > > >> + > >> +# Get the cross-compile option > >> +if args.crosscompile: > >> + cross_comp =3D args.crosscompile > >> + if not args.arch: > >> + sys.stderr.write('ERROR Arch must be set using -a when using = cross > compile\n') > >> + exit(1) > >> + cross_comp =3D os.path.abspath(cross_comp) > >> + cross_comp_meson =3D '--cross-file '+cross_comp > >> +else: > >> + cross_comp =3D '' > >> + cross_comp_meson =3D '' > >> + > >> +# Get the required system architecture if not supplied set as x86_64 > >> +if args.arch: > >> + arch =3D args.arch > >> +else: > >> + arch =3D os.popen('uname -m').read().strip() > > > > There's a built-in python library for this, i think it's called "platfo= rm". > > > >> + print('No system architecture supplied defaulting to '+arch) > >> + > >> +tag =3D '' > >> +# If the user did not supply tag or wants latest then get latest tag > >> +if user_tag =3D=3D 'latest': > >> + # Get latest quarterly build tag from git repo tag =3D > >> +os.popen('git ls-remote --tags http://dpdk.org/git/dpdk | > >> grep -v "rc\|{}" | tail -n 1 | sed "s/.*\///"').read().strip() > >> +else: > >> + tag =3D user_tag > >> + # If the user supplied tag is not in the DPDK repo then fail > >> +tag_check =3D int(os.popen('git ls-remote http://dpdk.org/git/dpdk > >> refs/tags/'+tag+' | wc -l').read().strip()) > >> + if tag_check !=3D 1: > >> + sys.stderr.write('ERROR supplied tag does not exist in DPDK r= epo\n') > >> + exit(1) > >> + > >> +# Get the specified compiler from system comp_env =3D 'CC' > >> +if comp_env in os.environ: > >> + comp =3D os.environ[comp_env] > >> + comp_default =3D '' > >> +else: > >> + print('No compiler specified, defaulting to gcc') > >> + comp =3D 'gcc' > >> + comp_default =3D 'CC=3Dgcc' > >> + > >> +# Print the configuration to the user print('\nSelected Build: > >> +'+tag+', Compiler: '+comp+', Architecture: > >> '+arch+', Cross Compile: '+cross_comp) > > > > Here and in similar places: please don't use string concatenation, use > > string formatting instead. > > > >> + > >> +# Store the base directory script is working from baseDir =3D > >> +os.getcwd() # Store devtools dir devtoolsDir =3D > >> +os.path.abspath(os.path.dirname(os.path.realpath(sys.argv[0]))) > >> + > >> +# Create directory for DPDK git repo and build > >> +try: > >> + os.mkdir('dump_dpdk') > >> +except OSError as error: > >> + sys.stderr.write('ERROR The dump_dpdk directory could not be > >> created, ensure it does not exist before start\n') > >> + exit(1) > >> +os.chdir('dump_dpdk') > >> +# Clone DPDK and switch to specified tag print('Cloning '+tag+' from > >> +DPDK git') os.popen('git clone --quiet http://dpdk.org/git/dpdk > >> +>/dev/null').read() > >> +os.chdir('dpdk') > >> +os.popen('git checkout --quiet '+tag+' >/dev/null').read() > >> + > >> +# Create build folder with meson and set debug build and cross > >> +compile (if needed) print('Configuring Meson') > >> +os.popen(comp_default+' meson dumpbuild '+cross_comp_meson+' > >> +>/dev/null').read() > > > > You do this os.popen(something > /dev/bull).read() quite a lot, maybe > > put it into a function? Also, since you're discarding the data anyway > > - > > why os.popen().read() instead of os.system()? > > > >> +os.chdir('dumpbuild') > >> +os.popen('meson configure -Dbuildtype=3Ddebug >/dev/null').read() > >> +print('Building DPDK . . .') #Build DPDK with ninja os.popen('ninja > >> +>/dev/null').read() gccDir =3D os.getcwd() > >> + > >> +# Create directory for abi dump files dumpDir =3D > >> +os.path.join(baseDir,tag+'-'+comp+'-'+arch+'-abi_dump') > >> +try: > >> + os.mkdir(dumpDir) > >> +except OSError as error: > >> + sys.stderr.write('ERROR The '+dumpDir+' directory could not be > created\n') > >> + os.popen('rm -rf '+os.path.join(baseDir,'dump_dpdk')).read() > >> + exit(1) > >> + > >> +# Create dump files and output to dump directory print('Generating > >> +ABI dump files') genabiout =3D > >> +os.popen(os.path.join(devtoolsDir,'gen-abi.sh')+' '+gccDir).read() > >> +os.popen('cp dump/* '+dumpDir).read() > >> + > >> +# Compress the dump directory > >> +print('Creating Tarball of dump files') > >> +os.chdir(baseDir) > >> +origSize =3D os.popen('du -sh '+dumpDir+' | sed "s/\s.*$//"').read() > >> +os.popen('tar -czf '+dumpDir.split('/')[-1]+'.tar.gz > >> +'+dumpDir.split('/')[-1]+' >/dev/null').read() newSize =3D > >> +os.popen('du -sh '+dumpDir+'.tar.gz | sed "s/\s.*$//"').read() > >> + > >> +# Remove all temporary directories > >> +print('Cleaning up temporary directories') os.popen('rm -rf > >> +'+dumpDir).read() os.popen('rm -rf > >> +'+os.path.join(baseDir,'dump_dpdk')).read() > >> + > >> +#Print output of the script to the user print('\nDump of DPDK ABI > >> +'+tag+' is available in > >> '+dumpDir.split('/')[-1]+'.tar.gz (Original Size: > >> '+origSize.strip()+', Compressed Size:'+newSize.strip()+')\n') > >> Thanks, Conor.