DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Walsh, Conor" <conor.walsh@intel.com>
To: Aaron Conole <aconole@redhat.com>,
	"Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"Kinsella, Ray" <ray.kinsella@intel.com>,
	"nhorman@tuxdriver.com" <nhorman@tuxdriver.com>,
	"maicolgabriel@hotmail.com" <maicolgabriel@hotmail.com>,
	"thomas@monjalon.net" <thomas@monjalon.net>,
	"Richardson, Bruce" <bruce.richardson@intel.com>
Subject: Re: [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives
Date: Tue, 15 Sep 2020 14:49:43 +0000
Message-ID: <BYAPR11MB37993FACD5B2AF27C2C882AEFF200@BYAPR11MB3799.namprd11.prod.outlook.com> (raw)
In-Reply-To: <f7tlfhbnm5e.fsf@dhcp-25.97.bos.redhat.com>

> >> This patch adds a script that generates a compressed archive
> >> containing .dump files which can be used to perform ABI breakage
> >> checking for the build specified in the parameters.
> >> Invoke using "./gen-abi-tarball.py -t <tag> -a <arch> [-cf <cross-file>]"
> >>   - <tag>: dpdk tag e.g. "v20.11"
> >>   - <arch>: required architecture e.g. "arm" or "x86_64"
> >>   - <cross-file>: configuration file for cross compiling for another
> >>                   system, this flag is not required.
> >>                   e.g. "config/arm/arm64_armv8_linux_gcc"
> >> E.g. "./gen-abi-tarball.py -t latest -a x86_64"
> >> If a compiler is not specified using the CC environmental variable
> >> then the script will default to using gcc.
> >> Using these parameters the script will produce a .tar.gz archive
> >> containing .dump files required to do ABI breakage checking
> >>
> >> Signed-off-by: Conor Walsh <conor.walsh@intel.com>
> >> ---
> >
> > Just a general comment: this script looks like it's bash translated to
> > python. If you're going to do that, you might as well just write it in
> > bash?

I am currently reworking this script to make it less BASH like and use more Python native functions and modules.
Thank you for the feedback Anatoly, it was very helpful.

> >
> >>   devtools/gen-abi-tarball.py | 125
> ++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 125 insertions(+)
> >>   create mode 100755 devtools/gen-abi-tarball.py
> >>
> >> diff --git a/devtools/gen-abi-tarball.py
> >> b/devtools/gen-abi-tarball.py new file mode 100755 index
> >> 000000000..06761fca6
> >> --- /dev/null
> >> +++ b/devtools/gen-abi-tarball.py
> >> @@ -0,0 +1,125 @@
> >> +#!/usr/bin/env python3
> >> +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Intel
> >> +Corporation
> >> +
> >> +import sys
> >> +import os
> >> +import argparse
> >> +
> >
> > It is preferred to wrap executable code in "if __name__ ==
> > "__main__"..., helps with importing code while avoiding executing it
> > on import.
> 
> I wonder, since DPDK is supporting windows, we should prefer python to
> bash?  I do prefer to use bash because I don't use windows, but maybe that
> is a good reason?

This patchset does not support Windows currently but if it did in the future having the scripts written in Python would probably make this easier to implement.

> 
> >> +# Get command line arguments
> >> +parser = argparse.ArgumentParser(usage='\rThis script is intended to
> generate ABI dump tarballs\n'+
> >> +                                       'Supported environmental
> >> +variables\n'+
> >> + '\t- CC: The required compiler will be determined using this
> >> environmental variable.\n')
> >> +parser.add_argument('-t', '--tag', type=str, dest='tag', help='DPDK
> >> +tag e.g. latest or v20.11') parser.add_argument('-cf',
> >> +'--cross-file', type=str,
> >> dest='crosscompile', help='Set the location of a cross compile
> >> config')
> >> +parser.add_argument('-a', '--arch', type=str, dest='arch',
> >> +help='Arch arm or x86_64') args = parser.parse_args()
> >> +
> >> +# Get the DPDK tag if not supplied set as latest if args.tag:
> >> +    user_tag = args.tag
> >> +else:
> >> +    user_tag = 'latest'
> >> +    print('No tag supplied defaulting to latest')
> >
> > There's a "default" option for arguments.
> >
> >> +
> >> +# Get the cross-compile option
> >> +if args.crosscompile:
> >> +    cross_comp = args.crosscompile
> >> +    if not args.arch:
> >> +        sys.stderr.write('ERROR Arch must be set using -a when using cross
> compile\n')
> >> +        exit(1)
> >> +    cross_comp = os.path.abspath(cross_comp)
> >> +    cross_comp_meson = '--cross-file '+cross_comp
> >> +else:
> >> +    cross_comp = ''
> >> +    cross_comp_meson = ''
> >> +
> >> +# Get the required system architecture if not supplied set as x86_64
> >> +if args.arch:
> >> +    arch = args.arch
> >> +else:
> >> +    arch = os.popen('uname -m').read().strip()
> >
> > There's a built-in python library for this, i think it's called "platform".
> >
> >> +    print('No system architecture supplied defaulting to '+arch)
> >> +
> >> +tag = ''
> >> +# If the user did not supply tag or wants latest then get latest tag
> >> +if user_tag == 'latest':
> >> +    # Get latest quarterly build tag from git repo  tag =
> >> +os.popen('git ls-remote --tags http://dpdk.org/git/dpdk |
> >> grep -v "rc\|{}" | tail -n 1 | sed "s/.*\///"').read().strip()
> >> +else:
> >> +    tag = user_tag
> >> +    # If the user supplied tag is not in the DPDK repo then fail
> >> +tag_check = int(os.popen('git ls-remote http://dpdk.org/git/dpdk
> >> refs/tags/'+tag+' | wc -l').read().strip())
> >> +    if tag_check != 1:
> >> +        sys.stderr.write('ERROR supplied tag does not exist in DPDK repo\n')
> >> +        exit(1)
> >> +
> >> +# Get the specified compiler from system comp_env = 'CC'
> >> +if comp_env in os.environ:
> >> +    comp = os.environ[comp_env]
> >> +    comp_default = ''
> >> +else:
> >> +    print('No compiler specified, defaulting to gcc')
> >> +    comp = 'gcc'
> >> +    comp_default = 'CC=gcc'
> >> +
> >> +# Print the configuration to the user print('\nSelected Build:
> >> +'+tag+', Compiler: '+comp+', Architecture:
> >> '+arch+', Cross Compile: '+cross_comp)
> >
> > Here and in similar places: please don't use string concatenation, use
> > string formatting instead.
> >
> >> +
> >> +# Store the base directory script is working from baseDir =
> >> +os.getcwd() # Store devtools dir devtoolsDir =
> >> +os.path.abspath(os.path.dirname(os.path.realpath(sys.argv[0])))
> >> +
> >> +# Create directory for DPDK git repo and build
> >> +try:
> >> +    os.mkdir('dump_dpdk')
> >> +except OSError as error:
> >> + sys.stderr.write('ERROR The dump_dpdk directory could not be
> >> created, ensure it does not exist before start\n')
> >> +    exit(1)
> >> +os.chdir('dump_dpdk')
> >> +# Clone DPDK and switch to specified tag print('Cloning '+tag+' from
> >> +DPDK git') os.popen('git clone --quiet http://dpdk.org/git/dpdk
> >> +>/dev/null').read()
> >> +os.chdir('dpdk')
> >> +os.popen('git checkout --quiet '+tag+' >/dev/null').read()
> >> +
> >> +# Create build folder with meson and set debug build and cross
> >> +compile (if needed) print('Configuring Meson')
> >> +os.popen(comp_default+' meson dumpbuild '+cross_comp_meson+'
> >> +>/dev/null').read()
> >
> > You do this os.popen(something > /dev/bull).read() quite a lot, maybe
> > put it into a function? Also, since you're discarding the data anyway
> > -
> > why os.popen().read() instead of os.system()?
> >
> >> +os.chdir('dumpbuild')
> >> +os.popen('meson configure -Dbuildtype=debug >/dev/null').read()
> >> +print('Building DPDK . . .') #Build DPDK with ninja os.popen('ninja
> >> +>/dev/null').read() gccDir = os.getcwd()
> >> +
> >> +# Create directory for abi dump files dumpDir =
> >> +os.path.join(baseDir,tag+'-'+comp+'-'+arch+'-abi_dump')
> >> +try:
> >> +    os.mkdir(dumpDir)
> >> +except OSError as error:
> >> +    sys.stderr.write('ERROR The '+dumpDir+' directory could not be
> created\n')
> >> +    os.popen('rm -rf '+os.path.join(baseDir,'dump_dpdk')).read()
> >> +    exit(1)
> >> +
> >> +# Create dump files and output to dump directory print('Generating
> >> +ABI dump files') genabiout =
> >> +os.popen(os.path.join(devtoolsDir,'gen-abi.sh')+' '+gccDir).read()
> >> +os.popen('cp dump/* '+dumpDir).read()
> >> +
> >> +# Compress the dump directory
> >> +print('Creating Tarball of dump files')
> >> +os.chdir(baseDir)
> >> +origSize = os.popen('du -sh '+dumpDir+' | sed "s/\s.*$//"').read()
> >> +os.popen('tar -czf '+dumpDir.split('/')[-1]+'.tar.gz
> >> +'+dumpDir.split('/')[-1]+' >/dev/null').read() newSize =
> >> +os.popen('du -sh '+dumpDir+'.tar.gz | sed "s/\s.*$//"').read()
> >> +
> >> +# Remove all temporary directories
> >> +print('Cleaning up temporary directories') os.popen('rm -rf
> >> +'+dumpDir).read() os.popen('rm -rf
> >> +'+os.path.join(baseDir,'dump_dpdk')).read()
> >> +
> >> +#Print output of the script to the user print('\nDump of DPDK ABI
> >> +'+tag+' is available in
> >> '+dumpDir.split('/')[-1]+'.tar.gz (Original Size:
> >> '+origSize.strip()+', Compressed Size:'+newSize.strip()+')\n')
> >>

Thanks,
Conor.


  reply	other threads:[~2020-09-15 15:57 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-10 14:01 [dpdk-dev] [PATCH 0/4] abi breakage checks for meson Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-10 14:21 ` [dpdk-dev] [PATCH v2 0/4] abi breakage checks for meson Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-11 16:03   ` [dpdk-dev] [PATCH v3 0/4] abi breakage checks for meson Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-14 12:50       ` Burakov, Anatoly
2020-09-15 14:35         ` Aaron Conole
2020-09-15 14:49           ` Walsh, Conor [this message]
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-14  8:08     ` [dpdk-dev] [PATCH v3 0/4] abi breakage checks for meson Thomas Monjalon
2020-09-14  8:30       ` Kinsella, Ray
2020-09-14  9:34       ` Kinsella, Ray
2020-09-18 12:11     ` [dpdk-dev] [PATCH v4 " Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 4/4] build: add abi breakage checks to meson Conor Walsh
2020-10-12  8:08       ` [dpdk-dev] [PATCH v5 0/4] devtools: abi breakage checks Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 3/4] devtools: change dump file not found to warning in check-abi.sh Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
2020-10-12 13:03         ` [dpdk-dev] [PATCH v6 0/4] devtools: abi breakage checks Conor Walsh
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-14  9:38             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-14  9:43             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 3/4] devtools: change dump file not found to warning in check-abi.sh Conor Walsh
2020-10-14  9:44             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
2020-10-14  9:46             ` Kinsella, Ray
2020-10-14  9:37           ` [dpdk-dev] [PATCH v6 0/4] devtools: abi breakage checks Kinsella, Ray
2020-10-14 10:33             ` Walsh, Conor
2020-10-14 10:41           ` [dpdk-dev] [PATCH v7 " Conor Walsh
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-15 10:15               ` Kinsella, Ray
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-15 10:16               ` Kinsella, Ray
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 3/4] devtools: change not found to warning check-abi.sh Conor Walsh
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
     [not found]             ` <7206c209-ed4a-2aeb-12d8-ee162ef92596@ashroe.eu>
     [not found]               ` <CAJFAV8wmpft6XLRg1RAL+d4ibbJVrR9C0ghkE-kqyig_q_Meeg@mail.gmail.com>
2020-11-03 10:07                 ` [dpdk-dev] [PATCH v7 0/4] devtools: abi breakage checks Kinsella, Ray
2020-11-10 12:53                   ` David Marchand
2020-11-10 13:54                     ` Kinsella, Ray
2020-11-10 13:57                       ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BYAPR11MB37993FACD5B2AF27C2C882AEFF200@BYAPR11MB3799.namprd11.prod.outlook.com \
    --to=conor.walsh@intel.com \
    --cc=aconole@redhat.com \
    --cc=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=maicolgabriel@hotmail.com \
    --cc=nhorman@tuxdriver.com \
    --cc=ray.kinsella@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git