DPDK patches and discussions
 help / color / mirror / Atom feed
From: Conor Walsh <conor.walsh@intel.com>
To: dev@dpdk.org
Cc: david.marchand@redhat.com, ray.kinsella@intel.com,
	nhorman@tuxdriver.com, aconole@redhat.com,
	maicolgabriel@hotmail.com, thomas@monjalon.net,
	bruce.richardson@intel.com, anatoly.burakov@intel.com,
	Conor Walsh <conor.walsh@intel.com>
Subject: [dpdk-dev] [PATCH v4 2/4] devtools: add generation of compressed abi dump archives
Date: Fri, 18 Sep 2020 12:11:35 +0000
Message-ID: <20200918121137.1370883-3-conor.walsh@intel.com> (raw)
In-Reply-To: <20200918121137.1370883-1-conor.walsh@intel.com>

This patch adds a script that generates a compressed archive
containing .dump files which can be used to perform ABI
breakage checking for the build specified in the parameters.
Invoke using "./gen-abi-tarball.py [-t <tag>] [-a <arch>]
              [-cf <cross-file>]"
 - <tag>: dpdk tag e.g. "v20.11", default: latest
 - <arch>: required architecture e.g. "arm" or "x86_64",
           default: current system's architecture
 - <cross-file>: configuration file for cross compiling for another
                 system, this flag is not required.
                 e.g. "config/arm/arm64_armv8_linux_gcc", default: None
E.g. "./gen-abi-tarball.py -t latest -a x86_64"
If a compiler is not specified using the CC environmental variable then
the script will default to using gcc.
Using these parameters the script will produce a .tar.gz archive
containing .dump files required to do ABI breakage checking

Signed-off-by: Conor Walsh <conor.walsh@intel.com>
---
 devtools/gen-abi-tarball.py | 142 ++++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)
 create mode 100755 devtools/gen-abi-tarball.py

diff --git a/devtools/gen-abi-tarball.py b/devtools/gen-abi-tarball.py
new file mode 100755
index 000000000..6c337104e
--- /dev/null
+++ b/devtools/gen-abi-tarball.py
@@ -0,0 +1,142 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Intel Corporation
+
+"""
+This Python script generates a compressed archive containing .dump
+files which can be used to perform ABI breakage checking for the
+build specified in the parameters.
+"""
+
+import os
+from os.path import abspath, realpath, dirname, basename, join, getsize
+import sys
+import argparse
+import platform
+import tarfile
+import subprocess
+import shutil
+import tempfile
+
+# Get command line options
+def args_parse():
+    parser = argparse.ArgumentParser(
+            description='This script is intended to generate ABI dump tarballs\n\n'+
+                        'Supported environmental variables:\n'+
+                        '\t- CC: The required compiler will be determined using this environmental variable.\n',
+            formatter_class=argparse.RawTextHelpFormatter)
+    parser.add_argument(
+            '-t', '--tag', type=str, dest='tag',
+            help='DPDK tag e.g. latest or v20.11', default='latest')
+    parser.add_argument(
+            '-cf', '--cross-file', type=str, dest='crosscompile',
+            help='Set the location of a cross compile config')
+    parser.add_argument(
+            '-a', '--arch', type=str, dest='arch',
+            help='Architecture arm or x86_64', default=platform.machine())
+    args = parser.parse_args()
+    return args
+
+# Function to execute git commands
+def call_git(args):
+    args = list(filter(None, args))
+    git_call = subprocess.run(['git'] + args, capture_output=True)
+    if git_call.returncode != 0:
+        print('ERROR Git returned an error', file=sys.stderr)
+        exit(1)
+    return git_call.stdout.decode('utf-8').strip()
+
+# Function to execute commands
+def call_exec(args):
+    args = list(filter(None, args))
+    exec_call = subprocess.run(args, stdout=subprocess.DEVNULL)
+    if exec_call.returncode != 0:
+        print('ERROR Script returned an error', file=sys.stderr)
+        exit(1)
+
+# Get the required git tag
+def get_tag(tag):
+    tags = call_git(['ls-remote', '--tags', 'http://dpdk.org/git/dpdk']).split('\n')
+    tags = [t.split('/')[-1].strip() for t in tags if 'rc' not in t and not t.endswith('{}')]
+    if tag == 'latest':
+        tag = tags[-1]
+    if tag not in tags:
+        print('ERROR supplied tag does not exist in DPDK repo', file=sys.stderr)
+        exit(1)
+    return tag
+
+def main():
+    args = args_parse()
+
+    # Get the cross-compile option
+    cross_comp_meson = [None, None]
+    if args.crosscompile:
+        cross_comp_meson = ['--cross-file', abspath(args.crosscompile)]
+
+    tag = get_tag(args.tag)
+
+    # Get the specified compiler from system
+    if 'CC' in os.environ:
+        comp = os.environ['CC']
+    else:
+        print('No compiler specified in environmental varibles, setting CC=gcc')
+        comp = 'gcc'
+        os.environ['CC'] = 'gcc'
+
+    # Print the configuration to the user
+    print('\nSelected Build: {}, Compiler: {}, Architecture: {}, Cross Compile: {}'.format(tag,comp,args.arch,cross_comp_meson[1]))
+
+    # Store the users working directory
+    baseDir = os.getcwd()
+    # Store devtools dir
+    devtoolsDir = abspath(dirname(realpath(sys.argv[0])))
+
+    # Create directory for DPDK git repo and build
+    tmpDir = tempfile.TemporaryDirectory(dir = "/tmp")
+
+    os.chdir(tmpDir.name)
+    # Clone DPDK and switch to specified tag
+    print('Cloning {} from DPDK git'.format(tag))
+    call_git(['clone', '--quiet', 'http://dpdk.org/git/dpdk'])
+    os.chdir('dpdk')
+    call_git(['checkout', '--quiet', tag])
+
+    # Create build folder with meson and set debug build and cross compile (if needed)
+    print('Configuring Meson')
+    call_exec(['meson', '-Dbuildtype=debug', 'dumpbuild'] + cross_comp_meson)
+    #os.system('meson -Dbuildtype=debug dumpbuild {} >/dev/null'.format(cross_comp_meson))
+    print('Building DPDK . . .')
+    #Build DPDK with ninja
+    call_exec(['ninja', '-C', 'dumpbuild'])
+
+    # Create dump files and output to dump directory
+    dumpDir = join(baseDir,'{}-{}-{}-abi_dump'.format(tag,comp,args.arch))
+    print('Generating ABI dump files')
+    call_exec([join(devtoolsDir,'gen-abi.sh'), 'dumpbuild'])
+    try:
+        shutil.copytree('dumpbuild/dump', dumpDir)
+    except FileExistsError as error:
+        print('ERROR The {} directory already exists, ensure it is not present before running script'.format(dumpDir), file=sys.stderr)
+        tmpDir.cleanup()
+        exit(1)
+
+    # Compress the dump directory
+    print('Creating Tarball of dump files')
+    os.chdir(baseDir)
+    origSize = 0
+    for f in os.scandir(dumpDir):
+        origSize += getsize(f)
+    with tarfile.open('{}.tar.gz'.format(dumpDir), "w:gz") as tar:
+        tar.add(dumpDir, arcname=basename(dumpDir))
+    newSize = getsize('{}.tar.gz'.format(dumpDir))
+
+    # Remove all temporary directories
+    print('Cleaning up temporary directories')
+    shutil.rmtree(dumpDir)
+    tmpDir.cleanup()
+
+    #Print output of the script to the user
+    print('\nDump of DPDK ABI {} is available in {}.tar.gz (Original Size: {:.1f}MB, Compressed Size: {:.1f}MB)\n'.format(tag,dumpDir.split('/')[-1],float(origSize)*1e-6,float(newSize)*1e-6))
+
+if __name__ == "__main__":
+    main()
-- 
2.25.1


  parent reply	other threads:[~2020-09-18 12:12 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-10 14:01 [dpdk-dev] [PATCH 0/4] abi breakage checks for meson Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-10 14:01 ` [dpdk-dev] [PATCH 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-10 14:21 ` [dpdk-dev] [PATCH v2 0/4] abi breakage checks for meson Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-10 14:21   ` [dpdk-dev] [PATCH v2 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-11 16:03   ` [dpdk-dev] [PATCH v3 0/4] abi breakage checks for meson Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 2/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-09-14 12:50       ` Burakov, Anatoly
2020-09-15 14:35         ` Aaron Conole
2020-09-15 14:49           ` Walsh, Conor
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-11 16:03     ` [dpdk-dev] [PATCH v3 4/4] build: add abi breakage checks to meson Conor Walsh
2020-09-14  8:08     ` [dpdk-dev] [PATCH v3 0/4] abi breakage checks for meson Thomas Monjalon
2020-09-14  8:30       ` Kinsella, Ray
2020-09-14  9:34       ` Kinsella, Ray
2020-09-18 12:11     ` [dpdk-dev] [PATCH v4 " Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 1/4] devtools: bug fix for gen-abi.sh Conor Walsh
2020-09-18 12:11       ` Conor Walsh [this message]
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 3/4] buildtools: add script to setup abi checks for meson Conor Walsh
2020-09-18 12:11       ` [dpdk-dev] [PATCH v4 4/4] build: add abi breakage checks to meson Conor Walsh
2020-10-12  8:08       ` [dpdk-dev] [PATCH v5 0/4] devtools: abi breakage checks Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 3/4] devtools: change dump file not found to warning in check-abi.sh Conor Walsh
2020-10-12  8:08         ` [dpdk-dev] [PATCH v5 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
2020-10-12 13:03         ` [dpdk-dev] [PATCH v6 0/4] devtools: abi breakage checks Conor Walsh
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-14  9:38             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-14  9:43             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 3/4] devtools: change dump file not found to warning in check-abi.sh Conor Walsh
2020-10-14  9:44             ` Kinsella, Ray
2020-10-12 13:03           ` [dpdk-dev] [PATCH v6 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
2020-10-14  9:46             ` Kinsella, Ray
2020-10-14  9:37           ` [dpdk-dev] [PATCH v6 0/4] devtools: abi breakage checks Kinsella, Ray
2020-10-14 10:33             ` Walsh, Conor
2020-10-14 10:41           ` [dpdk-dev] [PATCH v7 " Conor Walsh
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 1/4] devtools: add generation of compressed abi dump archives Conor Walsh
2020-10-15 10:15               ` Kinsella, Ray
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 2/4] devtools: abi and UX changes for test-meson-builds.sh Conor Walsh
2020-10-15 10:16               ` Kinsella, Ray
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 3/4] devtools: change not found to warning check-abi.sh Conor Walsh
2020-10-14 10:41             ` [dpdk-dev] [PATCH v7 4/4] doc: test-meson-builds.sh doc updates Conor Walsh
     [not found]             ` <7206c209-ed4a-2aeb-12d8-ee162ef92596@ashroe.eu>
     [not found]               ` <CAJFAV8wmpft6XLRg1RAL+d4ibbJVrR9C0ghkE-kqyig_q_Meeg@mail.gmail.com>
2020-11-03 10:07                 ` [dpdk-dev] [PATCH v7 0/4] devtools: abi breakage checks Kinsella, Ray
2020-11-10 12:53                   ` David Marchand
2020-11-10 13:54                     ` Kinsella, Ray
2020-11-10 13:57                       ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200918121137.1370883-3-conor.walsh@intel.com \
    --to=conor.walsh@intel.com \
    --cc=aconole@redhat.com \
    --cc=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=maicolgabriel@hotmail.com \
    --cc=nhorman@tuxdriver.com \
    --cc=ray.kinsella@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git