[dpdk-ci] [PATCH 0/7] first scripts for CI integration

[dpdk-ci] [PATCH 0/7] first scripts for CI integration

[Thomas Monjalon]

These scripts allow to check a patch received by email and
send a report in order to be integrated in patchwork.

The existing CI tests run by Intel could be converted to use
the script send-patch-report.sh so they will be seen in patchwork.

Next steps (to be implemented):
- script to apply a patch on the right tree
- script to apply dependencies (preceding in a series)

Thomas Monjalon (7):
  tools: add mail filter
  tools: add mail parser
  config: add loader and template
  tools: add patchwork client
  tools: add per-patch report mailer
  tools: add patchwork integration
  tests: add checkpatch

 config/ci.config            |  10 +
 config/pwclientrc           |   9 +
 tests/checkpatch.sh         |  42 +++
 tools/filter-patch-email.sh |  76 +++++
 tools/load-ci-config.sh     |  14 +
 tools/parse-email.sh        |  43 +++
 tools/pwclient              | 808 ++++++++++++++++++++++++++++++++++++++++++++
 tools/send-patch-report.sh  | 100 ++++++
 tools/update-pw.sh          |  57 ++++
 9 files changed, 1159 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 config/pwclientrc
 create mode 100755 tests/checkpatch.sh
 create mode 100755 tools/filter-patch-email.sh
 create mode 100644 tools/load-ci-config.sh
 create mode 100755 tools/parse-email.sh
 create mode 100755 tools/pwclient
 create mode 100755 tools/send-patch-report.sh
 create mode 100755 tools/update-pw.sh

-- 
2.7.0

[dpdk-ci] [PATCH 1/7] tools: add mail filter

[Thomas Monjalon]

This script acts as a pipe which blocks non-patch emails.
It can be used as a first filter before processing a patch.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/filter-patch-email.sh | 76 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100755 tools/filter-patch-email.sh

diff --git a/tools/filter-patch-email.sh b/tools/filter-patch-email.sh
new file mode 100755
index 0000000..fd75271
--- /dev/null
+++ b/tools/filter-patch-email.sh
@@ -0,0 +1,76 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) < email
+
+	Filter out email from stdin if does not match patch criterias.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+
+if [ -t 0 ] ; then
+	echo 'nothing to read on stdin' >&2
+	exit 0
+fi
+
+fifo=/tmp/$(basename $0 sh)$$
+mkfifo $fifo
+trap "rm -f $fifo" INT EXIT
+
+parse ()
+{
+	gitsend=false
+	patchsubject=false
+	content=false
+	minusline=false
+	plusline=false
+	atline=false
+	done=false
+	while IFS= read -r line ; do
+		printf '%s\n' "$line"
+		set -- $line
+		if ! $content ; then
+			[ "$1" != 'X-Mailer:' -o "$2" != 'git-send-email' ] || gitsend=true
+			if echo "$line" | grep -qa '^Subject:.*\[PATCH' ; then
+				subject=$(echo "$line" | sed 's,^Subject:[[:space:]]*,,')
+				while [ -n "$subject" ] ; do
+					echo "$subject" | grep -q '^\[' || break
+					if echo "$subject" | grep -q '^\[PATCH' ; then
+						patchsubject=true
+						break
+					fi
+					subject=$(echo "$subject" | sed 's,^[^]]*\][[:space:]]*,,')
+				done
+			fi
+			[ -n "$line" ] || content=true
+		elif ! $done ; then
+			$gitsend || $patchsubject || break
+			[ "$1" != '---' ] || minusline=true
+			[ "$1" != '+++' ] || plusline=true
+			[ "$1" != '@@' ] || atline=true
+			if $minusline && $plusline && $atline ; then
+				echo 1 >$fifo
+				done=true
+			fi
+		fi
+	done
+	$done || echo 0 >$fifo
+	exec >&-
+}
+
+waitparsing ()
+{
+	result=$(cat $fifo)
+	[ "$result" = 0 ] || cat
+}
+
+parse | waitparsing
-- 
2.7.0

[dpdk-ci] [PATCH 2/7] tools: add mail parser

[Thomas Monjalon]

This script get some mail headers from an email file.
The retrieved headers can be used for test and report in other scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/parse-email.sh | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 tools/parse-email.sh

diff --git a/tools/parse-email.sh b/tools/parse-email.sh
new file mode 100755
index 0000000..b6cd2df
--- /dev/null
+++ b/tools/parse-email.sh
@@ -0,0 +1,43 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <email_file>
+
+	Parse basic headers of the email
+	and print them as shell variable assignments to evaluate.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -z "$1" ] ; then
+	printf 'file argument is missing\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+getheader () # <header_name> <email_file>
+{
+	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2"
+}
+
+subject=$(getheader Subject "$1")
+from=$(getheader From "$1")
+msgid=$(getheader Message-Id "$1")
+[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+listid=$(getheader List-Id "$1")
+
+cat <<- END_OF_HEADERS
+	subject="$subject"
+	from="$from"
+	msgid="$msgid"
+	listid="$listid"
+END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH 3/7] config: add loader and template

[Thomas Monjalon]

The configuration file will allow to set some environment-specific
options and paths to be used by the scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config        |  4 ++++
 tools/load-ci-config.sh | 14 ++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 tools/load-ci-config.sh

diff --git a/config/ci.config b/config/ci.config
new file mode 100644
index 0000000..2aeff04
--- /dev/null
+++ b/config/ci.config
@@ -0,0 +1,4 @@
+# Configuration template to be copied in
+#         /etc/dpdk/ci.config
+#     or  ~/.config/dpdk/ci.config
+#     or  .ciconfig
diff --git a/tools/load-ci-config.sh b/tools/load-ci-config.sh
new file mode 100644
index 0000000..9ec18b4
--- /dev/null
+++ b/tools/load-ci-config.sh
@@ -0,0 +1,14 @@
+#! /bin/echo must be loaded with .
+
+# Load DPDK CI config and allow override
+# from system file
+test ! -r /etc/dpdk/ci.config ||
+        . /etc/dpdk/ci.config
+# from user file
+test ! -r ~/.config/dpdk/ci.config ||
+        . ~/.config/dpdk/ci.config
+# from local file
+test ! -r $(dirname $(readlink -m $0))/../.ciconfig ||
+        . $(dirname $(readlink -m $0))/../.ciconfig
+
+# The config files must export variables in the shell style
-- 
2.7.0

[dpdk-ci] [PATCH 4/7] tools: add patchwork client

[Thomas Monjalon]

The script pwclient is a frontend for patchwork.
It is imported from patchwork 1.1.
Another version can be used by setting DPDK_CI_PWCLIENT in config.

The config template pwclientrc must be copied in ~/.pwclientrc.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config  |   3 +
 config/pwclientrc |   9 +
 tools/pwclient    | 808 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 820 insertions(+)
 create mode 100644 config/pwclientrc
 create mode 100755 tools/pwclient

diff --git a/config/ci.config b/config/ci.config
index 2aeff04..722aeb5 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -2,3 +2,6 @@
 #         /etc/dpdk/ci.config
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
+
+# The pwclient script is part of patchwork and is copied in dpdk-ci
+# export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/config/pwclientrc b/config/pwclientrc
new file mode 100644
index 0000000..d73e87e
--- /dev/null
+++ b/config/pwclientrc
@@ -0,0 +1,9 @@
+[options]
+default=dpdk
+
+[dpdk]
+url=http://dpdk.org/dev/patchwork/xmlrpc/
+
+# credentials are useless for read-only access
+#username: myuser
+#password: mypass
diff --git a/tools/pwclient b/tools/pwclient
new file mode 100755
index 0000000..f437786
--- /dev/null
+++ b/tools/pwclient
@@ -0,0 +1,808 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Patchwork command line client
+# Copyright (C) 2008 Nate Case <ncase@xes-inc.com>
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+from __future__ import print_function
+
+import os
+import sys
+try:
+    import xmlrpclib
+except ImportError:
+    # Python 3 has merged/renamed things.
+    import xmlrpc.client as xmlrpclib
+import argparse
+import string
+import subprocess
+import base64
+try:
+    import ConfigParser
+except ImportError:
+    # Python 3 has renamed things.
+    import configparser as ConfigParser
+import shutil
+import re
+
+# Add a shim for Python 2's unicode() helper.
+try:
+    unicode
+except NameError:
+    # Python 3 does everything by unicode now.
+    unicode = str
+
+# Default Patchwork remote XML-RPC server URL
+# This script will check the PW_XMLRPC_URL environment variable
+# for the URL to access.  If that is unspecified, it will fallback to
+# the hardcoded default value specified here.
+DEFAULT_URL = "http://patchwork/xmlrpc/"
+CONFIG_FILE = os.path.expanduser('~/.pwclientrc')
+
+
+class Filter(object):
+
+    """Filter for selecting patches."""
+
+    def __init__(self):
+        # These fields refer to specific objects, so they are special
+        # because we have to resolve them to IDs before passing the
+        # filter to the server
+        self.state = ""
+        self.project = ""
+
+        # The dictionary that gets passed to via XML-RPC
+        self.d = {}
+
+    def add(self, field, value):
+        if field == 'state':
+            self.state = value
+        elif field == 'project':
+            self.project = value
+        else:
+            # OK to add directly
+            self.d[field] = value
+
+    def resolve_ids(self, rpc):
+        """Resolve State, Project, and Person IDs based on filter strings."""
+        if self.state != "":
+            id = state_id_by_name(rpc, self.state)
+            if id == 0:
+                sys.stderr.write("Note: No State found matching %s*, "
+                                 "ignoring filter\n" % self.state)
+            else:
+                self.d['state_id'] = id
+
+        if self.project is not None:
+            id = project_id_by_name(rpc, self.project)
+            if id == 0:
+                sys.stderr.write("Note: No Project found matching %s, "
+                                 "ignoring filter\n" % self.project)
+            else:
+                self.d['project_id'] = id
+
+    def __str__(self):
+        """Return human-readable description of the filter."""
+        return str(self.d)
+
+
+class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+
+    def __init__(self, username=None, password=None, use_https=False):
+        self.username = username
+        self.password = password
+        self.use_https = use_https
+        xmlrpclib.SafeTransport.__init__(self)
+
+    def authenticated(self):
+        return self.username is not None and self.password is not None
+
+    def send_host(self, connection, host):
+        xmlrpclib.Transport.send_host(self, connection, host)
+        if not self.authenticated():
+            return
+        credentials = '%s:%s' % (self.username, self.password)
+        auth = 'Basic ' + base64.encodestring(credentials).strip()
+        connection.putheader('Authorization', auth)
+
+    def make_connection(self, host):
+        if self.use_https:
+            fn = xmlrpclib.SafeTransport.make_connection
+        else:
+            fn = xmlrpclib.Transport.make_connection
+        return fn(self, host)
+
+
+def project_id_by_name(rpc, linkname):
+    """Given a project short name, look up the Project ID."""
+    if len(linkname) == 0:
+        return 0
+    projects = rpc.project_list(linkname, 0)
+    for project in projects:
+        if project['linkname'] == linkname:
+            return project['id']
+    return 0
+
+
+def state_id_by_name(rpc, name):
+    """Given a partial state name, look up the state ID."""
+    if len(name) == 0:
+        return 0
+    states = rpc.state_list(name, 0)
+    for state in states:
+        if state['name'].lower().startswith(name.lower()):
+            return state['id']
+    return 0
+
+
+def person_ids_by_name(rpc, name):
+    """Given a partial name or email address, return a list of the
+    person IDs that match."""
+    if len(name) == 0:
+        return []
+    people = rpc.person_list(name, 0)
+    return [x['id'] for x in people]
+
+
+def list_patches(patches, format_str=None):
+    """Dump a list of patches to stdout."""
+    if format_str:
+        format_field_re = re.compile("%{([a-z0-9_]+)}")
+
+        def patch_field(matchobj):
+            fieldname = matchobj.group(1)
+
+            if fieldname == "_msgid_":
+                # naive way to strip < and > from message-id
+                val = string.strip(str(patch["msgid"]), "<>")
+            else:
+                val = str(patch[fieldname])
+
+            return val
+
+        for patch in patches:
+            print(format_field_re.sub(patch_field, format_str))
+    else:
+        print("%-7s %-12s %s" % ("ID", "State", "Name"))
+        print("%-7s %-12s %s" % ("--", "-----", "----"))
+        for patch in patches:
+            print("%-7d %-12s %s" %
+                  (patch['id'], patch['state'], patch['name']))
+
+
+def action_list(rpc, filter, submitter_str, delegate_str, format_str=None):
+    filter.resolve_ids(rpc)
+
+    if submitter_str is not None:
+        ids = person_ids_by_name(rpc, submitter_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             submitter_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches submitted by %s <%s>:' %
+                      (unicode(person['name']).encode('utf-8'),
+                       unicode(person['email']).encode('utf-8')))
+                f = filter
+                f.add("submitter_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    if delegate_str is not None:
+        ids = person_ids_by_name(rpc, delegate_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             delegate_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches delegated to %s <%s>:' %
+                      (person['name'], person['email']))
+                f = filter
+                f.add("delegate_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    patches = rpc.patch_list(filter.d)
+    list_patches(patches, format_str)
+
+
+def action_projects(rpc):
+    projects = rpc.project_list("", 0)
+    print("%-5s %-24s %s" % ("ID", "Name", "Description"))
+    print("%-5s %-24s %s" % ("--", "----", "-----------"))
+    for project in projects:
+        print("%-5d %-24s %s" % (project['id'],
+                                 project['linkname'],
+                                 project['name']))
+
+
+def action_check_list(rpc):
+    checks = rpc.check_list()
+    print("%-5s %-16s %-8s %s" % ("ID", "Context", "State", "Patch"))
+    print("%-5s %-16s %-8s %s" % ("--", "-------", "-----", "-----"))
+    for check in checks:
+        print("%-5s %-16s %-8s %s" % (check['id'],
+                                      check['context'],
+                                      check['state'],
+                                      check['patch']))
+
+
+def action_check_info(rpc, check_id):
+    check = rpc.check_get(check_id)
+    s = "Information for check id %d" % (check_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(check.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_check_create(rpc, patch_id, context, state, url, description):
+    try:
+        rpc.check_create(patch_id, context, state, url, description)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error creating check: %s\n" % f.faultString)
+
+
+def action_states(rpc):
+    states = rpc.state_list("", 0)
+    print("%-5s %s" % ("ID", "Name"))
+    print("%-5s %s" % ("--", "----"))
+    for state in states:
+        print("%-5d %s" % (state['id'], state['name']))
+
+
+def action_info(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = "Information for patch id %d" % (patch_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(patch.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_get(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = rpc.patch_get_mbox(patch_id)
+
+    if patch == {} or len(s) == 0:
+        sys.stderr.write("Unable to get patch %d\n" % patch_id)
+        sys.exit(1)
+
+    base_fname = fname = os.path.basename(patch['filename'])
+    i = 0
+    while os.path.exists(fname):
+        fname = "%s.%d" % (base_fname, i)
+        i += 1
+
+    try:
+        f = open(fname, "w")
+    except:
+        sys.stderr.write("Unable to open %s for writing\n" % fname)
+        sys.exit(1)
+
+    try:
+        f.write(unicode(s).encode("utf-8"))
+        f.close()
+        print('Saved patch to %s' % fname)
+    except:
+        sys.stderr.write("Failed to write to %s\n" % fname)
+        sys.exit(1)
+
+
+def action_apply(rpc, patch_id, apply_cmd=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    if apply_cmd is None:
+        print('Applying patch #%d to current directory' % patch_id)
+        apply_cmd = ['patch', '-p1']
+    else:
+        print('Applying patch #%d using %s' %
+              (patch_id, repr(' '.join(apply_cmd))))
+
+    print('Description: %s' % patch['name'])
+    s = rpc.patch_get_mbox(patch_id)
+    if len(s) > 0:
+        proc = subprocess.Popen(apply_cmd, stdin=subprocess.PIPE)
+        proc.communicate(unicode(s).encode('utf-8'))
+        return proc.returncode
+    else:
+        sys.stderr.write("Error: No patch content found\n")
+        sys.exit(1)
+
+
+def action_update_patch(rpc, patch_id, state=None, archived=None, commit=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    params = {}
+
+    if state:
+        state_id = state_id_by_name(rpc, state)
+        if state_id == 0:
+            sys.stderr.write("Error: No State found matching %s*\n" % state)
+            sys.exit(1)
+        params['state'] = state_id
+
+    if commit:
+        params['commit_ref'] = commit
+
+    if archived:
+        params['archived'] = archived == 'yes'
+
+    success = False
+    try:
+        success = rpc.patch_set(patch_id, params)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error updating patch: %s\n" % f.faultString)
+
+    if not success:
+        sys.stderr.write("Patch not updated\n")
+
+
+def patch_id_from_hash(rpc, project, hash):
+    try:
+        patch = rpc.patch_get_by_project_hash(project, hash)
+    except xmlrpclib.Fault:
+        # the server may not have the newer patch_get_by_project_hash function,
+        # so fall back to hash-only.
+        patch = rpc.patch_get_by_hash(hash)
+
+    if patch == {}:
+        sys.stderr.write("No patch has the hash provided\n")
+        sys.exit(1)
+
+    patch_id = patch['id']
+    # be super paranoid
+    try:
+        patch_id = int(patch_id)
+    except:
+        sys.stderr.write("Invalid patch ID obtained from server\n")
+        sys.exit(1)
+    return patch_id
+
+auth_actions = ['check_create', 'update']
+
+
+def main():
+    hash_parser = argparse.ArgumentParser(add_help=False)
+    hash_parser.add_argument(
+        '-h', metavar='HASH', dest='hash', action='store',
+        help='''Lookup by patch hash'''
+    )
+    hash_parser.add_argument(
+        'id', metavar='ID', nargs='*', action='store', type=int,
+        help='Patch ID',
+    )
+    hash_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Lookup patch in project'''
+    )
+
+    filter_parser = argparse.ArgumentParser(add_help=False)
+    filter_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Filter by patch state (e.g., 'New', 'Accepted', etc.)'''
+    )
+    filter_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Filter by patch archived state'''
+    )
+    filter_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Filter by project name (see 'projects' for list)'''
+    )
+    filter_parser.add_argument(
+        '-w', metavar='WHO',
+        help='''Filter by submitter (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-d', metavar='WHO',
+        help='''Filter by delegate (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-n', metavar='MAX#',
+        type=int,
+        help='''Return first n results'''
+    )
+    filter_parser.add_argument(
+        '-N', metavar='MAX#',
+        type=int,
+        help='''Return last N results'''
+    )
+    filter_parser.add_argument(
+        '-m', metavar='MESSAGEID',
+        help='''Filter by Message-Id'''
+    )
+    filter_parser.add_argument(
+        '-f', metavar='FORMAT',
+        help='''Print output in the given format. You can use tags matching '''
+        '''fields, e.g. %%{id}, %%{state}, or %%{msgid}.'''
+    )
+    filter_parser.add_argument(
+        'patch_name', metavar='STR', nargs='?',
+        help='substring to search for patches by name',
+    )
+
+    action_parser = argparse.ArgumentParser(
+        prog='pwclient',
+        epilog='Use \'pwclient <command> --help\' for more info',
+    )
+
+    subparsers = action_parser.add_subparsers(
+        title='Commands',
+    )
+    apply_parser = subparsers.add_parser(
+        'apply', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch (in the current dir, using -p1)'''
+    )
+    apply_parser.set_defaults(subcmd='apply')
+    git_am_parser = subparsers.add_parser(
+        'git-am', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch to current git branch using "git am".'''
+    )
+    git_am_parser.set_defaults(subcmd='git_am')
+    git_am_parser.add_argument(
+        '-s', '--signoff',
+        action='store_true',
+        help='''pass --signoff to git-am'''
+    )
+    get_parser = subparsers.add_parser(
+        'get', parents=[hash_parser], conflict_handler='resolve',
+        help='''Download a patch and save it locally'''
+    )
+    get_parser.set_defaults(subcmd='get')
+    info_parser = subparsers.add_parser(
+        'info', parents=[hash_parser], conflict_handler='resolve',
+        help='''Display patchwork info about a given patch ID'''
+    )
+    info_parser.set_defaults(subcmd='info')
+    projects_parser = subparsers.add_parser(
+        'projects',
+        help='''List all projects'''
+    )
+    projects_parser.set_defaults(subcmd='projects')
+    check_list_parser = subparsers.add_parser(
+        'check-list',
+        add_help=False,
+        help='''List all checks'''
+    )
+    check_list_parser.set_defaults(subcmd='check_list')
+    check_info_parser = subparsers.add_parser(
+        'check-info',
+        add_help=False,
+        help='''Show information for a given check'''
+    )
+    check_info_parser.set_defaults(subcmd='check_info')
+    check_info_parser.add_argument(
+        'check_id', metavar='ID', action='store', type=int,
+        help='Check ID',)
+    check_create_parser = subparsers.add_parser(
+        'check-create', parents=[hash_parser], conflict_handler='resolve',
+        help='Add a check to a patch')
+    check_create_parser.set_defaults(subcmd='check_create')
+    check_create_parser.add_argument(
+        '-c', metavar='CONTEXT')
+    check_create_parser.add_argument(
+        '-s', choices=('pending', 'success', 'warning', 'fail'))
+    check_create_parser.add_argument(
+        '-u', metavar='TARGET_URL', default="")
+    check_create_parser.add_argument(
+        '-d', metavar='DESCRIPTION', default="")
+    states_parser = subparsers.add_parser(
+        'states',
+        help='''Show list of potential patch states'''
+    )
+    states_parser.set_defaults(subcmd='states')
+    view_parser = subparsers.add_parser(
+        'view', parents=[hash_parser], conflict_handler='resolve',
+        help='''View a patch'''
+    )
+    view_parser.set_defaults(subcmd='view')
+    update_parser = subparsers.add_parser(
+        'update', parents=[hash_parser], conflict_handler='resolve',
+        help='''Update patch''',
+        epilog='''Using a COMMIT-REF allows for only one ID to be specified''',
+    )
+    update_parser.add_argument(
+        '-c', metavar='COMMIT-REF',
+        help='''commit reference hash'''
+    )
+    update_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Set patch state (e.g., 'Accepted', 'Superseded' etc.)'''
+    )
+    update_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Set patch archived state'''
+    )
+    update_parser.set_defaults(subcmd='update')
+    list_parser = subparsers.add_parser("list",
+                                        # aliases=['search'],
+                                        parents=[filter_parser],
+                                        help='''List patches, using the optional filters specified
+        below and an optional substring to search for patches
+        by name'''
+                                        )
+    list_parser.set_defaults(subcmd='list')
+    search_parser = subparsers.add_parser("search",
+                                          parents=[filter_parser],
+                                          help='''Alias for "list"'''
+                                          )
+    # Poor man's argparse aliases:
+    # We register the "search" parser but effectively use "list" for the
+    # help-text.
+    search_parser.set_defaults(subcmd='list')
+    if len(sys.argv) < 2:
+        action_parser.print_help()
+        sys.exit(0)
+
+    args = action_parser.parse_args()
+    args = dict(vars(args))
+    action = args.get('subcmd')
+
+    if args.get('hash') and len(args.get('id')):
+        # mimic mutual exclusive group
+        locals()[action + '_parser'].error(
+            "[-h HASH] and [ID [ID ...]] are mutually exlusive")
+
+    # set defaults
+    filt = Filter()
+    commit_str = None
+    url = DEFAULT_URL
+
+    archived_str = args.get('a')
+    state_str = args.get('s')
+    project_str = args.get('p')
+    submitter_str = args.get('w')
+    delegate_str = args.get('d')
+    format_str = args.get('f')
+    hash_str = args.get('hash')
+    patch_ids = args.get('id')
+    msgid_str = args.get('m')
+    if args.get('c'):
+        # update multiple IDs with a single commit-hash does not make sense
+        if action == 'update' and patch_ids and len(patch_ids) > 1:
+            update_parser.error(
+                "Declining update with COMMIT-REF on multiple IDs")
+        commit_str = args.get('c')
+
+    if state_str is None and archived_str is None and action == 'update':
+        update_parser.error(
+            'Must specify one or more update options (-a or -s)')
+
+    if args.get('n') is not None:
+        try:
+            filt.add("max_count", args.get('n'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('n'))
+
+    if args.get('N') is not None:
+        try:
+            filt.add("max_count", 0 - args.get('N'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('N'))
+
+    do_signoff = args.get('signoff')
+
+    # grab settings from config files
+    config = ConfigParser.ConfigParser()
+    config.read([CONFIG_FILE])
+
+    if not config.has_section('options') and os.path.exists(CONFIG_FILE):
+        sys.stderr.write('~/.pwclientrc is in the old format. Migrating it...')
+
+        old_project = config.get('base', 'project')
+
+        new_config = ConfigParser.ConfigParser()
+        new_config.add_section('options')
+
+        new_config.set('options', 'default', old_project)
+        new_config.add_section(old_project)
+
+        new_config.set(old_project, 'url', config.get('base', 'url'))
+        if config.has_option('auth', 'username'):
+            new_config.set(
+                old_project, 'username', config.get('auth', 'username'))
+        if config.has_option('auth', 'password'):
+            new_config.set(
+                old_project, 'password', config.get('auth', 'password'))
+
+        old_config_file = CONFIG_FILE + '.orig'
+        shutil.copy2(CONFIG_FILE, old_config_file)
+
+        with open(CONFIG_FILE, 'wb') as fd:
+            new_config.write(fd)
+
+        sys.stderr.write(' Done.\n')
+        sys.stderr.write(
+            'Your old ~/.pwclientrc was saved to %s\n' % old_config_file)
+        sys.stderr.write(
+            'and was converted to the new format. You may want to\n')
+        sys.stderr.write('inspect it before continuing.\n')
+        sys.exit(1)
+
+    if not project_str:
+        try:
+            project_str = config.get('options', 'default')
+        except:
+            action_parser.error(
+                "No default project configured in ~/.pwclientrc")
+
+    if not config.has_section(project_str):
+        sys.stderr.write(
+            'No section for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not config.has_option(project_str, 'url'):
+        sys.stderr.write(
+            'No URL for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not do_signoff and config.has_option('options', 'signoff'):
+        do_signoff = config.getboolean('options', 'signoff')
+    if not do_signoff and config.has_option(project_str, 'signoff'):
+        do_signoff = config.getboolean(project_str, 'signoff')
+
+    url = config.get(project_str, 'url')
+
+    transport = None
+    if action in auth_actions:
+        if config.has_option(project_str, 'username') and \
+                config.has_option(project_str, 'password'):
+
+            use_https = url.startswith('https')
+
+            transport = BasicHTTPAuthTransport(
+                config.get(project_str, 'username'),
+                config.get(project_str, 'password'),
+                use_https)
+
+        else:
+            sys.stderr.write("The %s action requires authentication, but no "
+                             "username or password\nis configured\n" % action)
+            sys.exit(1)
+
+    if project_str:
+        filt.add("project", project_str)
+
+    if state_str:
+        filt.add("state", state_str)
+
+    if archived_str:
+        filt.add("archived", archived_str == 'yes')
+
+    if msgid_str:
+        filt.add("msgid", msgid_str)
+
+    try:
+        rpc = xmlrpclib.Server(url, transport=transport)
+    except:
+        sys.stderr.write("Unable to connect to %s\n" % url)
+        sys.exit(1)
+
+    # It should be safe to assume hash_str is not zero, but who knows..
+    if hash_str is not None:
+        patch_ids = [patch_id_from_hash(rpc, project_str, hash_str)]
+
+    # helper for non_empty() to print correct helptext
+    h = locals()[action + '_parser']
+
+    # Require either hash_str or IDs for
+    def non_empty(h, patch_ids):
+        """Error out if no patch IDs were specified"""
+        if patch_ids is None or len(patch_ids) < 1:
+            sys.stderr.write("Error: Missing Argument! Either [-h HASH] or "
+                             "[ID [ID ...]] are required\n")
+            if h:
+                h.print_help()
+            sys.exit(1)
+        return patch_ids
+
+    if action == 'list' or action == 'search':
+        if args.get('patch_name') is not None:
+            filt.add("name__icontains", args.get('patch_name'))
+        action_list(rpc, filt, submitter_str, delegate_str, format_str)
+
+    elif action.startswith('project'):
+        action_projects(rpc)
+
+
+    elif action.startswith('state'):
+        action_states(rpc)
+
+    elif action == 'view':
+        pager = os.environ.get('PAGER')
+        if pager:
+            pager = subprocess.Popen(
+                pager.split(), stdin=subprocess.PIPE
+            )
+        if pager:
+            i = list()
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    i.append(unicode(s).encode("utf-8"))
+            if len(i) > 0:
+                pager.communicate(input="\n".join(i))
+            pager.stdin.close()
+        else:
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    print(unicode(s).encode("utf-8"))
+
+    elif action == 'info':
+        for patch_id in non_empty(h, patch_ids):
+            action_info(rpc, patch_id)
+
+    elif action == 'get':
+        for patch_id in non_empty(h, patch_ids):
+            action_get(rpc, patch_id)
+
+    elif action == 'apply':
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id)
+            if ret:
+                sys.stderr.write("Apply failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'git_am':
+        cmd = ['git', 'am']
+        if do_signoff:
+            cmd.append('-s')
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id, cmd)
+            if ret:
+                sys.stderr.write("'git am' failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'update':
+        for patch_id in non_empty(h, patch_ids):
+            action_update_patch(rpc, patch_id, state=state_str,
+                                archived=archived_str, commit=commit_str
+                                )
+
+    elif action == 'check_list':
+        action_check_list(rpc)
+
+    elif action == 'check_info':
+        check_id = args['check_id']
+        action_check_info(rpc, check_id)
+
+    elif action == 'check_create':
+        for patch_id in non_empty(h, patch_ids):
+            action_check_create(
+                rpc, patch_id, args['c'], args['s'], args['u'], args['d'])
+
+    else:
+        sys.stderr.write("Unknown action '%s'\n" % action)
+        action_parser.print_help()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
-- 
2.7.0

[dpdk-ci] [PATCH 5/7] tools: add per-patch report mailer

[Thomas Monjalon]

The report sent by this script will be parsed and integrated in patchwork
if the patch was public.
Otherwise it will just send the report to the patch submitter.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config           |   3 ++
 tools/send-patch-report.sh | 100 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)
 create mode 100755 tools/send-patch-report.sh

diff --git a/config/ci.config b/config/ci.config
index 722aeb5..b3b0376 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -3,5 +3,8 @@
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
 
+# The mailer (sendmail, mail, mailx) must support the option -t
+# export DPDK_CI_MAILER=/usr/sbin/sendmail
+
 # The pwclient script is part of patchwork and is copied in dpdk-ci
 # export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/tools/send-patch-report.sh b/tools/send-patch-report.sh
new file mode 100755
index 0000000..34b451d
--- /dev/null
+++ b/tools/send-patch-report.sh
@@ -0,0 +1,100 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) [options] < report
+
+	Send test report in a properly formatted email for patchwork integration.
+	The report is submitted to this script via stdin.
+
+	options:
+	        -t title    subject of the patch email
+	        -f from     sender of the patch email
+	        -m msgid    id of the patch email
+	        -p listid   mailing list publishing the patch
+	        -l label    title of the test
+	        -s status   one of these test results: SUCCESS, WARNING, FAILURE
+	        -d desc     few words to better describe the status
+	        -h          this help
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+unset title
+unset from
+unset msgid
+unset listid
+unset label
+unset status
+unset desc
+while getopts d:f:hl:m:p:s:t: arg ; do
+	case $arg in
+		t ) title=$OPTARG ;;
+		f ) from=$OPTARG ;;
+		m ) msgid=$OPTARG ;;
+		p ) listid=$OPTARG ;;
+		l ) label=$OPTARG ;;
+		s ) status=$OPTARG ;;
+		d ) desc=$OPTARG ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -t 0 ] ; then
+	printf 'nothing to read on stdin\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+report=$(cat)
+
+writeheaders () # <subject> <ref> <to> [cc]
+{
+	echo "Subject: $1"
+	echo "In-Reply-To: $2"
+	echo "References: $2"
+	echo "To: $3"
+	[ -z "$4" ] || echo "Cc: $4"
+	echo
+}
+
+writeheadlines () # <label> <status> <description> [pwid]
+{
+	echo "Test-Label: $1"
+	echo "Test-Status: $2"
+	[ -z "$4" ] || echo "http://dpdk.org/patch/$4"
+	echo
+	echo "_${3}_"
+	echo
+}
+
+if echo "$listid" | grep -q 'dev.dpdk.org' ; then
+	# get patchwork id
+	if [ -n "$msgid" ] ; then
+		for try in $(seq 20) ; do
+			pwid=$($pwclient list -f '%{id}' -m "$msgid")
+			[ -n "$pwid" ] && break || sleep 7
+		done
+	fi
+	[ -n "$pwid" ] || pwid='?'
+	# send public report
+	subject=$(echo $title | sed 's,\[dpdk-dev\] ,,')
+	$failed && cc="$from" || cc=''
+	(
+	writeheaders "|$status| $subject" "$msgid" 'test-report@dpdk.org' "$cc"
+	writeheadlines "$label" "$status" "$desc" "$pwid"
+	echo "$report"
+	) | $sendmail -t
+else
+	# send private report
+	(
+		writeheaders "Re: $title" "$msgid" "$from"
+		writeheadlines "$label" "$status" "$desc"
+		echo "$report"
+	) | $sendmail -t
+fi
-- 
2.7.0

[dpdk-ci] [PATCH 6/7] tools: add patchwork integration

[Thomas Monjalon]

This script is run with patchwork admin credentials.
It is typically installed on dpdk.org and run for each mail
received in the test-report mailing list.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/update-pw.sh | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100755 tools/update-pw.sh

diff --git a/tools/update-pw.sh b/tools/update-pw.sh
new file mode 100755
index 0000000..7f10add
--- /dev/null
+++ b/tools/update-pw.sh
@@ -0,0 +1,57 @@
+#! /bin/sh
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <report_url>
+
+	Add or update a check in patchwork based on a test report.
+	The argument specifies only the last URL parts of the test-report
+	mailing list archives (month/id.html).
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+if [ -z "$1" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+url="http://dpdk.org/ml/archives/test-report/$1"
+mmarker='<!--beginarticle-->'
+for try in $(seq 20) ; do
+	[ -z "$report" -o $try -ge 19 ] || continue # 2 last tries if got something
+	[ $try -le 1 ] || sleep 3 # no delay before first try
+	report=$(curl -sf $url | grep -m1 -A9 "$mmarker") || continue
+	echo "$report" | grep -q '^_.*_$' && break || continue
+done
+if [ -z "$report" ] ; then
+	echo "cannot download report at $url" >&2
+	exit 2
+fi
+
+pwid=$(echo "$report" | sed -rn 's,.*http://.*dpdk.org/.*patch/([0-9]+).*,\1,p')
+label=$(echo "$report" | sed -n 's,.*Test-Label: *,,p')
+status=$(echo "$report" | sed -n 's,.*Test-Status: *,,p')
+desc=$(echo "$report" | sed -n 's,^_\(.*\)_$,\1,p')
+case $status in
+	'SUCCESS') pwstatus='success' ;;
+	'WARNING') pwstatus='warning' ;;
+	'FAILURE') pwstatus='fail' ;;
+esac
+printf 'id = %s\nlabel = %s\nstatus = %s/%s %s\nurl = %s\n' \
+	"$pwid" "$label" "$status" "$pwstatus" "$desc" "$url"
+[ -n "$pwid" -a -n "$label" -a -n "$status" -a -n "$desc" ] || exit 3
+
+pwclient=$(dirname $(readlink -m $0))/pwclient
+$pwclient check-create -c "$label" -s "$pwstatus" -d "$desc" -u "$url" $pwid
-- 
2.7.0

[dpdk-ci] [PATCH 7/7] tests: add checkpatch

[Thomas Monjalon]

This is the first test in this repository.
It runs on dpdk.org and use checkpatch.pl of Linux.

Note that the patch is not applied on a git tree for this basic test.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tests/checkpatch.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100755 tests/checkpatch.sh

diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh
new file mode 100755
index 0000000..b94aaac
--- /dev/null
+++ b/tests/checkpatch.sh
@@ -0,0 +1,42 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) dpdk_dir < email
+
+	Check email-formatted patch from stdin.
+	This test runs checkpatch.pl of Linux via a script in dpdk_dir.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+toolsdir=$(dirname $(readlink -m $0))/../tools
+dpdkdir=$1
+
+email=/tmp/$(basename $0 sh)$$
+$toolsdir/filter-patch-email.sh >$email
+trap "rm -f $email" INT EXIT
+
+eval $($toolsdir/parse-email.sh $email)
+# normal exit if no valid patch in the email
+[ -n "$subject" -a -n "$from" ] || exit 0
+
+failed=false
+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true
+report=$(echo "$report" | sed '1,/^###/d')
+
+label='checkpatch'
+$failed && status='WARNING' || status='SUCCESS'
+$failed && desc='coding style issues' || desc='coding style OK'
+
+echo "$report" | $toolsdir/send-patch-report.sh \
+	-t "$subject" -f "$from" -m "$msgid" -p "$listid" \
+	-l "$label" -s "$status" -d "$desc"
-- 
2.7.0

[dpdk-ci] [PATCH v2 0/7] first scripts for CI integration

[Thomas Monjalon]

These scripts allow to check a patch received by email and
send a report in order to be integrated in patchwork.

The existing CI tests run by Intel could be converted to use
the script send-patch-report.sh so they will be seen in patchwork.

Next steps (to be implemented):
- script to clean and update a git tree
- script to apply a patch on the right tree
- script to apply dependencies (preceding in a series)

Thomas Monjalon (7):
  tools: add mail filter
  tools: add mail parser
  config: add loader and template
  tools: add patchwork client
  tools: add per-patch report mailer
  tools: add patchwork integration
  tests: add checkpatch

changes in v2:
- fix mail parsing (bug with quotes in From:)
- fix public success report (no CC:)

 config/ci.config            |  10 +
 config/pwclientrc           |   9 +
 tests/checkpatch.sh         |  42 +++
 tools/filter-patch-email.sh |  76 +++++
 tools/load-ci-config.sh     |  14 +
 tools/parse-email.sh        |  44 +++
 tools/pwclient              | 808 ++++++++++++++++++++++++++++++++++++++++++++
 tools/send-patch-report.sh  | 100 ++++++
 tools/update-pw.sh          |  57 ++++
 9 files changed, 1160 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 config/pwclientrc
 create mode 100755 tests/checkpatch.sh
 create mode 100755 tools/filter-patch-email.sh
 create mode 100644 tools/load-ci-config.sh
 create mode 100755 tools/parse-email.sh
 create mode 100755 tools/pwclient
 create mode 100755 tools/send-patch-report.sh
 create mode 100755 tools/update-pw.sh

-- 
2.7.0

[dpdk-ci] [PATCH v2 1/7] tools: add mail filter

[Thomas Monjalon]

This script acts as a pipe which blocks non-patch emails.
It can be used as a first filter before processing a patch.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/filter-patch-email.sh | 76 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100755 tools/filter-patch-email.sh

diff --git a/tools/filter-patch-email.sh b/tools/filter-patch-email.sh
new file mode 100755
index 0000000..fd75271
--- /dev/null
+++ b/tools/filter-patch-email.sh
@@ -0,0 +1,76 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) < email
+
+	Filter out email from stdin if does not match patch criterias.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+
+if [ -t 0 ] ; then
+	echo 'nothing to read on stdin' >&2
+	exit 0
+fi
+
+fifo=/tmp/$(basename $0 sh)$$
+mkfifo $fifo
+trap "rm -f $fifo" INT EXIT
+
+parse ()
+{
+	gitsend=false
+	patchsubject=false
+	content=false
+	minusline=false
+	plusline=false
+	atline=false
+	done=false
+	while IFS= read -r line ; do
+		printf '%s\n' "$line"
+		set -- $line
+		if ! $content ; then
+			[ "$1" != 'X-Mailer:' -o "$2" != 'git-send-email' ] || gitsend=true
+			if echo "$line" | grep -qa '^Subject:.*\[PATCH' ; then
+				subject=$(echo "$line" | sed 's,^Subject:[[:space:]]*,,')
+				while [ -n "$subject" ] ; do
+					echo "$subject" | grep -q '^\[' || break
+					if echo "$subject" | grep -q '^\[PATCH' ; then
+						patchsubject=true
+						break
+					fi
+					subject=$(echo "$subject" | sed 's,^[^]]*\][[:space:]]*,,')
+				done
+			fi
+			[ -n "$line" ] || content=true
+		elif ! $done ; then
+			$gitsend || $patchsubject || break
+			[ "$1" != '---' ] || minusline=true
+			[ "$1" != '+++' ] || plusline=true
+			[ "$1" != '@@' ] || atline=true
+			if $minusline && $plusline && $atline ; then
+				echo 1 >$fifo
+				done=true
+			fi
+		fi
+	done
+	$done || echo 0 >$fifo
+	exec >&-
+}
+
+waitparsing ()
+{
+	result=$(cat $fifo)
+	[ "$result" = 0 ] || cat
+}
+
+parse | waitparsing
-- 
2.7.0

[dpdk-ci] [PATCH v2 2/7] tools: add mail parser

[Thomas Monjalon]

This script get some mail headers from an email file.
The retrieved headers can be used for test and report in other scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/parse-email.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100755 tools/parse-email.sh

diff --git a/tools/parse-email.sh b/tools/parse-email.sh
new file mode 100755
index 0000000..2292773
--- /dev/null
+++ b/tools/parse-email.sh
@@ -0,0 +1,44 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <email_file>
+
+	Parse basic headers of the email
+	and print them as shell variable assignments to evaluate.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -z "$1" ] ; then
+	printf 'file argument is missing\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+getheader () # <header_name> <email_file>
+{
+	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2" |
+	sed 's,",\\",g'
+}
+
+subject=$(getheader Subject "$1")
+from=$(getheader From "$1")
+msgid=$(getheader Message-Id "$1")
+[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+listid=$(getheader List-Id "$1")
+
+cat <<- END_OF_HEADERS
+	subject="$subject"
+	from="$from"
+	msgid="$msgid"
+	listid="$listid"
+END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH v2 3/7] config: add loader and template

[Thomas Monjalon]

The configuration file will allow to set some environment-specific
options and paths to be used by the scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config        |  4 ++++
 tools/load-ci-config.sh | 14 ++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 tools/load-ci-config.sh

diff --git a/config/ci.config b/config/ci.config
new file mode 100644
index 0000000..2aeff04
--- /dev/null
+++ b/config/ci.config
@@ -0,0 +1,4 @@
+# Configuration template to be copied in
+#         /etc/dpdk/ci.config
+#     or  ~/.config/dpdk/ci.config
+#     or  .ciconfig
diff --git a/tools/load-ci-config.sh b/tools/load-ci-config.sh
new file mode 100644
index 0000000..9ec18b4
--- /dev/null
+++ b/tools/load-ci-config.sh
@@ -0,0 +1,14 @@
+#! /bin/echo must be loaded with .
+
+# Load DPDK CI config and allow override
+# from system file
+test ! -r /etc/dpdk/ci.config ||
+        . /etc/dpdk/ci.config
+# from user file
+test ! -r ~/.config/dpdk/ci.config ||
+        . ~/.config/dpdk/ci.config
+# from local file
+test ! -r $(dirname $(readlink -m $0))/../.ciconfig ||
+        . $(dirname $(readlink -m $0))/../.ciconfig
+
+# The config files must export variables in the shell style
-- 
2.7.0

[dpdk-ci] [PATCH v2 4/7] tools: add patchwork client

[Thomas Monjalon]

The script pwclient is a frontend for patchwork.
It is imported from patchwork 1.1.
Another version can be used by setting DPDK_CI_PWCLIENT in config.

The config template pwclientrc must be copied in ~/.pwclientrc.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config  |   3 +
 config/pwclientrc |   9 +
 tools/pwclient    | 808 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 820 insertions(+)
 create mode 100644 config/pwclientrc
 create mode 100755 tools/pwclient

diff --git a/config/ci.config b/config/ci.config
index 2aeff04..722aeb5 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -2,3 +2,6 @@
 #         /etc/dpdk/ci.config
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
+
+# The pwclient script is part of patchwork and is copied in dpdk-ci
+# export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/config/pwclientrc b/config/pwclientrc
new file mode 100644
index 0000000..d73e87e
--- /dev/null
+++ b/config/pwclientrc
@@ -0,0 +1,9 @@
+[options]
+default=dpdk
+
+[dpdk]
+url=http://dpdk.org/dev/patchwork/xmlrpc/
+
+# credentials are useless for read-only access
+#username: myuser
+#password: mypass
diff --git a/tools/pwclient b/tools/pwclient
new file mode 100755
index 0000000..f437786
--- /dev/null
+++ b/tools/pwclient
@@ -0,0 +1,808 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Patchwork command line client
+# Copyright (C) 2008 Nate Case <ncase@xes-inc.com>
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+from __future__ import print_function
+
+import os
+import sys
+try:
+    import xmlrpclib
+except ImportError:
+    # Python 3 has merged/renamed things.
+    import xmlrpc.client as xmlrpclib
+import argparse
+import string
+import subprocess
+import base64
+try:
+    import ConfigParser
+except ImportError:
+    # Python 3 has renamed things.
+    import configparser as ConfigParser
+import shutil
+import re
+
+# Add a shim for Python 2's unicode() helper.
+try:
+    unicode
+except NameError:
+    # Python 3 does everything by unicode now.
+    unicode = str
+
+# Default Patchwork remote XML-RPC server URL
+# This script will check the PW_XMLRPC_URL environment variable
+# for the URL to access.  If that is unspecified, it will fallback to
+# the hardcoded default value specified here.
+DEFAULT_URL = "http://patchwork/xmlrpc/"
+CONFIG_FILE = os.path.expanduser('~/.pwclientrc')
+
+
+class Filter(object):
+
+    """Filter for selecting patches."""
+
+    def __init__(self):
+        # These fields refer to specific objects, so they are special
+        # because we have to resolve them to IDs before passing the
+        # filter to the server
+        self.state = ""
+        self.project = ""
+
+        # The dictionary that gets passed to via XML-RPC
+        self.d = {}
+
+    def add(self, field, value):
+        if field == 'state':
+            self.state = value
+        elif field == 'project':
+            self.project = value
+        else:
+            # OK to add directly
+            self.d[field] = value
+
+    def resolve_ids(self, rpc):
+        """Resolve State, Project, and Person IDs based on filter strings."""
+        if self.state != "":
+            id = state_id_by_name(rpc, self.state)
+            if id == 0:
+                sys.stderr.write("Note: No State found matching %s*, "
+                                 "ignoring filter\n" % self.state)
+            else:
+                self.d['state_id'] = id
+
+        if self.project is not None:
+            id = project_id_by_name(rpc, self.project)
+            if id == 0:
+                sys.stderr.write("Note: No Project found matching %s, "
+                                 "ignoring filter\n" % self.project)
+            else:
+                self.d['project_id'] = id
+
+    def __str__(self):
+        """Return human-readable description of the filter."""
+        return str(self.d)
+
+
+class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+
+    def __init__(self, username=None, password=None, use_https=False):
+        self.username = username
+        self.password = password
+        self.use_https = use_https
+        xmlrpclib.SafeTransport.__init__(self)
+
+    def authenticated(self):
+        return self.username is not None and self.password is not None
+
+    def send_host(self, connection, host):
+        xmlrpclib.Transport.send_host(self, connection, host)
+        if not self.authenticated():
+            return
+        credentials = '%s:%s' % (self.username, self.password)
+        auth = 'Basic ' + base64.encodestring(credentials).strip()
+        connection.putheader('Authorization', auth)
+
+    def make_connection(self, host):
+        if self.use_https:
+            fn = xmlrpclib.SafeTransport.make_connection
+        else:
+            fn = xmlrpclib.Transport.make_connection
+        return fn(self, host)
+
+
+def project_id_by_name(rpc, linkname):
+    """Given a project short name, look up the Project ID."""
+    if len(linkname) == 0:
+        return 0
+    projects = rpc.project_list(linkname, 0)
+    for project in projects:
+        if project['linkname'] == linkname:
+            return project['id']
+    return 0
+
+
+def state_id_by_name(rpc, name):
+    """Given a partial state name, look up the state ID."""
+    if len(name) == 0:
+        return 0
+    states = rpc.state_list(name, 0)
+    for state in states:
+        if state['name'].lower().startswith(name.lower()):
+            return state['id']
+    return 0
+
+
+def person_ids_by_name(rpc, name):
+    """Given a partial name or email address, return a list of the
+    person IDs that match."""
+    if len(name) == 0:
+        return []
+    people = rpc.person_list(name, 0)
+    return [x['id'] for x in people]
+
+
+def list_patches(patches, format_str=None):
+    """Dump a list of patches to stdout."""
+    if format_str:
+        format_field_re = re.compile("%{([a-z0-9_]+)}")
+
+        def patch_field(matchobj):
+            fieldname = matchobj.group(1)
+
+            if fieldname == "_msgid_":
+                # naive way to strip < and > from message-id
+                val = string.strip(str(patch["msgid"]), "<>")
+            else:
+                val = str(patch[fieldname])
+
+            return val
+
+        for patch in patches:
+            print(format_field_re.sub(patch_field, format_str))
+    else:
+        print("%-7s %-12s %s" % ("ID", "State", "Name"))
+        print("%-7s %-12s %s" % ("--", "-----", "----"))
+        for patch in patches:
+            print("%-7d %-12s %s" %
+                  (patch['id'], patch['state'], patch['name']))
+
+
+def action_list(rpc, filter, submitter_str, delegate_str, format_str=None):
+    filter.resolve_ids(rpc)
+
+    if submitter_str is not None:
+        ids = person_ids_by_name(rpc, submitter_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             submitter_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches submitted by %s <%s>:' %
+                      (unicode(person['name']).encode('utf-8'),
+                       unicode(person['email']).encode('utf-8')))
+                f = filter
+                f.add("submitter_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    if delegate_str is not None:
+        ids = person_ids_by_name(rpc, delegate_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             delegate_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches delegated to %s <%s>:' %
+                      (person['name'], person['email']))
+                f = filter
+                f.add("delegate_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    patches = rpc.patch_list(filter.d)
+    list_patches(patches, format_str)
+
+
+def action_projects(rpc):
+    projects = rpc.project_list("", 0)
+    print("%-5s %-24s %s" % ("ID", "Name", "Description"))
+    print("%-5s %-24s %s" % ("--", "----", "-----------"))
+    for project in projects:
+        print("%-5d %-24s %s" % (project['id'],
+                                 project['linkname'],
+                                 project['name']))
+
+
+def action_check_list(rpc):
+    checks = rpc.check_list()
+    print("%-5s %-16s %-8s %s" % ("ID", "Context", "State", "Patch"))
+    print("%-5s %-16s %-8s %s" % ("--", "-------", "-----", "-----"))
+    for check in checks:
+        print("%-5s %-16s %-8s %s" % (check['id'],
+                                      check['context'],
+                                      check['state'],
+                                      check['patch']))
+
+
+def action_check_info(rpc, check_id):
+    check = rpc.check_get(check_id)
+    s = "Information for check id %d" % (check_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(check.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_check_create(rpc, patch_id, context, state, url, description):
+    try:
+        rpc.check_create(patch_id, context, state, url, description)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error creating check: %s\n" % f.faultString)
+
+
+def action_states(rpc):
+    states = rpc.state_list("", 0)
+    print("%-5s %s" % ("ID", "Name"))
+    print("%-5s %s" % ("--", "----"))
+    for state in states:
+        print("%-5d %s" % (state['id'], state['name']))
+
+
+def action_info(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = "Information for patch id %d" % (patch_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(patch.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_get(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = rpc.patch_get_mbox(patch_id)
+
+    if patch == {} or len(s) == 0:
+        sys.stderr.write("Unable to get patch %d\n" % patch_id)
+        sys.exit(1)
+
+    base_fname = fname = os.path.basename(patch['filename'])
+    i = 0
+    while os.path.exists(fname):
+        fname = "%s.%d" % (base_fname, i)
+        i += 1
+
+    try:
+        f = open(fname, "w")
+    except:
+        sys.stderr.write("Unable to open %s for writing\n" % fname)
+        sys.exit(1)
+
+    try:
+        f.write(unicode(s).encode("utf-8"))
+        f.close()
+        print('Saved patch to %s' % fname)
+    except:
+        sys.stderr.write("Failed to write to %s\n" % fname)
+        sys.exit(1)
+
+
+def action_apply(rpc, patch_id, apply_cmd=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    if apply_cmd is None:
+        print('Applying patch #%d to current directory' % patch_id)
+        apply_cmd = ['patch', '-p1']
+    else:
+        print('Applying patch #%d using %s' %
+              (patch_id, repr(' '.join(apply_cmd))))
+
+    print('Description: %s' % patch['name'])
+    s = rpc.patch_get_mbox(patch_id)
+    if len(s) > 0:
+        proc = subprocess.Popen(apply_cmd, stdin=subprocess.PIPE)
+        proc.communicate(unicode(s).encode('utf-8'))
+        return proc.returncode
+    else:
+        sys.stderr.write("Error: No patch content found\n")
+        sys.exit(1)
+
+
+def action_update_patch(rpc, patch_id, state=None, archived=None, commit=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    params = {}
+
+    if state:
+        state_id = state_id_by_name(rpc, state)
+        if state_id == 0:
+            sys.stderr.write("Error: No State found matching %s*\n" % state)
+            sys.exit(1)
+        params['state'] = state_id
+
+    if commit:
+        params['commit_ref'] = commit
+
+    if archived:
+        params['archived'] = archived == 'yes'
+
+    success = False
+    try:
+        success = rpc.patch_set(patch_id, params)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error updating patch: %s\n" % f.faultString)
+
+    if not success:
+        sys.stderr.write("Patch not updated\n")
+
+
+def patch_id_from_hash(rpc, project, hash):
+    try:
+        patch = rpc.patch_get_by_project_hash(project, hash)
+    except xmlrpclib.Fault:
+        # the server may not have the newer patch_get_by_project_hash function,
+        # so fall back to hash-only.
+        patch = rpc.patch_get_by_hash(hash)
+
+    if patch == {}:
+        sys.stderr.write("No patch has the hash provided\n")
+        sys.exit(1)
+
+    patch_id = patch['id']
+    # be super paranoid
+    try:
+        patch_id = int(patch_id)
+    except:
+        sys.stderr.write("Invalid patch ID obtained from server\n")
+        sys.exit(1)
+    return patch_id
+
+auth_actions = ['check_create', 'update']
+
+
+def main():
+    hash_parser = argparse.ArgumentParser(add_help=False)
+    hash_parser.add_argument(
+        '-h', metavar='HASH', dest='hash', action='store',
+        help='''Lookup by patch hash'''
+    )
+    hash_parser.add_argument(
+        'id', metavar='ID', nargs='*', action='store', type=int,
+        help='Patch ID',
+    )
+    hash_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Lookup patch in project'''
+    )
+
+    filter_parser = argparse.ArgumentParser(add_help=False)
+    filter_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Filter by patch state (e.g., 'New', 'Accepted', etc.)'''
+    )
+    filter_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Filter by patch archived state'''
+    )
+    filter_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Filter by project name (see 'projects' for list)'''
+    )
+    filter_parser.add_argument(
+        '-w', metavar='WHO',
+        help='''Filter by submitter (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-d', metavar='WHO',
+        help='''Filter by delegate (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-n', metavar='MAX#',
+        type=int,
+        help='''Return first n results'''
+    )
+    filter_parser.add_argument(
+        '-N', metavar='MAX#',
+        type=int,
+        help='''Return last N results'''
+    )
+    filter_parser.add_argument(
+        '-m', metavar='MESSAGEID',
+        help='''Filter by Message-Id'''
+    )
+    filter_parser.add_argument(
+        '-f', metavar='FORMAT',
+        help='''Print output in the given format. You can use tags matching '''
+        '''fields, e.g. %%{id}, %%{state}, or %%{msgid}.'''
+    )
+    filter_parser.add_argument(
+        'patch_name', metavar='STR', nargs='?',
+        help='substring to search for patches by name',
+    )
+
+    action_parser = argparse.ArgumentParser(
+        prog='pwclient',
+        epilog='Use \'pwclient <command> --help\' for more info',
+    )
+
+    subparsers = action_parser.add_subparsers(
+        title='Commands',
+    )
+    apply_parser = subparsers.add_parser(
+        'apply', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch (in the current dir, using -p1)'''
+    )
+    apply_parser.set_defaults(subcmd='apply')
+    git_am_parser = subparsers.add_parser(
+        'git-am', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch to current git branch using "git am".'''
+    )
+    git_am_parser.set_defaults(subcmd='git_am')
+    git_am_parser.add_argument(
+        '-s', '--signoff',
+        action='store_true',
+        help='''pass --signoff to git-am'''
+    )
+    get_parser = subparsers.add_parser(
+        'get', parents=[hash_parser], conflict_handler='resolve',
+        help='''Download a patch and save it locally'''
+    )
+    get_parser.set_defaults(subcmd='get')
+    info_parser = subparsers.add_parser(
+        'info', parents=[hash_parser], conflict_handler='resolve',
+        help='''Display patchwork info about a given patch ID'''
+    )
+    info_parser.set_defaults(subcmd='info')
+    projects_parser = subparsers.add_parser(
+        'projects',
+        help='''List all projects'''
+    )
+    projects_parser.set_defaults(subcmd='projects')
+    check_list_parser = subparsers.add_parser(
+        'check-list',
+        add_help=False,
+        help='''List all checks'''
+    )
+    check_list_parser.set_defaults(subcmd='check_list')
+    check_info_parser = subparsers.add_parser(
+        'check-info',
+        add_help=False,
+        help='''Show information for a given check'''
+    )
+    check_info_parser.set_defaults(subcmd='check_info')
+    check_info_parser.add_argument(
+        'check_id', metavar='ID', action='store', type=int,
+        help='Check ID',)
+    check_create_parser = subparsers.add_parser(
+        'check-create', parents=[hash_parser], conflict_handler='resolve',
+        help='Add a check to a patch')
+    check_create_parser.set_defaults(subcmd='check_create')
+    check_create_parser.add_argument(
+        '-c', metavar='CONTEXT')
+    check_create_parser.add_argument(
+        '-s', choices=('pending', 'success', 'warning', 'fail'))
+    check_create_parser.add_argument(
+        '-u', metavar='TARGET_URL', default="")
+    check_create_parser.add_argument(
+        '-d', metavar='DESCRIPTION', default="")
+    states_parser = subparsers.add_parser(
+        'states',
+        help='''Show list of potential patch states'''
+    )
+    states_parser.set_defaults(subcmd='states')
+    view_parser = subparsers.add_parser(
+        'view', parents=[hash_parser], conflict_handler='resolve',
+        help='''View a patch'''
+    )
+    view_parser.set_defaults(subcmd='view')
+    update_parser = subparsers.add_parser(
+        'update', parents=[hash_parser], conflict_handler='resolve',
+        help='''Update patch''',
+        epilog='''Using a COMMIT-REF allows for only one ID to be specified''',
+    )
+    update_parser.add_argument(
+        '-c', metavar='COMMIT-REF',
+        help='''commit reference hash'''
+    )
+    update_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Set patch state (e.g., 'Accepted', 'Superseded' etc.)'''
+    )
+    update_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Set patch archived state'''
+    )
+    update_parser.set_defaults(subcmd='update')
+    list_parser = subparsers.add_parser("list",
+                                        # aliases=['search'],
+                                        parents=[filter_parser],
+                                        help='''List patches, using the optional filters specified
+        below and an optional substring to search for patches
+        by name'''
+                                        )
+    list_parser.set_defaults(subcmd='list')
+    search_parser = subparsers.add_parser("search",
+                                          parents=[filter_parser],
+                                          help='''Alias for "list"'''
+                                          )
+    # Poor man's argparse aliases:
+    # We register the "search" parser but effectively use "list" for the
+    # help-text.
+    search_parser.set_defaults(subcmd='list')
+    if len(sys.argv) < 2:
+        action_parser.print_help()
+        sys.exit(0)
+
+    args = action_parser.parse_args()
+    args = dict(vars(args))
+    action = args.get('subcmd')
+
+    if args.get('hash') and len(args.get('id')):
+        # mimic mutual exclusive group
+        locals()[action + '_parser'].error(
+            "[-h HASH] and [ID [ID ...]] are mutually exlusive")
+
+    # set defaults
+    filt = Filter()
+    commit_str = None
+    url = DEFAULT_URL
+
+    archived_str = args.get('a')
+    state_str = args.get('s')
+    project_str = args.get('p')
+    submitter_str = args.get('w')
+    delegate_str = args.get('d')
+    format_str = args.get('f')
+    hash_str = args.get('hash')
+    patch_ids = args.get('id')
+    msgid_str = args.get('m')
+    if args.get('c'):
+        # update multiple IDs with a single commit-hash does not make sense
+        if action == 'update' and patch_ids and len(patch_ids) > 1:
+            update_parser.error(
+                "Declining update with COMMIT-REF on multiple IDs")
+        commit_str = args.get('c')
+
+    if state_str is None and archived_str is None and action == 'update':
+        update_parser.error(
+            'Must specify one or more update options (-a or -s)')
+
+    if args.get('n') is not None:
+        try:
+            filt.add("max_count", args.get('n'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('n'))
+
+    if args.get('N') is not None:
+        try:
+            filt.add("max_count", 0 - args.get('N'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('N'))
+
+    do_signoff = args.get('signoff')
+
+    # grab settings from config files
+    config = ConfigParser.ConfigParser()
+    config.read([CONFIG_FILE])
+
+    if not config.has_section('options') and os.path.exists(CONFIG_FILE):
+        sys.stderr.write('~/.pwclientrc is in the old format. Migrating it...')
+
+        old_project = config.get('base', 'project')
+
+        new_config = ConfigParser.ConfigParser()
+        new_config.add_section('options')
+
+        new_config.set('options', 'default', old_project)
+        new_config.add_section(old_project)
+
+        new_config.set(old_project, 'url', config.get('base', 'url'))
+        if config.has_option('auth', 'username'):
+            new_config.set(
+                old_project, 'username', config.get('auth', 'username'))
+        if config.has_option('auth', 'password'):
+            new_config.set(
+                old_project, 'password', config.get('auth', 'password'))
+
+        old_config_file = CONFIG_FILE + '.orig'
+        shutil.copy2(CONFIG_FILE, old_config_file)
+
+        with open(CONFIG_FILE, 'wb') as fd:
+            new_config.write(fd)
+
+        sys.stderr.write(' Done.\n')
+        sys.stderr.write(
+            'Your old ~/.pwclientrc was saved to %s\n' % old_config_file)
+        sys.stderr.write(
+            'and was converted to the new format. You may want to\n')
+        sys.stderr.write('inspect it before continuing.\n')
+        sys.exit(1)
+
+    if not project_str:
+        try:
+            project_str = config.get('options', 'default')
+        except:
+            action_parser.error(
+                "No default project configured in ~/.pwclientrc")
+
+    if not config.has_section(project_str):
+        sys.stderr.write(
+            'No section for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not config.has_option(project_str, 'url'):
+        sys.stderr.write(
+            'No URL for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not do_signoff and config.has_option('options', 'signoff'):
+        do_signoff = config.getboolean('options', 'signoff')
+    if not do_signoff and config.has_option(project_str, 'signoff'):
+        do_signoff = config.getboolean(project_str, 'signoff')
+
+    url = config.get(project_str, 'url')
+
+    transport = None
+    if action in auth_actions:
+        if config.has_option(project_str, 'username') and \
+                config.has_option(project_str, 'password'):
+
+            use_https = url.startswith('https')
+
+            transport = BasicHTTPAuthTransport(
+                config.get(project_str, 'username'),
+                config.get(project_str, 'password'),
+                use_https)
+
+        else:
+            sys.stderr.write("The %s action requires authentication, but no "
+                             "username or password\nis configured\n" % action)
+            sys.exit(1)
+
+    if project_str:
+        filt.add("project", project_str)
+
+    if state_str:
+        filt.add("state", state_str)
+
+    if archived_str:
+        filt.add("archived", archived_str == 'yes')
+
+    if msgid_str:
+        filt.add("msgid", msgid_str)
+
+    try:
+        rpc = xmlrpclib.Server(url, transport=transport)
+    except:
+        sys.stderr.write("Unable to connect to %s\n" % url)
+        sys.exit(1)
+
+    # It should be safe to assume hash_str is not zero, but who knows..
+    if hash_str is not None:
+        patch_ids = [patch_id_from_hash(rpc, project_str, hash_str)]
+
+    # helper for non_empty() to print correct helptext
+    h = locals()[action + '_parser']
+
+    # Require either hash_str or IDs for
+    def non_empty(h, patch_ids):
+        """Error out if no patch IDs were specified"""
+        if patch_ids is None or len(patch_ids) < 1:
+            sys.stderr.write("Error: Missing Argument! Either [-h HASH] or "
+                             "[ID [ID ...]] are required\n")
+            if h:
+                h.print_help()
+            sys.exit(1)
+        return patch_ids
+
+    if action == 'list' or action == 'search':
+        if args.get('patch_name') is not None:
+            filt.add("name__icontains", args.get('patch_name'))
+        action_list(rpc, filt, submitter_str, delegate_str, format_str)
+
+    elif action.startswith('project'):
+        action_projects(rpc)
+
+
+    elif action.startswith('state'):
+        action_states(rpc)
+
+    elif action == 'view':
+        pager = os.environ.get('PAGER')
+        if pager:
+            pager = subprocess.Popen(
+                pager.split(), stdin=subprocess.PIPE
+            )
+        if pager:
+            i = list()
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    i.append(unicode(s).encode("utf-8"))
+            if len(i) > 0:
+                pager.communicate(input="\n".join(i))
+            pager.stdin.close()
+        else:
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    print(unicode(s).encode("utf-8"))
+
+    elif action == 'info':
+        for patch_id in non_empty(h, patch_ids):
+            action_info(rpc, patch_id)
+
+    elif action == 'get':
+        for patch_id in non_empty(h, patch_ids):
+            action_get(rpc, patch_id)
+
+    elif action == 'apply':
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id)
+            if ret:
+                sys.stderr.write("Apply failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'git_am':
+        cmd = ['git', 'am']
+        if do_signoff:
+            cmd.append('-s')
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id, cmd)
+            if ret:
+                sys.stderr.write("'git am' failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'update':
+        for patch_id in non_empty(h, patch_ids):
+            action_update_patch(rpc, patch_id, state=state_str,
+                                archived=archived_str, commit=commit_str
+                                )
+
+    elif action == 'check_list':
+        action_check_list(rpc)
+
+    elif action == 'check_info':
+        check_id = args['check_id']
+        action_check_info(rpc, check_id)
+
+    elif action == 'check_create':
+        for patch_id in non_empty(h, patch_ids):
+            action_check_create(
+                rpc, patch_id, args['c'], args['s'], args['u'], args['d'])
+
+    else:
+        sys.stderr.write("Unknown action '%s'\n" % action)
+        action_parser.print_help()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
-- 
2.7.0

[dpdk-ci] [PATCH v2 5/7] tools: add per-patch report mailer

[Thomas Monjalon]

The report sent by this script will be parsed and integrated in patchwork
if the patch was public.
Otherwise it will just send the report to the patch submitter.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config           |   3 ++
 tools/send-patch-report.sh | 100 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)
 create mode 100755 tools/send-patch-report.sh

diff --git a/config/ci.config b/config/ci.config
index 722aeb5..b3b0376 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -3,5 +3,8 @@
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
 
+# The mailer (sendmail, mail, mailx) must support the option -t
+# export DPDK_CI_MAILER=/usr/sbin/sendmail
+
 # The pwclient script is part of patchwork and is copied in dpdk-ci
 # export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/tools/send-patch-report.sh b/tools/send-patch-report.sh
new file mode 100755
index 0000000..afdf542
--- /dev/null
+++ b/tools/send-patch-report.sh
@@ -0,0 +1,100 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) [options] < report
+
+	Send test report in a properly formatted email for patchwork integration.
+	The report is submitted to this script via stdin.
+
+	options:
+	        -t title    subject of the patch email
+	        -f from     sender of the patch email
+	        -m msgid    id of the patch email
+	        -p listid   mailing list publishing the patch
+	        -l label    title of the test
+	        -s status   one of these test results: SUCCESS, WARNING, FAILURE
+	        -d desc     few words to better describe the status
+	        -h          this help
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+unset title
+unset from
+unset msgid
+unset listid
+unset label
+unset status
+unset desc
+while getopts d:f:hl:m:p:s:t: arg ; do
+	case $arg in
+		t ) title=$OPTARG ;;
+		f ) from=$OPTARG ;;
+		m ) msgid=$OPTARG ;;
+		p ) listid=$OPTARG ;;
+		l ) label=$OPTARG ;;
+		s ) status=$OPTARG ;;
+		d ) desc=$OPTARG ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -t 0 ] ; then
+	printf 'nothing to read on stdin\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+report=$(cat)
+
+writeheaders () # <subject> <ref> <to> [cc]
+{
+	echo "Subject: $1"
+	echo "In-Reply-To: $2"
+	echo "References: $2"
+	echo "To: $3"
+	[ -z "$4" ] || echo "Cc: $4"
+	echo
+}
+
+writeheadlines () # <label> <status> <description> [pwid]
+{
+	echo "Test-Label: $1"
+	echo "Test-Status: $2"
+	[ -z "$4" ] || echo "http://dpdk.org/patch/$4"
+	echo
+	echo "_${3}_"
+	echo
+}
+
+if echo "$listid" | grep -q 'dev.dpdk.org' ; then
+	# get patchwork id
+	if [ -n "$msgid" ] ; then
+		for try in $(seq 20) ; do
+			pwid=$($pwclient list -f '%{id}' -m "$msgid")
+			[ -n "$pwid" ] && break || sleep 7
+		done
+	fi
+	[ -n "$pwid" ] || pwid='?'
+	# send public report
+	subject=$(echo $title | sed 's,\[dpdk-dev\] ,,')
+	[ "$status" = 'SUCCESS' ] && cc='' || cc="$from"
+	(
+	writeheaders "|$status| $subject" "$msgid" 'test-report@dpdk.org' "$cc"
+	writeheadlines "$label" "$status" "$desc" "$pwid"
+	echo "$report"
+	) | $sendmail -t
+else
+	# send private report
+	(
+		writeheaders "Re: $title" "$msgid" "$from"
+		writeheadlines "$label" "$status" "$desc"
+		echo "$report"
+	) | $sendmail -t
+fi
-- 
2.7.0

[dpdk-ci] [PATCH v2 6/7] tools: add patchwork integration

[Thomas Monjalon]

This script is run with patchwork admin credentials.
It is typically installed on dpdk.org and run for each mail
received in the test-report mailing list.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/update-pw.sh | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100755 tools/update-pw.sh

diff --git a/tools/update-pw.sh b/tools/update-pw.sh
new file mode 100755
index 0000000..7f10add
--- /dev/null
+++ b/tools/update-pw.sh
@@ -0,0 +1,57 @@
+#! /bin/sh
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <report_url>
+
+	Add or update a check in patchwork based on a test report.
+	The argument specifies only the last URL parts of the test-report
+	mailing list archives (month/id.html).
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+if [ -z "$1" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+url="http://dpdk.org/ml/archives/test-report/$1"
+mmarker='<!--beginarticle-->'
+for try in $(seq 20) ; do
+	[ -z "$report" -o $try -ge 19 ] || continue # 2 last tries if got something
+	[ $try -le 1 ] || sleep 3 # no delay before first try
+	report=$(curl -sf $url | grep -m1 -A9 "$mmarker") || continue
+	echo "$report" | grep -q '^_.*_$' && break || continue
+done
+if [ -z "$report" ] ; then
+	echo "cannot download report at $url" >&2
+	exit 2
+fi
+
+pwid=$(echo "$report" | sed -rn 's,.*http://.*dpdk.org/.*patch/([0-9]+).*,\1,p')
+label=$(echo "$report" | sed -n 's,.*Test-Label: *,,p')
+status=$(echo "$report" | sed -n 's,.*Test-Status: *,,p')
+desc=$(echo "$report" | sed -n 's,^_\(.*\)_$,\1,p')
+case $status in
+	'SUCCESS') pwstatus='success' ;;
+	'WARNING') pwstatus='warning' ;;
+	'FAILURE') pwstatus='fail' ;;
+esac
+printf 'id = %s\nlabel = %s\nstatus = %s/%s %s\nurl = %s\n' \
+	"$pwid" "$label" "$status" "$pwstatus" "$desc" "$url"
+[ -n "$pwid" -a -n "$label" -a -n "$status" -a -n "$desc" ] || exit 3
+
+pwclient=$(dirname $(readlink -m $0))/pwclient
+$pwclient check-create -c "$label" -s "$pwstatus" -d "$desc" -u "$url" $pwid
-- 
2.7.0

[dpdk-ci] [PATCH v2 7/7] tests: add checkpatch

[Thomas Monjalon]

This is the first test in this repository.
It runs on dpdk.org and use checkpatch.pl of Linux.

Note that the patch is not applied on a git tree for this basic test.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tests/checkpatch.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100755 tests/checkpatch.sh

diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh
new file mode 100755
index 0000000..b94aaac
--- /dev/null
+++ b/tests/checkpatch.sh
@@ -0,0 +1,42 @@
+#! /bin/sh -e
+
+# This file is in the public domain.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) dpdk_dir < email
+
+	Check email-formatted patch from stdin.
+	This test runs checkpatch.pl of Linux via a script in dpdk_dir.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+toolsdir=$(dirname $(readlink -m $0))/../tools
+dpdkdir=$1
+
+email=/tmp/$(basename $0 sh)$$
+$toolsdir/filter-patch-email.sh >$email
+trap "rm -f $email" INT EXIT
+
+eval $($toolsdir/parse-email.sh $email)
+# normal exit if no valid patch in the email
+[ -n "$subject" -a -n "$from" ] || exit 0
+
+failed=false
+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true
+report=$(echo "$report" | sed '1,/^###/d')
+
+label='checkpatch'
+$failed && status='WARNING' || status='SUCCESS'
+$failed && desc='coding style issues' || desc='coding style OK'
+
+echo "$report" | $toolsdir/send-patch-report.sh \
+	-t "$subject" -f "$from" -m "$msgid" -p "$listid" \
+	-l "$label" -s "$status" -d "$desc"
-- 
2.7.0

[dpdk-ci] [PATCH v3 0/7] first scripts for CI integration

[Thomas Monjalon]

These scripts allow to check a patch received by email and
send a report in order to be integrated in patchwork.

The existing CI tests run by Intel could be converted to use
the script send-patch-report.sh so they will be seen in patchwork.

Next steps (to be implemented):
- script to clean and update a git tree
- script to apply a patch on the right tree
- script to apply dependencies (preceding in a series)

---

changes in v3:
- BSD licensing

changes in v2:
- fix mail parsing (bug with quotes in From:)
- fix public success report (no CC:)

---

Thomas Monjalon (7):
  tools: add mail filter
  tools: add mail parser
  config: add loader and template
  tools: add patchwork client
  tools: add per-patch report mailer
  tools: add patchwork integration
  tests: add checkpatch

 config/ci.config            |  10 +
 config/pwclientrc           |   9 +
 tests/checkpatch.sh         |  70 ++++
 tools/filter-patch-email.sh | 104 ++++++
 tools/load-ci-config.sh     |  14 +
 tools/parse-email.sh        |  72 ++++
 tools/pwclient              | 808 ++++++++++++++++++++++++++++++++++++++++++++
 tools/send-patch-report.sh  | 128 +++++++
 tools/update-pw.sh          |  85 +++++
 9 files changed, 1300 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 config/pwclientrc
 create mode 100755 tests/checkpatch.sh
 create mode 100755 tools/filter-patch-email.sh
 create mode 100644 tools/load-ci-config.sh
 create mode 100755 tools/parse-email.sh
 create mode 100755 tools/pwclient
 create mode 100755 tools/send-patch-report.sh
 create mode 100755 tools/update-pw.sh

-- 
2.7.0

[dpdk-ci] [PATCH v3 1/7] tools: add mail filter

[Thomas Monjalon]

This script acts as a pipe which blocks non-patch emails.
It can be used as a first filter before processing a patch.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/filter-patch-email.sh | 104 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)
 create mode 100755 tools/filter-patch-email.sh

diff --git a/tools/filter-patch-email.sh b/tools/filter-patch-email.sh
new file mode 100755
index 0000000..a59e739
--- /dev/null
+++ b/tools/filter-patch-email.sh
@@ -0,0 +1,104 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) < email
+
+	Filter out email from stdin if does not match patch criterias.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+
+if [ -t 0 ] ; then
+	echo 'nothing to read on stdin' >&2
+	exit 0
+fi
+
+fifo=/tmp/$(basename $0 sh)$$
+mkfifo $fifo
+trap "rm -f $fifo" INT EXIT
+
+parse ()
+{
+	gitsend=false
+	patchsubject=false
+	content=false
+	minusline=false
+	plusline=false
+	atline=false
+	done=false
+	while IFS= read -r line ; do
+		printf '%s\n' "$line"
+		set -- $line
+		if ! $content ; then
+			[ "$1" != 'X-Mailer:' -o "$2" != 'git-send-email' ] || gitsend=true
+			if echo "$line" | grep -qa '^Subject:.*\[PATCH' ; then
+				subject=$(echo "$line" | sed 's,^Subject:[[:space:]]*,,')
+				while [ -n "$subject" ] ; do
+					echo "$subject" | grep -q '^\[' || break
+					if echo "$subject" | grep -q '^\[PATCH' ; then
+						patchsubject=true
+						break
+					fi
+					subject=$(echo "$subject" | sed 's,^[^]]*\][[:space:]]*,,')
+				done
+			fi
+			[ -n "$line" ] || content=true
+		elif ! $done ; then
+			$gitsend || $patchsubject || break
+			[ "$1" != '---' ] || minusline=true
+			[ "$1" != '+++' ] || plusline=true
+			[ "$1" != '@@' ] || atline=true
+			if $minusline && $plusline && $atline ; then
+				echo 1 >$fifo
+				done=true
+			fi
+		fi
+	done
+	$done || echo 0 >$fifo
+	exec >&-
+}
+
+waitparsing ()
+{
+	result=$(cat $fifo)
+	[ "$result" = 0 ] || cat
+}
+
+parse | waitparsing
-- 
2.7.0

[dpdk-ci] [PATCH v3 2/7] tools: add mail parser

[Thomas Monjalon]

This script get some mail headers from an email file.
The retrieved headers can be used for test and report in other scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/parse-email.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100755 tools/parse-email.sh

diff --git a/tools/parse-email.sh b/tools/parse-email.sh
new file mode 100755
index 0000000..d92c246
--- /dev/null
+++ b/tools/parse-email.sh
@@ -0,0 +1,72 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <email_file>
+
+	Parse basic headers of the email
+	and print them as shell variable assignments to evaluate.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -z "$1" ] ; then
+	printf 'file argument is missing\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+getheader () # <header_name> <email_file>
+{
+	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2" |
+	sed 's,",\\",g'
+}
+
+subject=$(getheader Subject "$1")
+from=$(getheader From "$1")
+msgid=$(getheader Message-Id "$1")
+[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+listid=$(getheader List-Id "$1")
+
+cat <<- END_OF_HEADERS
+	subject="$subject"
+	from="$from"
+	msgid="$msgid"
+	listid="$listid"
+END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH v3 3/7] config: add loader and template

[Thomas Monjalon]

The configuration file will allow to set some environment-specific
options and paths to be used by the scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config        |  4 ++++
 tools/load-ci-config.sh | 14 ++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 tools/load-ci-config.sh

diff --git a/config/ci.config b/config/ci.config
new file mode 100644
index 0000000..2aeff04
--- /dev/null
+++ b/config/ci.config
@@ -0,0 +1,4 @@
+# Configuration template to be copied in
+#         /etc/dpdk/ci.config
+#     or  ~/.config/dpdk/ci.config
+#     or  .ciconfig
diff --git a/tools/load-ci-config.sh b/tools/load-ci-config.sh
new file mode 100644
index 0000000..9ec18b4
--- /dev/null
+++ b/tools/load-ci-config.sh
@@ -0,0 +1,14 @@
+#! /bin/echo must be loaded with .
+
+# Load DPDK CI config and allow override
+# from system file
+test ! -r /etc/dpdk/ci.config ||
+        . /etc/dpdk/ci.config
+# from user file
+test ! -r ~/.config/dpdk/ci.config ||
+        . ~/.config/dpdk/ci.config
+# from local file
+test ! -r $(dirname $(readlink -m $0))/../.ciconfig ||
+        . $(dirname $(readlink -m $0))/../.ciconfig
+
+# The config files must export variables in the shell style
-- 
2.7.0

[dpdk-ci] [PATCH v3 4/7] tools: add patchwork client

[Thomas Monjalon]

The script pwclient is a frontend for patchwork.
It is imported from patchwork 1.1.
Another version can be used by setting DPDK_CI_PWCLIENT in config.

The config template pwclientrc must be copied in ~/.pwclientrc.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config  |   3 +
 config/pwclientrc |   9 +
 tools/pwclient    | 808 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 820 insertions(+)
 create mode 100644 config/pwclientrc
 create mode 100755 tools/pwclient

diff --git a/config/ci.config b/config/ci.config
index 2aeff04..722aeb5 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -2,3 +2,6 @@
 #         /etc/dpdk/ci.config
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
+
+# The pwclient script is part of patchwork and is copied in dpdk-ci
+# export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/config/pwclientrc b/config/pwclientrc
new file mode 100644
index 0000000..d73e87e
--- /dev/null
+++ b/config/pwclientrc
@@ -0,0 +1,9 @@
+[options]
+default=dpdk
+
+[dpdk]
+url=http://dpdk.org/dev/patchwork/xmlrpc/
+
+# credentials are useless for read-only access
+#username: myuser
+#password: mypass
diff --git a/tools/pwclient b/tools/pwclient
new file mode 100755
index 0000000..f437786
--- /dev/null
+++ b/tools/pwclient
@@ -0,0 +1,808 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Patchwork command line client
+# Copyright (C) 2008 Nate Case <ncase@xes-inc.com>
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+from __future__ import print_function
+
+import os
+import sys
+try:
+    import xmlrpclib
+except ImportError:
+    # Python 3 has merged/renamed things.
+    import xmlrpc.client as xmlrpclib
+import argparse
+import string
+import subprocess
+import base64
+try:
+    import ConfigParser
+except ImportError:
+    # Python 3 has renamed things.
+    import configparser as ConfigParser
+import shutil
+import re
+
+# Add a shim for Python 2's unicode() helper.
+try:
+    unicode
+except NameError:
+    # Python 3 does everything by unicode now.
+    unicode = str
+
+# Default Patchwork remote XML-RPC server URL
+# This script will check the PW_XMLRPC_URL environment variable
+# for the URL to access.  If that is unspecified, it will fallback to
+# the hardcoded default value specified here.
+DEFAULT_URL = "http://patchwork/xmlrpc/"
+CONFIG_FILE = os.path.expanduser('~/.pwclientrc')
+
+
+class Filter(object):
+
+    """Filter for selecting patches."""
+
+    def __init__(self):
+        # These fields refer to specific objects, so they are special
+        # because we have to resolve them to IDs before passing the
+        # filter to the server
+        self.state = ""
+        self.project = ""
+
+        # The dictionary that gets passed to via XML-RPC
+        self.d = {}
+
+    def add(self, field, value):
+        if field == 'state':
+            self.state = value
+        elif field == 'project':
+            self.project = value
+        else:
+            # OK to add directly
+            self.d[field] = value
+
+    def resolve_ids(self, rpc):
+        """Resolve State, Project, and Person IDs based on filter strings."""
+        if self.state != "":
+            id = state_id_by_name(rpc, self.state)
+            if id == 0:
+                sys.stderr.write("Note: No State found matching %s*, "
+                                 "ignoring filter\n" % self.state)
+            else:
+                self.d['state_id'] = id
+
+        if self.project is not None:
+            id = project_id_by_name(rpc, self.project)
+            if id == 0:
+                sys.stderr.write("Note: No Project found matching %s, "
+                                 "ignoring filter\n" % self.project)
+            else:
+                self.d['project_id'] = id
+
+    def __str__(self):
+        """Return human-readable description of the filter."""
+        return str(self.d)
+
+
+class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+
+    def __init__(self, username=None, password=None, use_https=False):
+        self.username = username
+        self.password = password
+        self.use_https = use_https
+        xmlrpclib.SafeTransport.__init__(self)
+
+    def authenticated(self):
+        return self.username is not None and self.password is not None
+
+    def send_host(self, connection, host):
+        xmlrpclib.Transport.send_host(self, connection, host)
+        if not self.authenticated():
+            return
+        credentials = '%s:%s' % (self.username, self.password)
+        auth = 'Basic ' + base64.encodestring(credentials).strip()
+        connection.putheader('Authorization', auth)
+
+    def make_connection(self, host):
+        if self.use_https:
+            fn = xmlrpclib.SafeTransport.make_connection
+        else:
+            fn = xmlrpclib.Transport.make_connection
+        return fn(self, host)
+
+
+def project_id_by_name(rpc, linkname):
+    """Given a project short name, look up the Project ID."""
+    if len(linkname) == 0:
+        return 0
+    projects = rpc.project_list(linkname, 0)
+    for project in projects:
+        if project['linkname'] == linkname:
+            return project['id']
+    return 0
+
+
+def state_id_by_name(rpc, name):
+    """Given a partial state name, look up the state ID."""
+    if len(name) == 0:
+        return 0
+    states = rpc.state_list(name, 0)
+    for state in states:
+        if state['name'].lower().startswith(name.lower()):
+            return state['id']
+    return 0
+
+
+def person_ids_by_name(rpc, name):
+    """Given a partial name or email address, return a list of the
+    person IDs that match."""
+    if len(name) == 0:
+        return []
+    people = rpc.person_list(name, 0)
+    return [x['id'] for x in people]
+
+
+def list_patches(patches, format_str=None):
+    """Dump a list of patches to stdout."""
+    if format_str:
+        format_field_re = re.compile("%{([a-z0-9_]+)}")
+
+        def patch_field(matchobj):
+            fieldname = matchobj.group(1)
+
+            if fieldname == "_msgid_":
+                # naive way to strip < and > from message-id
+                val = string.strip(str(patch["msgid"]), "<>")
+            else:
+                val = str(patch[fieldname])
+
+            return val
+
+        for patch in patches:
+            print(format_field_re.sub(patch_field, format_str))
+    else:
+        print("%-7s %-12s %s" % ("ID", "State", "Name"))
+        print("%-7s %-12s %s" % ("--", "-----", "----"))
+        for patch in patches:
+            print("%-7d %-12s %s" %
+                  (patch['id'], patch['state'], patch['name']))
+
+
+def action_list(rpc, filter, submitter_str, delegate_str, format_str=None):
+    filter.resolve_ids(rpc)
+
+    if submitter_str is not None:
+        ids = person_ids_by_name(rpc, submitter_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             submitter_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches submitted by %s <%s>:' %
+                      (unicode(person['name']).encode('utf-8'),
+                       unicode(person['email']).encode('utf-8')))
+                f = filter
+                f.add("submitter_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    if delegate_str is not None:
+        ids = person_ids_by_name(rpc, delegate_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             delegate_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches delegated to %s <%s>:' %
+                      (person['name'], person['email']))
+                f = filter
+                f.add("delegate_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    patches = rpc.patch_list(filter.d)
+    list_patches(patches, format_str)
+
+
+def action_projects(rpc):
+    projects = rpc.project_list("", 0)
+    print("%-5s %-24s %s" % ("ID", "Name", "Description"))
+    print("%-5s %-24s %s" % ("--", "----", "-----------"))
+    for project in projects:
+        print("%-5d %-24s %s" % (project['id'],
+                                 project['linkname'],
+                                 project['name']))
+
+
+def action_check_list(rpc):
+    checks = rpc.check_list()
+    print("%-5s %-16s %-8s %s" % ("ID", "Context", "State", "Patch"))
+    print("%-5s %-16s %-8s %s" % ("--", "-------", "-----", "-----"))
+    for check in checks:
+        print("%-5s %-16s %-8s %s" % (check['id'],
+                                      check['context'],
+                                      check['state'],
+                                      check['patch']))
+
+
+def action_check_info(rpc, check_id):
+    check = rpc.check_get(check_id)
+    s = "Information for check id %d" % (check_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(check.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_check_create(rpc, patch_id, context, state, url, description):
+    try:
+        rpc.check_create(patch_id, context, state, url, description)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error creating check: %s\n" % f.faultString)
+
+
+def action_states(rpc):
+    states = rpc.state_list("", 0)
+    print("%-5s %s" % ("ID", "Name"))
+    print("%-5s %s" % ("--", "----"))
+    for state in states:
+        print("%-5d %s" % (state['id'], state['name']))
+
+
+def action_info(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = "Information for patch id %d" % (patch_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(patch.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_get(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = rpc.patch_get_mbox(patch_id)
+
+    if patch == {} or len(s) == 0:
+        sys.stderr.write("Unable to get patch %d\n" % patch_id)
+        sys.exit(1)
+
+    base_fname = fname = os.path.basename(patch['filename'])
+    i = 0
+    while os.path.exists(fname):
+        fname = "%s.%d" % (base_fname, i)
+        i += 1
+
+    try:
+        f = open(fname, "w")
+    except:
+        sys.stderr.write("Unable to open %s for writing\n" % fname)
+        sys.exit(1)
+
+    try:
+        f.write(unicode(s).encode("utf-8"))
+        f.close()
+        print('Saved patch to %s' % fname)
+    except:
+        sys.stderr.write("Failed to write to %s\n" % fname)
+        sys.exit(1)
+
+
+def action_apply(rpc, patch_id, apply_cmd=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    if apply_cmd is None:
+        print('Applying patch #%d to current directory' % patch_id)
+        apply_cmd = ['patch', '-p1']
+    else:
+        print('Applying patch #%d using %s' %
+              (patch_id, repr(' '.join(apply_cmd))))
+
+    print('Description: %s' % patch['name'])
+    s = rpc.patch_get_mbox(patch_id)
+    if len(s) > 0:
+        proc = subprocess.Popen(apply_cmd, stdin=subprocess.PIPE)
+        proc.communicate(unicode(s).encode('utf-8'))
+        return proc.returncode
+    else:
+        sys.stderr.write("Error: No patch content found\n")
+        sys.exit(1)
+
+
+def action_update_patch(rpc, patch_id, state=None, archived=None, commit=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    params = {}
+
+    if state:
+        state_id = state_id_by_name(rpc, state)
+        if state_id == 0:
+            sys.stderr.write("Error: No State found matching %s*\n" % state)
+            sys.exit(1)
+        params['state'] = state_id
+
+    if commit:
+        params['commit_ref'] = commit
+
+    if archived:
+        params['archived'] = archived == 'yes'
+
+    success = False
+    try:
+        success = rpc.patch_set(patch_id, params)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error updating patch: %s\n" % f.faultString)
+
+    if not success:
+        sys.stderr.write("Patch not updated\n")
+
+
+def patch_id_from_hash(rpc, project, hash):
+    try:
+        patch = rpc.patch_get_by_project_hash(project, hash)
+    except xmlrpclib.Fault:
+        # the server may not have the newer patch_get_by_project_hash function,
+        # so fall back to hash-only.
+        patch = rpc.patch_get_by_hash(hash)
+
+    if patch == {}:
+        sys.stderr.write("No patch has the hash provided\n")
+        sys.exit(1)
+
+    patch_id = patch['id']
+    # be super paranoid
+    try:
+        patch_id = int(patch_id)
+    except:
+        sys.stderr.write("Invalid patch ID obtained from server\n")
+        sys.exit(1)
+    return patch_id
+
+auth_actions = ['check_create', 'update']
+
+
+def main():
+    hash_parser = argparse.ArgumentParser(add_help=False)
+    hash_parser.add_argument(
+        '-h', metavar='HASH', dest='hash', action='store',
+        help='''Lookup by patch hash'''
+    )
+    hash_parser.add_argument(
+        'id', metavar='ID', nargs='*', action='store', type=int,
+        help='Patch ID',
+    )
+    hash_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Lookup patch in project'''
+    )
+
+    filter_parser = argparse.ArgumentParser(add_help=False)
+    filter_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Filter by patch state (e.g., 'New', 'Accepted', etc.)'''
+    )
+    filter_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Filter by patch archived state'''
+    )
+    filter_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Filter by project name (see 'projects' for list)'''
+    )
+    filter_parser.add_argument(
+        '-w', metavar='WHO',
+        help='''Filter by submitter (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-d', metavar='WHO',
+        help='''Filter by delegate (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-n', metavar='MAX#',
+        type=int,
+        help='''Return first n results'''
+    )
+    filter_parser.add_argument(
+        '-N', metavar='MAX#',
+        type=int,
+        help='''Return last N results'''
+    )
+    filter_parser.add_argument(
+        '-m', metavar='MESSAGEID',
+        help='''Filter by Message-Id'''
+    )
+    filter_parser.add_argument(
+        '-f', metavar='FORMAT',
+        help='''Print output in the given format. You can use tags matching '''
+        '''fields, e.g. %%{id}, %%{state}, or %%{msgid}.'''
+    )
+    filter_parser.add_argument(
+        'patch_name', metavar='STR', nargs='?',
+        help='substring to search for patches by name',
+    )
+
+    action_parser = argparse.ArgumentParser(
+        prog='pwclient',
+        epilog='Use \'pwclient <command> --help\' for more info',
+    )
+
+    subparsers = action_parser.add_subparsers(
+        title='Commands',
+    )
+    apply_parser = subparsers.add_parser(
+        'apply', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch (in the current dir, using -p1)'''
+    )
+    apply_parser.set_defaults(subcmd='apply')
+    git_am_parser = subparsers.add_parser(
+        'git-am', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch to current git branch using "git am".'''
+    )
+    git_am_parser.set_defaults(subcmd='git_am')
+    git_am_parser.add_argument(
+        '-s', '--signoff',
+        action='store_true',
+        help='''pass --signoff to git-am'''
+    )
+    get_parser = subparsers.add_parser(
+        'get', parents=[hash_parser], conflict_handler='resolve',
+        help='''Download a patch and save it locally'''
+    )
+    get_parser.set_defaults(subcmd='get')
+    info_parser = subparsers.add_parser(
+        'info', parents=[hash_parser], conflict_handler='resolve',
+        help='''Display patchwork info about a given patch ID'''
+    )
+    info_parser.set_defaults(subcmd='info')
+    projects_parser = subparsers.add_parser(
+        'projects',
+        help='''List all projects'''
+    )
+    projects_parser.set_defaults(subcmd='projects')
+    check_list_parser = subparsers.add_parser(
+        'check-list',
+        add_help=False,
+        help='''List all checks'''
+    )
+    check_list_parser.set_defaults(subcmd='check_list')
+    check_info_parser = subparsers.add_parser(
+        'check-info',
+        add_help=False,
+        help='''Show information for a given check'''
+    )
+    check_info_parser.set_defaults(subcmd='check_info')
+    check_info_parser.add_argument(
+        'check_id', metavar='ID', action='store', type=int,
+        help='Check ID',)
+    check_create_parser = subparsers.add_parser(
+        'check-create', parents=[hash_parser], conflict_handler='resolve',
+        help='Add a check to a patch')
+    check_create_parser.set_defaults(subcmd='check_create')
+    check_create_parser.add_argument(
+        '-c', metavar='CONTEXT')
+    check_create_parser.add_argument(
+        '-s', choices=('pending', 'success', 'warning', 'fail'))
+    check_create_parser.add_argument(
+        '-u', metavar='TARGET_URL', default="")
+    check_create_parser.add_argument(
+        '-d', metavar='DESCRIPTION', default="")
+    states_parser = subparsers.add_parser(
+        'states',
+        help='''Show list of potential patch states'''
+    )
+    states_parser.set_defaults(subcmd='states')
+    view_parser = subparsers.add_parser(
+        'view', parents=[hash_parser], conflict_handler='resolve',
+        help='''View a patch'''
+    )
+    view_parser.set_defaults(subcmd='view')
+    update_parser = subparsers.add_parser(
+        'update', parents=[hash_parser], conflict_handler='resolve',
+        help='''Update patch''',
+        epilog='''Using a COMMIT-REF allows for only one ID to be specified''',
+    )
+    update_parser.add_argument(
+        '-c', metavar='COMMIT-REF',
+        help='''commit reference hash'''
+    )
+    update_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Set patch state (e.g., 'Accepted', 'Superseded' etc.)'''
+    )
+    update_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Set patch archived state'''
+    )
+    update_parser.set_defaults(subcmd='update')
+    list_parser = subparsers.add_parser("list",
+                                        # aliases=['search'],
+                                        parents=[filter_parser],
+                                        help='''List patches, using the optional filters specified
+        below and an optional substring to search for patches
+        by name'''
+                                        )
+    list_parser.set_defaults(subcmd='list')
+    search_parser = subparsers.add_parser("search",
+                                          parents=[filter_parser],
+                                          help='''Alias for "list"'''
+                                          )
+    # Poor man's argparse aliases:
+    # We register the "search" parser but effectively use "list" for the
+    # help-text.
+    search_parser.set_defaults(subcmd='list')
+    if len(sys.argv) < 2:
+        action_parser.print_help()
+        sys.exit(0)
+
+    args = action_parser.parse_args()
+    args = dict(vars(args))
+    action = args.get('subcmd')
+
+    if args.get('hash') and len(args.get('id')):
+        # mimic mutual exclusive group
+        locals()[action + '_parser'].error(
+            "[-h HASH] and [ID [ID ...]] are mutually exlusive")
+
+    # set defaults
+    filt = Filter()
+    commit_str = None
+    url = DEFAULT_URL
+
+    archived_str = args.get('a')
+    state_str = args.get('s')
+    project_str = args.get('p')
+    submitter_str = args.get('w')
+    delegate_str = args.get('d')
+    format_str = args.get('f')
+    hash_str = args.get('hash')
+    patch_ids = args.get('id')
+    msgid_str = args.get('m')
+    if args.get('c'):
+        # update multiple IDs with a single commit-hash does not make sense
+        if action == 'update' and patch_ids and len(patch_ids) > 1:
+            update_parser.error(
+                "Declining update with COMMIT-REF on multiple IDs")
+        commit_str = args.get('c')
+
+    if state_str is None and archived_str is None and action == 'update':
+        update_parser.error(
+            'Must specify one or more update options (-a or -s)')
+
+    if args.get('n') is not None:
+        try:
+            filt.add("max_count", args.get('n'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('n'))
+
+    if args.get('N') is not None:
+        try:
+            filt.add("max_count", 0 - args.get('N'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('N'))
+
+    do_signoff = args.get('signoff')
+
+    # grab settings from config files
+    config = ConfigParser.ConfigParser()
+    config.read([CONFIG_FILE])
+
+    if not config.has_section('options') and os.path.exists(CONFIG_FILE):
+        sys.stderr.write('~/.pwclientrc is in the old format. Migrating it...')
+
+        old_project = config.get('base', 'project')
+
+        new_config = ConfigParser.ConfigParser()
+        new_config.add_section('options')
+
+        new_config.set('options', 'default', old_project)
+        new_config.add_section(old_project)
+
+        new_config.set(old_project, 'url', config.get('base', 'url'))
+        if config.has_option('auth', 'username'):
+            new_config.set(
+                old_project, 'username', config.get('auth', 'username'))
+        if config.has_option('auth', 'password'):
+            new_config.set(
+                old_project, 'password', config.get('auth', 'password'))
+
+        old_config_file = CONFIG_FILE + '.orig'
+        shutil.copy2(CONFIG_FILE, old_config_file)
+
+        with open(CONFIG_FILE, 'wb') as fd:
+            new_config.write(fd)
+
+        sys.stderr.write(' Done.\n')
+        sys.stderr.write(
+            'Your old ~/.pwclientrc was saved to %s\n' % old_config_file)
+        sys.stderr.write(
+            'and was converted to the new format. You may want to\n')
+        sys.stderr.write('inspect it before continuing.\n')
+        sys.exit(1)
+
+    if not project_str:
+        try:
+            project_str = config.get('options', 'default')
+        except:
+            action_parser.error(
+                "No default project configured in ~/.pwclientrc")
+
+    if not config.has_section(project_str):
+        sys.stderr.write(
+            'No section for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not config.has_option(project_str, 'url'):
+        sys.stderr.write(
+            'No URL for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not do_signoff and config.has_option('options', 'signoff'):
+        do_signoff = config.getboolean('options', 'signoff')
+    if not do_signoff and config.has_option(project_str, 'signoff'):
+        do_signoff = config.getboolean(project_str, 'signoff')
+
+    url = config.get(project_str, 'url')
+
+    transport = None
+    if action in auth_actions:
+        if config.has_option(project_str, 'username') and \
+                config.has_option(project_str, 'password'):
+
+            use_https = url.startswith('https')
+
+            transport = BasicHTTPAuthTransport(
+                config.get(project_str, 'username'),
+                config.get(project_str, 'password'),
+                use_https)
+
+        else:
+            sys.stderr.write("The %s action requires authentication, but no "
+                             "username or password\nis configured\n" % action)
+            sys.exit(1)
+
+    if project_str:
+        filt.add("project", project_str)
+
+    if state_str:
+        filt.add("state", state_str)
+
+    if archived_str:
+        filt.add("archived", archived_str == 'yes')
+
+    if msgid_str:
+        filt.add("msgid", msgid_str)
+
+    try:
+        rpc = xmlrpclib.Server(url, transport=transport)
+    except:
+        sys.stderr.write("Unable to connect to %s\n" % url)
+        sys.exit(1)
+
+    # It should be safe to assume hash_str is not zero, but who knows..
+    if hash_str is not None:
+        patch_ids = [patch_id_from_hash(rpc, project_str, hash_str)]
+
+    # helper for non_empty() to print correct helptext
+    h = locals()[action + '_parser']
+
+    # Require either hash_str or IDs for
+    def non_empty(h, patch_ids):
+        """Error out if no patch IDs were specified"""
+        if patch_ids is None or len(patch_ids) < 1:
+            sys.stderr.write("Error: Missing Argument! Either [-h HASH] or "
+                             "[ID [ID ...]] are required\n")
+            if h:
+                h.print_help()
+            sys.exit(1)
+        return patch_ids
+
+    if action == 'list' or action == 'search':
+        if args.get('patch_name') is not None:
+            filt.add("name__icontains", args.get('patch_name'))
+        action_list(rpc, filt, submitter_str, delegate_str, format_str)
+
+    elif action.startswith('project'):
+        action_projects(rpc)
+
+
+    elif action.startswith('state'):
+        action_states(rpc)
+
+    elif action == 'view':
+        pager = os.environ.get('PAGER')
+        if pager:
+            pager = subprocess.Popen(
+                pager.split(), stdin=subprocess.PIPE
+            )
+        if pager:
+            i = list()
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    i.append(unicode(s).encode("utf-8"))
+            if len(i) > 0:
+                pager.communicate(input="\n".join(i))
+            pager.stdin.close()
+        else:
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    print(unicode(s).encode("utf-8"))
+
+    elif action == 'info':
+        for patch_id in non_empty(h, patch_ids):
+            action_info(rpc, patch_id)
+
+    elif action == 'get':
+        for patch_id in non_empty(h, patch_ids):
+            action_get(rpc, patch_id)
+
+    elif action == 'apply':
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id)
+            if ret:
+                sys.stderr.write("Apply failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'git_am':
+        cmd = ['git', 'am']
+        if do_signoff:
+            cmd.append('-s')
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id, cmd)
+            if ret:
+                sys.stderr.write("'git am' failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'update':
+        for patch_id in non_empty(h, patch_ids):
+            action_update_patch(rpc, patch_id, state=state_str,
+                                archived=archived_str, commit=commit_str
+                                )
+
+    elif action == 'check_list':
+        action_check_list(rpc)
+
+    elif action == 'check_info':
+        check_id = args['check_id']
+        action_check_info(rpc, check_id)
+
+    elif action == 'check_create':
+        for patch_id in non_empty(h, patch_ids):
+            action_check_create(
+                rpc, patch_id, args['c'], args['s'], args['u'], args['d'])
+
+    else:
+        sys.stderr.write("Unknown action '%s'\n" % action)
+        action_parser.print_help()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
-- 
2.7.0

[dpdk-ci] [PATCH v3 5/7] tools: add per-patch report mailer

[Thomas Monjalon]

The report sent by this script will be parsed and integrated in patchwork
if the patch was public.
Otherwise it will just send the report to the patch submitter.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config           |   3 ++
 tools/send-patch-report.sh | 128 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
 create mode 100755 tools/send-patch-report.sh

diff --git a/config/ci.config b/config/ci.config
index 722aeb5..b3b0376 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -3,5 +3,8 @@
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
 
+# The mailer (sendmail, mail, mailx) must support the option -t
+# export DPDK_CI_MAILER=/usr/sbin/sendmail
+
 # The pwclient script is part of patchwork and is copied in dpdk-ci
 # export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/tools/send-patch-report.sh b/tools/send-patch-report.sh
new file mode 100755
index 0000000..716e8e3
--- /dev/null
+++ b/tools/send-patch-report.sh
@@ -0,0 +1,128 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) [options] < report
+
+	Send test report in a properly formatted email for patchwork integration.
+	The report is submitted to this script via stdin.
+
+	options:
+	        -t title    subject of the patch email
+	        -f from     sender of the patch email
+	        -m msgid    id of the patch email
+	        -p listid   mailing list publishing the patch
+	        -l label    title of the test
+	        -s status   one of these test results: SUCCESS, WARNING, FAILURE
+	        -d desc     few words to better describe the status
+	        -h          this help
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+unset title
+unset from
+unset msgid
+unset listid
+unset label
+unset status
+unset desc
+while getopts d:f:hl:m:p:s:t: arg ; do
+	case $arg in
+		t ) title=$OPTARG ;;
+		f ) from=$OPTARG ;;
+		m ) msgid=$OPTARG ;;
+		p ) listid=$OPTARG ;;
+		l ) label=$OPTARG ;;
+		s ) status=$OPTARG ;;
+		d ) desc=$OPTARG ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -t 0 ] ; then
+	printf 'nothing to read on stdin\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+report=$(cat)
+
+writeheaders () # <subject> <ref> <to> [cc]
+{
+	echo "Subject: $1"
+	echo "In-Reply-To: $2"
+	echo "References: $2"
+	echo "To: $3"
+	[ -z "$4" ] || echo "Cc: $4"
+	echo
+}
+
+writeheadlines () # <label> <status> <description> [pwid]
+{
+	echo "Test-Label: $1"
+	echo "Test-Status: $2"
+	[ -z "$4" ] || echo "http://dpdk.org/patch/$4"
+	echo
+	echo "_${3}_"
+	echo
+}
+
+if echo "$listid" | grep -q 'dev.dpdk.org' ; then
+	# get patchwork id
+	if [ -n "$msgid" ] ; then
+		for try in $(seq 20) ; do
+			pwid=$($pwclient list -f '%{id}' -m "$msgid")
+			[ -n "$pwid" ] && break || sleep 7
+		done
+	fi
+	[ -n "$pwid" ] || pwid='?'
+	# send public report
+	subject=$(echo $title | sed 's,\[dpdk-dev\] ,,')
+	[ "$status" = 'SUCCESS' ] && cc='' || cc="$from"
+	(
+	writeheaders "|$status| $subject" "$msgid" 'test-report@dpdk.org' "$cc"
+	writeheadlines "$label" "$status" "$desc" "$pwid"
+	echo "$report"
+	) | $sendmail -t
+else
+	# send private report
+	(
+		writeheaders "Re: $title" "$msgid" "$from"
+		writeheadlines "$label" "$status" "$desc"
+		echo "$report"
+	) | $sendmail -t
+fi
-- 
2.7.0

[dpdk-ci] [PATCH v3 6/7] tools: add patchwork integration

[Thomas Monjalon]

This script is run with patchwork admin credentials.
It is typically installed on dpdk.org and run for each mail
received in the test-report mailing list.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/update-pw.sh | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 tools/update-pw.sh

diff --git a/tools/update-pw.sh b/tools/update-pw.sh
new file mode 100755
index 0000000..1a60b63
--- /dev/null
+++ b/tools/update-pw.sh
@@ -0,0 +1,85 @@
+#! /bin/sh
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <report_url>
+
+	Add or update a check in patchwork based on a test report.
+	The argument specifies only the last URL parts of the test-report
+	mailing list archives (month/id.html).
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+if [ -z "$1" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+url="http://dpdk.org/ml/archives/test-report/$1"
+mmarker='<!--beginarticle-->'
+for try in $(seq 20) ; do
+	[ -z "$report" -o $try -ge 19 ] || continue # 2 last tries if got something
+	[ $try -le 1 ] || sleep 3 # no delay before first try
+	report=$(curl -sf $url | grep -m1 -A9 "$mmarker") || continue
+	echo "$report" | grep -q '^_.*_$' && break || continue
+done
+if [ -z "$report" ] ; then
+	echo "cannot download report at $url" >&2
+	exit 2
+fi
+
+pwid=$(echo "$report" | sed -rn 's,.*http://.*dpdk.org/.*patch/([0-9]+).*,\1,p')
+label=$(echo "$report" | sed -n 's,.*Test-Label: *,,p')
+status=$(echo "$report" | sed -n 's,.*Test-Status: *,,p')
+desc=$(echo "$report" | sed -n 's,^_\(.*\)_$,\1,p')
+case $status in
+	'SUCCESS') pwstatus='success' ;;
+	'WARNING') pwstatus='warning' ;;
+	'FAILURE') pwstatus='fail' ;;
+esac
+printf 'id = %s\nlabel = %s\nstatus = %s/%s %s\nurl = %s\n' \
+	"$pwid" "$label" "$status" "$pwstatus" "$desc" "$url"
+[ -n "$pwid" -a -n "$label" -a -n "$status" -a -n "$desc" ] || exit 3
+
+pwclient=$(dirname $(readlink -m $0))/pwclient
+$pwclient check-create -c "$label" -s "$pwstatus" -d "$desc" -u "$url" $pwid
-- 
2.7.0

[dpdk-ci] [PATCH v3 7/7] tests: add checkpatch

[Thomas Monjalon]

This is the first test in this repository.
It runs on dpdk.org and use checkpatch.pl of Linux.

Note that the patch is not applied on a git tree for this basic test.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tests/checkpatch.sh | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100755 tests/checkpatch.sh

diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh
new file mode 100755
index 0000000..319da1e
--- /dev/null
+++ b/tests/checkpatch.sh
@@ -0,0 +1,70 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) dpdk_dir < email
+
+	Check email-formatted patch from stdin.
+	This test runs checkpatch.pl of Linux via a script in dpdk_dir.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+toolsdir=$(dirname $(readlink -m $0))/../tools
+dpdkdir=$1
+
+email=/tmp/$(basename $0 sh)$$
+$toolsdir/filter-patch-email.sh >$email
+trap "rm -f $email" INT EXIT
+
+eval $($toolsdir/parse-email.sh $email)
+# normal exit if no valid patch in the email
+[ -n "$subject" -a -n "$from" ] || exit 0
+
+failed=false
+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true
+report=$(echo "$report" | sed '1,/^###/d')
+
+label='checkpatch'
+$failed && status='WARNING' || status='SUCCESS'
+$failed && desc='coding style issues' || desc='coding style OK'
+
+echo "$report" | $toolsdir/send-patch-report.sh \
+	-t "$subject" -f "$from" -m "$msgid" -p "$listid" \
+	-l "$label" -s "$status" -d "$desc"
-- 
2.7.0

[dpdk-ci] [PATCH v4 0/7] first scripts for CI integration

[Thomas Monjalon]

These scripts allow to check a patch received by email and
send a report in order to be integrated in patchwork.

The existing CI tests run by Intel could be converted to use
the script send-patch-report.sh so they will be seen in patchwork.

Next steps (to be implemented):
- script to clean and update a git tree
- script to apply a patch on the right tree
- script to apply dependencies (preceding in a series)

---

changes in v4:
- fortify mail parsing for binary patches and long emails

changes in v3:
- BSD licensing

changes in v2:
- fix mail parsing (bug with quotes in From:)
- fix public success report (no CC:)

---

Thomas Monjalon (7):
  tools: add mail filter
  tools: add mail parser
  config: add loader and template
  tools: add patchwork client
  tools: add per-patch report mailer
  tools: add patchwork integration
  tests: add checkpatch

 config/ci.config            |  10 +
 config/pwclientrc           |   9 +
 tests/checkpatch.sh         |  70 ++++
 tools/filter-patch-email.sh | 111 ++++++
 tools/load-ci-config.sh     |  14 +
 tools/parse-email.sh        |  72 ++++
 tools/pwclient              | 808 ++++++++++++++++++++++++++++++++++++++++++++
 tools/send-patch-report.sh  | 128 +++++++
 tools/update-pw.sh          |  85 +++++
 9 files changed, 1307 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 config/pwclientrc
 create mode 100755 tests/checkpatch.sh
 create mode 100755 tools/filter-patch-email.sh
 create mode 100644 tools/load-ci-config.sh
 create mode 100755 tools/parse-email.sh
 create mode 100755 tools/pwclient
 create mode 100755 tools/send-patch-report.sh
 create mode 100755 tools/update-pw.sh

-- 
2.7.0

[dpdk-ci] [PATCH v4 1/7] tools: add mail filter

[Thomas Monjalon]

This script acts as a pipe which blocks non-patch emails.
It can be used as a first filter before processing a patch.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/filter-patch-email.sh | 111 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 111 insertions(+)
 create mode 100755 tools/filter-patch-email.sh

diff --git a/tools/filter-patch-email.sh b/tools/filter-patch-email.sh
new file mode 100755
index 0000000..821786f
--- /dev/null
+++ b/tools/filter-patch-email.sh
@@ -0,0 +1,111 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) < email
+
+	Filter out email from stdin if does not match patch criterias.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+
+if [ -t 0 ] ; then
+	echo 'nothing to read on stdin' >&2
+	exit 0
+fi
+
+fifo=/tmp/$(basename $0 sh)$$
+mkfifo $fifo
+trap "rm -f $fifo" INT EXIT
+
+parse ()
+{
+	gitsend=false
+	patchsubject=false
+	content=false
+	linenum=0
+	minusline=false
+	plusline=false
+	atline=false
+	binary=false
+	done=false
+	while IFS= read -r line ; do
+		printf '%s\n' "$line"
+		set -- $line
+		if ! $content ; then
+			[ "$1" != 'X-Mailer:' -o "$2" != 'git-send-email' ] || gitsend=true
+			if echo "$line" | grep -qa '^Subject:.*\[PATCH' ; then
+				subject=$(echo "$line" | sed 's,^Subject:[[:space:]]*,,')
+				while [ -n "$subject" ] ; do
+					echo "$subject" | grep -q '^\[' || break
+					if echo "$subject" | grep -q '^\[PATCH' ; then
+						patchsubject=true
+						break
+					fi
+					subject=$(echo "$subject" | sed 's,^[^]]*\][[:space:]]*,,')
+				done
+			fi
+			[ -n "$line" ] || content=true
+		elif ! $done ; then
+			$gitsend || $patchsubject || break
+			[ "$1" != '---' ] || minusline=true
+			[ "$1" != '+++' ] || plusline=true
+			[ "$1" != '@@' ] || atline=true
+			[ "$1 $2 $3" != 'GIT binary patch' ] || binary=true
+			if ($minusline && $plusline && $atline) || $binary ; then
+				echo 1 >$fifo
+				done=true
+				cat
+				break
+			fi
+			linenum=$(($linenum + 1))
+			[ $linenum -lt 999 ] || break
+		fi
+	done
+	$done || echo 0 >$fifo
+	exec >&-
+}
+
+waitparsing ()
+{
+	result=$(cat $fifo)
+	[ "$result" = 0 ] || cat
+}
+
+parse | waitparsing
-- 
2.7.0

[dpdk-ci] [PATCH v4 2/7] tools: add mail parser

[Thomas Monjalon]

This script get some mail headers from an email file.
The retrieved headers can be used for test and report in other scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/parse-email.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100755 tools/parse-email.sh

diff --git a/tools/parse-email.sh b/tools/parse-email.sh
new file mode 100755
index 0000000..d92c246
--- /dev/null
+++ b/tools/parse-email.sh
@@ -0,0 +1,72 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <email_file>
+
+	Parse basic headers of the email
+	and print them as shell variable assignments to evaluate.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -z "$1" ] ; then
+	printf 'file argument is missing\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+getheader () # <header_name> <email_file>
+{
+	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2" |
+	sed 's,",\\",g'
+}
+
+subject=$(getheader Subject "$1")
+from=$(getheader From "$1")
+msgid=$(getheader Message-Id "$1")
+[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+listid=$(getheader List-Id "$1")
+
+cat <<- END_OF_HEADERS
+	subject="$subject"
+	from="$from"
+	msgid="$msgid"
+	listid="$listid"
+END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH v4 3/7] config: add loader and template

[Thomas Monjalon]

The configuration file will allow to set some environment-specific
options and paths to be used by the scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config        |  4 ++++
 tools/load-ci-config.sh | 14 ++++++++++++++
 2 files changed, 18 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 tools/load-ci-config.sh

diff --git a/config/ci.config b/config/ci.config
new file mode 100644
index 0000000..2aeff04
--- /dev/null
+++ b/config/ci.config
@@ -0,0 +1,4 @@
+# Configuration template to be copied in
+#         /etc/dpdk/ci.config
+#     or  ~/.config/dpdk/ci.config
+#     or  .ciconfig
diff --git a/tools/load-ci-config.sh b/tools/load-ci-config.sh
new file mode 100644
index 0000000..9ec18b4
--- /dev/null
+++ b/tools/load-ci-config.sh
@@ -0,0 +1,14 @@
+#! /bin/echo must be loaded with .
+
+# Load DPDK CI config and allow override
+# from system file
+test ! -r /etc/dpdk/ci.config ||
+        . /etc/dpdk/ci.config
+# from user file
+test ! -r ~/.config/dpdk/ci.config ||
+        . ~/.config/dpdk/ci.config
+# from local file
+test ! -r $(dirname $(readlink -m $0))/../.ciconfig ||
+        . $(dirname $(readlink -m $0))/../.ciconfig
+
+# The config files must export variables in the shell style
-- 
2.7.0

[dpdk-ci] [PATCH v4 4/7] tools: add patchwork client

[Thomas Monjalon]

The script pwclient is a frontend for patchwork.
It is imported from patchwork 1.1.
Another version can be used by setting DPDK_CI_PWCLIENT in config.

The config template pwclientrc must be copied in ~/.pwclientrc.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config  |   3 +
 config/pwclientrc |   9 +
 tools/pwclient    | 808 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 820 insertions(+)
 create mode 100644 config/pwclientrc
 create mode 100755 tools/pwclient

diff --git a/config/ci.config b/config/ci.config
index 2aeff04..722aeb5 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -2,3 +2,6 @@
 #         /etc/dpdk/ci.config
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
+
+# The pwclient script is part of patchwork and is copied in dpdk-ci
+# export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/config/pwclientrc b/config/pwclientrc
new file mode 100644
index 0000000..d73e87e
--- /dev/null
+++ b/config/pwclientrc
@@ -0,0 +1,9 @@
+[options]
+default=dpdk
+
+[dpdk]
+url=http://dpdk.org/dev/patchwork/xmlrpc/
+
+# credentials are useless for read-only access
+#username: myuser
+#password: mypass
diff --git a/tools/pwclient b/tools/pwclient
new file mode 100755
index 0000000..f437786
--- /dev/null
+++ b/tools/pwclient
@@ -0,0 +1,808 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Patchwork command line client
+# Copyright (C) 2008 Nate Case <ncase@xes-inc.com>
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+from __future__ import print_function
+
+import os
+import sys
+try:
+    import xmlrpclib
+except ImportError:
+    # Python 3 has merged/renamed things.
+    import xmlrpc.client as xmlrpclib
+import argparse
+import string
+import subprocess
+import base64
+try:
+    import ConfigParser
+except ImportError:
+    # Python 3 has renamed things.
+    import configparser as ConfigParser
+import shutil
+import re
+
+# Add a shim for Python 2's unicode() helper.
+try:
+    unicode
+except NameError:
+    # Python 3 does everything by unicode now.
+    unicode = str
+
+# Default Patchwork remote XML-RPC server URL
+# This script will check the PW_XMLRPC_URL environment variable
+# for the URL to access.  If that is unspecified, it will fallback to
+# the hardcoded default value specified here.
+DEFAULT_URL = "http://patchwork/xmlrpc/"
+CONFIG_FILE = os.path.expanduser('~/.pwclientrc')
+
+
+class Filter(object):
+
+    """Filter for selecting patches."""
+
+    def __init__(self):
+        # These fields refer to specific objects, so they are special
+        # because we have to resolve them to IDs before passing the
+        # filter to the server
+        self.state = ""
+        self.project = ""
+
+        # The dictionary that gets passed to via XML-RPC
+        self.d = {}
+
+    def add(self, field, value):
+        if field == 'state':
+            self.state = value
+        elif field == 'project':
+            self.project = value
+        else:
+            # OK to add directly
+            self.d[field] = value
+
+    def resolve_ids(self, rpc):
+        """Resolve State, Project, and Person IDs based on filter strings."""
+        if self.state != "":
+            id = state_id_by_name(rpc, self.state)
+            if id == 0:
+                sys.stderr.write("Note: No State found matching %s*, "
+                                 "ignoring filter\n" % self.state)
+            else:
+                self.d['state_id'] = id
+
+        if self.project is not None:
+            id = project_id_by_name(rpc, self.project)
+            if id == 0:
+                sys.stderr.write("Note: No Project found matching %s, "
+                                 "ignoring filter\n" % self.project)
+            else:
+                self.d['project_id'] = id
+
+    def __str__(self):
+        """Return human-readable description of the filter."""
+        return str(self.d)
+
+
+class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+
+    def __init__(self, username=None, password=None, use_https=False):
+        self.username = username
+        self.password = password
+        self.use_https = use_https
+        xmlrpclib.SafeTransport.__init__(self)
+
+    def authenticated(self):
+        return self.username is not None and self.password is not None
+
+    def send_host(self, connection, host):
+        xmlrpclib.Transport.send_host(self, connection, host)
+        if not self.authenticated():
+            return
+        credentials = '%s:%s' % (self.username, self.password)
+        auth = 'Basic ' + base64.encodestring(credentials).strip()
+        connection.putheader('Authorization', auth)
+
+    def make_connection(self, host):
+        if self.use_https:
+            fn = xmlrpclib.SafeTransport.make_connection
+        else:
+            fn = xmlrpclib.Transport.make_connection
+        return fn(self, host)
+
+
+def project_id_by_name(rpc, linkname):
+    """Given a project short name, look up the Project ID."""
+    if len(linkname) == 0:
+        return 0
+    projects = rpc.project_list(linkname, 0)
+    for project in projects:
+        if project['linkname'] == linkname:
+            return project['id']
+    return 0
+
+
+def state_id_by_name(rpc, name):
+    """Given a partial state name, look up the state ID."""
+    if len(name) == 0:
+        return 0
+    states = rpc.state_list(name, 0)
+    for state in states:
+        if state['name'].lower().startswith(name.lower()):
+            return state['id']
+    return 0
+
+
+def person_ids_by_name(rpc, name):
+    """Given a partial name or email address, return a list of the
+    person IDs that match."""
+    if len(name) == 0:
+        return []
+    people = rpc.person_list(name, 0)
+    return [x['id'] for x in people]
+
+
+def list_patches(patches, format_str=None):
+    """Dump a list of patches to stdout."""
+    if format_str:
+        format_field_re = re.compile("%{([a-z0-9_]+)}")
+
+        def patch_field(matchobj):
+            fieldname = matchobj.group(1)
+
+            if fieldname == "_msgid_":
+                # naive way to strip < and > from message-id
+                val = string.strip(str(patch["msgid"]), "<>")
+            else:
+                val = str(patch[fieldname])
+
+            return val
+
+        for patch in patches:
+            print(format_field_re.sub(patch_field, format_str))
+    else:
+        print("%-7s %-12s %s" % ("ID", "State", "Name"))
+        print("%-7s %-12s %s" % ("--", "-----", "----"))
+        for patch in patches:
+            print("%-7d %-12s %s" %
+                  (patch['id'], patch['state'], patch['name']))
+
+
+def action_list(rpc, filter, submitter_str, delegate_str, format_str=None):
+    filter.resolve_ids(rpc)
+
+    if submitter_str is not None:
+        ids = person_ids_by_name(rpc, submitter_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             submitter_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches submitted by %s <%s>:' %
+                      (unicode(person['name']).encode('utf-8'),
+                       unicode(person['email']).encode('utf-8')))
+                f = filter
+                f.add("submitter_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    if delegate_str is not None:
+        ids = person_ids_by_name(rpc, delegate_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             delegate_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches delegated to %s <%s>:' %
+                      (person['name'], person['email']))
+                f = filter
+                f.add("delegate_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    patches = rpc.patch_list(filter.d)
+    list_patches(patches, format_str)
+
+
+def action_projects(rpc):
+    projects = rpc.project_list("", 0)
+    print("%-5s %-24s %s" % ("ID", "Name", "Description"))
+    print("%-5s %-24s %s" % ("--", "----", "-----------"))
+    for project in projects:
+        print("%-5d %-24s %s" % (project['id'],
+                                 project['linkname'],
+                                 project['name']))
+
+
+def action_check_list(rpc):
+    checks = rpc.check_list()
+    print("%-5s %-16s %-8s %s" % ("ID", "Context", "State", "Patch"))
+    print("%-5s %-16s %-8s %s" % ("--", "-------", "-----", "-----"))
+    for check in checks:
+        print("%-5s %-16s %-8s %s" % (check['id'],
+                                      check['context'],
+                                      check['state'],
+                                      check['patch']))
+
+
+def action_check_info(rpc, check_id):
+    check = rpc.check_get(check_id)
+    s = "Information for check id %d" % (check_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(check.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_check_create(rpc, patch_id, context, state, url, description):
+    try:
+        rpc.check_create(patch_id, context, state, url, description)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error creating check: %s\n" % f.faultString)
+
+
+def action_states(rpc):
+    states = rpc.state_list("", 0)
+    print("%-5s %s" % ("ID", "Name"))
+    print("%-5s %s" % ("--", "----"))
+    for state in states:
+        print("%-5d %s" % (state['id'], state['name']))
+
+
+def action_info(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = "Information for patch id %d" % (patch_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(patch.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_get(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = rpc.patch_get_mbox(patch_id)
+
+    if patch == {} or len(s) == 0:
+        sys.stderr.write("Unable to get patch %d\n" % patch_id)
+        sys.exit(1)
+
+    base_fname = fname = os.path.basename(patch['filename'])
+    i = 0
+    while os.path.exists(fname):
+        fname = "%s.%d" % (base_fname, i)
+        i += 1
+
+    try:
+        f = open(fname, "w")
+    except:
+        sys.stderr.write("Unable to open %s for writing\n" % fname)
+        sys.exit(1)
+
+    try:
+        f.write(unicode(s).encode("utf-8"))
+        f.close()
+        print('Saved patch to %s' % fname)
+    except:
+        sys.stderr.write("Failed to write to %s\n" % fname)
+        sys.exit(1)
+
+
+def action_apply(rpc, patch_id, apply_cmd=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    if apply_cmd is None:
+        print('Applying patch #%d to current directory' % patch_id)
+        apply_cmd = ['patch', '-p1']
+    else:
+        print('Applying patch #%d using %s' %
+              (patch_id, repr(' '.join(apply_cmd))))
+
+    print('Description: %s' % patch['name'])
+    s = rpc.patch_get_mbox(patch_id)
+    if len(s) > 0:
+        proc = subprocess.Popen(apply_cmd, stdin=subprocess.PIPE)
+        proc.communicate(unicode(s).encode('utf-8'))
+        return proc.returncode
+    else:
+        sys.stderr.write("Error: No patch content found\n")
+        sys.exit(1)
+
+
+def action_update_patch(rpc, patch_id, state=None, archived=None, commit=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    params = {}
+
+    if state:
+        state_id = state_id_by_name(rpc, state)
+        if state_id == 0:
+            sys.stderr.write("Error: No State found matching %s*\n" % state)
+            sys.exit(1)
+        params['state'] = state_id
+
+    if commit:
+        params['commit_ref'] = commit
+
+    if archived:
+        params['archived'] = archived == 'yes'
+
+    success = False
+    try:
+        success = rpc.patch_set(patch_id, params)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error updating patch: %s\n" % f.faultString)
+
+    if not success:
+        sys.stderr.write("Patch not updated\n")
+
+
+def patch_id_from_hash(rpc, project, hash):
+    try:
+        patch = rpc.patch_get_by_project_hash(project, hash)
+    except xmlrpclib.Fault:
+        # the server may not have the newer patch_get_by_project_hash function,
+        # so fall back to hash-only.
+        patch = rpc.patch_get_by_hash(hash)
+
+    if patch == {}:
+        sys.stderr.write("No patch has the hash provided\n")
+        sys.exit(1)
+
+    patch_id = patch['id']
+    # be super paranoid
+    try:
+        patch_id = int(patch_id)
+    except:
+        sys.stderr.write("Invalid patch ID obtained from server\n")
+        sys.exit(1)
+    return patch_id
+
+auth_actions = ['check_create', 'update']
+
+
+def main():
+    hash_parser = argparse.ArgumentParser(add_help=False)
+    hash_parser.add_argument(
+        '-h', metavar='HASH', dest='hash', action='store',
+        help='''Lookup by patch hash'''
+    )
+    hash_parser.add_argument(
+        'id', metavar='ID', nargs='*', action='store', type=int,
+        help='Patch ID',
+    )
+    hash_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Lookup patch in project'''
+    )
+
+    filter_parser = argparse.ArgumentParser(add_help=False)
+    filter_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Filter by patch state (e.g., 'New', 'Accepted', etc.)'''
+    )
+    filter_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Filter by patch archived state'''
+    )
+    filter_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Filter by project name (see 'projects' for list)'''
+    )
+    filter_parser.add_argument(
+        '-w', metavar='WHO',
+        help='''Filter by submitter (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-d', metavar='WHO',
+        help='''Filter by delegate (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-n', metavar='MAX#',
+        type=int,
+        help='''Return first n results'''
+    )
+    filter_parser.add_argument(
+        '-N', metavar='MAX#',
+        type=int,
+        help='''Return last N results'''
+    )
+    filter_parser.add_argument(
+        '-m', metavar='MESSAGEID',
+        help='''Filter by Message-Id'''
+    )
+    filter_parser.add_argument(
+        '-f', metavar='FORMAT',
+        help='''Print output in the given format. You can use tags matching '''
+        '''fields, e.g. %%{id}, %%{state}, or %%{msgid}.'''
+    )
+    filter_parser.add_argument(
+        'patch_name', metavar='STR', nargs='?',
+        help='substring to search for patches by name',
+    )
+
+    action_parser = argparse.ArgumentParser(
+        prog='pwclient',
+        epilog='Use \'pwclient <command> --help\' for more info',
+    )
+
+    subparsers = action_parser.add_subparsers(
+        title='Commands',
+    )
+    apply_parser = subparsers.add_parser(
+        'apply', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch (in the current dir, using -p1)'''
+    )
+    apply_parser.set_defaults(subcmd='apply')
+    git_am_parser = subparsers.add_parser(
+        'git-am', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch to current git branch using "git am".'''
+    )
+    git_am_parser.set_defaults(subcmd='git_am')
+    git_am_parser.add_argument(
+        '-s', '--signoff',
+        action='store_true',
+        help='''pass --signoff to git-am'''
+    )
+    get_parser = subparsers.add_parser(
+        'get', parents=[hash_parser], conflict_handler='resolve',
+        help='''Download a patch and save it locally'''
+    )
+    get_parser.set_defaults(subcmd='get')
+    info_parser = subparsers.add_parser(
+        'info', parents=[hash_parser], conflict_handler='resolve',
+        help='''Display patchwork info about a given patch ID'''
+    )
+    info_parser.set_defaults(subcmd='info')
+    projects_parser = subparsers.add_parser(
+        'projects',
+        help='''List all projects'''
+    )
+    projects_parser.set_defaults(subcmd='projects')
+    check_list_parser = subparsers.add_parser(
+        'check-list',
+        add_help=False,
+        help='''List all checks'''
+    )
+    check_list_parser.set_defaults(subcmd='check_list')
+    check_info_parser = subparsers.add_parser(
+        'check-info',
+        add_help=False,
+        help='''Show information for a given check'''
+    )
+    check_info_parser.set_defaults(subcmd='check_info')
+    check_info_parser.add_argument(
+        'check_id', metavar='ID', action='store', type=int,
+        help='Check ID',)
+    check_create_parser = subparsers.add_parser(
+        'check-create', parents=[hash_parser], conflict_handler='resolve',
+        help='Add a check to a patch')
+    check_create_parser.set_defaults(subcmd='check_create')
+    check_create_parser.add_argument(
+        '-c', metavar='CONTEXT')
+    check_create_parser.add_argument(
+        '-s', choices=('pending', 'success', 'warning', 'fail'))
+    check_create_parser.add_argument(
+        '-u', metavar='TARGET_URL', default="")
+    check_create_parser.add_argument(
+        '-d', metavar='DESCRIPTION', default="")
+    states_parser = subparsers.add_parser(
+        'states',
+        help='''Show list of potential patch states'''
+    )
+    states_parser.set_defaults(subcmd='states')
+    view_parser = subparsers.add_parser(
+        'view', parents=[hash_parser], conflict_handler='resolve',
+        help='''View a patch'''
+    )
+    view_parser.set_defaults(subcmd='view')
+    update_parser = subparsers.add_parser(
+        'update', parents=[hash_parser], conflict_handler='resolve',
+        help='''Update patch''',
+        epilog='''Using a COMMIT-REF allows for only one ID to be specified''',
+    )
+    update_parser.add_argument(
+        '-c', metavar='COMMIT-REF',
+        help='''commit reference hash'''
+    )
+    update_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Set patch state (e.g., 'Accepted', 'Superseded' etc.)'''
+    )
+    update_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Set patch archived state'''
+    )
+    update_parser.set_defaults(subcmd='update')
+    list_parser = subparsers.add_parser("list",
+                                        # aliases=['search'],
+                                        parents=[filter_parser],
+                                        help='''List patches, using the optional filters specified
+        below and an optional substring to search for patches
+        by name'''
+                                        )
+    list_parser.set_defaults(subcmd='list')
+    search_parser = subparsers.add_parser("search",
+                                          parents=[filter_parser],
+                                          help='''Alias for "list"'''
+                                          )
+    # Poor man's argparse aliases:
+    # We register the "search" parser but effectively use "list" for the
+    # help-text.
+    search_parser.set_defaults(subcmd='list')
+    if len(sys.argv) < 2:
+        action_parser.print_help()
+        sys.exit(0)
+
+    args = action_parser.parse_args()
+    args = dict(vars(args))
+    action = args.get('subcmd')
+
+    if args.get('hash') and len(args.get('id')):
+        # mimic mutual exclusive group
+        locals()[action + '_parser'].error(
+            "[-h HASH] and [ID [ID ...]] are mutually exlusive")
+
+    # set defaults
+    filt = Filter()
+    commit_str = None
+    url = DEFAULT_URL
+
+    archived_str = args.get('a')
+    state_str = args.get('s')
+    project_str = args.get('p')
+    submitter_str = args.get('w')
+    delegate_str = args.get('d')
+    format_str = args.get('f')
+    hash_str = args.get('hash')
+    patch_ids = args.get('id')
+    msgid_str = args.get('m')
+    if args.get('c'):
+        # update multiple IDs with a single commit-hash does not make sense
+        if action == 'update' and patch_ids and len(patch_ids) > 1:
+            update_parser.error(
+                "Declining update with COMMIT-REF on multiple IDs")
+        commit_str = args.get('c')
+
+    if state_str is None and archived_str is None and action == 'update':
+        update_parser.error(
+            'Must specify one or more update options (-a or -s)')
+
+    if args.get('n') is not None:
+        try:
+            filt.add("max_count", args.get('n'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('n'))
+
+    if args.get('N') is not None:
+        try:
+            filt.add("max_count", 0 - args.get('N'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('N'))
+
+    do_signoff = args.get('signoff')
+
+    # grab settings from config files
+    config = ConfigParser.ConfigParser()
+    config.read([CONFIG_FILE])
+
+    if not config.has_section('options') and os.path.exists(CONFIG_FILE):
+        sys.stderr.write('~/.pwclientrc is in the old format. Migrating it...')
+
+        old_project = config.get('base', 'project')
+
+        new_config = ConfigParser.ConfigParser()
+        new_config.add_section('options')
+
+        new_config.set('options', 'default', old_project)
+        new_config.add_section(old_project)
+
+        new_config.set(old_project, 'url', config.get('base', 'url'))
+        if config.has_option('auth', 'username'):
+            new_config.set(
+                old_project, 'username', config.get('auth', 'username'))
+        if config.has_option('auth', 'password'):
+            new_config.set(
+                old_project, 'password', config.get('auth', 'password'))
+
+        old_config_file = CONFIG_FILE + '.orig'
+        shutil.copy2(CONFIG_FILE, old_config_file)
+
+        with open(CONFIG_FILE, 'wb') as fd:
+            new_config.write(fd)
+
+        sys.stderr.write(' Done.\n')
+        sys.stderr.write(
+            'Your old ~/.pwclientrc was saved to %s\n' % old_config_file)
+        sys.stderr.write(
+            'and was converted to the new format. You may want to\n')
+        sys.stderr.write('inspect it before continuing.\n')
+        sys.exit(1)
+
+    if not project_str:
+        try:
+            project_str = config.get('options', 'default')
+        except:
+            action_parser.error(
+                "No default project configured in ~/.pwclientrc")
+
+    if not config.has_section(project_str):
+        sys.stderr.write(
+            'No section for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not config.has_option(project_str, 'url'):
+        sys.stderr.write(
+            'No URL for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not do_signoff and config.has_option('options', 'signoff'):
+        do_signoff = config.getboolean('options', 'signoff')
+    if not do_signoff and config.has_option(project_str, 'signoff'):
+        do_signoff = config.getboolean(project_str, 'signoff')
+
+    url = config.get(project_str, 'url')
+
+    transport = None
+    if action in auth_actions:
+        if config.has_option(project_str, 'username') and \
+                config.has_option(project_str, 'password'):
+
+            use_https = url.startswith('https')
+
+            transport = BasicHTTPAuthTransport(
+                config.get(project_str, 'username'),
+                config.get(project_str, 'password'),
+                use_https)
+
+        else:
+            sys.stderr.write("The %s action requires authentication, but no "
+                             "username or password\nis configured\n" % action)
+            sys.exit(1)
+
+    if project_str:
+        filt.add("project", project_str)
+
+    if state_str:
+        filt.add("state", state_str)
+
+    if archived_str:
+        filt.add("archived", archived_str == 'yes')
+
+    if msgid_str:
+        filt.add("msgid", msgid_str)
+
+    try:
+        rpc = xmlrpclib.Server(url, transport=transport)
+    except:
+        sys.stderr.write("Unable to connect to %s\n" % url)
+        sys.exit(1)
+
+    # It should be safe to assume hash_str is not zero, but who knows..
+    if hash_str is not None:
+        patch_ids = [patch_id_from_hash(rpc, project_str, hash_str)]
+
+    # helper for non_empty() to print correct helptext
+    h = locals()[action + '_parser']
+
+    # Require either hash_str or IDs for
+    def non_empty(h, patch_ids):
+        """Error out if no patch IDs were specified"""
+        if patch_ids is None or len(patch_ids) < 1:
+            sys.stderr.write("Error: Missing Argument! Either [-h HASH] or "
+                             "[ID [ID ...]] are required\n")
+            if h:
+                h.print_help()
+            sys.exit(1)
+        return patch_ids
+
+    if action == 'list' or action == 'search':
+        if args.get('patch_name') is not None:
+            filt.add("name__icontains", args.get('patch_name'))
+        action_list(rpc, filt, submitter_str, delegate_str, format_str)
+
+    elif action.startswith('project'):
+        action_projects(rpc)
+
+
+    elif action.startswith('state'):
+        action_states(rpc)
+
+    elif action == 'view':
+        pager = os.environ.get('PAGER')
+        if pager:
+            pager = subprocess.Popen(
+                pager.split(), stdin=subprocess.PIPE
+            )
+        if pager:
+            i = list()
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    i.append(unicode(s).encode("utf-8"))
+            if len(i) > 0:
+                pager.communicate(input="\n".join(i))
+            pager.stdin.close()
+        else:
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    print(unicode(s).encode("utf-8"))
+
+    elif action == 'info':
+        for patch_id in non_empty(h, patch_ids):
+            action_info(rpc, patch_id)
+
+    elif action == 'get':
+        for patch_id in non_empty(h, patch_ids):
+            action_get(rpc, patch_id)
+
+    elif action == 'apply':
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id)
+            if ret:
+                sys.stderr.write("Apply failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'git_am':
+        cmd = ['git', 'am']
+        if do_signoff:
+            cmd.append('-s')
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id, cmd)
+            if ret:
+                sys.stderr.write("'git am' failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'update':
+        for patch_id in non_empty(h, patch_ids):
+            action_update_patch(rpc, patch_id, state=state_str,
+                                archived=archived_str, commit=commit_str
+                                )
+
+    elif action == 'check_list':
+        action_check_list(rpc)
+
+    elif action == 'check_info':
+        check_id = args['check_id']
+        action_check_info(rpc, check_id)
+
+    elif action == 'check_create':
+        for patch_id in non_empty(h, patch_ids):
+            action_check_create(
+                rpc, patch_id, args['c'], args['s'], args['u'], args['d'])
+
+    else:
+        sys.stderr.write("Unknown action '%s'\n" % action)
+        action_parser.print_help()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
-- 
2.7.0

[dpdk-ci] [PATCH v4 5/7] tools: add per-patch report mailer

[Thomas Monjalon]

The report sent by this script will be parsed and integrated in patchwork
if the patch was public.
Otherwise it will just send the report to the patch submitter.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 config/ci.config           |   3 ++
 tools/send-patch-report.sh | 128 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
 create mode 100755 tools/send-patch-report.sh

diff --git a/config/ci.config b/config/ci.config
index 722aeb5..b3b0376 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -3,5 +3,8 @@
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
 
+# The mailer (sendmail, mail, mailx) must support the option -t
+# export DPDK_CI_MAILER=/usr/sbin/sendmail
+
 # The pwclient script is part of patchwork and is copied in dpdk-ci
 # export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/tools/send-patch-report.sh b/tools/send-patch-report.sh
new file mode 100755
index 0000000..716e8e3
--- /dev/null
+++ b/tools/send-patch-report.sh
@@ -0,0 +1,128 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) [options] < report
+
+	Send test report in a properly formatted email for patchwork integration.
+	The report is submitted to this script via stdin.
+
+	options:
+	        -t title    subject of the patch email
+	        -f from     sender of the patch email
+	        -m msgid    id of the patch email
+	        -p listid   mailing list publishing the patch
+	        -l label    title of the test
+	        -s status   one of these test results: SUCCESS, WARNING, FAILURE
+	        -d desc     few words to better describe the status
+	        -h          this help
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+unset title
+unset from
+unset msgid
+unset listid
+unset label
+unset status
+unset desc
+while getopts d:f:hl:m:p:s:t: arg ; do
+	case $arg in
+		t ) title=$OPTARG ;;
+		f ) from=$OPTARG ;;
+		m ) msgid=$OPTARG ;;
+		p ) listid=$OPTARG ;;
+		l ) label=$OPTARG ;;
+		s ) status=$OPTARG ;;
+		d ) desc=$OPTARG ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -t 0 ] ; then
+	printf 'nothing to read on stdin\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+report=$(cat)
+
+writeheaders () # <subject> <ref> <to> [cc]
+{
+	echo "Subject: $1"
+	echo "In-Reply-To: $2"
+	echo "References: $2"
+	echo "To: $3"
+	[ -z "$4" ] || echo "Cc: $4"
+	echo
+}
+
+writeheadlines () # <label> <status> <description> [pwid]
+{
+	echo "Test-Label: $1"
+	echo "Test-Status: $2"
+	[ -z "$4" ] || echo "http://dpdk.org/patch/$4"
+	echo
+	echo "_${3}_"
+	echo
+}
+
+if echo "$listid" | grep -q 'dev.dpdk.org' ; then
+	# get patchwork id
+	if [ -n "$msgid" ] ; then
+		for try in $(seq 20) ; do
+			pwid=$($pwclient list -f '%{id}' -m "$msgid")
+			[ -n "$pwid" ] && break || sleep 7
+		done
+	fi
+	[ -n "$pwid" ] || pwid='?'
+	# send public report
+	subject=$(echo $title | sed 's,\[dpdk-dev\] ,,')
+	[ "$status" = 'SUCCESS' ] && cc='' || cc="$from"
+	(
+	writeheaders "|$status| $subject" "$msgid" 'test-report@dpdk.org' "$cc"
+	writeheadlines "$label" "$status" "$desc" "$pwid"
+	echo "$report"
+	) | $sendmail -t
+else
+	# send private report
+	(
+		writeheaders "Re: $title" "$msgid" "$from"
+		writeheadlines "$label" "$status" "$desc"
+		echo "$report"
+	) | $sendmail -t
+fi
-- 
2.7.0

[dpdk-ci] [PATCH v4 6/7] tools: add patchwork integration

[Thomas Monjalon]

This script is run with patchwork admin credentials.
It is typically installed on dpdk.org and run for each mail
received in the test-report mailing list.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/update-pw.sh | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 tools/update-pw.sh

diff --git a/tools/update-pw.sh b/tools/update-pw.sh
new file mode 100755
index 0000000..1a60b63
--- /dev/null
+++ b/tools/update-pw.sh
@@ -0,0 +1,85 @@
+#! /bin/sh
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <report_url>
+
+	Add or update a check in patchwork based on a test report.
+	The argument specifies only the last URL parts of the test-report
+	mailing list archives (month/id.html).
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+if [ -z "$1" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+url="http://dpdk.org/ml/archives/test-report/$1"
+mmarker='<!--beginarticle-->'
+for try in $(seq 20) ; do
+	[ -z "$report" -o $try -ge 19 ] || continue # 2 last tries if got something
+	[ $try -le 1 ] || sleep 3 # no delay before first try
+	report=$(curl -sf $url | grep -m1 -A9 "$mmarker") || continue
+	echo "$report" | grep -q '^_.*_$' && break || continue
+done
+if [ -z "$report" ] ; then
+	echo "cannot download report at $url" >&2
+	exit 2
+fi
+
+pwid=$(echo "$report" | sed -rn 's,.*http://.*dpdk.org/.*patch/([0-9]+).*,\1,p')
+label=$(echo "$report" | sed -n 's,.*Test-Label: *,,p')
+status=$(echo "$report" | sed -n 's,.*Test-Status: *,,p')
+desc=$(echo "$report" | sed -n 's,^_\(.*\)_$,\1,p')
+case $status in
+	'SUCCESS') pwstatus='success' ;;
+	'WARNING') pwstatus='warning' ;;
+	'FAILURE') pwstatus='fail' ;;
+esac
+printf 'id = %s\nlabel = %s\nstatus = %s/%s %s\nurl = %s\n' \
+	"$pwid" "$label" "$status" "$pwstatus" "$desc" "$url"
+[ -n "$pwid" -a -n "$label" -a -n "$status" -a -n "$desc" ] || exit 3
+
+pwclient=$(dirname $(readlink -m $0))/pwclient
+$pwclient check-create -c "$label" -s "$pwstatus" -d "$desc" -u "$url" $pwid
-- 
2.7.0

[dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

This is the first test in this repository.
It runs on dpdk.org and use checkpatch.pl of Linux.

Note that the patch is not applied on a git tree for this basic test.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tests/checkpatch.sh | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100755 tests/checkpatch.sh

diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh
new file mode 100755
index 0000000..319da1e
--- /dev/null
+++ b/tests/checkpatch.sh
@@ -0,0 +1,70 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) dpdk_dir < email
+
+	Check email-formatted patch from stdin.
+	This test runs checkpatch.pl of Linux via a script in dpdk_dir.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+toolsdir=$(dirname $(readlink -m $0))/../tools
+dpdkdir=$1
+
+email=/tmp/$(basename $0 sh)$$
+$toolsdir/filter-patch-email.sh >$email
+trap "rm -f $email" INT EXIT
+
+eval $($toolsdir/parse-email.sh $email)
+# normal exit if no valid patch in the email
+[ -n "$subject" -a -n "$from" ] || exit 0
+
+failed=false
+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true
+report=$(echo "$report" | sed '1,/^###/d')
+
+label='checkpatch'
+$failed && status='WARNING' || status='SUCCESS'
+$failed && desc='coding style issues' || desc='coding style OK'
+
+echo "$report" | $toolsdir/send-patch-report.sh \
+	-t "$subject" -f "$from" -m "$msgid" -p "$listid" \
+	-l "$label" -s "$status" -d "$desc"
-- 
2.7.0

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

[-- Attachment #1.1: Type: text/plain, Size: 6068 bytes --]

Hi Thomas,



Here are some questiones about your scripts:

1. With the script "send-patch-report.sh", Can I send my perpatch build results to patchwork as checkpatch result in patchwork?

Such as, generate the GUI as blow:

[cid:image001.png@01D24FCC.01E4A4D0]

2. Must I create /etc/dpdk/ci.config before using your scripts?

   Because when I run it, it prompted following error message:

   /root/dpdk-cii/tools/load-ci-config.sh: line 5: /etc/dpdk/ci.config: No such file or directory

   /root/dpdk-cii/tools/load-ci-config.sh: line 6: /etc/dpdk/ci.config: No such file or directory



3. Some confuse about the parameters in "send-patch-report.sh":

           options:

                -t title    subject of the patch email

                -f from     sender of the patch email

                -m msgid    id of the patch email

                -p listid   mailing list publishing the patch

                -l label    title of the test

                -s status   one of these test results: SUCCESS, WARNING, FAILURE

                -d desc     few words to better describe the status



For example, about patch http://www.dpdk.org/dev/patchwork/patch/17673/

-t title    subject of the patch email

   Is it "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations."?

-f from     sender of the patch email

   Is it the author who send the patch? In patch 17673, is it akhil.goyal@nxp.com<mailto:akhil.goyal@nxp.com>?

-m msgid    id of the patch email

   Is it the message-id of this patch? In patch 17673, is it 20161205125540.6419-3-akhil.goyal@nxp.com<mailto:20161205125540.6419-3-akhil.goyal@nxp.com>?

-p listid   mailing list publishing the patch

   Is it the receiver about this patch? In patch 17673, is it dev@dpdk.org<mailto:dev@dpdk.org>?



I try to send my result with script "send-patch-report.sh" with below command, but nothing happened.

echo "$report" | /root/dpdk-cii/tools/send-patch-report.sh -t "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations." -f "akhil.goyal@nxp.com" -m "20161205125540.6419-3-akhil.goyal@nxp.com" -p "dev@dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"





Best Regards

Fangfang





-----Original Message-----
From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Thomas Monjalon
Sent: Monday, December 5, 2016 9:26 PM
To: ci@dpdk.org
Subject: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch



This is the first test in this repository.

It runs on dpdk.org and use checkpatch.pl of Linux.



Note that the patch is not applied on a git tree for this basic test.



Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>>

---

tests/checkpatch.sh | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 70 insertions(+)

create mode 100755 tests/checkpatch.sh



diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh new file mode 100755 index 0000000..319da1e

--- /dev/null

+++ b/tests/checkpatch.sh

@@ -0,0 +1,70 @@

+#! /bin/sh -e

+

+# BSD LICENSE

+#

+# Copyright 2016 6WIND S.A.

+#

+# Redistribution and use in source and binary forms, with or without #

+modification, are permitted provided that the following conditions #

+are met:

+#

+#   * Redistributions of source code must retain the above copyright

+#     notice, this list of conditions and the following disclaimer.

+#   * Redistributions in binary form must reproduce the above copyright

+#     notice, this list of conditions and the following disclaimer in

+#     the documentation and/or other materials provided with the

+#     distribution.

+#   * Neither the name of 6WIND S.A. nor the names of its

+#     contributors may be used to endorse or promote products derived

+#     from this software without specific prior written permission.

+#

+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS #

+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT #

+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR #

+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT #

+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, #

+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT #

+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, #

+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY #

+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT #

+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE #

+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

+

+print_usage () {

+    cat <<- END_OF_HELP

+    usage: $(basename $0) dpdk_dir < email

+

+    Check email-formatted patch from stdin.

+    This test runs checkpatch.pl of Linux via a script in dpdk_dir.

+    END_OF_HELP

+}

+

+while getopts h arg ; do

+    case $arg in

+          h ) print_usage ; exit 0 ;;

+          ? ) print_usage >&2 ; exit 1 ;;

+    esac

+done

+shift $(($OPTIND - 1))

+toolsdir=$(dirname $(readlink -m $0))/../tools

+dpdkdir=$1

+

+email=/tmp/$(basename $0 sh)$$

+$toolsdir/filter-patch-email.sh >$email trap "rm -f $email" INT EXIT

+

+eval $($toolsdir/parse-email.sh $email) # normal exit if no valid patch

+in the email [ -n "$subject" -a -n "$from" ] || exit 0

+

+failed=false

+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true

+report=$(echo "$report" | sed '1,/^###/d')

+

+label='checkpatch'

+$failed && status='WARNING' || status='SUCCESS'

+$failed && desc='coding style issues' || desc='coding style OK'

+

+echo "$report" | $toolsdir/send-patch-report.sh \

+    -t "$subject" -f "$from" -m "$msgid" -p "$listid" \

+    -l "$label" -s "$status" -d "$desc"

--

2.7.0



[-- Attachment #1.2: Type: text/html, Size: 15482 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 7765 bytes --]

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

Hi Fangfang,

2016-12-06 06:34, Wei, FangfangX:
> Hi Thomas,
> 
> Here are some questiones about your scripts:
> 
> 1. With the script "send-patch-report.sh", Can I send my perpatch build results to patchwork as checkpatch result in patchwork?
> 
> Such as, generate the GUI as blow:
> 
> [cid:image001.png@01D24FCC.01E4A4D0]

Yes, your report will be properly integrated into patchwork if you use
send-patch-report.sh.

> 2. Must I create /etc/dpdk/ci.config before using your scripts?

No it is not required.

>    Because when I run it, it prompted following error message:
>    /root/dpdk-cii/tools/load-ci-config.sh: line 5: /etc/dpdk/ci.config: No such file or directory
>    /root/dpdk-cii/tools/load-ci-config.sh: line 6: /etc/dpdk/ci.config: No such file or directory

It is strange. This configuration file is read only if it exists:
	test ! -r /etc/dpdk/ci.config || . /etc/dpdk/ci.config

> 3. Some confuse about the parameters in "send-patch-report.sh":
> 
>            options:
>                 -t title    subject of the patch email
>                 -f from     sender of the patch email
>                 -m msgid    id of the patch email
>                 -p listid   mailing list publishing the patch

These 4 options can be filled with the help of parse-email.sh.

>                 -l label    title of the test
>                 -s status   one of these test results: SUCCESS, WARNING, FAILURE
>                 -d desc     few words to better describe the status
> 
> For example, about patch http://www.dpdk.org/dev/patchwork/patch/17673/
> 
> -t title    subject of the patch email
>    Is it "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations."?

It should be the original subject from the email, not the one converted by patchwork (as above):
	[dpdk-dev] [PATCH 2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations.

> -f from     sender of the patch email
>    Is it the author who send the patch? In patch 17673, is it akhil.goyal@nxp.com<mailto:akhil.goyal@nxp.com>?

Not exactly. You should refer to the original mail:
	Akhil Goyal <akhil.goyal@nxp.com>

> -m msgid    id of the patch email
>    Is it the message-id of this patch? In patch 17673, is it 20161205125540.6419-3-akhil.goyal@nxp.com<mailto:20161205125540.6419-3-akhil.goyal@nxp.com>?

No it is <20161205125540.6419-3-akhil.goyal@nxp.com>

> -p listid   mailing list publishing the patch
>    Is it the receiver about this patch? In patch 17673, is it dev@dpdk.org<mailto:dev@dpdk.org>?

No, it is the List-Id header:
	DPDK patches and discussions <dev.dpdk.org>

> I try to send my result with script "send-patch-report.sh" with below command, but nothing happened.
> 
> echo "$report" | /root/dpdk-cii/tools/send-patch-report.sh -t "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations." -f "akhil.goyal@nxp.com" -m "20161205125540.6419-3-akhil.goyal@nxp.com" -p "dev@dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

The list-id is wrong so it was detected as a private patch.
It has probably sent a private report to Akhil.

I think your issue is to get the original email.
If I understand well, you are getting the patch from patchwork.
If you want to continue getting the patch from patchwork, you must use this URL:
	http://www.dpdk.org/dev/patchwork/patch/17673/mbox/
and fake listid and from.
For the listid, you can use -p dev.dpdk.org
For the from, you must convert [dpdk-dev,v2,1/4] to [dpdk-dev] [PATCH v2 1/4]
or just [PATCH v2 1/4] (without [dpdk-dev]) would be sufficient.

For the checkpatch example, it fetches emails from patchwork@dpdk.org mailbox
which is registered in the dev mailing list.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

Thank you for your reply, I'll try it again.

Best Regards
Fangfang Wei

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, December 6, 2016 4:40 PM
To: Wei, FangfangX <fangfangx.wei@intel.com>
Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>
Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

Hi Fangfang,

2016-12-06 06:34, Wei, FangfangX:
> Hi Thomas,
> 
> Here are some questiones about your scripts:
> 
> 1. With the script "send-patch-report.sh", Can I send my perpatch build results to patchwork as checkpatch result in patchwork?
> 
> Such as, generate the GUI as blow:
> 
> [cid:image001.png@01D24FCC.01E4A4D0]

Yes, your report will be properly integrated into patchwork if you use send-patch-report.sh.

> 2. Must I create /etc/dpdk/ci.config before using your scripts?

No it is not required.

>    Because when I run it, it prompted following error message:
>    /root/dpdk-cii/tools/load-ci-config.sh: line 5: /etc/dpdk/ci.config: No such file or directory
>    /root/dpdk-cii/tools/load-ci-config.sh: line 6: 
> /etc/dpdk/ci.config: No such file or directory

It is strange. This configuration file is read only if it exists:
	test ! -r /etc/dpdk/ci.config || . /etc/dpdk/ci.config

> 3. Some confuse about the parameters in "send-patch-report.sh":
> 
>            options:
>                 -t title    subject of the patch email
>                 -f from     sender of the patch email
>                 -m msgid    id of the patch email
>                 -p listid   mailing list publishing the patch

These 4 options can be filled with the help of parse-email.sh.

>                 -l label    title of the test
>                 -s status   one of these test results: SUCCESS, WARNING, FAILURE
>                 -d desc     few words to better describe the status
> 
> For example, about patch 
> http://www.dpdk.org/dev/patchwork/patch/17673/
> 
> -t title    subject of the patch email
>    Is it "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations."?

It should be the original subject from the email, not the one converted by patchwork (as above):
	[dpdk-dev] [PATCH 2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations.

> -f from     sender of the patch email
>    Is it the author who send the patch? In patch 17673, is it akhil.goyal@nxp.com<mailto:akhil.goyal@nxp.com>?

Not exactly. You should refer to the original mail:
	Akhil Goyal <akhil.goyal@nxp.com>

> -m msgid    id of the patch email
>    Is it the message-id of this patch? In patch 17673, is it 20161205125540.6419-3-akhil.goyal@nxp.com<mailto:20161205125540.6419-3-akhil.goyal@nxp.com>?

No it is <20161205125540.6419-3-akhil.goyal@nxp.com>

> -p listid   mailing list publishing the patch
>    Is it the receiver about this patch? In patch 17673, is it dev@dpdk.org<mailto:dev@dpdk.org>?

No, it is the List-Id header:
	DPDK patches and discussions <dev.dpdk.org>

> I try to send my result with script "send-patch-report.sh" with below command, but nothing happened.
> 
> echo "$report" | /root/dpdk-cii/tools/send-patch-report.sh -t "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations." -f "akhil.goyal@nxp.com" -m "20161205125540.6419-3-akhil.goyal@nxp.com" -p "dev@dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

The list-id is wrong so it was detected as a private patch.
It has probably sent a private report to Akhil.

I think your issue is to get the original email.
If I understand well, you are getting the patch from patchwork.
If you want to continue getting the patch from patchwork, you must use this URL:
	http://www.dpdk.org/dev/patchwork/patch/17673/mbox/
and fake listid and from.
For the listid, you can use -p dev.dpdk.org For the from, you must convert [dpdk-dev,v2,1/4] to [dpdk-dev] [PATCH v2 1/4] or just [PATCH v2 1/4] (without [dpdk-dev]) would be sufficient.

For the checkpatch example, it fetches emails from patchwork@dpdk.org mailbox which is registered in the dev mailing list.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

Hi Thomas,
I try to use " send-patch-report.sh" with your suggestion, but after run it, nothing happened.
The command shows as below:

echo $report | tools/send-patch-report.sh -t "[PATCH, v3] net/i40evf: fix reporting of imissed packets" -f "tcrugnale@sandvine.com" -m "1481055381-14243-1-git-send-email-tcrugnale@sandvine.com" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

In case it was detected as a private patch, I also try to run it with my email address: -f "fangfangx.wei@intel.com", nothing happened, and I didn't receive this email either.

Is there still any error when I use the script? (BTW: This is the report about patch 17720)

Best Regards
Fangfang Wei

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, December 6, 2016 4:40 PM
To: Wei, FangfangX <fangfangx.wei@intel.com>
Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>
Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

Hi Fangfang,

2016-12-06 06:34, Wei, FangfangX:
> Hi Thomas,
> 
> Here are some questiones about your scripts:
> 
> 1. With the script "send-patch-report.sh", Can I send my perpatch build results to patchwork as checkpatch result in patchwork?
> 
> Such as, generate the GUI as blow:
> 
> [cid:image001.png@01D24FCC.01E4A4D0]

Yes, your report will be properly integrated into patchwork if you use send-patch-report.sh.

> 2. Must I create /etc/dpdk/ci.config before using your scripts?

No it is not required.

>    Because when I run it, it prompted following error message:
>    /root/dpdk-cii/tools/load-ci-config.sh: line 5: /etc/dpdk/ci.config: No such file or directory
>    /root/dpdk-cii/tools/load-ci-config.sh: line 6: 
> /etc/dpdk/ci.config: No such file or directory

It is strange. This configuration file is read only if it exists:
	test ! -r /etc/dpdk/ci.config || . /etc/dpdk/ci.config

> 3. Some confuse about the parameters in "send-patch-report.sh":
> 
>            options:
>                 -t title    subject of the patch email
>                 -f from     sender of the patch email
>                 -m msgid    id of the patch email
>                 -p listid   mailing list publishing the patch

These 4 options can be filled with the help of parse-email.sh.

>                 -l label    title of the test
>                 -s status   one of these test results: SUCCESS, WARNING, FAILURE
>                 -d desc     few words to better describe the status
> 
> For example, about patch 
> http://www.dpdk.org/dev/patchwork/patch/17673/
> 
> -t title    subject of the patch email
>    Is it "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations."?

It should be the original subject from the email, not the one converted by patchwork (as above):
	[dpdk-dev] [PATCH 2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations.

> -f from     sender of the patch email
>    Is it the author who send the patch? In patch 17673, is it akhil.goyal@nxp.com<mailto:akhil.goyal@nxp.com>?

Not exactly. You should refer to the original mail:
	Akhil Goyal <akhil.goyal@nxp.com>

> -m msgid    id of the patch email
>    Is it the message-id of this patch? In patch 17673, is it 20161205125540.6419-3-akhil.goyal@nxp.com<mailto:20161205125540.6419-3-akhil.goyal@nxp.com>?

No it is <20161205125540.6419-3-akhil.goyal@nxp.com>

> -p listid   mailing list publishing the patch
>    Is it the receiver about this patch? In patch 17673, is it dev@dpdk.org<mailto:dev@dpdk.org>?

No, it is the List-Id header:
	DPDK patches and discussions <dev.dpdk.org>

> I try to send my result with script "send-patch-report.sh" with below command, but nothing happened.
> 
> echo "$report" | /root/dpdk-cii/tools/send-patch-report.sh -t "[dpdk-dev,2/8] drivers/common/dpaa2: Sample descriptors for NXP DPAA2 SEC operations." -f "akhil.goyal@nxp.com" -m "20161205125540.6419-3-akhil.goyal@nxp.com" -p "dev@dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

The list-id is wrong so it was detected as a private patch.
It has probably sent a private report to Akhil.

I think your issue is to get the original email.
If I understand well, you are getting the patch from patchwork.
If you want to continue getting the patch from patchwork, you must use this URL:
	http://www.dpdk.org/dev/patchwork/patch/17673/mbox/
and fake listid and from.
For the listid, you can use -p dev.dpdk.org For the from, you must convert [dpdk-dev,v2,1/4] to [dpdk-dev] [PATCH v2 1/4] or just [PATCH v2 1/4] (without [dpdk-dev]) would be sufficient.

For the checkpatch example, it fetches emails from patchwork@dpdk.org mailbox which is registered in the dev mailing list.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-07 05:48, Wei, FangfangX:
> Hi Thomas,
> I try to use " send-patch-report.sh" with your suggestion, but after run it, nothing happened.
> The command shows as below:
> 
> echo $report | tools/send-patch-report.sh -t "[PATCH, v3] net/i40evf: fix reporting of imissed packets" -f "tcrugnale@sandvine.com" -m "1481055381-14243-1-git-send-email-tcrugnale@sandvine.com" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"
> 
> In case it was detected as a private patch, I also try to run it with my email address: -f "fangfangx.wei@intel.com", nothing happened, and I didn't receive this email either.
> 
> Is there still any error when I use the script? (BTW: This is the report about patch 17720)

You should debug the script to understand what happens.
I suggest you to replace "$sendmail -t" by "cat" in order to print the email
instead of trying to send it.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

Hi Thomas,
Following is the debug information today, please help to review and give some suggestion.

1. replace "$sendmail -t" by "cat":
>echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

usage: pwclient [-h]
                {apply,git-am,get,info,projects,check-list,check-info,check-create,states,view,update,list,search}
                ...
pwclient: error: No default project configured in ~/.pwclientrc


There is no ~/.pwclientrc, should I copy config/pwclientrc to ~/.pwclientrc?


2. replace "$sendmail -t" by "cat" then copy config/pwclientrc to ~/.pwclientrc:
>echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

Traceback (most recent call last):
  File "/root/dpdk-cii/tools/pwclient", line 818, in <module>
    main()
  File "/root/dpdk-cii/tools/pwclient", line 738, in main
    action_list(rpc, filt, submitter_str, delegate_str, format_str)
  File "/root/dpdk-cii/tools/pwclient", line 193, in action_list
    filt.resolve_ids(rpc)
  File "/root/dpdk-cii/tools/pwclient", line 94, in resolve_ids
    id = project_id_by_name(rpc, self.project)
  File "/root/dpdk-cii/tools/pwclient", line 138, in project_id_by_name
    projects = rpc.project_list(linkname, 0)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1240, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1599, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1280, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1308, in single_request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1456, in send_content
    connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
    raise err
socket.error: [Errno 110] Connection timed out


3. The ci.config hasn't been loaded, because the environment $DPDK_CI_PWCLIENT and $DPDK_CI_MAILER are None.


4. replace "$sendmail -t" by "cat" then replace " pwid=$($pwclient list -f '%{id}' -m "$msgid " by "#pwid=$($pwclient list -f '%{id}' -m "$msgid")":
>echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

Subject: |SUCCESS| pw17762 [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach
In-Reply-To: 20161208014751.24285-2-stephen@networkplumber.org
References: 20161208014751.24285-2-stephen@networkplumber.org
To: test-report@dpdk.org
Cc: fangfangx.wei@intel.com

Test-Label: Intel Per-patch compilation check
Test-Status: SUCCESS
http://dpdk.org/patch/?

_Compilation OK_

Test-Label: Intel Per-patch compilation check Test-Status: SUCCESS http://www.dpdk.org/dev/patchwork/patch/17762 Submitter: Stephen Hemminger <stephen@networkplumber.org> Date: Wed, 7 Dec 2016 17:47:50 -0800 DPDK git baseline: Repo:dpdk, Branch:master, CommitID:c431384c8fbf8503693bcae1bdcd58d6fa459b8a Patch17762-17762 --> compile pass Build Summary: 18 Builds Done, 18 Successful, 0 Failures Test environment and configuration as below: OS: FreeBSD10.3_64 Kernel Version:10.3-RELEASE CPU info:CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (2194.97-MHz K8-class CPU) GCC Version:gcc (FreeBSD Ports Collection) 4.8.5 Clang Version:3.4.1 x86_64-native-bsdapp-clang x86_64-native-bsdapp-gcc OS: RHEL7.2_64 Kernel Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-gcc x86_64-native-linuxapp-gcc-shared OS: UB1604_64 Kernel Version:4.4.0-47-generic CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 Clang Version:3.8.0 i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc OS: CentOS7_64 Kernel Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc OS: FC24_64 Kernel Version:4.8.6-201.fc24.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2) Clang Version:3.8.0 x86_64-native-linuxapp-gcc i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc-debug


Any suggestion about my debug?

Best Regards
Fangfang Wei

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Wednesday, December 7, 2016 5:33 PM
To: Wei, FangfangX <fangfangx.wei@intel.com>
Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>
Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

2016-12-07 05:48, Wei, FangfangX:
> Hi Thomas,
> I try to use " send-patch-report.sh" with your suggestion, but after run it, nothing happened.
> The command shows as below:
> 
> echo $report | tools/send-patch-report.sh -t "[PATCH, v3] net/i40evf: fix reporting of imissed packets" -f "tcrugnale@sandvine.com" -m "1481055381-14243-1-git-send-email-tcrugnale@sandvine.com" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"
> 
> In case it was detected as a private patch, I also try to run it with my email address: -f "fangfangx.wei@intel.com", nothing happened, and I didn't receive this email either.
> 
> Is there still any error when I use the script? (BTW: This is the 
> report about patch 17720)

You should debug the script to understand what happens.
I suggest you to replace "$sendmail -t" by "cat" in order to print the email instead of trying to send it.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-08 09:02, Wei, FangfangX:
> Hi Thomas,
> Following is the debug information today, please help to review and give some suggestion.
> 
> 1. replace "$sendmail -t" by "cat":
> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"
> 
> usage: pwclient [-h]
>                 {apply,git-am,get,info,projects,check-list,check-info,check-create,states,view,update,list,search}
>                 ...
> pwclient: error: No default project configured in ~/.pwclientrc
> 
> 
> There is no ~/.pwclientrc, should I copy config/pwclientrc to ~/.pwclientrc?

Yes
We could add a section about configuration in the README.

> 2. replace "$sendmail -t" by "cat" then copy config/pwclientrc to ~/.pwclientrc:
> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"
> 
> Traceback (most recent call last):
[...]
> socket.error: [Errno 110] Connection timed out

Is your Internet connection working fine? Do you need some proxy configuration?

> 3. The ci.config hasn't been loaded, because the environment $DPDK_CI_PWCLIENT and $DPDK_CI_MAILER are None.

They are not needed.
Some default values are used in send-patch-report.sh:
sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}

> 4. replace "$sendmail -t" by "cat" then replace " pwid=$($pwclient list -f '%{id}' -m "$msgid " by "#pwid=$($pwclient list -f '%{id}' -m "$msgid")":
> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com" -m "20161208014751.24285-2-stephen@networkplumber.org" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"
> 
> Subject: |SUCCESS| pw17762 [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach
> In-Reply-To: 20161208014751.24285-2-stephen@networkplumber.org
> References: 20161208014751.24285-2-stephen@networkplumber.org
> To: test-report@dpdk.org
> Cc: fangfangx.wei@intel.com
> 
> Test-Label: Intel Per-patch compilation check
> Test-Status: SUCCESS
> http://dpdk.org/patch/?
> 
> _Compilation OK_
> 
> Test-Label: Intel Per-patch compilation check Test-Status: SUCCESS http://www.dpdk.org/dev/patchwork/patch/17762 Submitter: Stephen Hemminger <stephen@networkplumber.org> Date: Wed, 7 Dec 2016 17:47:50 -0800 DPDK git baseline: Repo:dpdk, Branch:master, CommitID:c431384c8fbf8503693bcae1bdcd58d6fa459b8a Patch17762-17762 --> compile pass Build Summary: 18 Builds Done, 18 Successful, 0 Failures Test environment and configuration as below: OS: FreeBSD10.3_64 Kernel Version:10.3-RELEASE CPU info:CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (2194.97-MHz K8-class CPU) GCC Version:gcc (FreeBSD Ports Collection) 4.8.5 Clang Version:3.4.1 x86_64-native-bsdapp-clang x86_64-native-bsdapp-gcc OS: RHEL7.2_64 Kernel Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-gcc x86_64-native-linuxapp-gcc-shared OS: UB1604_64 Kernel Version:4.4.0-47-generic CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 Clang Version:3.8.0 i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc OS: CentOS7_64 Kernel Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc OS: FC24_64 Kernel Version:4.8.6-201.fc24.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2) Clang Version:3.8.0 x86_64-native-linuxapp-gcc i686-native-linuxapp-gcc x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc-debug

You have a wrapping issue ;)
You can remove Test-Label, Test-Status and other lines as they are already
printed above.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

[-- Attachment #1.1: Type: text/plain, Size: 6597 bytes --]

Hi Thomas,



My internet connection is working fine, and also configure http_proxy and https_proxy, but the connection is still timeout.

I think that's because the rpc client can't connect to your rpc server. Is there any other configuration should be configured?



According to my understanding, the "pwid=$($pwclient list -f '%{id}' -m "$msgid " in the script "send-patch-report.sh" is just trying to get the patchwork id, is that right?

Then I just define pwid="17772" in "send-patch-report.sh" directly, run the script again, nothing happened. The email hasn't been sent to test-report@dpdk.org.



I think the function about send-patch-report.sh is just to send patch report to test-report@dpdk.org<mailto:test-report@dpdk.org>. It can't generate the following content in patchwork as checkpatch. Am I right?

[cid:image001.png@01D25227.03650540]





Best Regards

Fangfang Wei



-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
Sent: Thursday, December 8, 2016 9:11 PM
To: Wei, FangfangX <fangfangx.wei@intel.com>
Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>; Chen, WeichunX <weichunx.chen@intel.com>
Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch



2016-12-08 09:02, Wei, FangfangX:

> Hi Thomas,

> Following is the debug information today, please help to review and give some suggestion.

>

> 1. replace "$sendmail -t" by "cat":

> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com<mailto:fangfangx.wei@intel.com>" -m "20161208014751.24285-2-stephen@networkplumber.org<mailto:20161208014751.24285-2-stephen@networkplumber.org>" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

>

> usage: pwclient [-h]

>                 {apply,git-am,get,info,projects,check-list,check-info,check-create,states,view,update,list,search}

>                 ...

> pwclient: error: No default project configured in ~/.pwclientrc

>

>

> There is no ~/.pwclientrc, should I copy config/pwclientrc to ~/.pwclientrc?



Yes

We could add a section about configuration in the README.



> 2. replace "$sendmail -t" by "cat" then copy config/pwclientrc to ~/.pwclientrc:

> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com<mailto:fangfangx.wei@intel.com>" -m "20161208014751.24285-2-stephen@networkplumber.org<mailto:20161208014751.24285-2-stephen@networkplumber.org>" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

>

> Traceback (most recent call last):

[...]

> socket.error: [Errno 110] Connection timed out



Is your Internet connection working fine? Do you need some proxy configuration?



> 3. The ci.config hasn't been loaded, because the environment $DPDK_CI_PWCLIENT and $DPDK_CI_MAILER are None.



They are not needed.

Some default values are used in send-patch-report.sh:

sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}

pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}



> 4. replace "$sendmail -t" by "cat" then replace " pwid=$($pwclient list -f '%{id}' -m "$msgid " by "#pwid=$($pwclient list -f '%{id}' -m "$msgid")":

> >echo $report | tools/send-patch-report.sh -t "pw17762  [PATCH, 1/2] eth: get rid of goto's in rte_eth_dev_detach" -f "fangfangx.wei@intel.com<mailto:fangfangx.wei@intel.com>" -m "20161208014751.24285-2-stephen@networkplumber.org<mailto:20161208014751.24285-2-stephen@networkplumber.org>" -p "dev.dpdk.org" -l "Intel Per-patch compilation check" -s "SUCCESS" -d "Compilation OK"

>

> Subject: |SUCCESS| pw17762 [PATCH, 1/2] eth: get rid of goto's in

> rte_eth_dev_detach

> In-Reply-To: 20161208014751.24285-2-stephen@networkplumber.org<mailto:20161208014751.24285-2-stephen@networkplumber.org>

> References: 20161208014751.24285-2-stephen@networkplumber.org<mailto:20161208014751.24285-2-stephen@networkplumber.org>

> To: test-report@dpdk.org<mailto:test-report@dpdk.org>

> Cc: fangfangx.wei@intel.com<mailto:fangfangx.wei@intel.com>

>

> Test-Label: Intel Per-patch compilation check

> Test-Status: SUCCESS

> http://dpdk.org/patch/?

>

> _Compilation OK_

>

> Test-Label: Intel Per-patch compilation check Test-Status: SUCCESS

> http://www.dpdk.org/dev/patchwork/patch/17762 Submitter: Stephen

> Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>> Date: Wed, 7 Dec 2016 17:47:50

> -0800 DPDK git baseline: Repo:dpdk, Branch:master,

> CommitID:c431384c8fbf8503693bcae1bdcd58d6fa459b8a Patch17762-17762 -->

> compile pass Build Summary: 18 Builds Done, 18 Successful, 0 Failures

> Test environment and configuration as below: OS: FreeBSD10.3_64 Kernel

> Version:10.3-RELEASE CPU info:CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @

> 2.20GHz (2194.97-MHz K8-class CPU) GCC Version:gcc (FreeBSD Ports

> Collection) 4.8.5 Clang Version:3.4.1 x86_64-native-bsdapp-clang

> x86_64-native-bsdapp-gcc OS: RHEL7.2_64 Kernel

> Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v4

> @ 2.20GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang

> Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-gcc

> x86_64-native-linuxapp-gcc-shared OS: UB1604_64 Kernel

> Version:4.4.0-47-generic CPU info:Intel(R) Xeon(R) CPU E5-2699 v3 @

> 2.30GHz GCC Version:gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609

> Clang Version:3.8.0 i686-native-linuxapp-gcc

> x86_64-native-linuxapp-clang x86_64-native-linuxapp-gcc-shared

> x86_64-native-linuxapp-gcc OS: CentOS7_64 Kernel

> Version:3.10.0-327.el7.x86_64 CPU info:Intel(R) Xeon(R) CPU E5-2699 v3

> @ 2.30GHz GCC Version:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Clang

> Version:3.4.2 i686-native-linuxapp-gcc x86_64-native-linuxapp-clang

> x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc OS:

> FC24_64 Kernel Version:4.8.6-201.fc24.x86_64 CPU info:Intel(R) Xeon(R)

> CPU E5-2699 v3 @ 2.30GHz GCC Version:gcc (GCC) 6.2.1 20160916 (Red Hat

> 6.2.1-2) Clang Version:3.8.0 x86_64-native-linuxapp-gcc

> i686-native-linuxapp-gcc x86_64-native-linuxapp-clang

> x86_64-native-linuxapp-gcc-shared x86_64-native-linuxapp-gcc-debug



You have a wrapping issue ;)

You can remove Test-Label, Test-Status and other lines as they are already printed above.

[-- Attachment #1.2: Type: text/html, Size: 14529 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 7624 bytes --]

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-09 08:51, Wei, FangfangX:
> My internet connection is working fine, and also configure http_proxy and https_proxy, but the connection is still timeout.
> 
> I think that's because the rpc client can't connect to your rpc server. Is there any other configuration should be configured?

I've found it is an open issue. pwclient does not work with proxy:
	https://github.com/getpatchwork/patchwork/issues/47

> According to my understanding, the "pwid=$($pwclient list -f '%{id}' -m "$msgid " in the script "send-patch-report.sh" is just trying to get the patchwork id, is that right?

Yes, if you already have to patchwork id, you can put it.
It makes me think that the patchwork id should be an option of this script.

> Then I just define pwid="17772" in "send-patch-report.sh" directly, run the script again, nothing happened. The email hasn't been sent to test-report@dpdk.org.

Are you sure "sendmail -t" is working fine in your environment?

> I think the function about send-patch-report.sh is just to send patch report to test-report@dpdk.org. It can't generate the following content in patchwork as checkpatch. Am I right?

Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed
and integrated in patchwork by update-pw.sh running on the server.
That's how the checkpatch test works (see tests/checkpatch.sh).

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Mcnamara, John]

> -----Original Message-----
> From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Friday, December 9, 2016 9:17 AM
> To: Wei, FangfangX <fangfangx.wei@intel.com>
> Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong
> <yong.liu@intel.com>; Chen, WeichunX <weichunx.chen@intel.com>
> Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch
> 
> 2016-12-09 08:51, Wei, FangfangX:
> > My internet connection is working fine, and also configure http_proxy
> and https_proxy, but the connection is still timeout.
> >
> > I think that's because the rpc client can't connect to your rpc server.
> Is there any other configuration should be configured?
> 
> I've found it is an open issue. pwclient does not work with proxy:
> 	https://github.com/getpatchwork/patchwork/issues/47
> 

Hi,

I have a hardcoded patch that allows pwclient to work within the Intel firewall. I'll send it on to the @intel.com people.

John

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-09 10:07, Mcnamara, John:
> From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > 2016-12-09 08:51, Wei, FangfangX:
> > > My internet connection is working fine, and also configure http_proxy
> > and https_proxy, but the connection is still timeout.
> > >
> > > I think that's because the rpc client can't connect to your rpc server.
> > Is there any other configuration should be configured?
> > 
> > I've found it is an open issue. pwclient does not work with proxy:
> > 	https://github.com/getpatchwork/patchwork/issues/47
> > 
> 
> Hi,
> 
> I have a hardcoded patch that allows pwclient to work within the Intel firewall. I'll send it on to the @intel.com people.

In the meantime, I'm trying to fix the issue upstream.
And I'll send a v5 of these patches to be able to get the patch from patchwork
without requiring pwclient.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Mcnamara, John]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Friday, December 9, 2016 10:11 AM
> To: Mcnamara, John <john.mcnamara@intel.com>
> Cc: Wei, FangfangX <fangfangx.wei@intel.com>; ci@dpdk.org; Xu, Qian Q
> <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>; Chen, WeichunX
> <weichunx.chen@intel.com>
> Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch
> 
> 2016-12-09 10:07, Mcnamara, John:
> > From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> > > 2016-12-09 08:51, Wei, FangfangX:
> > > > My internet connection is working fine, and also configure
> > > > http_proxy
> > > and https_proxy, but the connection is still timeout.
> > > >
> > > > I think that's because the rpc client can't connect to your rpc
> server.
> > > Is there any other configuration should be configured?
> > >
> > > I've found it is an open issue. pwclient does not work with proxy:
> > > 	https://github.com/getpatchwork/patchwork/issues/47
> > >
> >
> > Hi,
> >
> > I have a hardcoded patch that allows pwclient to work within the Intel
> firewall. I'll send it on to the @intel.com people.
> 
> In the meantime, I'm trying to fix the issue upstream.
> And I'll send a v5 of these patches to be able to get the patch from
> patchwork without requiring pwclient.


Thomas,

For what it is worth here is my patch. It isn't a general solution since the proxy is hardcoded and also it doesn't work with a proxy that requires a password.

John



diff --git a/pwclient b/pwclient
index dfbea30..1652a7e 100755
--- a/pwclient
+++ b/pwclient
@@ -29,6 +29,7 @@ import subprocess
 import base64
 import ConfigParser
 import shutil
+import httplib
 
 # Default Patchwork remote XML-RPC server URL
 # This script will check the PW_XMLRPC_URL environment variable
@@ -80,6 +81,22 @@ class Filter:
         """Return human-readable description of the filter."""
         return str(self.d)
 
+class ProxiedTransport(xmlrpclib.Transport):
+
+    def set_proxy(self, proxy):
+        self.proxy = proxy
+
+    def make_connection(self, host):
+        self.realhost = host
+        h = httplib.HTTPConnection(self.proxy)
+        return h
+
+    def send_request(self, connection, handler, request_body):
+        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
+
+    def send_host(self, connection, host):
+        connection.putheader('Host', self.realhost)
+
 class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
 
     def __init__(self, username = None, password = None, use_https = False):
@@ -431,7 +448,8 @@ def main():
     url = config.get(project_str, 'url')
 
     (username, password) = (None, None)
-    transport = None
+    transport = ProxiedTransport()
+    transport.set_proxy('proxy.mydomain.com:123')
     if action in auth_actions:
         if config.has_option(project_str, 'username') and \
                 config.has_option(project_str, 'password'):

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

2016-12-09 08:51, Wei, FangfangX:
>> My internet connection is working fine, and also configure http_proxy and https_proxy, but the connection is still timeout.
>> 
>> I think that's because the rpc client can't connect to your rpc server. Is there any other configuration should be configured?

>I've found it is an open issue. pwclient does not work with proxy:
>	https://github.com/getpatchwork/patchwork/issues/47

With John's patch, pwclient is working via intel firewall now.
Thanks, John!


>> According to my understanding, the "pwid=$($pwclient list -f '%{id}' -m "$msgid " in the script "send-patch-report.sh" is just trying to get the patchwork id, is that right?

>Yes, if you already have to patchwork id, you can put it.
>It makes me think that the patchwork id should be an option of this script.

I think adding patchwork id as an option of this script is a good idea.
Because in my per-patch build report, the patchwork id is known. There's no necessary to get it through patchwork with message id.


>> Then I just define pwid="17772" in "send-patch-report.sh" directly, run the script again, nothing happened. The email hasn't been sent to test-report@dpdk.org.

>Are you sure "sendmail -t" is working fine in your environment?

Actually, I'm not sure about it. In your environment, is there any configuration about sendmail in Linux?


>> I think the function about send-patch-report.sh is just to send patch report to test-report@dpdk.org. It can't generate the following content in patchwork as checkpatch. Am I right?

>Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed and integrated in patchwork by update-pw.sh running on the server.
>That's how the checkpatch test works (see tests/checkpatch.sh).

If I send email with my own script, the receiver is test-report@dpdk.org, and the report format shows as below. Is this email also automatically parsed and integrated in patchwork on the server?

Test-Label: Intel Per-patch compilation check
Test-Status: SUCCESS
http://dpdk.org/patch/17859

_Compilation OK_

The content about this compilation


Best Regards
Fangfang Wei

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

Hi Thomas,

>>Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed and integrated in patchwork by update-pw.sh running on the server.
>>That's how the checkpatch test works (see tests/checkpatch.sh).

>If I send email with my own script, the receiver is test-report@dpdk.org, and the report format shows as below. Is this email also automatically parsed and integrated in patchwork on the server?

>Test-Label: Intel Per-patch compilation check
>Test-Status: SUCCESS
>http://dpdk.org/patch/17859

>_Compilation OK_

>The content about this compilation

It works now!
Please refer to http://dpdk.org/patch/17860

Thomas, thanks for your kindly help!
John, thanks for your help about pwclient!


Any comments please contacts me, thanks!

Best Regards
Fangfang Wei

-----Original Message-----
From: ci [mailto:ci-bounces@dpdk.org] On Behalf Of Wei, FangfangX
Sent: Monday, December 12, 2016 5:27 PM
To: Thomas Monjalon <thomas.monjalon@6wind.com>
Cc: ci@dpdk.org; Xu, Qian Q <qian.q.xu@intel.com>; Liu, Yong <yong.liu@intel.com>; Chen, WeichunX <weichunx.chen@intel.com>
Subject: Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch


2016-12-09 08:51, Wei, FangfangX:
>> My internet connection is working fine, and also configure http_proxy and https_proxy, but the connection is still timeout.
>> 
>> I think that's because the rpc client can't connect to your rpc server. Is there any other configuration should be configured?

>I've found it is an open issue. pwclient does not work with proxy:
>	https://github.com/getpatchwork/patchwork/issues/47

With John's patch, pwclient is working via intel firewall now.
Thanks, John!


>> According to my understanding, the "pwid=$($pwclient list -f '%{id}' -m "$msgid " in the script "send-patch-report.sh" is just trying to get the patchwork id, is that right?

>Yes, if you already have to patchwork id, you can put it.
>It makes me think that the patchwork id should be an option of this script.

I think adding patchwork id as an option of this script is a good idea.
Because in my per-patch build report, the patchwork id is known. There's no necessary to get it through patchwork with message id.


>> Then I just define pwid="17772" in "send-patch-report.sh" directly, run the script again, nothing happened. The email hasn't been sent to test-report@dpdk.org.

>Are you sure "sendmail -t" is working fine in your environment?

Actually, I'm not sure about it. In your environment, is there any configuration about sendmail in Linux?


>> I think the function about send-patch-report.sh is just to send patch report to test-report@dpdk.org. It can't generate the following content in patchwork as checkpatch. Am I right?

>Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed and integrated in patchwork by update-pw.sh running on the server.
>That's how the checkpatch test works (see tests/checkpatch.sh).

If I send email with my own script, the receiver is test-report@dpdk.org, and the report format shows as below. Is this email also automatically parsed and integrated in patchwork on the server?

Test-Label: Intel Per-patch compilation check
Test-Status: SUCCESS
http://dpdk.org/patch/17859

_Compilation OK_

The content about this compilation


Best Regards
Fangfang Wei

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-12 09:27, Wei, FangfangX:
> 
> 2016-12-09 08:51, Wei, FangfangX:
> >> My internet connection is working fine, and also configure http_proxy and https_proxy, but the connection is still timeout.
> >> 
> >> I think that's because the rpc client can't connect to your rpc server. Is there any other configuration should be configured?
> 
> >I've found it is an open issue. pwclient does not work with proxy:
> >	https://github.com/getpatchwork/patchwork/issues/47
> 
> With John's patch, pwclient is working via intel firewall now.
> Thanks, John!

I've submitted a generic proxy support to patchwork:
	https://patchwork.ozlabs.org/patch/704687/
	https://patchwork.ozlabs.org/patch/704688/

> >> According to my understanding, the "pwid=$($pwclient list -f '%{id}' -m "$msgid " in the script "send-patch-report.sh" is just trying to get the patchwork id, is that right?
> 
> >Yes, if you already have to patchwork id, you can put it.
> >It makes me think that the patchwork id should be an option of this script.
> 
> I think adding patchwork id as an option of this script is a good idea.
> Because in my per-patch build report, the patchwork id is known. There's no necessary to get it through patchwork with message id.

It will be in the next revision of this patchset.

> >> Then I just define pwid="17772" in "send-patch-report.sh" directly, run the script again, nothing happened. The email hasn't been sent to test-report@dpdk.org.
> 
> >Are you sure "sendmail -t" is working fine in your environment?
> 
> Actually, I'm not sure about it. In your environment, is there any configuration about sendmail in Linux?

Nothing special.

> >> I think the function about send-patch-report.sh is just to send patch report to test-report@dpdk.org. It can't generate the following content in patchwork as checkpatch. Am I right?
> 
> >Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed and integrated in patchwork by update-pw.sh running on the server.
> >That's how the checkpatch test works (see tests/checkpatch.sh).
> 
> If I send email with my own script, the receiver is test-report@dpdk.org, and the report format shows as below. Is this email also automatically parsed and integrated in patchwork on the server?
> 
> Test-Label: Intel Per-patch compilation check
> Test-Status: SUCCESS
> http://dpdk.org/patch/17859
> 
> _Compilation OK_
> 
> The content about this compilation

Yes it would work.
What are you using to send this email?
I'd like to understand why sendmail does not work for you.

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-12 09:34, Wei, FangfangX:
> It works now!
> Please refer to http://dpdk.org/patch/17860

Great!

About the title:
	[dpdk-test-report] |SUCCESS| pw17860 [PATCH, v2] vmxnet3: fix Rx deadlock
Please remove the patchwork id pw17860,
and remove the comma in [PATCH, v2].

> Test-Label: Intel Per-patch compilation check

The label could be shorter.
"Per-patch" is not needed as every patchwork checks are "per-patch".
"check" is also redundant.
I suggest "Intel compilation" as Test-Label.

> Test-Status: SUCCESS
> http://dpdk.org/patch/17859
> 
> _Compilation OK_

It would be better to have a blank line between the status description
and the full report.

Thanks

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

>> Actually, I'm not sure about it. In your environment, is there any configuration about sendmail in Linux?

>Nothing special.

>> >Yes it just an email to test-report@dpdk.org. Then this email is automatically parsed and integrated in patchwork by update-pw.sh running on the server.
>> >That's how the checkpatch test works (see tests/checkpatch.sh).
>> 
>> If I send email with my own script, the receiver is test-report@dpdk.org, and the report format shows as below. Is this email also automatically parsed and integrated in patchwork on the server?
>> 
>> Test-Label: Intel Per-patch compilation check
>> Test-Status: SUCCESS
>> http://dpdk.org/patch/17859
>> 
>> _Compilation OK_
>> 
>> The content about this compilation

>Yes it would work.
>What are you using to send this email?
>I'd like to understand why sendmail does not work for you.

I wrote a python script, which using smtplib module to send email:
smtp = smtplib.SMTP('Intel SMTP HOST')

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

>About the title:
>	[dpdk-test-report] |SUCCESS| pw17860 [PATCH, v2] vmxnet3: fix Rx deadlock Please remove the patchwork id pw17860, and remove the comma in [PATCH, v2].

I have remove the comma. 
For the patchwork id "pw17860", I think it would be better to be remained. Because if it's removed from the subject, receivers can't be directly told which patch has been complicated, they must open the email to check it. 

>The label could be shorter.
>"Per-patch" is not needed as every patchwork checks are "per-patch".
>"check" is also redundant.
>I suggest "Intel compilation" as Test-Label.

The label has been changed to "Intel compilation".

>It would be better to have a blank line between the status description and the full report.

A blank line has been added between the status description and the full report.


Best Regards
Fangfang Wei

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-13 08:29, Wei, FangfangX:
> 
> > About the title:
> >	[dpdk-test-report] |SUCCESS| pw17860 [PATCH, v2] vmxnet3: fix Rx deadlock
> >	Please remove the patchwork id pw17860, and remove the comma in [PATCH, v2].
> 
> I have remove the comma. 
> For the patchwork id "pw17860", I think it would be better to be remained. Because if it's removed from the subject, receivers can't be directly told which patch has been complicated, they must open the email to check it. 

I don't understand.
With the title "|SUCCESS| [PATCH v2] vmxnet3: fix Rx deadlock]",
the receiver see which patch was tested, isn't it?

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Wei, FangfangX]

>> > About the title:
>> >	[dpdk-test-report] |SUCCESS| pw17860 [PATCH, v2] vmxnet3: fix Rx deadlock
>> >	Please remove the patchwork id pw17860, and remove the comma in [PATCH, v2].
>> 
>> I have remove the comma. 
>> For the patchwork id "pw17860", I think it would be better to be remained. Because if it's removed from the subject, receivers can't be directly told which patch has been complicated, they must open the email to check it. 

>I don't understand.
>With the title "|SUCCESS| [PATCH v2] vmxnet3: fix Rx deadlock]", the receiver see which patch was tested, isn't it?

I mean if removed patchwork id, the receiver know one patch was tested, but they don't see the patch id unless open the email.
But we need the patchwork id for two reasons:
1. For debug, I want to see if the patches were tested in case of leakage tested.
2. We summary the tested patches status weekly. It is intuitive to see the patchwork id in subject.

Best Regards
Fangfang Wei

[dpdk-ci] [PATCH v5 0/9] first scripts for CI integration

[Thomas Monjalon]

These scripts allow to check a patch received by email and
send a report in order to be integrated in patchwork.

The existing CI tests run by Intel could be converted to use
the script send-patch-report.sh so they will be seen in patchwork.

Next steps (to be implemented):
- script to clean and update a git tree
- script to apply a patch on the right tree
- script to apply dependencies (preceding in a series)

---

changes in v5:
- update README
- add pwid option to send-report
- add pwid in report subject
- fix pwclient for proxy and python 3
- allow to download patch from patchwork

changes in v4:
- fortify mail parsing for binary patches and long emails

changes in v3:
- BSD licensing

changes in v2:
- fix mail parsing (bug with quotes in From:)
- fix public success report (no CC:)

---

Thomas Monjalon (9):
  tools: add mail filter
  tools: add mail parser
  config: add loader and template
  tools: add patchwork client
  tools: fix pwclient for proxy and python 3
  tools: add patch mail download
  tools: add per-patch report mailer
  tools: add patchwork integration
  tests: add checkpatch

 README                      |  27 ++
 config/ci.config            |  10 +
 config/pwclientrc           |   9 +
 tests/checkpatch.sh         |  75 ++++
 tools/download-patch.sh     |  67 ++++
 tools/filter-patch-email.sh | 111 ++++++
 tools/load-ci-config.sh     |  14 +
 tools/parse-email.sh        |  74 ++++
 tools/pwclient              | 819 ++++++++++++++++++++++++++++++++++++++++++++
 tools/send-patch-report.sh  | 131 +++++++
 tools/update-pw.sh          |  84 +++++
 11 files changed, 1421 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 config/pwclientrc
 create mode 100755 tests/checkpatch.sh
 create mode 100755 tools/download-patch.sh
 create mode 100755 tools/filter-patch-email.sh
 create mode 100644 tools/load-ci-config.sh
 create mode 100755 tools/parse-email.sh
 create mode 100755 tools/pwclient
 create mode 100755 tools/send-patch-report.sh
 create mode 100755 tools/update-pw.sh

-- 
2.7.0

[dpdk-ci] [PATCH v5 1/9] tools: add mail filter

[Thomas Monjalon]

This script acts as a pipe which blocks non-patch emails.
It can be used as a first filter before processing a patch.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README                      |   8 ++++
 tools/filter-patch-email.sh | 111 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 119 insertions(+)
 create mode 100755 tools/filter-patch-email.sh

diff --git a/README b/README
index b4bf001..806ce8f 100644
--- a/README
+++ b/README
@@ -44,3 +44,11 @@ The sender must be trusted (whitelisted) by dpdk.org.
 
 If a report has the format of a "per-patch" test,
 it is automatically integrated in patchwork.
+
+
+Scripts help
+------------
+
+Some scripts in this repository (dpdk-ci) may help to build a test.
+
+Their purpose and options are described when using the option -h.
diff --git a/tools/filter-patch-email.sh b/tools/filter-patch-email.sh
new file mode 100755
index 0000000..821786f
--- /dev/null
+++ b/tools/filter-patch-email.sh
@@ -0,0 +1,111 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) < email
+
+	Filter out email from stdin if does not match patch criterias.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage ; exit 1 ;;
+	esac
+done
+
+if [ -t 0 ] ; then
+	echo 'nothing to read on stdin' >&2
+	exit 0
+fi
+
+fifo=/tmp/$(basename $0 sh)$$
+mkfifo $fifo
+trap "rm -f $fifo" INT EXIT
+
+parse ()
+{
+	gitsend=false
+	patchsubject=false
+	content=false
+	linenum=0
+	minusline=false
+	plusline=false
+	atline=false
+	binary=false
+	done=false
+	while IFS= read -r line ; do
+		printf '%s\n' "$line"
+		set -- $line
+		if ! $content ; then
+			[ "$1" != 'X-Mailer:' -o "$2" != 'git-send-email' ] || gitsend=true
+			if echo "$line" | grep -qa '^Subject:.*\[PATCH' ; then
+				subject=$(echo "$line" | sed 's,^Subject:[[:space:]]*,,')
+				while [ -n "$subject" ] ; do
+					echo "$subject" | grep -q '^\[' || break
+					if echo "$subject" | grep -q '^\[PATCH' ; then
+						patchsubject=true
+						break
+					fi
+					subject=$(echo "$subject" | sed 's,^[^]]*\][[:space:]]*,,')
+				done
+			fi
+			[ -n "$line" ] || content=true
+		elif ! $done ; then
+			$gitsend || $patchsubject || break
+			[ "$1" != '---' ] || minusline=true
+			[ "$1" != '+++' ] || plusline=true
+			[ "$1" != '@@' ] || atline=true
+			[ "$1 $2 $3" != 'GIT binary patch' ] || binary=true
+			if ($minusline && $plusline && $atline) || $binary ; then
+				echo 1 >$fifo
+				done=true
+				cat
+				break
+			fi
+			linenum=$(($linenum + 1))
+			[ $linenum -lt 999 ] || break
+		fi
+	done
+	$done || echo 0 >$fifo
+	exec >&-
+}
+
+waitparsing ()
+{
+	result=$(cat $fifo)
+	[ "$result" = 0 ] || cat
+}
+
+parse | waitparsing
-- 
2.7.0

[dpdk-ci] [PATCH v5 2/9] tools: add mail parser

[Thomas Monjalon]

This script get some mail headers from an email file.
The retrieved headers can be used for test and report in other scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/parse-email.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100755 tools/parse-email.sh

diff --git a/tools/parse-email.sh b/tools/parse-email.sh
new file mode 100755
index 0000000..d92c246
--- /dev/null
+++ b/tools/parse-email.sh
@@ -0,0 +1,72 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <email_file>
+
+	Parse basic headers of the email
+	and print them as shell variable assignments to evaluate.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -z "$1" ] ; then
+	printf 'file argument is missing\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+getheader () # <header_name> <email_file>
+{
+	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2" |
+	sed 's,",\\",g'
+}
+
+subject=$(getheader Subject "$1")
+from=$(getheader From "$1")
+msgid=$(getheader Message-Id "$1")
+[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+listid=$(getheader List-Id "$1")
+
+cat <<- END_OF_HEADERS
+	subject="$subject"
+	from="$from"
+	msgid="$msgid"
+	listid="$listid"
+END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH v5 3/9] config: add loader and template

[Thomas Monjalon]

The configuration file will allow to set some environment-specific
options and paths to be used by the scripts.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README                  | 11 +++++++++++
 config/ci.config        |  4 ++++
 tools/load-ci-config.sh | 14 ++++++++++++++
 3 files changed, 29 insertions(+)
 create mode 100644 config/ci.config
 create mode 100644 tools/load-ci-config.sh

diff --git a/README b/README
index 806ce8f..12a208c 100644
--- a/README
+++ b/README
@@ -52,3 +52,14 @@ Scripts help
 Some scripts in this repository (dpdk-ci) may help to build a test.
 
 Their purpose and options are described when using the option -h.
+
+
+Scripts configuration
+---------------------
+
+Some scripts may need some configuration.
+
+The file config/ci.config will be read if it is copied to /etc/dpdk/ci.config,
+~/.config/dpdk/ci.config or .ciconfig in this directory.
+The optional options can be uncommented to change the default values used in
+the scripts.
diff --git a/config/ci.config b/config/ci.config
new file mode 100644
index 0000000..2aeff04
--- /dev/null
+++ b/config/ci.config
@@ -0,0 +1,4 @@
+# Configuration template to be copied in
+#         /etc/dpdk/ci.config
+#     or  ~/.config/dpdk/ci.config
+#     or  .ciconfig
diff --git a/tools/load-ci-config.sh b/tools/load-ci-config.sh
new file mode 100644
index 0000000..9ec18b4
--- /dev/null
+++ b/tools/load-ci-config.sh
@@ -0,0 +1,14 @@
+#! /bin/echo must be loaded with .
+
+# Load DPDK CI config and allow override
+# from system file
+test ! -r /etc/dpdk/ci.config ||
+        . /etc/dpdk/ci.config
+# from user file
+test ! -r ~/.config/dpdk/ci.config ||
+        . ~/.config/dpdk/ci.config
+# from local file
+test ! -r $(dirname $(readlink -m $0))/../.ciconfig ||
+        . $(dirname $(readlink -m $0))/../.ciconfig
+
+# The config files must export variables in the shell style
-- 
2.7.0

[dpdk-ci] [PATCH v5 4/9] tools: add patchwork client

[Thomas Monjalon]

The script pwclient is a frontend for patchwork.
It is imported from patchwork 1.1.
Another version can be used by setting DPDK_CI_PWCLIENT in config.

The config template pwclientrc must be copied in ~/.pwclientrc.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README            |   3 +
 config/ci.config  |   3 +
 config/pwclientrc |   9 +
 tools/pwclient    | 808 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 823 insertions(+)
 create mode 100644 config/pwclientrc
 create mode 100755 tools/pwclient

diff --git a/README b/README
index 12a208c..00e1415 100644
--- a/README
+++ b/README
@@ -63,3 +63,6 @@ The file config/ci.config will be read if it is copied to /etc/dpdk/ci.config,
 ~/.config/dpdk/ci.config or .ciconfig in this directory.
 The optional options can be uncommented to change the default values used in
 the scripts.
+
+The file pwclientrc must be copied in ~/.pwclientrc in order to access to
+the XML-RPC interface of patchwork with the script pwclient.
diff --git a/config/ci.config b/config/ci.config
index 2aeff04..722aeb5 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -2,3 +2,6 @@
 #         /etc/dpdk/ci.config
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
+
+# The pwclient script is part of patchwork and is copied in dpdk-ci
+# export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/config/pwclientrc b/config/pwclientrc
new file mode 100644
index 0000000..d73e87e
--- /dev/null
+++ b/config/pwclientrc
@@ -0,0 +1,9 @@
+[options]
+default=dpdk
+
+[dpdk]
+url=http://dpdk.org/dev/patchwork/xmlrpc/
+
+# credentials are useless for read-only access
+#username: myuser
+#password: mypass
diff --git a/tools/pwclient b/tools/pwclient
new file mode 100755
index 0000000..f437786
--- /dev/null
+++ b/tools/pwclient
@@ -0,0 +1,808 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Patchwork command line client
+# Copyright (C) 2008 Nate Case <ncase@xes-inc.com>
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+from __future__ import print_function
+
+import os
+import sys
+try:
+    import xmlrpclib
+except ImportError:
+    # Python 3 has merged/renamed things.
+    import xmlrpc.client as xmlrpclib
+import argparse
+import string
+import subprocess
+import base64
+try:
+    import ConfigParser
+except ImportError:
+    # Python 3 has renamed things.
+    import configparser as ConfigParser
+import shutil
+import re
+
+# Add a shim for Python 2's unicode() helper.
+try:
+    unicode
+except NameError:
+    # Python 3 does everything by unicode now.
+    unicode = str
+
+# Default Patchwork remote XML-RPC server URL
+# This script will check the PW_XMLRPC_URL environment variable
+# for the URL to access.  If that is unspecified, it will fallback to
+# the hardcoded default value specified here.
+DEFAULT_URL = "http://patchwork/xmlrpc/"
+CONFIG_FILE = os.path.expanduser('~/.pwclientrc')
+
+
+class Filter(object):
+
+    """Filter for selecting patches."""
+
+    def __init__(self):
+        # These fields refer to specific objects, so they are special
+        # because we have to resolve them to IDs before passing the
+        # filter to the server
+        self.state = ""
+        self.project = ""
+
+        # The dictionary that gets passed to via XML-RPC
+        self.d = {}
+
+    def add(self, field, value):
+        if field == 'state':
+            self.state = value
+        elif field == 'project':
+            self.project = value
+        else:
+            # OK to add directly
+            self.d[field] = value
+
+    def resolve_ids(self, rpc):
+        """Resolve State, Project, and Person IDs based on filter strings."""
+        if self.state != "":
+            id = state_id_by_name(rpc, self.state)
+            if id == 0:
+                sys.stderr.write("Note: No State found matching %s*, "
+                                 "ignoring filter\n" % self.state)
+            else:
+                self.d['state_id'] = id
+
+        if self.project is not None:
+            id = project_id_by_name(rpc, self.project)
+            if id == 0:
+                sys.stderr.write("Note: No Project found matching %s, "
+                                 "ignoring filter\n" % self.project)
+            else:
+                self.d['project_id'] = id
+
+    def __str__(self):
+        """Return human-readable description of the filter."""
+        return str(self.d)
+
+
+class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+
+    def __init__(self, username=None, password=None, use_https=False):
+        self.username = username
+        self.password = password
+        self.use_https = use_https
+        xmlrpclib.SafeTransport.__init__(self)
+
+    def authenticated(self):
+        return self.username is not None and self.password is not None
+
+    def send_host(self, connection, host):
+        xmlrpclib.Transport.send_host(self, connection, host)
+        if not self.authenticated():
+            return
+        credentials = '%s:%s' % (self.username, self.password)
+        auth = 'Basic ' + base64.encodestring(credentials).strip()
+        connection.putheader('Authorization', auth)
+
+    def make_connection(self, host):
+        if self.use_https:
+            fn = xmlrpclib.SafeTransport.make_connection
+        else:
+            fn = xmlrpclib.Transport.make_connection
+        return fn(self, host)
+
+
+def project_id_by_name(rpc, linkname):
+    """Given a project short name, look up the Project ID."""
+    if len(linkname) == 0:
+        return 0
+    projects = rpc.project_list(linkname, 0)
+    for project in projects:
+        if project['linkname'] == linkname:
+            return project['id']
+    return 0
+
+
+def state_id_by_name(rpc, name):
+    """Given a partial state name, look up the state ID."""
+    if len(name) == 0:
+        return 0
+    states = rpc.state_list(name, 0)
+    for state in states:
+        if state['name'].lower().startswith(name.lower()):
+            return state['id']
+    return 0
+
+
+def person_ids_by_name(rpc, name):
+    """Given a partial name or email address, return a list of the
+    person IDs that match."""
+    if len(name) == 0:
+        return []
+    people = rpc.person_list(name, 0)
+    return [x['id'] for x in people]
+
+
+def list_patches(patches, format_str=None):
+    """Dump a list of patches to stdout."""
+    if format_str:
+        format_field_re = re.compile("%{([a-z0-9_]+)}")
+
+        def patch_field(matchobj):
+            fieldname = matchobj.group(1)
+
+            if fieldname == "_msgid_":
+                # naive way to strip < and > from message-id
+                val = string.strip(str(patch["msgid"]), "<>")
+            else:
+                val = str(patch[fieldname])
+
+            return val
+
+        for patch in patches:
+            print(format_field_re.sub(patch_field, format_str))
+    else:
+        print("%-7s %-12s %s" % ("ID", "State", "Name"))
+        print("%-7s %-12s %s" % ("--", "-----", "----"))
+        for patch in patches:
+            print("%-7d %-12s %s" %
+                  (patch['id'], patch['state'], patch['name']))
+
+
+def action_list(rpc, filter, submitter_str, delegate_str, format_str=None):
+    filter.resolve_ids(rpc)
+
+    if submitter_str is not None:
+        ids = person_ids_by_name(rpc, submitter_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             submitter_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches submitted by %s <%s>:' %
+                      (unicode(person['name']).encode('utf-8'),
+                       unicode(person['email']).encode('utf-8')))
+                f = filter
+                f.add("submitter_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    if delegate_str is not None:
+        ids = person_ids_by_name(rpc, delegate_str)
+        if len(ids) == 0:
+            sys.stderr.write("Note: Nobody found matching *%s*\n" %
+                             delegate_str)
+        else:
+            for id in ids:
+                person = rpc.person_get(id)
+                print('Patches delegated to %s <%s>:' %
+                      (person['name'], person['email']))
+                f = filter
+                f.add("delegate_id", id)
+                patches = rpc.patch_list(f.d)
+                list_patches(patches, format_str)
+        return
+
+    patches = rpc.patch_list(filter.d)
+    list_patches(patches, format_str)
+
+
+def action_projects(rpc):
+    projects = rpc.project_list("", 0)
+    print("%-5s %-24s %s" % ("ID", "Name", "Description"))
+    print("%-5s %-24s %s" % ("--", "----", "-----------"))
+    for project in projects:
+        print("%-5d %-24s %s" % (project['id'],
+                                 project['linkname'],
+                                 project['name']))
+
+
+def action_check_list(rpc):
+    checks = rpc.check_list()
+    print("%-5s %-16s %-8s %s" % ("ID", "Context", "State", "Patch"))
+    print("%-5s %-16s %-8s %s" % ("--", "-------", "-----", "-----"))
+    for check in checks:
+        print("%-5s %-16s %-8s %s" % (check['id'],
+                                      check['context'],
+                                      check['state'],
+                                      check['patch']))
+
+
+def action_check_info(rpc, check_id):
+    check = rpc.check_get(check_id)
+    s = "Information for check id %d" % (check_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(check.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_check_create(rpc, patch_id, context, state, url, description):
+    try:
+        rpc.check_create(patch_id, context, state, url, description)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error creating check: %s\n" % f.faultString)
+
+
+def action_states(rpc):
+    states = rpc.state_list("", 0)
+    print("%-5s %s" % ("ID", "Name"))
+    print("%-5s %s" % ("--", "----"))
+    for state in states:
+        print("%-5d %s" % (state['id'], state['name']))
+
+
+def action_info(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = "Information for patch id %d" % (patch_id)
+    print(s)
+    print('-' * len(s))
+    for key, value in sorted(patch.items()):
+        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+
+
+def action_get(rpc, patch_id):
+    patch = rpc.patch_get(patch_id)
+    s = rpc.patch_get_mbox(patch_id)
+
+    if patch == {} or len(s) == 0:
+        sys.stderr.write("Unable to get patch %d\n" % patch_id)
+        sys.exit(1)
+
+    base_fname = fname = os.path.basename(patch['filename'])
+    i = 0
+    while os.path.exists(fname):
+        fname = "%s.%d" % (base_fname, i)
+        i += 1
+
+    try:
+        f = open(fname, "w")
+    except:
+        sys.stderr.write("Unable to open %s for writing\n" % fname)
+        sys.exit(1)
+
+    try:
+        f.write(unicode(s).encode("utf-8"))
+        f.close()
+        print('Saved patch to %s' % fname)
+    except:
+        sys.stderr.write("Failed to write to %s\n" % fname)
+        sys.exit(1)
+
+
+def action_apply(rpc, patch_id, apply_cmd=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    if apply_cmd is None:
+        print('Applying patch #%d to current directory' % patch_id)
+        apply_cmd = ['patch', '-p1']
+    else:
+        print('Applying patch #%d using %s' %
+              (patch_id, repr(' '.join(apply_cmd))))
+
+    print('Description: %s' % patch['name'])
+    s = rpc.patch_get_mbox(patch_id)
+    if len(s) > 0:
+        proc = subprocess.Popen(apply_cmd, stdin=subprocess.PIPE)
+        proc.communicate(unicode(s).encode('utf-8'))
+        return proc.returncode
+    else:
+        sys.stderr.write("Error: No patch content found\n")
+        sys.exit(1)
+
+
+def action_update_patch(rpc, patch_id, state=None, archived=None, commit=None):
+    patch = rpc.patch_get(patch_id)
+    if patch == {}:
+        sys.stderr.write("Error getting information on patch ID %d\n" %
+                         patch_id)
+        sys.exit(1)
+
+    params = {}
+
+    if state:
+        state_id = state_id_by_name(rpc, state)
+        if state_id == 0:
+            sys.stderr.write("Error: No State found matching %s*\n" % state)
+            sys.exit(1)
+        params['state'] = state_id
+
+    if commit:
+        params['commit_ref'] = commit
+
+    if archived:
+        params['archived'] = archived == 'yes'
+
+    success = False
+    try:
+        success = rpc.patch_set(patch_id, params)
+    except xmlrpclib.Fault as f:
+        sys.stderr.write("Error updating patch: %s\n" % f.faultString)
+
+    if not success:
+        sys.stderr.write("Patch not updated\n")
+
+
+def patch_id_from_hash(rpc, project, hash):
+    try:
+        patch = rpc.patch_get_by_project_hash(project, hash)
+    except xmlrpclib.Fault:
+        # the server may not have the newer patch_get_by_project_hash function,
+        # so fall back to hash-only.
+        patch = rpc.patch_get_by_hash(hash)
+
+    if patch == {}:
+        sys.stderr.write("No patch has the hash provided\n")
+        sys.exit(1)
+
+    patch_id = patch['id']
+    # be super paranoid
+    try:
+        patch_id = int(patch_id)
+    except:
+        sys.stderr.write("Invalid patch ID obtained from server\n")
+        sys.exit(1)
+    return patch_id
+
+auth_actions = ['check_create', 'update']
+
+
+def main():
+    hash_parser = argparse.ArgumentParser(add_help=False)
+    hash_parser.add_argument(
+        '-h', metavar='HASH', dest='hash', action='store',
+        help='''Lookup by patch hash'''
+    )
+    hash_parser.add_argument(
+        'id', metavar='ID', nargs='*', action='store', type=int,
+        help='Patch ID',
+    )
+    hash_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Lookup patch in project'''
+    )
+
+    filter_parser = argparse.ArgumentParser(add_help=False)
+    filter_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Filter by patch state (e.g., 'New', 'Accepted', etc.)'''
+    )
+    filter_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Filter by patch archived state'''
+    )
+    filter_parser.add_argument(
+        '-p', metavar='PROJECT',
+        help='''Filter by project name (see 'projects' for list)'''
+    )
+    filter_parser.add_argument(
+        '-w', metavar='WHO',
+        help='''Filter by submitter (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-d', metavar='WHO',
+        help='''Filter by delegate (name, e-mail substring search)'''
+    )
+    filter_parser.add_argument(
+        '-n', metavar='MAX#',
+        type=int,
+        help='''Return first n results'''
+    )
+    filter_parser.add_argument(
+        '-N', metavar='MAX#',
+        type=int,
+        help='''Return last N results'''
+    )
+    filter_parser.add_argument(
+        '-m', metavar='MESSAGEID',
+        help='''Filter by Message-Id'''
+    )
+    filter_parser.add_argument(
+        '-f', metavar='FORMAT',
+        help='''Print output in the given format. You can use tags matching '''
+        '''fields, e.g. %%{id}, %%{state}, or %%{msgid}.'''
+    )
+    filter_parser.add_argument(
+        'patch_name', metavar='STR', nargs='?',
+        help='substring to search for patches by name',
+    )
+
+    action_parser = argparse.ArgumentParser(
+        prog='pwclient',
+        epilog='Use \'pwclient <command> --help\' for more info',
+    )
+
+    subparsers = action_parser.add_subparsers(
+        title='Commands',
+    )
+    apply_parser = subparsers.add_parser(
+        'apply', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch (in the current dir, using -p1)'''
+    )
+    apply_parser.set_defaults(subcmd='apply')
+    git_am_parser = subparsers.add_parser(
+        'git-am', parents=[hash_parser], conflict_handler='resolve',
+        help='''Apply a patch to current git branch using "git am".'''
+    )
+    git_am_parser.set_defaults(subcmd='git_am')
+    git_am_parser.add_argument(
+        '-s', '--signoff',
+        action='store_true',
+        help='''pass --signoff to git-am'''
+    )
+    get_parser = subparsers.add_parser(
+        'get', parents=[hash_parser], conflict_handler='resolve',
+        help='''Download a patch and save it locally'''
+    )
+    get_parser.set_defaults(subcmd='get')
+    info_parser = subparsers.add_parser(
+        'info', parents=[hash_parser], conflict_handler='resolve',
+        help='''Display patchwork info about a given patch ID'''
+    )
+    info_parser.set_defaults(subcmd='info')
+    projects_parser = subparsers.add_parser(
+        'projects',
+        help='''List all projects'''
+    )
+    projects_parser.set_defaults(subcmd='projects')
+    check_list_parser = subparsers.add_parser(
+        'check-list',
+        add_help=False,
+        help='''List all checks'''
+    )
+    check_list_parser.set_defaults(subcmd='check_list')
+    check_info_parser = subparsers.add_parser(
+        'check-info',
+        add_help=False,
+        help='''Show information for a given check'''
+    )
+    check_info_parser.set_defaults(subcmd='check_info')
+    check_info_parser.add_argument(
+        'check_id', metavar='ID', action='store', type=int,
+        help='Check ID',)
+    check_create_parser = subparsers.add_parser(
+        'check-create', parents=[hash_parser], conflict_handler='resolve',
+        help='Add a check to a patch')
+    check_create_parser.set_defaults(subcmd='check_create')
+    check_create_parser.add_argument(
+        '-c', metavar='CONTEXT')
+    check_create_parser.add_argument(
+        '-s', choices=('pending', 'success', 'warning', 'fail'))
+    check_create_parser.add_argument(
+        '-u', metavar='TARGET_URL', default="")
+    check_create_parser.add_argument(
+        '-d', metavar='DESCRIPTION', default="")
+    states_parser = subparsers.add_parser(
+        'states',
+        help='''Show list of potential patch states'''
+    )
+    states_parser.set_defaults(subcmd='states')
+    view_parser = subparsers.add_parser(
+        'view', parents=[hash_parser], conflict_handler='resolve',
+        help='''View a patch'''
+    )
+    view_parser.set_defaults(subcmd='view')
+    update_parser = subparsers.add_parser(
+        'update', parents=[hash_parser], conflict_handler='resolve',
+        help='''Update patch''',
+        epilog='''Using a COMMIT-REF allows for only one ID to be specified''',
+    )
+    update_parser.add_argument(
+        '-c', metavar='COMMIT-REF',
+        help='''commit reference hash'''
+    )
+    update_parser.add_argument(
+        '-s', metavar='STATE',
+        help='''Set patch state (e.g., 'Accepted', 'Superseded' etc.)'''
+    )
+    update_parser.add_argument(
+        '-a', choices=['yes', 'no'],
+        help='''Set patch archived state'''
+    )
+    update_parser.set_defaults(subcmd='update')
+    list_parser = subparsers.add_parser("list",
+                                        # aliases=['search'],
+                                        parents=[filter_parser],
+                                        help='''List patches, using the optional filters specified
+        below and an optional substring to search for patches
+        by name'''
+                                        )
+    list_parser.set_defaults(subcmd='list')
+    search_parser = subparsers.add_parser("search",
+                                          parents=[filter_parser],
+                                          help='''Alias for "list"'''
+                                          )
+    # Poor man's argparse aliases:
+    # We register the "search" parser but effectively use "list" for the
+    # help-text.
+    search_parser.set_defaults(subcmd='list')
+    if len(sys.argv) < 2:
+        action_parser.print_help()
+        sys.exit(0)
+
+    args = action_parser.parse_args()
+    args = dict(vars(args))
+    action = args.get('subcmd')
+
+    if args.get('hash') and len(args.get('id')):
+        # mimic mutual exclusive group
+        locals()[action + '_parser'].error(
+            "[-h HASH] and [ID [ID ...]] are mutually exlusive")
+
+    # set defaults
+    filt = Filter()
+    commit_str = None
+    url = DEFAULT_URL
+
+    archived_str = args.get('a')
+    state_str = args.get('s')
+    project_str = args.get('p')
+    submitter_str = args.get('w')
+    delegate_str = args.get('d')
+    format_str = args.get('f')
+    hash_str = args.get('hash')
+    patch_ids = args.get('id')
+    msgid_str = args.get('m')
+    if args.get('c'):
+        # update multiple IDs with a single commit-hash does not make sense
+        if action == 'update' and patch_ids and len(patch_ids) > 1:
+            update_parser.error(
+                "Declining update with COMMIT-REF on multiple IDs")
+        commit_str = args.get('c')
+
+    if state_str is None and archived_str is None and action == 'update':
+        update_parser.error(
+            'Must specify one or more update options (-a or -s)')
+
+    if args.get('n') is not None:
+        try:
+            filt.add("max_count", args.get('n'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('n'))
+
+    if args.get('N') is not None:
+        try:
+            filt.add("max_count", 0 - args.get('N'))
+        except:
+            action_parser.error("Invalid maximum count '%s'" % args.get('N'))
+
+    do_signoff = args.get('signoff')
+
+    # grab settings from config files
+    config = ConfigParser.ConfigParser()
+    config.read([CONFIG_FILE])
+
+    if not config.has_section('options') and os.path.exists(CONFIG_FILE):
+        sys.stderr.write('~/.pwclientrc is in the old format. Migrating it...')
+
+        old_project = config.get('base', 'project')
+
+        new_config = ConfigParser.ConfigParser()
+        new_config.add_section('options')
+
+        new_config.set('options', 'default', old_project)
+        new_config.add_section(old_project)
+
+        new_config.set(old_project, 'url', config.get('base', 'url'))
+        if config.has_option('auth', 'username'):
+            new_config.set(
+                old_project, 'username', config.get('auth', 'username'))
+        if config.has_option('auth', 'password'):
+            new_config.set(
+                old_project, 'password', config.get('auth', 'password'))
+
+        old_config_file = CONFIG_FILE + '.orig'
+        shutil.copy2(CONFIG_FILE, old_config_file)
+
+        with open(CONFIG_FILE, 'wb') as fd:
+            new_config.write(fd)
+
+        sys.stderr.write(' Done.\n')
+        sys.stderr.write(
+            'Your old ~/.pwclientrc was saved to %s\n' % old_config_file)
+        sys.stderr.write(
+            'and was converted to the new format. You may want to\n')
+        sys.stderr.write('inspect it before continuing.\n')
+        sys.exit(1)
+
+    if not project_str:
+        try:
+            project_str = config.get('options', 'default')
+        except:
+            action_parser.error(
+                "No default project configured in ~/.pwclientrc")
+
+    if not config.has_section(project_str):
+        sys.stderr.write(
+            'No section for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not config.has_option(project_str, 'url'):
+        sys.stderr.write(
+            'No URL for project %s in ~/.pwclientrc\n' % project_str)
+        sys.exit(1)
+    if not do_signoff and config.has_option('options', 'signoff'):
+        do_signoff = config.getboolean('options', 'signoff')
+    if not do_signoff and config.has_option(project_str, 'signoff'):
+        do_signoff = config.getboolean(project_str, 'signoff')
+
+    url = config.get(project_str, 'url')
+
+    transport = None
+    if action in auth_actions:
+        if config.has_option(project_str, 'username') and \
+                config.has_option(project_str, 'password'):
+
+            use_https = url.startswith('https')
+
+            transport = BasicHTTPAuthTransport(
+                config.get(project_str, 'username'),
+                config.get(project_str, 'password'),
+                use_https)
+
+        else:
+            sys.stderr.write("The %s action requires authentication, but no "
+                             "username or password\nis configured\n" % action)
+            sys.exit(1)
+
+    if project_str:
+        filt.add("project", project_str)
+
+    if state_str:
+        filt.add("state", state_str)
+
+    if archived_str:
+        filt.add("archived", archived_str == 'yes')
+
+    if msgid_str:
+        filt.add("msgid", msgid_str)
+
+    try:
+        rpc = xmlrpclib.Server(url, transport=transport)
+    except:
+        sys.stderr.write("Unable to connect to %s\n" % url)
+        sys.exit(1)
+
+    # It should be safe to assume hash_str is not zero, but who knows..
+    if hash_str is not None:
+        patch_ids = [patch_id_from_hash(rpc, project_str, hash_str)]
+
+    # helper for non_empty() to print correct helptext
+    h = locals()[action + '_parser']
+
+    # Require either hash_str or IDs for
+    def non_empty(h, patch_ids):
+        """Error out if no patch IDs were specified"""
+        if patch_ids is None or len(patch_ids) < 1:
+            sys.stderr.write("Error: Missing Argument! Either [-h HASH] or "
+                             "[ID [ID ...]] are required\n")
+            if h:
+                h.print_help()
+            sys.exit(1)
+        return patch_ids
+
+    if action == 'list' or action == 'search':
+        if args.get('patch_name') is not None:
+            filt.add("name__icontains", args.get('patch_name'))
+        action_list(rpc, filt, submitter_str, delegate_str, format_str)
+
+    elif action.startswith('project'):
+        action_projects(rpc)
+
+
+    elif action.startswith('state'):
+        action_states(rpc)
+
+    elif action == 'view':
+        pager = os.environ.get('PAGER')
+        if pager:
+            pager = subprocess.Popen(
+                pager.split(), stdin=subprocess.PIPE
+            )
+        if pager:
+            i = list()
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    i.append(unicode(s).encode("utf-8"))
+            if len(i) > 0:
+                pager.communicate(input="\n".join(i))
+            pager.stdin.close()
+        else:
+            for patch_id in non_empty(h, patch_ids):
+                s = rpc.patch_get_mbox(patch_id)
+                if len(s) > 0:
+                    print(unicode(s).encode("utf-8"))
+
+    elif action == 'info':
+        for patch_id in non_empty(h, patch_ids):
+            action_info(rpc, patch_id)
+
+    elif action == 'get':
+        for patch_id in non_empty(h, patch_ids):
+            action_get(rpc, patch_id)
+
+    elif action == 'apply':
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id)
+            if ret:
+                sys.stderr.write("Apply failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'git_am':
+        cmd = ['git', 'am']
+        if do_signoff:
+            cmd.append('-s')
+        for patch_id in non_empty(h, patch_ids):
+            ret = action_apply(rpc, patch_id, cmd)
+            if ret:
+                sys.stderr.write("'git am' failed with exit status %d\n" % ret)
+                sys.exit(1)
+
+    elif action == 'update':
+        for patch_id in non_empty(h, patch_ids):
+            action_update_patch(rpc, patch_id, state=state_str,
+                                archived=archived_str, commit=commit_str
+                                )
+
+    elif action == 'check_list':
+        action_check_list(rpc)
+
+    elif action == 'check_info':
+        check_id = args['check_id']
+        action_check_info(rpc, check_id)
+
+    elif action == 'check_create':
+        for patch_id in non_empty(h, patch_ids):
+            action_check_create(
+                rpc, patch_id, args['c'], args['s'], args['u'], args['d'])
+
+    else:
+        sys.stderr.write("Unknown action '%s'\n" % action)
+        action_parser.print_help()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()
-- 
2.7.0

[dpdk-ci] [PATCH v5 5/9] tools: fix pwclient for proxy and python 3

[Thomas Monjalon]

Python 3 can be used.

The environment variables http_proxy and https_proxy can be used.

These fixes have been sent to the patchwork project.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README         |  1 +
 tools/pwclient | 77 +++++++++++++++++++++++++++++++++-------------------------
 2 files changed, 45 insertions(+), 33 deletions(-)

diff --git a/README b/README
index 00e1415..f662f6d 100644
--- a/README
+++ b/README
@@ -66,3 +66,4 @@ the scripts.
 
 The file pwclientrc must be copied in ~/.pwclientrc in order to access to
 the XML-RPC interface of patchwork with the script pwclient.
+A proxy can be specified in the environment variables http_proxy and https_proxy.
diff --git a/tools/pwclient b/tools/pwclient
index f437786..3d5be69 100755
--- a/tools/pwclient
+++ b/tools/pwclient
@@ -102,31 +102,47 @@ class Filter(object):
         return str(self.d)
 
 
-class BasicHTTPAuthTransport(xmlrpclib.SafeTransport):
+class Transport(xmlrpclib.SafeTransport):
 
-    def __init__(self, username=None, password=None, use_https=False):
-        self.username = username
-        self.password = password
-        self.use_https = use_https
+    def __init__(self, url):
         xmlrpclib.SafeTransport.__init__(self)
+        self.credentials = None
+        self.host = None
+        self.proxy = None
+        self.scheme = url.split('://', 1)[0]
+        self.https = url.startswith('https')
+        if self.https:
+            self.proxy = os.environ.get('https_proxy')
+        else:
+            self.proxy = os.environ.get('http_proxy')
+        if self.proxy:
+            self.https = self.proxy.startswith('https')
 
-    def authenticated(self):
-        return self.username is not None and self.password is not None
-
-    def send_host(self, connection, host):
-        xmlrpclib.Transport.send_host(self, connection, host)
-        if not self.authenticated():
-            return
-        credentials = '%s:%s' % (self.username, self.password)
-        auth = 'Basic ' + base64.encodestring(credentials).strip()
-        connection.putheader('Authorization', auth)
+    def set_credentials(self, username=None, password=None):
+        self.credentials = '%s:%s' % (username, password)
 
     def make_connection(self, host):
-        if self.use_https:
-            fn = xmlrpclib.SafeTransport.make_connection
+        self.host = host
+        if self.proxy:
+            host = self.proxy.split('://', 1)[-1]
+        if self.credentials:
+            host = '@'.join([self.credentials, host])
+        if self.https:
+            return xmlrpclib.SafeTransport.make_connection(self, host)
         else:
-            fn = xmlrpclib.Transport.make_connection
-        return fn(self, host)
+            return xmlrpclib.Transport.make_connection(self, host)
+
+    if sys.version_info[0] == 2:
+
+        def send_request(self, connection, handler, request_body):
+            handler = '%s://%s%s' % (self.scheme, self.host, handler)
+            xmlrpclib.Transport.send_request(self, connection, handler, request_body)
+
+    else: # Python 3
+
+        def send_request(self, host, handler, request_body, debug):
+            handler = '%s://%s%s' % (self.scheme, host, handler)
+            return xmlrpclib.Transport.send_request(self, host, handler, request_body, debug)
 
 
 def project_id_by_name(rpc, linkname):
@@ -253,7 +269,7 @@ def action_check_info(rpc, check_id):
     print(s)
     print('-' * len(s))
     for key, value in sorted(check.items()):
-        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+        print("- %- 14s: %s" % (key, unicode(value)))
 
 
 def action_check_create(rpc, patch_id, context, state, url, description):
@@ -277,7 +293,7 @@ def action_info(rpc, patch_id):
     print(s)
     print('-' * len(s))
     for key, value in sorted(patch.items()):
-        print("- %- 14s: %s" % (key, unicode(value).encode("utf-8")))
+        print("- %- 14s: %s" % (key, unicode(value)))
 
 
 def action_get(rpc, patch_id):
@@ -301,7 +317,7 @@ def action_get(rpc, patch_id):
         sys.exit(1)
 
     try:
-        f.write(unicode(s).encode("utf-8"))
+        f.write(unicode(s))
         f.close()
         print('Saved patch to %s' % fname)
     except:
@@ -670,18 +686,13 @@ def main():
 
     url = config.get(project_str, 'url')
 
-    transport = None
+    transport = Transport(url)
     if action in auth_actions:
         if config.has_option(project_str, 'username') and \
                 config.has_option(project_str, 'password'):
-
-            use_https = url.startswith('https')
-
-            transport = BasicHTTPAuthTransport(
+            transport.set_credentials(
                 config.get(project_str, 'username'),
-                config.get(project_str, 'password'),
-                use_https)
-
+                config.get(project_str, 'password'))
         else:
             sys.stderr.write("The %s action requires authentication, but no "
                              "username or password\nis configured\n" % action)
@@ -746,15 +757,15 @@ def main():
             for patch_id in non_empty(h, patch_ids):
                 s = rpc.patch_get_mbox(patch_id)
                 if len(s) > 0:
-                    i.append(unicode(s).encode("utf-8"))
+                    i.append(unicode(s))
             if len(i) > 0:
-                pager.communicate(input="\n".join(i))
+                pager.communicate(input="\n".join(i).encode("utf-8"))
             pager.stdin.close()
         else:
             for patch_id in non_empty(h, patch_ids):
                 s = rpc.patch_get_mbox(patch_id)
                 if len(s) > 0:
-                    print(unicode(s).encode("utf-8"))
+                    print(unicode(s))
 
     elif action == 'info':
         for patch_id in non_empty(h, patch_ids):
-- 
2.7.0

[dpdk-ci] [PATCH v5 6/9] tools: add patch mail download

[Thomas Monjalon]

This script downloads a patch from patchwork with some email headers.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tools/{parse-email.sh => download-patch.sh} | 43 +++++++++++++----------------
 tools/parse-email.sh                        |  2 ++
 2 files changed, 21 insertions(+), 24 deletions(-)
 copy tools/{parse-email.sh => download-patch.sh} (73%)

diff --git a/tools/parse-email.sh b/tools/download-patch.sh
similarity index 73%
copy from tools/parse-email.sh
copy to tools/download-patch.sh
index d92c246..86d08c2 100755
--- a/tools/parse-email.sh
+++ b/tools/download-patch.sh
@@ -32,41 +32,36 @@
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) <email_file>
+	usage: $(basename $0) [-g] <patchwork_id>
 
-	Parse basic headers of the email
-	and print them as shell variable assignments to evaluate.
+	Download a patch from patchwork through pwclient XML-RPC (default)
+	or curl HTTP GET (option -g).
 	END_OF_HELP
 }
 
-while getopts h arg ; do
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+http_get=false
+while getopts gh arg ; do
 	case $arg in
+		g ) http_get=true ;;
 		h ) print_usage ; exit 0 ;;
 		? ) print_usage >&2 ; exit 1 ;;
 	esac
 done
 shift $(($OPTIND - 1))
-if [ -z "$1" ] ; then
-	printf 'file argument is missing\n\n' >&2
+pwid=$1
+if [ -z "$pwid" ] ; then
+	printf 'missing argument\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 
-getheader () # <header_name> <email_file>
-{
-	sed "/^$1: */!d;s///;N;s,\n[[:space:]]\+, ,;s,\n.*,,;q" "$2" |
-	sed 's,",\\",g'
-}
-
-subject=$(getheader Subject "$1")
-from=$(getheader From "$1")
-msgid=$(getheader Message-Id "$1")
-[ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
-listid=$(getheader List-Id "$1")
-
-cat <<- END_OF_HEADERS
-	subject="$subject"
-	from="$from"
-	msgid="$msgid"
-	listid="$listid"
-END_OF_HEADERS
+if $http_get ; then
+	url="http://dpdk.org/dev/patchwork/patch/$pwid/mbox/"
+	curl -sf $url
+else
+	$pwclient view $pwid
+fi |
+sed '/^Subject:/{s/\(\[[^],]*\)/\1] [PATCH/;s/,/ /g}'
diff --git a/tools/parse-email.sh b/tools/parse-email.sh
index d92c246..d997f5e 100755
--- a/tools/parse-email.sh
+++ b/tools/parse-email.sh
@@ -62,11 +62,13 @@ subject=$(getheader Subject "$1")
 from=$(getheader From "$1")
 msgid=$(getheader Message-Id "$1")
 [ -n "$msgid" ] || msgid=$(getheader Message-ID "$1")
+pwid=$(getheader X-Patchwork-Id "$1")
 listid=$(getheader List-Id "$1")
 
 cat <<- END_OF_HEADERS
 	subject="$subject"
 	from="$from"
 	msgid="$msgid"
+	pwid="$pwid"
 	listid="$listid"
 END_OF_HEADERS
-- 
2.7.0

[dpdk-ci] [PATCH v5 7/9] tools: add per-patch report mailer

[Thomas Monjalon]

The report sent by this script will be parsed and integrated in patchwork
if the patch was public.
Otherwise it will just send the report to the patch submitter.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README                     |   1 +
 config/ci.config           |   3 ++
 tools/send-patch-report.sh | 131 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 135 insertions(+)
 create mode 100755 tools/send-patch-report.sh

diff --git a/README b/README
index f662f6d..b030175 100644
--- a/README
+++ b/README
@@ -44,6 +44,7 @@ The sender must be trusted (whitelisted) by dpdk.org.
 
 If a report has the format of a "per-patch" test,
 it is automatically integrated in patchwork.
+The script send-patch-report.sh should be used.
 
 
 Scripts help
diff --git a/config/ci.config b/config/ci.config
index 722aeb5..b3b0376 100644
--- a/config/ci.config
+++ b/config/ci.config
@@ -3,5 +3,8 @@
 #     or  ~/.config/dpdk/ci.config
 #     or  .ciconfig
 
+# The mailer (sendmail, mail, mailx) must support the option -t
+# export DPDK_CI_MAILER=/usr/sbin/sendmail
+
 # The pwclient script is part of patchwork and is copied in dpdk-ci
 # export DPDK_CI_PWCLIENT=tools/pwclient
diff --git a/tools/send-patch-report.sh b/tools/send-patch-report.sh
new file mode 100755
index 0000000..7f78316
--- /dev/null
+++ b/tools/send-patch-report.sh
@@ -0,0 +1,131 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) [options] < report
+
+	Send test report in a properly formatted email for patchwork integration.
+	The report is submitted to this script via stdin.
+
+	options:
+	        -t title    subject of the patch email
+	        -f from     sender of the patch email
+	        -m msgid    id of the patch email
+	        -p pwid     id of the patch in patchwork (retrieved from msgid otherwise)
+	        -o listid   origin of the patch
+	        -l label    title of the test
+	        -s status   one of these test results: SUCCESS, WARNING, FAILURE
+	        -d desc     few words to better describe the status
+	        -h          this help
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+sendmail=${DPDK_CI_MAILER:-/usr/sbin/sendmail}
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+unset title
+unset from
+unset msgid
+unset pwid
+unset listid
+unset label
+unset status
+unset desc
+while getopts d:f:hl:m:o:p:s:t: arg ; do
+	case $arg in
+		t ) title=$OPTARG ;;
+		f ) from=$OPTARG ;;
+		m ) msgid=$OPTARG ;;
+		p ) pwid=$OPTARG ;;
+		o ) listid=$OPTARG ;;
+		l ) label=$OPTARG ;;
+		s ) status=$OPTARG ;;
+		d ) desc=$OPTARG ;;
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+if [ -t 0 ] ; then
+	printf 'nothing to read on stdin\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+report=$(cat)
+
+writeheaders () # <subject> <ref> <to> [cc]
+{
+	echo "Subject: $1"
+	echo "In-Reply-To: $2"
+	echo "References: $2"
+	echo "To: $3"
+	[ -z "$4" ] || echo "Cc: $4"
+	echo
+}
+
+writeheadlines () # <label> <status> <description> [pwid]
+{
+	echo "Test-Label: $1"
+	echo "Test-Status: $2"
+	[ -z "$4" ] || echo "http://dpdk.org/patch/$4"
+	echo
+	echo "_${3}_"
+	echo
+}
+
+if echo "$listid" | grep -q 'dev.dpdk.org' ; then
+	# get patchwork id
+	if [ -z "$pwid" -a -n "$msgid" ] ; then
+		for try in $(seq 20) ; do
+			pwid=$($pwclient list -f '%{id}' -m "$msgid")
+			[ -n "$pwid" ] && break || sleep 7
+		done
+	fi
+	[ -n "$pwid" ] || pwid='?'
+	# send public report
+	subject=$(echo $title | sed 's,\[dpdk-dev\] ,,')
+	[ "$status" = 'SUCCESS' ] && cc='' || cc="$from"
+	(
+	writeheaders "|$status| pw$pwid $subject" "$msgid" 'test-report@dpdk.org' "$cc"
+	writeheadlines "$label" "$status" "$desc" "$pwid"
+	echo "$report"
+	) | $sendmail -t
+else
+	# send private report
+	(
+		writeheaders "Re: $title" "$msgid" "$from"
+		writeheadlines "$label" "$status" "$desc"
+		echo "$report"
+	) | $sendmail -t
+fi
-- 
2.7.0

[dpdk-ci] [PATCH v5 8/9] tools: add patchwork integration

[Thomas Monjalon]

This script is run with patchwork admin credentials.
It is typically installed on dpdk.org and run for each mail
received in the test-report mailing list.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 README             |  3 ++
 tools/update-pw.sh | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
 create mode 100755 tools/update-pw.sh

diff --git a/README b/README
index b030175..c5f3d09 100644
--- a/README
+++ b/README
@@ -46,6 +46,9 @@ If a report has the format of a "per-patch" test,
 it is automatically integrated in patchwork.
 The script send-patch-report.sh should be used.
 
+If a report has the same Test-Label and patchwork id as a prior report,
+then it replaces the old one. It allows to re-run a test.
+
 
 Scripts help
 ------------
diff --git a/tools/update-pw.sh b/tools/update-pw.sh
new file mode 100755
index 0000000..61b239b
--- /dev/null
+++ b/tools/update-pw.sh
@@ -0,0 +1,84 @@
+#! /bin/sh
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) <report_url>
+
+	Add or update a check in patchwork based on a test report.
+	The argument specifies only the last URL parts of the test-report
+	mailing list archives (month/id.html).
+	END_OF_HELP
+}
+
+. $(dirname $(readlink -e $0))/load-ci-config.sh
+pwclient=${DPDK_CI_PWCLIENT:-$(dirname $(readlink -m $0))/pwclient}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+if [ -z "$1" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+url="http://dpdk.org/ml/archives/test-report/$1"
+mmarker='<!--beginarticle-->'
+for try in $(seq 20) ; do
+	[ -z "$report" -o $try -ge 19 ] || continue # 2 last tries if got something
+	[ $try -le 1 ] || sleep 3 # no delay before first try
+	report=$(curl -sf $url | grep -m1 -A9 "$mmarker") || continue
+	echo "$report" | grep -q '^_.*_$' && break || continue
+done
+if [ -z "$report" ] ; then
+	echo "cannot download report at $url" >&2
+	exit 2
+fi
+
+pwid=$(echo "$report" | sed -rn 's,.*http://.*dpdk.org/.*patch/([0-9]+).*,\1,p')
+label=$(echo "$report" | sed -n 's,.*Test-Label: *,,p')
+status=$(echo "$report" | sed -n 's,.*Test-Status: *,,p')
+desc=$(echo "$report" | sed -n 's,^_\(.*\)_$,\1,p')
+case $status in
+	'SUCCESS') pwstatus='success' ;;
+	'WARNING') pwstatus='warning' ;;
+	'FAILURE') pwstatus='fail' ;;
+esac
+printf 'id = %s\nlabel = %s\nstatus = %s/%s %s\nurl = %s\n' \
+	"$pwid" "$label" "$status" "$pwstatus" "$desc" "$url"
+[ -n "$pwid" -a -n "$label" -a -n "$status" -a -n "$desc" ] || exit 3
+
+$pwclient check-create -c "$label" -s "$pwstatus" -d "$desc" -u "$url" $pwid
-- 
2.7.0

[dpdk-ci] [PATCH v5 9/9] tests: add checkpatch

[Thomas Monjalon]

This is the first test in this repository.
It runs on dpdk.org and use checkpatch.pl of Linux.

Note that the patch is not applied on a git tree for this basic test.

Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 tests/checkpatch.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
 create mode 100755 tests/checkpatch.sh

diff --git a/tests/checkpatch.sh b/tests/checkpatch.sh
new file mode 100755
index 0000000..6bbfb7b
--- /dev/null
+++ b/tests/checkpatch.sh
@@ -0,0 +1,75 @@
+#! /bin/sh -e
+
+# BSD LICENSE
+#
+# Copyright 2016 6WIND S.A.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of 6WIND S.A. nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+print_usage () {
+	cat <<- END_OF_HELP
+	usage: $(basename $0) dpdk_dir < email
+
+	Check email-formatted patch from stdin.
+	This test runs checkpatch.pl of Linux via a script in dpdk_dir.
+	END_OF_HELP
+}
+
+while getopts h arg ; do
+	case $arg in
+		h ) print_usage ; exit 0 ;;
+		? ) print_usage >&2 ; exit 1 ;;
+	esac
+done
+shift $(($OPTIND - 1))
+toolsdir=$(dirname $(readlink -m $0))/../tools
+dpdkdir=$1
+if [ -z "$dpdkdir" ] ; then
+	printf 'missing argument\n\n' >&2
+	print_usage >&2
+	exit 1
+fi
+
+email=/tmp/$(basename $0 sh)$$
+$toolsdir/filter-patch-email.sh >$email
+trap "rm -f $email" INT EXIT
+
+eval $($toolsdir/parse-email.sh $email)
+# normal exit if no valid patch in the email
+[ -n "$subject" -a -n "$from" ] || exit 0
+
+failed=false
+report=$($dpdkdir/scripts/checkpatches.sh -q $email) || failed=true
+report=$(echo "$report" | sed '1,/^###/d')
+
+label='checkpatch'
+$failed && status='WARNING' || status='SUCCESS'
+$failed && desc='coding style issues' || desc='coding style OK'
+
+echo "$report" | $toolsdir/send-patch-report.sh \
+	-t "$subject" -f "$from" -m "$msgid" -p "$pwid" -o "$listid" \
+	-l "$label" -s "$status" -d "$desc"
-- 
2.7.0

Re: [dpdk-ci] [PATCH v4 7/7] tests: add checkpatch

[Thomas Monjalon]

2016-12-13 09:24, Wei, FangfangX:
> >> > About the title:
> >> >	[dpdk-test-report] |SUCCESS| pw17860 [PATCH, v2] vmxnet3: fix Rx deadlock
> >> >	Please remove the patchwork id pw17860, and remove the comma in [PATCH, v2].
> >> 
> >> I have remove the comma. 
> >> For the patchwork id "pw17860", I think it would be better to be remained. Because if it's removed from the subject, receivers can't be directly told which patch has been complicated, they must open the email to check it. 
> 
> >I don't understand.
> >With the title "|SUCCESS| [PATCH v2] vmxnet3: fix Rx deadlock]", the receiver see which patch was tested, isn't it?
> 
> I mean if removed patchwork id, the receiver know one patch was tested, but they don't see the patch id unless open the email.
> But we need the patchwork id for two reasons:
> 1. For debug, I want to see if the patches were tested in case of leakage tested.
> 2. We summary the tested patches status weekly. It is intuitive to see the patchwork id in subject.

The script in dpdk-ci is doing the same now.
Thanks for your comments

Re: [dpdk-ci] [PATCH v5 0/9] first scripts for CI integration

[Thomas Monjalon]

2016-12-15 00:05, Thomas Monjalon:
> changes in v5:
> - update README
> - add pwid option to send-report
> - add pwid in report subject
> - fix pwclient for proxy and python 3
> - allow to download patch from patchwork
> 
> changes in v4:
> - fortify mail parsing for binary patches and long emails
> 
> changes in v3:
> - BSD licensing
> 
> changes in v2:
> - fix mail parsing (bug with quotes in From:)
> - fix public success report (no CC:)
> 
> ---
> 
> Thomas Monjalon (9):
>   tools: add mail filter
>   tools: add mail parser
>   config: add loader and template
>   tools: add patchwork client
>   tools: fix pwclient for proxy and python 3
>   tools: add patch mail download
>   tools: add per-patch report mailer
>   tools: add patchwork integration
>   tests: add checkpatch

Applied