From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id D47917CE3 for ; Wed, 16 Aug 2017 04:16:09 +0200 (CEST) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2017 19:16:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,380,1498546800"; d="scan'208";a="1163041596" Received: from unknown (HELO dpdk-fedora20.icx.intel.com) ([10.240.176.135]) by orsmga001.jf.intel.com with ESMTP; 15 Aug 2017 19:16:06 -0700 From: Fangfang Wei To: ci@dpdk.org Cc: Fangfang Wei Date: Wed, 16 Aug 2017 10:17:22 +0800 Message-Id: <1502849842-75021-1-git-send-email-fangfangx.wei@intel.com> X-Mailer: git-send-email 1.9.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-ci] [PATCH V1] tools: add patchset performance CI doc and scripts X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Aug 2017 02:16:10 -0000 Add patchset performance test CI reference document, and the scripts used in CI. It includes applying one patchset to repo, and sending performance reports to patchwork. Signed-off-by: Fangfang Wei --- tools/intel_apply_patchset.py | 408 +++++++++++++++++++++ tools/intel_send_performance_report.py | 353 ++++++++++++++++++ ...atchset_performance_test_CI_reference_guide.txt | 216 +++++++++++ 3 files changed, 977 insertions(+) create mode 100644 tools/intel_apply_patchset.py create mode 100644 tools/intel_send_performance_report.py create mode 100644 tools/per_patchset_performance_test_CI_reference_guide.txt diff --git a/tools/intel_apply_patchset.py b/tools/intel_apply_patchset.py new file mode 100644 index 0000000..40dcd46 --- /dev/null +++ b/tools/intel_apply_patchset.py @@ -0,0 +1,408 @@ +# BSD LICENSE +# +# Copyright(c) 2010-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# -*- coding: utf-8 -*- +import os +import sys +import time +import re +import shutil +import pexpect +import logging +import json +import argparse + +reload(sys) +sys.setdefaultencoding('utf-8') + +REPOES = ["dpdk", "dpdk-next-net", "dpdk-next-crypto", "dpdk-next-virtio", + "dpdk-next-eventdev"] + + +def GetPatchInfo(patch): + patchSet = int(patch[0]) + patchID = int(patch[1]) + patchName = patch[2] + submitterMail = patch[3] + submitTime = patch[4] + submitComment = patch[5] + subject = patch[6] + return (patchSet, patchID, patchName, submitterMail, submitTime, + submitComment, subject) + + +class PatchWorkValidation(object): + ''' + You can get the path of test results, set submit + ''' + + def __init__(self, workPath, beginId, endId): + self.share_folder = workPath + self.beginId = beginId + self.endId = endId + + self.curPatchworkPath = "" + self.curPatchPath = "" + self.curReportFolderName = "" + self.reportsPath = "{}/patch_performance_result".format(self.share_folder) + self.dpdkPath = "{}/dpdk".format(self.share_folder) + self.patchesPath = "{}/patches".format(self.share_folder) + self.patchListDict = {} + self.curMasterCID = "" + self.switchGitMasterBranch() + + def getCommitID(self): + # get commit ID + handle = os.popen("git log -1") + firstLog = handle.read() + handle.close() + commitIdPat = r'(?<=commit ).+?(?=Author:)' + commitID = re.findall(commitIdPat, firstLog, re.S) + + return "".join(commitID)[:-1] + + def updateBranch_GetCommitID(self, repo): + os.chdir(self.dpdkPath) + lockFile = "".join([self.dpdkPath, "/", ".git", "/", "index.lock"]) + if os.path.exists(lockFile): + os.remove(lockFile) + os.system("git checkout -f {}".format(repo)) + os.system("git clean -fd") + os.system("git checkout -B {} {}/master".format(repo, repo)) + os.system("git pull") + commitID = self.getCommitID() + logging.info("commitID = {}".format(commitID)) + + return "".join(commitID) + + def makeTempBranch(self, repo): + os.chdir(self.dpdkPath) + os.system("git clean -fd") + os.system("git checkout -f {}".format(repo)) + os.system("git branch -D temp") + os.system("git checkout -B temp") + + def switchGitMasterBranch(self): + logging.info("switchGitMasterBranch......") + os.chdir(self.dpdkPath) + pattern = "^\* master" + handle = os.popen("git branch") + out = handle.read() + handle.close() + logging.info(out) + matchGr = re.findall(pattern, out, re.MULTILINE) + if len(matchGr) == 0: + handle = os.popen("git checkout -f master") + handle.close() + if self.curMasterCID != "": + os.system("git reset --hard {}".format(self.curMasterCID)) + else: + logging.info(matchGr) + + def logStatus(self): + handle = os.popen("date") + out = handle.read() + handle.close() + logPath = self.reportsPath + os.sep + "runningLog.log" + if os.path.exists(logPath): + fp = open(logPath, 'r') + out = "".join(["".join(fp.readlines()), out]) + fp.close() + + fp = open(logPath, 'w') + fp.writelines(out) + fp.close() + + def setCurSubmitInfo(self, submitInfo): + pattern = "<(.*)>" + if re.findall(pattern, submitInfo["submitter"]): + submitInfo["mail"] = "".join(re.findall(pattern, + submitInfo["submitter"])) + else: + submitInfo["mail"] = submitInfo["submitter"] + + self.submitInfo = submitInfo + + def makePatchworkReportsFolder(self, name, baseCommitID): + logging.info("make reports history folder......") + + self.curReportFolderName = "patch_{}_{}".format(name, baseCommitID) + self.curPatchworkPath = "".join([self.reportsPath, "/", self.curReportFolderName]) + if os.path.exists(self.curPatchworkPath) == False: + os.makedirs(self.curPatchworkPath) + os.makedirs(self.curPatchworkPath + os.sep + "performance_results") + + self.curPatchPath = self.curPatchworkPath + os.sep + "patches" + if os.path.exists(self.curPatchPath) == False: + os.makedirs(self.curPatchPath) + + def recordSubmitInfo(self, submitInfo): + submitInfoFile = self.curPatchworkPath + os.sep + "submitInfo.txt" + print submitInfoFile + if os.path.exists(submitInfoFile): + os.remove(submitInfoFile) + + content = "" + for item, value in submitInfo.iteritems(): + if value is None or len(value) == 0: + continue + + valueStr = "" + if isinstance(value, tuple) or isinstance(value, list): + for subItem in value: + valueStr += "{}".format(subItem) + else: + valueStr = value + content += "".join(["{}::".format(item), + "{}".format(valueStr), os.linesep]) + + with open(submitInfoFile, "wb") as submitInfo: + submitInfo.write(content) + + def recordTarMatchIdInfo(self, matchInfo): + matchInfoFile = self.curPatchworkPath + os.sep + "TarMatchIdInfo.txt" + if os.path.exists(matchInfoFile): + os.remove(matchInfoFile) + json.dump(matchInfo, open(matchInfoFile, 'w')) + + def makePatchErrorInfo(self, malformedPatch): + patchErrFile = "".join([self.curPatchworkPath, os.sep, + "patches", os.sep, + "patchError.txt"]) + if os.path.exists(patchErrFile): + os.remove(patchErrFile) + json.dump(malformedPatch, open(patchErrFile, 'w')) + os.system("cp {} {}/applyPatch.log".format(patchErrFile, self.share_folder)) + + def apply_patch_in_repo(self, patchset, patchsetInfo, repo, CommitID): + """ + apply patch into repo, if patch success, push to git server and + generate dpdk tar files, or generate dictionary malPatchPerRepo + which may contain the malformedpatch information. + """ + logging.info("Start patch file........") + malPatchPerRepo = {} + patchNo = 0 + matchInfo = {} + + first_patch = patchsetInfo[sorted(patchsetInfo.keys())[0]] + (first_patchSet, first_patchID, first_patchName, + first_submitterMail, first_submitTime, + first_submitComment, first_subject) = GetPatchInfo(first_patch) + + for item in sorted(patchsetInfo.keys()): + patchNo += 1 + patch = patchsetInfo[item] + (patchSet, patchID, patchName, submitterMail, submitTime, + submitComment, subject) = GetPatchInfo(patch) + + if patchNo == 1: + matchInfo[patchNo] = "Patch%s-%s" % (patchID, patchID) + else: + matchInfo[patchNo] = "Patch%s-%s" % (first_patchID, patchID) + + # Get the abosulte path of patchfile. + patchSetPath = self.patchesPath + os.sep + patchset + patchFile = patchSetPath + os.sep + patchName + + os.chdir(self.dpdkPath) + cmd = "patch -d {} -p1 < {}".format(self.dpdkPath, + patchFile) + logging.info(cmd) + outStream = os.popen(cmd) + patchingLog = outStream.read() + logging.info(patchingLog) + outStream.close() + malformedError = "malformed patch" + hunkFailError = "FAILED at" + missFileError = "can't find file to patch at input" + patchResult = "success" + if malformedError in patchingLog: + patchResult = "malformed patch" + elif hunkFailError in patchingLog: + patchResult = "hunk failed" + elif missFileError in patchingLog: + patchResult = "miss file" + + if patchResult != "success": + logging.info("apply patch error!") + malPatchPerRepo[patchID] = patchingLog + + # push to git server + os.chdir(self.dpdkPath) + os.system("git add --all") + os.system("git commit -m \" test \"") + submitCommentExd = submitComment.replace("\"", "") + # os.system("git commit --amend --author=\'" + submitterMail + + # "\' --date=\'" + submitTime + "\' -m \"" + submitCommentExd + "\"") + commit_cmd = "git commit --amend --author=\'{}\' --date=\'{}\' \ + -m \"{}\"".format(submitterMail, submitTime, + submitCommentExd) + os.system(commit_cmd) + + os.chdir(self.dpdkPath + "/../") + tar_cmd = "tar zcvf {}/dpdk.tar.gz dpdk".format(share_folder) + os.popen(tar_cmd) + shutil.copy2("{}/dpdk.tar.gz".format(share_folder), self.curPatchPath) + + self.recordTarMatchIdInfo(matchInfo) + + # Copy patch file into curPatchPath to backup. + shutil.copy2(patchFile, self.curPatchPath) + + return malPatchPerRepo + + def run_patchwork_validation(self, patchset, patchsetInfo, RepoCommitID, + baseCommitID): + malformedPatch = dict() + + # Write patchsetID and commitID into share_folder/info.txt. + # This file include the patch about where to store test results. + dpdk_info = file("{}/info.txt".format(share_folder), "w+") + dpdk_info.write(patchset + "_" + baseCommitID) + dpdk_info.close() + + # Remove dpdk tar file which may be left after last build. + os.system("rm -rf {}/dpdk.tar.gz".format(share_folder)) + + # Try to apply patch files into repoes, if patch success then break, + # else continue to patch files to next repo + repo = "dpdk" + for item, patch in patchsetInfo.iteritems(): + patchSet, patchID, patchName, submitterMail, submitTime, \ + submitComment, subject = GetPatchInfo(patch) + if re.search("net/", subject): + repo = "dpdk-next-net" + elif (re.search("crypto/", subject) or + re.search("cryptodev:", subject)): + repo = "dpdk-next-crypto" + elif re.search("event/", subject): + repo = "dpdk-next-eventdev" + elif re.search("vfio:", subject) or re.search("vfio/", subject): + repo = "dpdk-next-virtio" + + self.makeTempBranch(repo) + CommitID = RepoCommitID[repo] + malPatchPerRepo = self.apply_patch_in_repo(patchset, patchsetInfo, + repo, CommitID) + logging.info("malPatchPerRepo is {}".format(malPatchPerRepo)) + + if len(malPatchPerRepo) > 0: + malformedPatch[repo] = malPatchPerRepo + self.makePatchErrorInfo(malformedPatch) + for item, patch in patchsetInfo.iteritems(): + (patchSet, patchID, patchName, submitterMail, submitTime, + submitComment, subject) = GetPatchInfo(patch) + patchInfo = dict() + patchInfo["subject"] = re.sub("\[dpdk-dev.*\]", "", subject) + patchInfo["submitter"] = submitterMail + patchInfo["date"] = submitTime + patchInfo["patchworkId"] = patchset + patchInfo["baseline"] = "Repo:{}, Branch:master, \ + CommitID:{}".format(repo, CommitID) + self.setCurSubmitInfo(patchInfo) + self.recordSubmitInfo(patchInfo) + break + + def execute(self): + patchsetId = "".join([self.beginId, '-', self.endId]) + # Remove applyPatch.log which may be remained from last build. + os.system("rm -rf {}/applyPatch.log".format(self.share_folder)) + + if os.path.exists("{}/{}".format(self.patchesPath, patchsetId)) is False: + patchsets = os.listdir(self.patchesPath) + patchsetId = "" + if patchsets: + for patchset in patchsets: + if patchset.startswith(str(self.beginId)): + patchsetId = patchset + + if patchsetId == "": + print "patchset {}-{} not exist".format(self.beginId, self.endId) + with open("{}/applyPatch.log".format(self.share_folder), "wb") as ap: + ap.write("no patch to test!") + + else: + self.logStatus() + logging.info("Need to be tested patchset is {}".format(patchsetId)) + + # Get patchsetInfo which includes patchfilename, pasetfileId, + # Author, Time, Comments and etc. + file_path = "{}/{}/patchsetInfo.txt".format(self.patchesPath, patchsetId) + patchsetFile = open(file_path) + patchsetInfo = patchsetFile.read() + patchsetFile.close() + patchsetInfo = eval(patchsetInfo) + if patchsetInfo is not None and len(patchsetInfo) > 0: + RepoCommitID = {repo: self.updateBranch_GetCommitID( + repo) for repo in REPOES} + baseCommitID = RepoCommitID["dpdk"] + self.makePatchworkReportsFolder(patchsetId, baseCommitID) + # Apply patchset with per patch into repoes + self.run_patchwork_validation(patchsetId, patchsetInfo, + RepoCommitID, baseCommitID) + + # backup patchset files and patchset information + cp_cmd = "cp {} {}/patch_{}_{}".format(file_path, self.reportsPath, + patchsetId, baseCommitID) + os.system(cp_cmd) + patcheset_path = "{}/{}".format(self.patchesPath, patchsetId) + for parent, dirnames, filenames in os.walk(patcheset_path): + for dirname in dirnames: + dir_path = os.path.join(parent, dirname) + os.system("rm -rf {}".format(dir_path)) + mv_cmd = "mv {} {}/PatchSets".format(patcheset_path, self.share_folder) + os.system(mv_cmd) + + +if __name__ == "__main__": + """ + argument: + -p: the full path about share folder + -b: the first patch id in one patch set + -e: the last patch id in one patch set + usage: + python apply_patchset.py -p the/full/path/of/share/folder -b xxx -e xxx + """ + + parser = argparse.ArgumentParser(description='apply patch set to correct dpdk repo') + parser.add_argument('-p', '--path', help='the absolute path of share folder') + parser.add_argument('-b', '--begin-id', default=0, type=str, help='the begin id of the patch set') + parser.add_argument('-e', '--end-id', default=0, type=str, help='the end id of the patch set') + + args = parser.parse_args() + + InputBeginID = args.begin_id + InputEndID = args.end_id + share_folder = args.path + + autoValidation = PatchWorkValidation(share_folder, InputBeginID, InputBeginID) + autoValidation.execute() diff --git a/tools/intel_send_performance_report.py b/tools/intel_send_performance_report.py new file mode 100644 index 0000000..9c4d691 --- /dev/null +++ b/tools/intel_send_performance_report.py @@ -0,0 +1,353 @@ +# BSD LICENSE +# +# Copyright(c) 2010-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# -*- coding:utf-8 -*- +import os +import re +import sys +import datetime +import exceptions +import argparse +import smtplib +import email +from email.mime.text import MIMEText +from email.mime.multipart import MIMEMultipart +import json +import xlrd + +reload(sys) +sys.setdefaultencoding('utf8') + + +class PatchworkReport(object): + def __init__(self, report_folder): + self.report_folder = report_folder + self.patchInfoDict = dict() + self.testStatus = "SUCCESS" + + def getPatchSetInfo(self): + InfoFile = "".join([self.report_folder, "/patchsetInfo.txt"]) + with open(InfoFile, "rb") as pi: + patchInfo = pi.read() + self.patchSetInfoDict = eval(patchInfo) + return self.patchSetInfoDict + + def fetch_keys_values(self, content): + retDict = dict() + if content == "": + return retDict + patKey = "(.*)::(.*)" + patValue = "" + + result = re.findall(patKey, content, re.M) + for item in result: + key = item[0] + value = item[1].split(patValue)[1:] + if len(value) > 1: + retDict[key] = value + else: + retDict[key] = "".join(value) + return retDict + + def getPatchInfo(self): + submitInfoFile = "".join([self.report_folder, "/submitInfo.txt"]) + if os.path.exists(submitInfoFile): + with open(submitInfoFile, "rb") as info: + content = info.read() + infoDict = self.fetch_keys_values(content) + for item in infoDict.keys(): + self.patchInfoDict[item] = infoDict[item] + + def make_patchinfo_content(self): + content = "" + self.getPatchInfo() + content += "Submitter: {}\n".format(self.patchInfoDict["submitter"]) + content += "Date: {}\n".format(self.patchInfoDict["date"]) + content += "DPDK git baseline: {}\n".format(self.patchInfoDict["baseline"].replace(";", "\n ")) + content += os.linesep + return content + + def get_test_info(self, platform_name): + ''' + platform_name is named as "OS-NIC" + ''' + m = platform_name.split("_") + os = m[0] + nic = "_".join(m[1:]) + return os, nic + + def get_system_info(self, folder): + '''Kernel-GCC''' + f = None + kernel = 'NA' + gcc = 'NA' + system_filename = 'system.conf' + vm_system_filename = 'virtual_system.conf' + if os.path.exists(folder + os.sep + system_filename): + f = open(folder + os.sep + system_filename) + else: + f = None + + if f is not None: + line = f.readline() + s = line.split(',') + pattern = re.compile(r':.*') + kernel = '' + gcc = '' + k = pattern.search(s[0]) + if k: + kernel = k.group()[1:-7] + g = pattern.search(s[2]) + if g: + gcc = g.group()[1:-4] + gcc = gcc.split(" ")[2] + + return kernel, gcc + + def get_sheet(self, result_filename): + if os.path.exists(result_filename): + d = xlrd.open_workbook(result_filename) + sheet = d.sheets()[0] + return sheet + + def get_failed_cases_info(self, sheet): + column = len(sheet.col_values(1)) + m = 1 + total_number = 0 + failed_number = 0 + failed_case_list = [] + target = sheet.cell(1, 1).value + if sheet.cell(1, 5).value is not None and sheet.cell(1, 5).value != '': + failed_total_number = 'N/A: {}'.format(sheet.cell(1, 5).value) + else: + while m < column: + if sheet.cell(m, 4).value is not None and sheet.cell(m, 4).value != '': + total_number += 1 + if (sheet.cell(m, 5).value.startswith('FAILED') + or sheet.cell(m, 5).value.startswith('BLOCKED')): + failed_number += 1 + failed_case_list.append(sheet.cell(m, 4).value) + m += 1 + failed_total_number = str(failed_number)+'/'+str(total_number) + + return failed_total_number, failed_case_list, failed_number, target + + def get_all_info(self, resultPath, platform_name): + all_info = [] + fail_number = 0 + + test_info = self.get_test_info(platform_name) + os_info = test_info[0] + nic_info = test_info[1].lower() + + result_filename = '{}_single_core_perf.txt'.format(nic_info) + + folder = resultPath + os.sep + platform_name + if os.path.exists(folder + os.sep + "test_results.xls"): + system_info = self.get_system_info(folder) + kernel_info = system_info[0] + gcc_info = system_info[1] + + row = [] + perf_content = "" + row.append(os_info) + row.append(nic_info) + row.append(kernel_info) + row.append(gcc_info) + + sheet = self.get_sheet(folder + os.sep + "test_results.xls") + failed_cases_info = self.get_failed_cases_info(sheet) + failed_total_number = failed_cases_info[0] + failed_cases_list = failed_cases_info[1] + failed_number = failed_cases_info[2] + target = failed_cases_info[3] + + if failed_number != 0: + fail_number += 1 + + if os.path.exists(folder + os.sep + result_filename): + with open(folder + os.sep + result_filename) as perf_file: + for line in perf_file.readlines(): + perf_content += line + if 'N/A' in failed_total_number: + perf_content += failed_total_number.split(':')[-1] + fail_number = 1 + row.append(target) + row.append(perf_content) + row.append(failed_total_number) + row.append(failed_cases_list) + all_info.append(row) + return all_info, fail_number + + def execute_local(self, cmd): + print cmd + outStream = os.popen(cmd) + out = outStream.read() + outStream.close() + print out + return out + + def get_report_content(self): + report_content = [] + resultPath = "{}/regression_results".format(self.report_folder) + filenames = [] + flag = 0 + filenames = os.listdir(resultPath) + for filename in filenames: + if os.path.isdir("".join([resultPath, os.sep, filename])): + all_info = self.get_all_info(resultPath, filename) + report_content.extend(all_info[0]) + flag += all_info[1] + return report_content, flag + + def content_combine(self): + content = '' + report_content, flag = self.get_report_content() + for i in report_content: + content += '\n' + i[0] + '\n' + content += "Kernel: {}\n".format(i[2]) + content += "GCC: {}\n".format(i[3]) + content += "NIC: {}\n".format(i[1]) + content += "Target: {}\n".format(i[4]) + if 'N/A' in i[6]: + content += "Fail/Total: N/A\n" + else: + content += "Fail/Total: {}\n".format(i[6]) + if len(i[7]): + content += "Failed cases list:\n" + for case in i[7]: + content += " - DTS {}\n".format(case) + content += os.linesep + content += "Detail performance results: \n{}\n".format(i[5]) + return content, flag + + def send_email(self, receivers, cc_list, subject, content): + msg = MIMEMultipart('alternative') + msg['Subject'] = subject + msg['From'] = "the sender's email address" + msg['To'] = ", ".join(receivers) + msg['CC'] = ", ".join(cc_list) + part2 = MIMEText(content, "plain", "utf-8") + msg.attach(part2) + smtp = smtplib.SMTP('the smtp server') + smtp.sendmail("the sender's email", receivers, msg.as_string()) + smtp.quit() + + def send_report(self, status): + self.getPatchInfo() + TarMatchfile = "".join([self.report_folder, "/TarMatchIdInfo.txt"]) + matchDict = json.load(open(TarMatchfile, "r")) + + sortedKey = sorted(int(key) for key in matchDict.keys()) + whole_patchsetID = matchDict[str(sortedKey[-1])] + first_patchID = whole_patchsetID.split("-")[0] + + self.patchSetInfoDict = self.getPatchSetInfo() + + if status: + performance_content, flag = self.content_combine() + patchset_content = self.make_patchinfo_content() + self.testStatus = "FAILURE" + if flag == 0: + self.testStatus = "SUCCESS" + else: + self.testStatus = "FAILURE" + patchErrorfile = "".join([self.report_folder, + "/patches/patchError.txt"]) + patch_error_dict = json.load(open(patchErrorfile, "r")) + + content = "" + content += "Test-Label: Performance-Testing" + os.linesep + content += "Test-Status: {}\n".format(self.testStatus) + content += "http://dpdk.org/patch/{}\n\n".format(first_patchID) + if status: + if self.testStatus == "SUCCESS": + content += "_Performance Testing PASS_\n\n" + content += patchset_content + content += "{} --> performance testing pass\n".format(whole_patchsetID) + else: + content += "_Performance Testing issues_\n\n" + content += patchset_content + content += "{} --> performance testing fail\n".format(whole_patchsetID) + content += os.linesep + content += "Test environment and result as below:\n" + content += performance_content + else: + content += "_apply patch file failure_\n\n" + content += "Submitter: {}\n".format(self.patchInfoDict["submitter"]) + content += "Date: {}\n".format(self.patchInfoDict["date"]) + content += "DPDK git baseline: {}".format(self.patchInfoDict["baseline"].replace(";","\n ")) + content += os.linesep + content += "Apply patch set {} failed:\n".format(whole_patchsetID) + for repo, error_log in patch_error_dict.iteritems(): + for error_key, error_content in error_log.iteritems(): + content += "Repo: {}\n".format(repo) + content += "{}:\n".format(error_key) + content += error_log[error_key] + content += os.linesep + + content += os.linesep + "DPDK STV team" + os.linesep + print "content is {}".format(content) + + subject = "|{}| {} DPDK PatchSet Performance Test Report".format(self.testStatus, whole_patchsetID) + rece_maillist = ["receiver's email address"] + if status == "SUCCESS": + cc_list = [] + else: + cc_list = [self.patchInfoDict["submitter"]] + + self.send_email(rece_maillist, cc_list, subject, content) + + +if __name__ == "__main__": + """ + Send report to patchwork. + If patchError.txt exists, it maybe apply patch set failed or no patch + to tests. + If patchError.txt doesn't exists, it apply patch set successfully. + Arguments: + -p/--dst-path: the full path of share folder which store test result + """ + parser = argparse.ArgumentParser(description='Send report to patchwork') + parser.add_argument('-p', '--path', type=str, help='the absolute path of result folder') + args = parser.parse_args() + reporter = PatchworkReport(args.path) + + flag_file = "{}/patches/patchError.txt".format(args.path) + if os.path.exists(flag_file): + with open(flag_file, 'rb') as ff: + content = ff.read() + if "no patch" in content: + pass + else: + reporter.send_report(False) + else: + reporter.send_report(True) diff --git a/tools/per_patchset_performance_test_CI_reference_guide.txt b/tools/per_patchset_performance_test_CI_reference_guide.txt new file mode 100644 index 0000000..9dde040 --- /dev/null +++ b/tools/per_patchset_performance_test_CI_reference_guide.txt @@ -0,0 +1,216 @@ +# BSD LICENSE +# +# Copyright(c) 2010-2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +This document is to present the continuous integration deployment practice +from Intel on the per patch set performance testing. + +The purpose of patch set performance testing is to test each patch set's +performance, check if there is regression and integrate the results into +patchwork. + +This document includes following parts: +1. Prerequisites +2. Work Flow +3. Create Jenkins jobs +4. Intel CI scripts + + +Prerequisites +============= + +You need one machine which installs Jenkins; +One machine which can get source code, store scripts and store test results; +One or more DUT to run performance testing. +The machine which will get source code, it will be generate a dpdk.tar.gz. +If DUT want to do testing, they must get the dpdk.tar.gz file. +You can use the share strategy to share this dpdk.tar.gz to all DUT. You can +also use ssh to copy dpdk.tar.gz to DUT. + +1. Install Jenkins: On RPM-based distributions, such as Red Hat Enterprise + Linux (RHEL), CentOS, Fedora or Scientific Linux, you can install Jenkins + through yum. +2. Install neccessary plugins for patchset performance testing: + 1) BuildResultTrigger Plug-in + 2) Conditional BuildStep + 3) Files Found Trigger + 4) Multijob plugin + 5) SSH Slaves plugin + 6) SSH Credential Plugin + 7) Parameterized Trigger plugin +3. Create three nodes(machines) in Jenkins: + - source node: this node will get source code and store scripts. + - result node: This node will store all test results. + - tester node: This node will run performance testing. + +4. Get dpdk original repo to local : +You can create a dpdk source code repository on your local server, which +includes master branch, dpdk, dpdk-next-net, dpdk-next-crypto, +dpdk-next-eventdev and dpdk-next-virtio. Pull these repoes from dpdk.org. + + +CI Work Flow Introduction +========================= + +Step 1. Download patch set from dpdk.org. +Step 2. Apply one patch set to latest dpdk repo(will go through master repo and all + next-* repos). +Step 3. If applying patchset fails, then the build result will be marked as Failed, + and the result will be related to the patchwork. +Step 4. If applying patchset succeeds, then it will continue to trigger the + peroformance test and send performance report to patchwork. +Step 5. The result of performance test will be integrated to patchwork and showed on + the first patch of the whole patch set. And if the performance test is failed, + then the result will be sent to the patch authors for the notification. + +Related Jobs and scripts in Jenkins for each steps are as below: + +Step 1. Jenkins job:　"Get-Patch-Sets"; script: download_patchset.py +Step 2. Jenkins job "Apply-One-Patch-Set"; script: "apply-patchset.py" +Step 3. Jenkins job "DPDK-Send-Apply-Failed-Report"; script: N/A. +Step 4. Jenkins job "Apply-One-Patch-Set";script: performance test script. +Step 5. Jenkins job "performance testing and send report" which includes + "DPDK-Performance-Test" job and "DPDK-Send-Performance-Report" job. + These two jobs are running sequentially; Script "send_report.py". + + +Details on Jenkins Jobs +======================= + +1. Create "Get-Patch-Sets" job: + a. New items, enter an item name: "Get-Patch-Sets". Select + "Freestyle project" then press "OK"; + b. Restrict where this project can be run, write the label of the created + host into Label Expression; + c. Build Triggers: Build periodically, Schedule is "H/20 * * * *"; + d. Build: Runs a shell script (defaults to sh, but this is configurable) + for building the project. The script will be run with the workspace as the + current directory. Type in the contents of your shell script. + In this windows, use script download-patchset.py to download patchset from + dpdk.org. + e. This job works on source node. + +2. Create "Apply-One-Patch-Set" job: + a. New items, enter an item name: "Apply-One-Patch-Set". Select + "Freestyle project" then press "OK"; + b. General: Restrict where this project can be run, write the label of the + created host into Label Expression; + c. General: Block Build when downstream project is building; + d. Build Triggers: Build periodically, Schedule is "H/30 * * * *"; + e. Build: Runs a shell script (defaults to sh, but this is configurable) + for building the project. The script will be run with the workspace as the + current directory. Type in the contents of your shell script. + In this windows, use script apply_patchset.py to apply one patchset. + f. Add build step -> Conditional step(single). Run? is "File exists". + Fill the file name in "File". Base directory is "Workspace". + On evaluation failure is "Fail the build"; + g. Add post-build action -> Trigger parameterized build on other projects. + Build Triggers -> Projects to build "performance testing and send report", + Trigger when build is "Stable or unstable but not failed". + checkout "Trigger build without parameters". + h. Add Parameters. Projects to build is "DPDK-Send-Apply-Failed-Reports", + Trigger when build is "Failed". checkout "Trigger build without parameters". + i. This job works on source node. + +3. Create "DPDK-Performance-Test" + a. New items, enter an item name: "DPDK Performance Test". Select + "Freestyle project" then press "OK"; + b. General: Restrict where this project can be run, write the label of the + created host into Label Expression; + c. Build: Runs a shell script (defaults to sh, but this is configurable) + for building the project. The script will be run with the workspace as the + current directory. Type in the contents of your shell script. + In this windows, use script auto-regression.py to run performance testing. + d. This job works on test node. + +4. Create "DPDK-Send-Apply-Failed-Report" + a. New items, enter an item name: "DPDK-Send-Apply-Failed-Report". + Select "Freestyle project" then press "OK: + b. General: Restrict where this project can be run, write the label of the + created host into Label Expression + c. Build: Runs a shell script (defaults to sh, but this is configurable) + for building the project. The script will be run with the workspace as the + current directory. Type in the contents of your shell script. + In this windows, use script send-report.py. + d. This job works on source node. + +5. Create "DPDK-Send-Performance-Report" job + a. New items, enter an item name: "DPDK-Send-Performance-Report". + Select "Freestyle project" then press "OK"; + b. General: Restrict where this project can be run, write the label of the + created host into Label Expression; + c. Build: Runs a shell script (defaults to sh, but this is configurable) + for building the project. The script will be run with the workspace as the + current directory. Type in the contents of your shell script. + In this windows, use script send-report.py to send performance report. + d. This job works on source node. + +6. Create "performance testing and send report" job + a. New items, enter an item name: "performance testing and send report". + Select "MultiJob Project" then press "OK"; + b. Build -> MultiJob Phase: Phase name is the name you want to write. + Phase jobs -> Job, Job name is "DPDK-Performance-Test". Job execution type + is "Running phase jobs in parallel". Note: If you want to add more + performance testing environment, you can create these performance testing + jobs, then add them to this steps. + c. Add build step -> MultiJob Phase: hase name is the name you want to + write. Phase jobs -> Job, job name is "DPDK-Send-Performance-Report". + Job execution type is "Running phase jobs in sequentially". + + +Introductions of Intel CI Scripts +================================= + +Some example scripts for performance CI testing can be found in dpdk-ci repo via +http://dpdk.org/browse/tools/dpdk-ci/ +Such as download_patchset.py, apply_one_patchset.py, send_report.py and etc. + +Intel_download_patchset.py: +This script can download patchset and patchset information from patchwork. And +store this patchset into a directory which named as +"-". +Usage: +python download_patchset.py -p \ + -b \ + -e . +If end patch id is 0, this script will get the latest one from patchwork. + +Intel_apply_patchset.py: +This script can apply one patchset to correct repo, such as dpdk, +dpdk-next-net, dpdk-next-virtio, dpdk-next-eventdev and dpdk-next-crypto. +The subject prefix on the patches should provide the hint. If the majority of +the patches start with "net/" then it's for next-net, and similarly for crypto. +Patches with virtio or vhost go to that tree, etc. +After apply patchset to correct repo, it will generate a dpdk.tar.gz. + +Intel_send_report.py: +This script will parse the test results which are stored in a shared folder. +Then send performance report to patchwork. + -- 2.7.4