From: Jianbo Liu <jianbo.liu@linaro.org>
To: Radoslaw Biernacki <radoslaw.biernacki@linaro.org>
Cc: dts@dpdk.org, Herbert Guan <herbert.guan@arm.com>
Subject: Re: [dts] [PATCH] framework: Adding JSON reporter
Date: Wed, 6 Sep 2017 10:25:03 +0800 [thread overview]
Message-ID: <CAP4Qi3970AKkfOwcp+gojJz1ivA+xhSM-3VpTDYoX2oURR_OJA@mail.gmail.com> (raw)
In-Reply-To: <1504625174-17845-1-git-send-email-radoslaw.biernacki@linaro.org>
On 5 September 2017 at 23:26, Radoslaw Biernacki
<radoslaw.biernacki@linaro.org> wrote:
> This patch adds the JSON reporter class which puts the results
> into output/test_results.json file
> Having JSON file format for results is usefull for CI integration.
>
> Signed-off-by: Radoslaw Biernacki <radoslaw.biernacki@linaro.org>
> ---
> framework/dts.py | 5 ++++
> framework/json_reporter.py | 75 ++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 80 insertions(+)
> create mode 100644 framework/json_reporter.py
>
> diff --git a/framework/dts.py b/framework/dts.py
> index 931bf38..b38deb7 100644
> --- a/framework/dts.py
> +++ b/framework/dts.py
> @@ -51,6 +51,7 @@ from test_case import TestCase
> from test_result import Result
> from stats_reporter import StatsReporter
> from excel_reporter import ExcelReporter
> +from json_reporter import JSONReporter
> from exception import TimeoutException, ConfigParseException, VerifyFailure
> from logger import getLogger
> import logger
> @@ -66,6 +67,7 @@ sys.setdefaultencoding('UTF8')
> requested_tests = None
> result = None
> excel_report = None
> +json_report = None
> stats_report = None
> log_handler = None
>
> @@ -443,6 +445,7 @@ def run_all(config_file, pkgName, git, patch, skip_setup,
> global requested_tests
> global result
> global excel_report
> + global json_report
> global stats_report
> global log_handler
> global check_case_inst
> @@ -506,6 +509,7 @@ def run_all(config_file, pkgName, git, patch, skip_setup,
>
> # report objects
> excel_report = ExcelReporter(output_dir + '/test_results.xls')
> + json_report = JSONReporter(output_dir + '/test_results.json')
> stats_report = StatsReporter(output_dir + '/statistics.txt')
> result = Result()
>
> @@ -574,6 +578,7 @@ def save_all_results():
> Save all result to files.
> """
> excel_report.save(result)
> + json_report.save(result)
> stats_report.save(result)
>
>
> diff --git a/framework/json_reporter.py b/framework/json_reporter.py
> new file mode 100644
> index 0000000..47e6869
> --- /dev/null
> +++ b/framework/json_reporter.py
> @@ -0,0 +1,75 @@
> +# BSD LICENSE
> +#
> +# Copyright(c) 2017 Linaro. All rights reserved.
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions
> +# are met:
> +#
> +# * Redistributions of source code must retain the above copyright
> +# notice, this list of conditions and the following disclaimer.
> +# * Redistributions in binary form must reproduce the above copyright
> +# notice, this list of conditions and the following disclaimer in
> +# the documentation and/or other materials provided with the
> +# distribution.
> +# * Neither the name of Intel Corporation nor the names of its
> +# contributors may be used to endorse or promote products derived
> +# from this software without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +import json
> +import os
> +
> +class JSONReporter(object):
> +
> + def __init__(self, filename):
> + self.filename = filename
> +
> + def __scan_cases(self, result, dut, target, suite):
> + case_results = {}
> + for case in result.all_test_cases(dut, target, suite):
> + test_result = result.result_for(dut, target, suite, case)
> + case_name = '{}/{}'.format(suite,case)
> + if 'PASSED' in test_result:
> + case_results[case_name] = 'pass'
> + else:
> + case_results[case_name] = 'fail'
Is it reasonable to consider "N/A" or "SKIP" as fail?
> + return case_results
> +
> + def __scan_target(self, result, dut, target):
> + if result.is_target_failed(dut, target):
> + return {'Target failed', 'fail'}
> + case_results = {}
> + for suite in result.all_test_suites(dut, target):
> + case_results.update(self.__scan_cases(result, dut, target, suite))
> + abspath = os.path.abspath(self.filename)
> + filename = os.path.basename(abspath)
> + dirname = os.path.dirname(abspath)
> + splitname = os.path.splitext(filename)
> + extfilename = '{}/{}_{}_{}{}'.format(dirname, splitname[0], dut, target, splitname[1])
> + with open(extfilename, 'w') as outfile:
> + json.dump(case_results, outfile, indent=4, separators=(',', ': '), encoding="utf-8", sort_keys=True)
> +
> + def __scan_dut(self, result, dut):
> + if result.is_dut_failed(dut):
> + return {'DUT failed', 'fail'}
> + case_results = {}
> + for target in result.all_targets(dut):
> + self.__scan_target(result, dut, target)
> +
> + def save(self, result):
> + case_results = {}
> + for dut in result.all_duts():
> + self.__scan_dut(result, dut)
> --
> 1.9.1
>
next prev parent reply other threads:[~2017-09-06 2:25 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-05 15:26 Radoslaw Biernacki
2017-09-06 2:25 ` Jianbo Liu [this message]
2017-09-06 13:28 ` Radoslaw Biernacki
2017-09-06 3:09 ` Liu, Yong
2017-09-06 13:28 ` Radoslaw Biernacki
2017-09-06 17:50 ` [dts] [PATCH v2] " Radoslaw Biernacki
2017-09-07 10:54 ` Liu, Yong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAP4Qi3970AKkfOwcp+gojJz1ivA+xhSM-3VpTDYoX2oURR_OJA@mail.gmail.com \
--to=jianbo.liu@linaro.org \
--cc=dts@dpdk.org \
--cc=herbert.guan@arm.com \
--cc=radoslaw.biernacki@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).