[PATCH v1] dts: fix pass rate edge case in results json

[PATCH v1] dts: fix pass rate edge case in results json

[Dean Marx]

Add condition to results.json pass rate generation
method which returns 0 as the pass rate when the suite
is skipped, rather than causing a divide by 0 error.

Fixes: 9f8a257235ac ("dts: improve test run result statistics")

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/test_result.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index ba7c1c9804..aed20981b2 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -324,13 +324,15 @@ def generate_pass_rate_dict(self, test_run_summary) -> dict[str, float]:
         Returns:
             A dictionary with the PASS/FAIL ratio of all test cases.
         """
-        return {
-            "PASS_RATE": (
-                float(test_run_summary[Result.PASS.name])
-                * 100
-                / sum(test_run_summary[result.name] for result in Result if result != Result.SKIP)
-            )
-        }
+        cases_not_skipped = sum(
+            test_run_summary[result.name] for result in Result if result != Result.SKIP
+        )
+        if cases_not_skipped == 0:
+            return {"PASS_RATE": 0.0}
+        else:
+            return {
+                "PASS_RATE": (float(test_run_summary[Result.PASS.name]) * 100 / cases_not_skipped)
+            }
 
 
 class DTSResult(BaseResult):
-- 
2.44.0

Re: [PATCH v1] dts: fix pass rate edge case in results json

[Luca Vizzarro]

Hi Dean,

good timing! I actually noticed this issue last week and was going to 
tackle it soon.


On 13/01/2025 21:52, Dean Marx wrote:
> @@ -324,13 +324,15 @@ def generate_pass_rate_dict(self, test_run_summary) -> dict[str, float]:
>           Returns:
>               A dictionary with the PASS/FAIL ratio of all test cases.
>           """
> -        return {
> -            "PASS_RATE": (
> -                float(test_run_summary[Result.PASS.name])
> -                * 100
> -                / sum(test_run_summary[result.name] for result in Result if result != Result.SKIP)
> -            )
> -        }
> +        cases_not_skipped = sum(
> +            test_run_summary[result.name] for result in Result if result != Result.SKIP
> +        )
> +        if cases_not_skipped == 0:
> +            return {"PASS_RATE": 0.0}
> +        else:
> +            return {
> +                "PASS_RATE": (float(test_run_summary[Result.PASS.name]) * 100 / cases_not_skipped)
> +            }

The solution feels a bit overcomplicated, you can just throw a 
max(sum(..), 1), so:

     ran_tests = sum(test_run_summary[result.name] for result in Result
                     if result != Result.SKIP)
     return {
         "PASS_RATE": test_run_summary[Result.PASS.name] * 100.0 / 
max(ran_tests, 1)
     }

also while you are at it, changing the 100 to 100.0 will spare us from 
writing a longer float(..) expression.

Re: [PATCH v1] dts: fix pass rate edge case in results json

[Dean Marx]

Got it, I can implement this asap. I also noticed after submitting
that test suites with skipped cases still report a 100% pass rate if
the non skipped cases still pass, so I'll probably change the pass
rate to 100% for when an entire suite is skipped unless anyone
disagrees.

Re: [PATCH v1] dts: fix pass rate edge case in results json

[Dean Marx]

Got it, I can implement this asap

Re: [PATCH v1] dts: fix pass rate edge case in results json

[Dean Marx]

Also, I noticed after submitting this first version that suites where
some cases are skipped and the rest pass have a 100% pass rate in the
results report. Should this also mean that suites where every case is
skipped have a 100% pass rate? Right now this reports a 0% when the
entire suite is skipped, if anyone thinks this should be changed let
me know.

[PATCH v2] dts: fix pass rate edge case in results json

[Dean Marx]

Add condition to results.json pass rate generation
method which returns 0 as the pass rate when the suite
is skipped, rather than causing a divide by 0 error.

Fixes: 9f8a257235ac ("dts: improve test run result statistics")

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/test_result.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index ba7c1c9804..45fc2e8241 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -324,13 +324,10 @@ def generate_pass_rate_dict(self, test_run_summary) -> dict[str, float]:
         Returns:
             A dictionary with the PASS/FAIL ratio of all test cases.
         """
-        return {
-            "PASS_RATE": (
-                float(test_run_summary[Result.PASS.name])
-                * 100
-                / sum(test_run_summary[result.name] for result in Result if result != Result.SKIP)
-            )
-        }
+        cases_not_skipped = sum(
+            test_run_summary[result.name] for result in Result if result != Result.SKIP
+        )
+        return {"PASS_RATE": test_run_summary[Result.PASS.name] * 100.0 / max(cases_not_skipped, 1)}
 
 
 class DTSResult(BaseResult):
-- 
2.44.0

Re: [PATCH v1] dts: fix pass rate edge case in results json

[Luca Vizzarro]

On 15/01/2025 15:41, Dean Marx wrote:
> Also, I noticed after submitting this first version that suites where
> some cases are skipped and the rest pass have a 100% pass rate in the
> results report. Should this also mean that suites where every case is
> skipped have a 100% pass rate? Right now this reports a 0% when the
> entire suite is skipped, if anyone thinks this should be changed let
> me know.

yes, I believe this sounds reasonable. Good catch

Re: [PATCH v2] dts: fix pass rate edge case in results json

[Luca Vizzarro]

lgtm. thanks!

Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>