[dpdk-ci] [Bug 431] Missing performance results in Patchwork checks, and possible false positives

[dpdk-ci] [Bug 431] Missing performance results in Patchwork checks, and possible false positives

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=431

            Bug ID: 431
           Summary: Missing performance results in Patchwork checks, and
                    possible false positives
           Product: lab
           Version: unspecified
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: job scripts
          Assignee: ci@dpdk.org
          Reporter: alialnu@mellanox.com
                CC: dpdklab@iol.unh.edu
  Target Milestone: ---

Sometimes, not all of the performance results for a vendor are sent in the same
email to the mailing list. For example, for patch 67243
(http://dpdk.org/patch/67243), I see 2 reports:

1. http://mails.dpdk.org/archives/test-report/2020-March/122345.html (Mellanox
ConnectX-5 100000 Mbps & Mellanox ConnectX-4 Lx 25000 Mbps).
2. http://mails.dpdk.org/archives/test-report/2020-March/122348.html (Mellanox
ConnectX-4 Lx 40000 Mbps).

This might be a problem because only the last check will appear in Patchwork.
Is this an expected behavior?
Would a second report with "success" status be sent after a report with
"failure" status? Because this would cause the Patchwork check to be a false
positive.

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [dpdk-ci] [Bug 431] Missing performance results in Patchwork checks, and possible false positives

[Ali Alnubani]

Hello,

> -----Original Message-----
> From: ci <ci-bounces@dpdk.org> On Behalf Of bugzilla@dpdk.org
> Sent: Tuesday, March 31, 2020 5:43 PM
> To: ci@dpdk.org
> Subject: [dpdk-ci] [Bug 431] Missing performance results in Patchwork checks,
> and possible false positives
>

Patchwork is reporting a false positive on patch https://patches.dpdk.org/patch/68908/ (at least) for iol-mellanox-Performance because of this.

Two reports were sent by the lab for this patch in the following order:
https://mails.dpdk.org/archives/test-report/2020-April/127163.html (Failing, Sun Apr 19 13:57:32 CEST 2020).
http://mails.dpdk.org/archives/test-report/2020-April/127165.html (Passing, Sun Apr 19 14:06:51 CEST 2020).
And even though one of the reports is failing, the current status says passing on Patchwork.

What's the reason why aren't all the performance test results for a vendor sent in the same report?

Thanks,
Ali

[dpdk-ci] [Bug 431] Missing performance results in Patchwork checks, and possible false positives

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=431

Brandon Lo (blo@iol.unh.edu) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|IN_PROGRESS                 |RESOLVED

--- Comment #5 from Brandon Lo (blo@iol.unh.edu) ---
Fixed.

This issue is due to concurrent running tests.

Our internal testing job would run a job to send a performance report email to
vendors for a given patch. The problem is that this email would be sent after
all of a given vendor's machines are done testing (vendor A), but it could
occur before all of the machines of another vendor finishes (vendor B). 

This results in all of the results from the first vendor (A) being sent, but
the internal script will pick up the existing results created by the unfinished
vendor B and send those out as well.

This can be seen with email timings. An unfinished Mellanox report email could
have one single testbed's result in the email, and it will always be sent at
the same time as another vendor's email (Intel for example). 

The fix was to add support for sending a specific vendor's email only when all
of the given vendor's testbeds are finished testing. 
We have tested this by using dry runs of the email sending script with
successful separation of vendor testbed results.

-- 
You are receiving this mail because:
You are the assignee for the bug.