https://bugs.dpdk.org/show_bug.cgi?id=431 Bug ID: 431 Summary: Missing performance results in Patchwork checks, and possible false positives Product: lab Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: Normal Component: job scripts Assignee: ci@dpdk.org Reporter: alialnu@mellanox.com CC: dpdklab@iol.unh.edu Target Milestone: --- Sometimes, not all of the performance results for a vendor are sent in the same email to the mailing list. For example, for patch 67243 (http://dpdk.org/patch/67243), I see 2 reports: 1. http://mails.dpdk.org/archives/test-report/2020-March/122345.html (Mellanox ConnectX-5 100000 Mbps & Mellanox ConnectX-4 Lx 25000 Mbps). 2. http://mails.dpdk.org/archives/test-report/2020-March/122348.html (Mellanox ConnectX-4 Lx 40000 Mbps). This might be a problem because only the last check will appear in Patchwork. Is this an expected behavior? Would a second report with "success" status be sent after a report with "failure" status? Because this would cause the Patchwork check to be a false positive. -- You are receiving this mail because: You are the assignee for the bug.
Hello, > -----Original Message----- > From: ci <ci-bounces@dpdk.org> On Behalf Of bugzilla@dpdk.org > Sent: Tuesday, March 31, 2020 5:43 PM > To: ci@dpdk.org > Subject: [dpdk-ci] [Bug 431] Missing performance results in Patchwork checks, > and possible false positives > Patchwork is reporting a false positive on patch https://patches.dpdk.org/patch/68908/ (at least) for iol-mellanox-Performance because of this. Two reports were sent by the lab for this patch in the following order: https://mails.dpdk.org/archives/test-report/2020-April/127163.html (Failing, Sun Apr 19 13:57:32 CEST 2020). http://mails.dpdk.org/archives/test-report/2020-April/127165.html (Passing, Sun Apr 19 14:06:51 CEST 2020). And even though one of the reports is failing, the current status says passing on Patchwork. What's the reason why aren't all the performance test results for a vendor sent in the same report? Thanks, Ali
https://bugs.dpdk.org/show_bug.cgi?id=431 Brandon Lo (blo@iol.unh.edu) changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|IN_PROGRESS |RESOLVED --- Comment #5 from Brandon Lo (blo@iol.unh.edu) --- Fixed. This issue is due to concurrent running tests. Our internal testing job would run a job to send a performance report email to vendors for a given patch. The problem is that this email would be sent after all of a given vendor's machines are done testing (vendor A), but it could occur before all of the machines of another vendor finishes (vendor B). This results in all of the results from the first vendor (A) being sent, but the internal script will pick up the existing results created by the unfinished vendor B and send those out as well. This can be seen with email timings. An unfinished Mellanox report email could have one single testbed's result in the email, and it will always be sent at the same time as another vendor's email (Intel for example). The fix was to add support for sending a specific vendor's email only when all of the given vendor's testbeds are finished testing. We have tested this by using dry runs of the email sending script with successful separation of vendor testbed results. -- You are receiving this mail because: You are the assignee for the bug.