[dpdk-ci] Minutes of DPDK Lab Meeting, July 3rd

[dpdk-ci] Minutes of DPDK Lab Meeting, July 3rd

[O'Driscoll, Tim]

Lab Policies and Procedures:
- Bob's draft doc is at:  https://docs.google.com/document/d/1rtSwpNKltVNyDKNWgeTV5gaYeDoAPlK-sfm8XE7o_5s/edit?usp=sharing. Please review and comment.
- Ferruh suggested adding some form of change control for the lab. This can be included in the Policies and Procedures doc.

Test Setup:
- Current dashboard is at: https://dpdkdashboard.iol.unh.edu/results/dashboard/
- At the moment, all test cases are reporting errors (either Possible Regression or Apply Error). We need to reduce the number of failures to a manageable level, so that we're only highlighting genuine problems.
- At the moment, test results are compared with a hardcoded value provided by each vendor to determine whether a test passes or not. Agreed that we should be comparing performance versus master to determine if a patch passes or fails. Each vendor needs to update their scripts so that test results are compared against master. Vendors also need to review the tolerance (% deviation allowed for test to pass) for the tests to make sure it's correct.
- Vendors also need to confirm if they're happy for the relative performance data (what % patch improved/degraded performance by compared to master) for their platforms to be made public. This will allow the next level of test results to be accessible to anybody, so they can see which platform tests are failing on, and how much degradation in performance there is.
- When the previous two items are complete (reduce false positives, make next level of performance data public), we'll be able to link to the dashboard from DPDK.org. The dashboard can also be integrated with Patchwork at that time.