Below are the agreed objectives for the DPDK Community Lab for 2021. Please note, for the dates and milestones, the first draft of these objectives were not shared with the lab until February 4, 2021, and were not finalized until early March. As such the Q1 deliverables are aiming for the end of April 2021 to get caught up.
DPDK Community Lab Objectives
##############################################################################
Objective I
UNH-IOL will host the DPDK Performance/Regression Test Lab and provide the test infrastructure to execute, record and report the tests developed and defined by the DPDK Project.
Key Result(s)
* Maintain continuous performance and function testing systems, reporting results into patchworks systems. Tests are defined as the following:
* Compile Testing: Complete DPDK build / compile run, monitoring for errors / warnings and ABI test.
* Unit Testing: Hardware and infrastructure independent tests run through the make / compile tool chains
* Functional Tests: Hardware and infrastructure dependent tests, focused on verification of features or functionality of DPDK, reported per hardware system
* Performance Tests: Hardware and infrastructure dependent tests focused on throughput, reported per hardware system
* Tests will be run per-patch as patches become available
* Tests will be run weekly against the official ‘main’ and “next-” branches. The tech board will provide a specific list of these branches to test.
* Tests will be run weekly against supported LTS branches (Ex: 20.11 and 19.11 stable)
* Maintain graphical results report/dashboard. The graphic results should report the following items:
* Per NIC performance trend over time graph, based on the weekly periodic (not per patch) testing of the main DPDK branch. Note, the reported data shall be relative (i.e. performance lost or gained relative to an agreed baseline.
* Number of tests run within the lab per time (day / week), including test outcomes (i.e. pass, fail, etc.).
* Report/display periodic branch tests in the dashboard by the end of Q2
* Continue to support DPDK participants in deploying new hardware into the test lab.
##############################################################################
Objective II
Increase involvement and visibility of community lab to community
Key Result(s)
* Host bi-weekly meeting of lab stakeholders with Techboard participation
* Appoint Technical and Govboard liaisons for lab oversight
* Publish the meeting minutes from the lab meeting to CI and dev mailing list
* Publish quarterly lab update on blog and the DPDK announce email list
* Document every job as part of the dashboard or provide links to appropriate documentation (i.e. into a git repo)
* Publish the scripts and make them easy to consume for other labs
* Utilize repositories hosted by DPDK for DPDK-CI (common work), UNH-jenkins (lab jobs and runners), Dashboard (ui code)
* Aim to have scripts generalized and available in DPDK-CI repository for work items that are common across labs and different testing infrastructure.
##############################################################################
Objective III
Maintain existing tests (limit regressions) as new tests are developed
Key Result(s)
* Expedited/prioritization of any regressions introduced with new test development, that break existing DTS tests running the lab.
* Enable monitoring system to raise an alert to the community for repeating failures (more than 5) of the same test, automatically notify CI technical lead for appropriate follow up or tracking by the end of Q3.
CI Technical lead, and other members of the lab, should have some ability to pause testing or disable tests in the case that infrastructure is failed.
* Maintain compile and unit testing on the following operating systems: FreeBSD latest, Windows Server latest, Redhat 8 & 7, Ubuntu latest LTS and LTS-1, SUSE latest, CentOS Stream
* Break out the iol-testing test report into a some smaller reports:
* Per-OS would be best so Linux, BSD, and Windows by end of Q3
* This will also need to account for architecture here as well (better support of x86_64 and aarch64).
* Review and generate a report on execution time and pipeline flow of various testing running in the community lab.
* Techboard will review and provide specific recommendations for testing time targets, such as a goal to deliver x results within y hours of a patch submission. Note, some backlog will always occur when multiple simultaneous patch submissions occur.
##############################################################################
Objective IV (Highest - I)
Test Coverage: Add more checks static analysis (Coverity), ABI, compilation
Key Result(s)
* Compilation:
* Use lcov to generate a report on testing code coverage (Q2)
* Utilize ubsan and asan for early bug detection during test execution (Q3)
* Add ABI check tasks on all OS distributions, except Windows (Q1)
* Add spell checker to patch tests for documentation (needs input from tech board to define official DPDK documentation language requirement) by end of Q2
* Investigate alternatives to coverity and make a recommendation to the test board by end of Q1.
* List:
* Cppcheck
* SonarQube
* Clang static analyzer
* Flawfinder
* Splint
* infer
* Add 3 more static analyzers. as directed by the tech board by the end of Q4.
##############################################################################
Objective V (High - II - Continuous)
Test Coverage: Enable the unit tests (maintain function & enable micro benchmark)
Key Result(s)
* Work with community to enable the other included unit test suites (perf, debug, etc) (Q1)
* Use compilation profilers (where available, such as gcov) and report performance information (Q2)
##############################################################################
Objective VI (High - II - Year long)
Test Coverage: Extend the Functional features and performance scenarios (Preference for broad coverage vs deep dive into specific areas)
Key Result(s)
* Milestone 1: Enable DTS based RTEflow test cases to run in the lab on the bare-metal systems by end of Q2 2021 on the bare-metal pmds available in the lab.
* Milestone 2: Enable DTS virtio test suit to run on the bare metal systems by the end of 2021.
* Milestone 3: Add basic Crypto (execute FIPS Validation application for certification of crypto implementation and Crypto Performance application)
* Reach out to NIST re: acquiring .req files for FIPS validation application (
http://code.dpdk.org/dpdk/latest/source/examples/fips_validation/fips_validation.c,
http://code.dpdk.org/dpdk/latest/source/doc/guides/sample_app_ug/fips_validation.rst); report back to project Tech and Governing board for costs and agreements associated
* On the condition that the Board votes to continue the work after UNH reports back from NIST, acquire .req files from NIST and run the FIPS validation application included with DPDK apps on patches that make changes to the crypto code based on file locations.
* Milestone 4 (Bonus): Compression (execute Compress Performance application)
* Run the compress-perf test application in verify mode per-patch using the zlib vdev