[dpdk-ci] DPDK Community Lab 2021 Objectives

[dpdk-ci] DPDK Community Lab 2021 Objectives

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 7761 bytes --]

Hello All,

Below are the agreed objectives for the DPDK Community Lab for 2021.
Please note, for the dates and milestones, the first draft of these
objectives were not shared with the lab until February 4, 2021, and were
not finalized until early March.  As such the Q1 deliverables are aiming
for the end of April 2021 to get caught up.

DPDK Community Lab Objectives

##############################################################################
Objective I

UNH-IOL will host the DPDK Performance/Regression Test Lab and provide the
test infrastructure to execute, record and report the tests developed and
defined by the DPDK Project.

Key Result(s)

* Maintain continuous performance and function testing systems, reporting
results into patchworks systems. Tests are defined as the following:
   * Compile Testing: Complete DPDK build / compile run, monitoring for
errors / warnings and ABI test.
   * Unit Testing: Hardware and infrastructure independent tests run
through the make / compile tool chains
   * Functional Tests: Hardware and infrastructure dependent tests, focused
on verification of features or functionality of DPDK, reported per hardware
system
   * Performance Tests: Hardware and infrastructure dependent tests focused
on throughput, reported per hardware system
   * Tests will be run per-patch as patches become available
   * Tests will be run weekly against the official ‘main’ and “next-”
branches. The tech board will provide a specific list of these branches to
test.
   * Tests will be run weekly against supported LTS branches  (Ex: 20.11
and 19.11 stable)

* Maintain graphical results report/dashboard. The graphic results should
report the following items:
   * Per NIC performance trend over time graph, based on the weekly
periodic (not per patch) testing of the main DPDK branch. Note, the
reported data shall be relative (i.e. performance lost or gained relative
to an agreed baseline.
   * Number of tests run within the lab per time (day / week), including
test outcomes (i.e. pass, fail, etc.).

* Report/display periodic branch tests in the dashboard by the end of Q2

* Continue to support DPDK participants in deploying new hardware into the
test lab.



##############################################################################
Objective II

Increase involvement and visibility of community lab to community

Key Result(s)

* Host bi-weekly meeting of lab stakeholders with Techboard participation

* Appoint Technical and Govboard liaisons for lab oversight

* Publish the meeting minutes from the lab meeting to CI and dev mailing
list

* Publish quarterly lab update on blog and the DPDK announce email list

* Document every job as part of the dashboard or provide links to
appropriate documentation (i.e. into a git repo)

* Publish the scripts and make them easy to consume for other labs
   * Utilize repositories hosted by DPDK for DPDK-CI (common work),
UNH-jenkins (lab jobs and runners), Dashboard (ui code)
   * Aim to have scripts generalized and available in DPDK-CI repository
for work items that are common across labs and different testing
infrastructure.



##############################################################################
Objective  III

Maintain existing tests (limit regressions) as new tests are developed

Key Result(s)

* Expedited/prioritization of any regressions introduced with new test
development, that break existing DTS tests running the lab.

* Enable monitoring system to raise an alert to the community for repeating
failures (more than 5) of the same test, automatically notify CI technical
lead for appropriate follow up or tracking by the end of Q3.
CI Technical lead, and other members of the lab, should have some ability
to pause testing or disable tests in the case that infrastructure is failed.

* Maintain compile and unit testing on the following operating systems:
FreeBSD latest, Windows Server latest, Redhat 8 & 7, Ubuntu latest LTS and
LTS-1, SUSE latest, CentOS Stream

* Break out the iol-testing test report into a some smaller reports:
   * Per-OS would be best so Linux, BSD, and Windows by end of Q3
   * This will also need to account for architecture here as well (better
support of x86_64 and aarch64).

* Review and generate a report on execution time and pipeline flow of
various testing running in the community lab.
   * Techboard will review and provide specific recommendations for testing
time targets, such as a goal to deliver x results within y hours of a patch
submission.  Note, some backlog will always occur when multiple
simultaneous patch submissions occur.


##############################################################################
Objective  IV (Highest - I)

Test Coverage: Add more checks static analysis (Coverity), ABI, compilation


Key Result(s)

* Compilation:
   * Use lcov to generate a report on testing code coverage (Q2)
   * Utilize ubsan and asan for early bug detection during test execution
(Q3)

* Add ABI check tasks on all OS distributions, except Windows (Q1)

* Add spell checker to patch tests for documentation (needs  input from
tech board to define official DPDK documentation language requirement) by
end of Q2

* Investigate alternatives to coverity and make a recommendation to the
test board by end of Q1.
   * List:
      * Cppcheck
      * SonarQube
      * Clang static analyzer
      * Flawfinder
      * Splint
      * infer
   * Add 3 more static analyzers. as directed by the tech board by the end
of Q4.



##############################################################################
Objective  V (High - II - Continuous)

Test Coverage: Enable the unit tests (maintain function & enable micro
benchmark)

Key Result(s)

* Work with community to enable the other included unit test suites (perf,
debug, etc) (Q1)

* Use compilation profilers (where available, such as gcov) and report
performance information (Q2)

##############################################################################
Objective  VI (High - II - Year long)

Test Coverage: Extend the Functional features and performance scenarios
(Preference for broad coverage vs deep dive into specific areas)

Key Result(s)

* Milestone 1: Enable DTS based RTEflow test cases to run in the lab on the
bare-metal systems by end of Q2 2021 on the bare-metal pmds available in
the lab.

* Milestone 2: Enable DTS virtio test suit to run on the bare metal systems
by the end of 2021.

* Milestone 3: Add basic Crypto (execute FIPS Validation application for
certification of crypto implementation and Crypto Performance application)
   * Reach out to NIST re: acquiring .req files for FIPS validation
application (
http://code.dpdk.org/dpdk/latest/source/examples/fips_validation/fips_validation.c,
http://code.dpdk.org/dpdk/latest/source/doc/guides/sample_app_ug/fips_validation.rst);
report back to project Tech and Governing board for costs and agreements
associated
   * On the condition that the Board votes to continue the work after UNH
reports back from NIST, acquire .req files from NIST and run the FIPS
validation application included with DPDK apps on patches that make changes
to the crypto code based on file locations.

* Milestone 4 (Bonus): Compression (execute Compress Performance
application)
   * Run the compress-perf test application in verify mode per-patch using
the zlib vdev

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 9235 bytes --]