From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 170D4A0548 for ; Thu, 22 Apr 2021 18:07:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0B73241CAF; Thu, 22 Apr 2021 18:07:31 +0200 (CEST) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mails.dpdk.org (Postfix) with ESMTP id CE9FF41C9D for ; Thu, 22 Apr 2021 18:07:29 +0200 (CEST) Received: by mail-ej1-f51.google.com with SMTP id u21so69626487ejo.13 for ; Thu, 22 Apr 2021 09:07:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:from:date:message-id:subject:to; bh=gcBhS3Y85EA5jFaYo9AgKbVEsHyj1kZX/DvUHej/joo=; b=gypGGHK/DPWVYzjfo8+lANDkmZxwgKldOgoJytWRt6YUwaCqG1t9A1gqGcB0gtWOKM OknD9t6QUcdgcWvmaWIeKxBHeweW9Wj1LsCfAKWK8VZWnHVkLK/e7Bco00te1vYRk7dp pzN6iiyjdVVqqaO2BYIa6MXJfXH61g1II/450= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=gcBhS3Y85EA5jFaYo9AgKbVEsHyj1kZX/DvUHej/joo=; b=SkNZkKxEE0IsaofnwiV8ppByX5WfuaKx7sFXXi2y1oi4j7eBQ/ShL2o8uWJqLGhgPv Z64V9vxn+IdGOVQ0/Zauc/9QPFvZJbYXtRwzGG2nWxTepiQWX5dNlxempiwV60JQa+Kc xmKKXrBUoFMK2nWMs6cvShAzRqOqdI2uHmrr5nrcb4Eg65+cbHQYbYgbO6Ik6bUG2YqQ JFhlXBH9pTydJ3+UirbvQDS3IpDegaDMqsAByV5l7qyNv5vby8fc5ZoW4E53pHva8tVR 2zqeSYPG0Nl0qP8v0Y1ogIhVn+GOQWQgyLXAJ8woHPfBmZTrnrhaI1ovPoW0RfYYNxhh RfHA== X-Gm-Message-State: AOAM531GzG2gAmY5UjT+I4VHA2wBITErlUbFHJo3eQPYxBe7O00/1YZ2 Ghzv3n1aW145cLC7YRpcOOpWfCS/P8UxTNHu5ObNB6L1/Rizjcdm X-Google-Smtp-Source: ABdhPJwh8Bjybc1u+GD7MUE/+NUisxyF02V7wgpoDQljj+N7cJC/Q6XXGjZplHepszfEsDu8uExXGigKsqA7+FL5uSY= X-Received: by 2002:a17:907:3f08:: with SMTP id hq8mr4255495ejc.90.1619107647806; Thu, 22 Apr 2021 09:07:27 -0700 (PDT) MIME-Version: 1.0 From: Lincoln Lavoie Date: Thu, 22 Apr 2021 12:04:09 -0400 Message-ID: To: ci@dpdk.org Content-Type: multipart/alternative; boundary="00000000000033871805c091e098" Subject: [dpdk-ci] DPDK Community Lab 2021 Objectives X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --00000000000033871805c091e098 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello All, Below are the agreed objectives for the DPDK Community Lab for 2021. Please note, for the dates and milestones, the first draft of these objectives were not shared with the lab until February 4, 2021, and were not finalized until early March. As such the Q1 deliverables are aiming for the end of April 2021 to get caught up. DPDK Community Lab Objectives ###########################################################################= ### Objective I UNH-IOL will host the DPDK Performance/Regression Test Lab and provide the test infrastructure to execute, record and report the tests developed and defined by the DPDK Project. Key Result(s) * Maintain continuous performance and function testing systems, reporting results into patchworks systems. Tests are defined as the following: * Compile Testing: Complete DPDK build / compile run, monitoring for errors / warnings and ABI test. * Unit Testing: Hardware and infrastructure independent tests run through the make / compile tool chains * Functional Tests: Hardware and infrastructure dependent tests, focused on verification of features or functionality of DPDK, reported per hardware system * Performance Tests: Hardware and infrastructure dependent tests focused on throughput, reported per hardware system * Tests will be run per-patch as patches become available * Tests will be run weekly against the official =E2=80=98main=E2=80=99 a= nd =E2=80=9Cnext-=E2=80=9D branches. The tech board will provide a specific list of these branches to test. * Tests will be run weekly against supported LTS branches (Ex: 20.11 and 19.11 stable) * Maintain graphical results report/dashboard. The graphic results should report the following items: * Per NIC performance trend over time graph, based on the weekly periodic (not per patch) testing of the main DPDK branch. Note, the reported data shall be relative (i.e. performance lost or gained relative to an agreed baseline. * Number of tests run within the lab per time (day / week), including test outcomes (i.e. pass, fail, etc.). * Report/display periodic branch tests in the dashboard by the end of Q2 * Continue to support DPDK participants in deploying new hardware into the test lab. ###########################################################################= ### Objective II Increase involvement and visibility of community lab to community Key Result(s) * Host bi-weekly meeting of lab stakeholders with Techboard participation * Appoint Technical and Govboard liaisons for lab oversight * Publish the meeting minutes from the lab meeting to CI and dev mailing list * Publish quarterly lab update on blog and the DPDK announce email list * Document every job as part of the dashboard or provide links to appropriate documentation (i.e. into a git repo) * Publish the scripts and make them easy to consume for other labs * Utilize repositories hosted by DPDK for DPDK-CI (common work), UNH-jenkins (lab jobs and runners), Dashboard (ui code) * Aim to have scripts generalized and available in DPDK-CI repository for work items that are common across labs and different testing infrastructure. ###########################################################################= ### Objective III Maintain existing tests (limit regressions) as new tests are developed Key Result(s) * Expedited/prioritization of any regressions introduced with new test development, that break existing DTS tests running the lab. * Enable monitoring system to raise an alert to the community for repeating failures (more than 5) of the same test, automatically notify CI technical lead for appropriate follow up or tracking by the end of Q3. CI Technical lead, and other members of the lab, should have some ability to pause testing or disable tests in the case that infrastructure is failed= . * Maintain compile and unit testing on the following operating systems: FreeBSD latest, Windows Server latest, Redhat 8 & 7, Ubuntu latest LTS and LTS-1, SUSE latest, CentOS Stream * Break out the iol-testing test report into a some smaller reports: * Per-OS would be best so Linux, BSD, and Windows by end of Q3 * This will also need to account for architecture here as well (better support of x86_64 and aarch64). * Review and generate a report on execution time and pipeline flow of various testing running in the community lab. * Techboard will review and provide specific recommendations for testing time targets, such as a goal to deliver x results within y hours of a patch submission. Note, some backlog will always occur when multiple simultaneous patch submissions occur. ###########################################################################= ### Objective IV (Highest - I) Test Coverage: Add more checks static analysis (Coverity), ABI, compilation Key Result(s) * Compilation: * Use lcov to generate a report on testing code coverage (Q2) * Utilize ubsan and asan for early bug detection during test execution (Q3) * Add ABI check tasks on all OS distributions, except Windows (Q1) * Add spell checker to patch tests for documentation (needs input from tech board to define official DPDK documentation language requirement) by end of Q2 * Investigate alternatives to coverity and make a recommendation to the test board by end of Q1. * List: * Cppcheck * SonarQube * Clang static analyzer * Flawfinder * Splint * infer * Add 3 more static analyzers. as directed by the tech board by the end of Q4. ###########################################################################= ### Objective V (High - II - Continuous) Test Coverage: Enable the unit tests (maintain function & enable micro benchmark) Key Result(s) * Work with community to enable the other included unit test suites (perf, debug, etc) (Q1) * Use compilation profilers (where available, such as gcov) and report performance information (Q2) ###########################################################################= ### Objective VI (High - II - Year long) Test Coverage: Extend the Functional features and performance scenarios (Preference for broad coverage vs deep dive into specific areas) Key Result(s) * Milestone 1: Enable DTS based RTEflow test cases to run in the lab on the bare-metal systems by end of Q2 2021 on the bare-metal pmds available in the lab. * Milestone 2: Enable DTS virtio test suit to run on the bare metal systems by the end of 2021. * Milestone 3: Add basic Crypto (execute FIPS Validation application for certification of crypto implementation and Crypto Performance application) * Reach out to NIST re: acquiring .req files for FIPS validation application ( http://code.dpdk.org/dpdk/latest/source/examples/fips_validation/fips_valid= ation.c, http://code.dpdk.org/dpdk/latest/source/doc/guides/sample_app_ug/fips_valid= ation.rst); report back to project Tech and Governing board for costs and agreements associated * On the condition that the Board votes to continue the work after UNH reports back from NIST, acquire .req files from NIST and run the FIPS validation application included with DPDK apps on patches that make changes to the crypto code based on file locations. * Milestone 4 (Bonus): Compression (execute Compress Performance application) * Run the compress-perf test application in verify mode per-patch using the zlib vdev --=20 *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) --00000000000033871805c091e098 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hel= lo All,

Below are the agr= eed objectives for the DPDK Community Lab for 2021.=C2=A0 Please note, for = the dates and milestones, the first draft of these objectives were not shar= ed with the lab until February 4, 2021, and were not finalized until early = March.=C2=A0 As such the Q1 deliverables are aiming for the end of April 20= 21 to get caught up.

DPDK= Community Lab Objectives

##########################################= ####################################
Objective I

UNH-IOL will hos= t the DPDK Performance/Regression Test Lab and provide the test infrastruct= ure to execute, record and report the tests developed and defined by the DP= DK Project.

Key Result(s)

* Maintain continuous performance a= nd function testing systems, reporting results into patchworks systems. Tes= ts are defined as the following:
=C2=A0 =C2=A0* Compile Testing: Complet= e DPDK build / compile run, monitoring for errors / warnings and ABI test.<= br>=C2=A0 =C2=A0* Unit Testing: Hardware and infrastructure independent tes= ts run through the make / compile tool chains
=C2=A0 =C2=A0* Functional = Tests: Hardware and infrastructure dependent tests, focused on verification= of features or functionality of DPDK, reported per hardware system
=C2= =A0 =C2=A0* Performance Tests: Hardware and infrastructure dependent tests = focused on throughput, reported per hardware system
=C2=A0 =C2=A0* Tests= will be run per-patch as patches become available
=C2=A0 =C2=A0* Tests = will be run weekly against the official =E2=80=98main=E2=80=99 and =E2=80= =9Cnext-=E2=80=9D branches. The tech board will provide a specific list of = these branches to test.
=C2=A0 =C2=A0* Tests will be run weekly against = supported LTS branches =C2=A0(Ex: 20.11 and 19.11 stable)

* Maintain= graphical results report/dashboard. The graphic results should report the = following items:
=C2=A0 =C2=A0* Per NIC performance trend over time grap= h, based on the weekly periodic (not per patch) testing of the main DPDK br= anch. Note, the reported data shall be relative (i.e. performance lost or g= ained relative to an agreed baseline.
=C2=A0 =C2=A0* Number of tests run= within the lab per time (day / week), including test outcomes (i.e. pass, = fail, etc.).

* Report/display periodic branch tests in the dashboard= by the end of Q2

* Continue to support DPDK participants in deployi= ng new hardware into the test lab.



########################= ######################################################
Objective II
=
Increase involvement and visibility of community lab to community
Key Result(s)

* Host bi-weekly meeting of lab stakeholders with Te= chboard participation

* Appoint Technical and Govboard liaisons for = lab oversight

* Publish the meeting minutes from the lab meeting to= CI and dev mailing list

* Publish quarterly lab update on blog and= the DPDK announce email list

* Document every job as part of the d= ashboard or provide links to appropriate documentation (i.e. into a git rep= o)

* Publish the scripts and make them easy to consume for other lab= s
=C2=A0 =C2=A0* Utilize repositories hosted by DPDK for DPDK-CI (common= work), UNH-jenkins (lab jobs and runners), Dashboard (ui code)
=C2=A0 = =C2=A0* Aim to have scripts generalized and available in DPDK-CI repository= for work items that are common across labs and different testing infrastru= cture. =C2=A0



#############################################= #################################
Objective =C2=A0III

Maintain ex= isting tests (limit regressions) as new tests are developed

Key Resu= lt(s)

* Expedited/prioritization of any regressions introduced with = new test development, that break existing DTS tests running the lab.
* Enable monitoring system to raise an alert to the community for repeatin= g failures (more than 5) of the same test, automatically notify CI technica= l lead for appropriate follow up or tracking by the end of Q3.
CI Techni= cal lead, and other members of the lab, should have some ability to pause t= esting or disable tests in the case that infrastructure is failed.

*= Maintain compile and unit testing on the following operating systems: Free= BSD latest, Windows Server latest, Redhat 8 & 7, Ubuntu latest LTS and = LTS-1, SUSE latest, CentOS Stream

* Break out the iol-testing test = report into a some smaller reports:
=C2=A0 =C2=A0* Per-OS would be best = so Linux, BSD, and Windows by end of Q3
=C2=A0 =C2=A0* This will also ne= ed to account for architecture here as well (better support of x86_64 and a= arch64).

* Review and generate a report on execution time and pipeli= ne flow of various testing running in the community lab.
=C2=A0 =C2=A0* = Techboard will review and provide specific recommendations for testing time= targets, such as a goal to deliver x results within y hours of a patch sub= mission.=C2=A0 Note, some backlog will always occur when multiple simultane= ous patch submissions occur.


##################################= ############################################
Objective =C2=A0IV (Highest= - I)

Test Coverage: Add more checks static analysis (Coverity), ABI= , compilation =C2=A0

Key Result(s)

* Compilation:
=C2=A0 = =C2=A0* Use lcov to generate a report on testing code coverage (Q2)
=C2= =A0 =C2=A0* Utilize ubsan and asan for early bug detection during test exec= ution (Q3)

* Add ABI check tasks on all OS distributions, except Win= dows (Q1)

* Add spell checker to patch tests for documentation (need= s =C2=A0input from tech board to define official DPDK documentation languag= e requirement) by end of Q2

* Investigate alternatives to coverity a= nd make a recommendation to the test board by end of Q1.
=C2=A0 =C2=A0*= List:
=C2=A0 =C2=A0 =C2=A0 * Cppcheck
=C2=A0 =C2=A0 =C2=A0 * SonarQu= be
=C2=A0 =C2=A0 =C2=A0 * Clang static analyzer
=C2=A0 =C2=A0 =C2=A0 = * Flawfinder
=C2=A0 =C2=A0 =C2=A0 * Splint
=C2=A0 =C2=A0 =C2=A0 * inf= er
=C2=A0 =C2=A0* Add 3 more static analyzers. as directed by the tech b= oard by the end of Q4.



###################################= ###########################################
Objective =C2=A0V (High - II= - Continuous)

Test Coverage: Enable the unit tests (maintain functi= on & enable micro benchmark)

Key Result(s)

* Work with co= mmunity to enable the other included unit test suites (perf, debug, etc) (Q= 1)

* Use compilation profilers (where available, such as gcov) and r= eport performance information (Q2)

################################= ##############################################
Objective =C2=A0VI (High = - II - Year long)

Test Coverage: Extend the Functional features and = performance scenarios (Preference for broad coverage vs deep dive into spec= ific areas)

Key Result(s)

* Milestone 1: Enable DTS based RT= Eflow test cases to run in the lab on the bare-metal systems by end of Q2 2= 021 on the bare-metal pmds available in the lab.

* Milestone 2: Enab= le DTS virtio test suit to run on the bare metal systems by the end of 2021= .

* Milestone 3: Add basic Crypto (execute FIPS Validation applicati= on for certification of crypto implementation and Crypto Performance applic= ation)
=C2=A0 =C2=A0* Reach out to NIST re: acquiring .req files for FIP= S validation application (http://code.dpdk.org/dpdk/lat= est/source/examples/fips_validation/fips_validation.c, http://code.dpdk.org/dpdk/latest/source/doc/guides/sample_app_ug/fips= _validation.rst); report back to project Tech and Governing board for c= osts and agreements associated
=C2=A0 =C2=A0* On the condition that the = Board votes to continue the work after UNH reports back from NIST, acquire = .req files from NIST and run the FIPS validation application included with = DPDK apps on patches that make changes to the crypto code based on file loc= ations.

* Milestone 4 (Bonus): Compression (execute Compress Perform= ance application)
=C2=A0 =C2=A0* Run the compress-perf test application = in verify mode per-patch using the zlib vdev

--
Lincoln Lavoie
Principal Engineer, Broad= band Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
+1-603-674-2755 (m)

<= /div>
--00000000000033871805c091e098--