[dpdk-ci] Community CI Meeting Minutes

[dpdk-ci] Community CI Meeting Minutes

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 4569 bytes --]

DPDK Community CI Meeting Minutes

July 15, 2021

###############################################################################
Attendees

1. Lincoln Lavoie
2. Brandon Lo
3. Owen Hilyard
4. David Marchand
5. Ashley Waltz
6. Lijuan Tu
7. Ali Alnubani
8. Aaron Conole
9. Juraj Linkeš

###############################################################################
Agenda

1. Coverity Testing
2. CI Status
3. Test Development
4. Any other business

###############################################################################
Minutes

===============================================================================
Coverity Testing

* The requested tool “Coverity Desktop” is not available as part of the
“Coverity Scan” that is available to open source projects.  This is
available as part of their pair product offerings.
* The “Coverity Desktop” would support a per-patch scanning, to try to
catch issues before they are merged.
* Lincoln will look into the costs required and discuss with Thomas
Monjalon about getting this recommended to the Tech Board.

===============================================================================
CI Status

-------------------------------------------------------------------------------
Intel Lab

* Working on infrastructure upgrades that are causing some downtime. They
are working on getting this stable again.
* Is Intel running ABI?  Running as a daily regression test.  They are
aware of the libattomic needs for ABI.

-------------------------------------------------------------------------------
Github Actions / OBS

* Michael submitted a rework of the OBS patches, and Aaron is reviewing
them, they hope to turn it on after they finish going back and forth on any
code or design changes needed.
* Aaron will look into moving Github Actions to run testing per patch over
the next couple of days.

-------------------------------------------------------------------------------
UNH-IOL Community Lab

* FreeBSD 13 unit testing, waiting on updates from the community to fix
issues with FreeBSD.  There may be some changes needed for what tests can
be run on FreeBSD.
* Rolled out the percentage changes required within the dashboard
* Bare metal test machines (DTS runners) are now running the same commit
version, this version includes the change to the percentage reporting.
Moving forward, all bare metal systems will run the same versions of DTS
(i.e. upgrades will roll out across all hardware).
* Broadcom 25G (performance & functional) failures investigated and put
back into production, with the new DTS.
* Intel changes (DTS upgrades) were done successfully, put back into
production.
* Mellanox & ARM systems are still offline, as the updates are being
applied for the DTS upgrades. Ali will look into this over the next couple
of days to get the right driver versions installed.
* Installed libatomic on all container images, VMs, and bare metal systems
   * Reran the patchset to get all passing ABI tests
* FreeBSD 13.0 ABI testing is failing because of disabled driver in that
OS, will disable the ABI jobs for now on this OS. David is looking into
this with the Marvell team.
* Need to investigate failures of a couple of DTS cases on the systems,
will report back to the CI list on their status.

===============================================================================
Test Development

* func_reentrancy_autotest, et. al. Looks like the failing unit tests are
all isolated to ARM systems, and the tests which fail appear to be random.
We should have someone from ARM investigate the systems.  In the meantime,
Aaron is converting some of these tests to use the parent/sub testsuite
framework that Ciara Power submitted.  UNH team will reach out to the ARM
teams.
* DTS Improvement
   * Discussions are progressing. Work is being tracked here:
https://docs.google.com/document/d/1c5S0_mZzFvzZfYkqyORLT2-qNvUb-fBdjA6DGusy4yM/edit
   * Next Meeting: Jul 21, 2021
* DPDK CI Blog:
https://docs.google.com/document/d/1BmYiJK_Ndo21bgjCZYQS2HUupHpNjkXnwsu6aj8AG1U/edit#
   * Published here: https://www.dpdk.org/news/blog/

===============================================================================
Any other business

* Next CI Community Meeting, July 22, 2021

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 5734 bytes --]

[dpdk-ci] Community CI Meeting Minutes

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 4369 bytes --]

DPDK Community CI Meeting Minutes

August 26, 2021

#####################################################################
Attendees

1. Lincoln Lavoie
2. Brandon Lo
3. Owen Hilyard
4. Henry Nadeau
5. Juraj Linkeš
6. Aaron Conole
7. Michael Santana
8. Ali Alnubani
9. Ashley Weltz
10. Lijuan Tu

#####################################################################
Agenda

1. CI Status
2. Test Development
3. Any other business

#####################################################################
Minutes

=====================================================================
CI Status

---------------------------------------------------------------------
UNH-IOL Community Lab

* Failure reports from UNH are now also sent to the patch submitter
(BUG-716).
* ARM SVE emulation is now running lpm_autotest and compile testing.
* Periodic ABI test for v20.11 is now enabled as part of LTS branch testing.
* Mellanox performance testing requirements updated to 1% variation.
* TRex on Mellanox systems updated to v91.
* Various fixes to DTS, including updates to fix issues in RTE flow test
case with SCAPY.
   * Work is ongoing to fix issues in other test cases
* Work on sending periodic test reports to maintainers is underway.
   * Also working on a new view for the lab dashboard to better show and
sort periodic testing runs (i.e. view by LTS release or specific
branch/repo)
* We are starting to move to a “failure by default” model
   * This means that if DTS doesn’t produce the JSON artifacts with the
testing reports, or those reports are unreadable, that we will submit a
test named “infrastructure failure”, with a warning status.
   * This is due to hard crashes we’ve seen in DTS creating situations
where having no output is possible.
   * This also triggers an internal notification within the lab, so the lab
staff can investigate the cause.

---------------------------------------------------------------------
Intel Lab

* Working through / for 21.11 release, ABI testing has been disabled in
Intel as well.
* For DTS CI, working to now run all test suites that are supported by the
CI environment / servers. Working to dry run some of the CI scripts
provided by Owen.  CI system for DTS is now online and reporting to
patchworks.
* For DTS CI, should authors be notified of skipped testing (i.e. CI
infrastructure doesn’t support that test suite)?  Should this be marked as
a warning in patchwork?  Agreed it should do both of these actions, notify
the author and mark the patch as warning in patchworks.
* Should DTS CI require mandatory support for the test suites being run in
CI for DPDK, to ensure those suites are not broken by a patch?  Owen will
provide the list, for Intel to confirm the coverage.
* Also working on running the DTS patches through the formatter (Python
BLACK).

---------------------------------------------------------------------
Github Actions & OBS

* Update sent for OBS support, 1 comment, expecting to apply this next
week. Need to confirm the OBS scripts (build files, etc.) are correct (i.e.
meson vs make issues).
* Arm is looking to provide an application for the OVS robot and DPDK
accounts, to “call-in” to a remote lab to enable support for working in the
arm architecture.  Need to investigate what changes will be required in
Github actions definitions (i.e. yaml, etc.) to enable the application.
Details for how this integration will be set up are being worked out now.
This is likely before the end of the year, but likely not for the 21.11
release time-frame.

=====================================================================
Test Development

* From the DTS improvements team, the review of the initial input (word
document) has been completed.  Will create an excel document to sort the
agreed items, status / priority and next steps to present this to the
upcoming tech board, followed by bugzilla ticket creation and work
assignments.

=====================================================================
Any other business

* Next Meeting: September 9, 2021


-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 5273 bytes --]

Re: [dpdk-ci] Community CI Meeting Minutes

[Thomas Monjalon]

26/08/2021 15:33, Lincoln Lavoie:
> * For DTS CI, should authors be notified of skipped testing (i.e. CI infrastructure doesn’t support that test suite)?  Should this be marked as a warning in patchwork?  Agreed it should do both of these actions, notify the author and mark the patch as warning in patchworks.

Not sure to understand.
If a test is not supported, it is not an issue of the patch,
so why would it be reported in patchwork?

Re: [dpdk-ci] Community CI Meeting Minutes

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

This is specific to patches for DTS, where Intel doesn't have the
infrastructure to run every possible test suite that is included in DTS.
So, the warning is a notice to the submitter and the maintainer(s) the
patch couldn't be tested. It's not really an issue related to the content
of the patch itself (i.e. not about a breaking change or something like
that).

Cheers,
Lincoln

On Thu, Aug 26, 2021 at 12:05 PM Thomas Monjalon <thomas@monjalon.net>
wrote:

> 26/08/2021 15:33, Lincoln Lavoie:
> > * For DTS CI, should authors be notified of skipped testing (i.e. CI
> infrastructure doesn’t support that test suite)?  Should this be marked as
> a warning in patchwork?  Agreed it should do both of these actions, notify
> the author and mark the patch as warning in patchworks.
>
> Not sure to understand.
> If a test is not supported, it is not an issue of the patch,
> so why would it be reported in patchwork?
>
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 2287 bytes --]

Re: [dpdk-ci] Community CI Meeting Minutes

[Thomas Monjalon]

Then it should not be reported in patchwork.
Please let's not add more noise in patchwork.


26/08/2021 18:27, Lincoln Lavoie:
> This is specific to patches for DTS, where Intel doesn't have the
> infrastructure to run every possible test suite that is included in DTS.
> So, the warning is a notice to the submitter and the maintainer(s) the
> patch couldn't be tested. It's not really an issue related to the content
> of the patch itself (i.e. not about a breaking change or something like
> that).
> 
> Cheers,
> Lincoln
> 
> On Thu, Aug 26, 2021 at 12:05 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > 26/08/2021 15:33, Lincoln Lavoie:
> > > * For DTS CI, should authors be notified of skipped testing (i.e. CI
> > infrastructure doesn’t support that test suite)?  Should this be marked as
> > a warning in patchwork?  Agreed it should do both of these actions, notify
> > the author and mark the patch as warning in patchworks.
> >
> > Not sure to understand.
> > If a test is not supported, it is not an issue of the patch,
> > so why would it be reported in patchwork?

Re: [dpdk-ci] Community CI Meeting Minutes

[Owen Hilyard]

[-- Attachment #1: Type: text/plain, Size: 3340 bytes --]

Hello Thomas,

I was the one who asked for that feature, and I wanted to explain my
reasoning.

First, it is not possible to run every test in DTS for every patch with a
reasonable amount of hardware. The last time I tried to run everything it
took well over 2 days. As such, the DTS working group decided that it would
be best to target our testing. I wrote a script that walks the python
module paths and will use the files changed in a patch to determine which
DTS tests could possibly be affected by the change. However, since many of
the tests in DTS require special configuration or particular hardware, the
Intel Lab has a limited set of tests that can be run. I asked that these
warnings be sent out for a test that could be affected by a change, but
which was not run. In part, this is due to a situation that can occur like
with a recent patch I made to the rte flow test suite, where I basically
rewrote a test suite but it was not tested in CI. All of the smoke tests
pass, so it looks fine, but given how DTS is set up, I could have placed
code that wouldn't compile in the test suite and it would still have passed
CI. Since we don't have a good way to prevent that, these notifications are
a way for maintainers to quickly check if there were changes in a test
suite that was not tested, and either manually test the patch or pay
special attention to it.

As for your concern about noise in patchwork, this would be run on DTS
patches only, and would not report onto the main DPDK section. There is
very little going on in the DTS patchworks right now (many patches have no
tests at all) and this provides valuable information for maintainers to
help streamline patch review. The only time this would generate more than 1
or 2 warnings would be if a change is made to somewhere very deep in the
DTS framework. When that happens, I personally would prefer that it
generates a lot of output and makes itself noticable, since framework level
changes need to be very carefully reviewed to avoid breaking DPDK CI.

Owen Hilyard

On Tue, Aug 31, 2021 at 11:47 AM Thomas Monjalon <thomas@monjalon.net>
wrote:

> Then it should not be reported in patchwork.
> Please let's not add more noise in patchwork.
>
>
> 26/08/2021 18:27, Lincoln Lavoie:
> > This is specific to patches for DTS, where Intel doesn't have the
> > infrastructure to run every possible test suite that is included in DTS.
> > So, the warning is a notice to the submitter and the maintainer(s) the
> > patch couldn't be tested. It's not really an issue related to the content
> > of the patch itself (i.e. not about a breaking change or something like
> > that).
> >
> > Cheers,
> > Lincoln
> >
> > On Thu, Aug 26, 2021 at 12:05 PM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 26/08/2021 15:33, Lincoln Lavoie:
> > > > * For DTS CI, should authors be notified of skipped testing (i.e. CI
> > > infrastructure doesn’t support that test suite)?  Should this be
> marked as
> > > a warning in patchwork?  Agreed it should do both of these actions,
> notify
> > > the author and mark the patch as warning in patchworks.
> > >
> > > Not sure to understand.
> > > If a test is not supported, it is not an issue of the patch,
> > > so why would it be reported in patchwork?
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3908 bytes --]

Re: [dpdk-ci] Community CI Meeting Minutes

[Thomas Monjalon]

Thanks for the explanation.
OK for extra warnings in DTS patchwork only.


01/09/2021 14:49, Owen Hilyard:
> Hello Thomas,
> 
> I was the one who asked for that feature, and I wanted to explain my
> reasoning.
> 
> First, it is not possible to run every test in DTS for every patch with a
> reasonable amount of hardware. The last time I tried to run everything it
> took well over 2 days. As such, the DTS working group decided that it would
> be best to target our testing. I wrote a script that walks the python
> module paths and will use the files changed in a patch to determine which
> DTS tests could possibly be affected by the change. However, since many of
> the tests in DTS require special configuration or particular hardware, the
> Intel Lab has a limited set of tests that can be run. I asked that these
> warnings be sent out for a test that could be affected by a change, but
> which was not run. In part, this is due to a situation that can occur like
> with a recent patch I made to the rte flow test suite, where I basically
> rewrote a test suite but it was not tested in CI. All of the smoke tests
> pass, so it looks fine, but given how DTS is set up, I could have placed
> code that wouldn't compile in the test suite and it would still have passed
> CI. Since we don't have a good way to prevent that, these notifications are
> a way for maintainers to quickly check if there were changes in a test
> suite that was not tested, and either manually test the patch or pay
> special attention to it.
> 
> As for your concern about noise in patchwork, this would be run on DTS
> patches only, and would not report onto the main DPDK section. There is
> very little going on in the DTS patchworks right now (many patches have no
> tests at all) and this provides valuable information for maintainers to
> help streamline patch review. The only time this would generate more than 1
> or 2 warnings would be if a change is made to somewhere very deep in the
> DTS framework. When that happens, I personally would prefer that it
> generates a lot of output and makes itself noticable, since framework level
> changes need to be very carefully reviewed to avoid breaking DPDK CI.
> 
> Owen Hilyard
> 
> On Tue, Aug 31, 2021 at 11:47 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > Then it should not be reported in patchwork.
> > Please let's not add more noise in patchwork.
> >
> >
> > 26/08/2021 18:27, Lincoln Lavoie:
> > > This is specific to patches for DTS, where Intel doesn't have the
> > > infrastructure to run every possible test suite that is included in DTS.
> > > So, the warning is a notice to the submitter and the maintainer(s) the
> > > patch couldn't be tested. It's not really an issue related to the content
> > > of the patch itself (i.e. not about a breaking change or something like
> > > that).
> > >
> > > Cheers,
> > > Lincoln
> > >
> > > On Thu, Aug 26, 2021 at 12:05 PM Thomas Monjalon <thomas@monjalon.net>
> > > wrote:
> > >
> > > > 26/08/2021 15:33, Lincoln Lavoie:
> > > > > * For DTS CI, should authors be notified of skipped testing (i.e. CI
> > > > infrastructure doesn’t support that test suite)?  Should this be
> > marked as
> > > > a warning in patchwork?  Agreed it should do both of these actions,
> > notify
> > > > the author and mark the patch as warning in patchworks.
> > > >
> > > > Not sure to understand.
> > > > If a test is not supported, it is not an issue of the patch,
> > > > so why would it be reported in patchwork?

Re: [dpdk-ci] Community CI Meeting Minutes

[David Marchand]

On Thu, Aug 26, 2021 at 3:34 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
> * Periodic ABI test for v20.11 is now enabled as part of LTS branch testing.

Replying here, as I don't know if there is a separate thread on this topic.

Looking at the dashboard history, I see the ABI test for ARM (v20.11
tag) never passed successfully.
The logs show:
Error: reference directory
'/opt/dpdklab/abi_references/"v20.11"/armgigabyteref' does not exist.

It could be the extra double quotes, but in any case, please fix.


-- 
David Marchand