Hello Thomas,

I was the one who asked for that feature, and I wanted to explain my
reasoning.

First, it is not possible to run every test in DTS for every patch with a
reasonable amount of hardware. The last time I tried to run everything it
took well over 2 days. As such, the DTS working group decided that it would
be best to target our testing. I wrote a script that walks the python
module paths and will use the files changed in a patch to determine which
DTS tests could possibly be affected by the change. However, since many of
the tests in DTS require special configuration or particular hardware, the
Intel Lab has a limited set of tests that can be run. I asked that these
warnings be sent out for a test that could be affected by a change, but
which was not run. In part, this is due to a situation that can occur like
with a recent patch I made to the rte flow test suite, where I basically
rewrote a test suite but it was not tested in CI. All of the smoke tests
pass, so it looks fine, but given how DTS is set up, I could have placed
code that wouldn't compile in the test suite and it would still have passed
CI. Since we don't have a good way to prevent that, these notifications are
a way for maintainers to quickly check if there were changes in a test
suite that was not tested, and either manually test the patch or pay
special attention to it.

As for your concern about noise in patchwork, this would be run on DTS
patches only, and would not report onto the main DPDK section. There is
very little going on in the DTS patchworks right now (many patches have no
tests at all) and this provides valuable information for maintainers to
help streamline patch review. The only time this would generate more than 1
or 2 warnings would be if a change is made to somewhere very deep in the
DTS framework. When that happens, I personally would prefer that it
generates a lot of output and makes itself noticable, since framework level
changes need to be very carefully reviewed to avoid breaking DPDK CI.

Owen Hilyard

On Tue, Aug 31, 2021 at 11:47 AM Thomas Monjalon <thomas@monjalon.net>
wrote:

> Then it should not be reported in patchwork.
> Please let's not add more noise in patchwork.
>
>
> 26/08/2021 18:27, Lincoln Lavoie:
> > This is specific to patches for DTS, where Intel doesn't have the
> > infrastructure to run every possible test suite that is included in DTS.
> > So, the warning is a notice to the submitter and the maintainer(s) the
> > patch couldn't be tested. It's not really an issue related to the content
> > of the patch itself (i.e. not about a breaking change or something like
> > that).
> >
> > Cheers,
> > Lincoln
> >
> > On Thu, Aug 26, 2021 at 12:05 PM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 26/08/2021 15:33, Lincoln Lavoie:
> > > > * For DTS CI, should authors be notified of skipped testing (i.e. CI
> > > infrastructure doesn’t support that test suite)?  Should this be
> marked as
> > > a warning in patchwork?  Agreed it should do both of these actions,
> notify
> > > the author and mark the patch as warning in patchworks.
> > >
> > > Not sure to understand.
> > > If a test is not supported, it is not an issue of the patch,
> > > so why would it be reported in patchwork?
>
>
>
>