Re: UNH CI skipped tests

[Kevin Traynor <ktraynor@redhat.com>]

On 03/03/2023 14:33, Patrick Robb wrote:
> Hi Kevin,
> 
> My thinking was that allowing new testruns simply due to a jenkinsfile
> change might produce testrun results which yielded no value for you or
> others. But, it sounds like you do not feel this way, and I agree having a
> testrun kick off due to Jenkinsfile changes would provide some debugging
> information to us UNH folks. So, since it sounds like it's alright with
> you, I'm going to go ahead and undo my previous changes on this front.
> 

Fine by me.

> And yes I'm sure all of those new tests popping up all of a sudden was
> unexpected... Some of those were tests run on a server which was
> temporarily down (ARM-Gigabyte), so once that went back online we were able
> to run additional tests. And then you know the others which were failing
> were never intended to run on 21.11.
> 

ok, I think that's all the differences explained.

It might be worth flagging in the dashboard if some valid tests were 
skipped for a case like this. I've no idea if that's easy to do, so not 
sure if it's worth the effort.

Thanks Lincoln and Patrick for the quick response and explanations,

Kevin.

> Thanks,
> Patrick
> 
> 
> 
> On Fri, Mar 3, 2023 at 4:59 AM Kevin Traynor <ktraynor@redhat.com> wrote:
> 
>> On 02/03/2023 18:49, Patrick Robb wrote:
>>> Hi Kevin,
>>>
>>
>> Hi Patrick,
>>
>>> The FIPS test has the same issue as the cryptodev tests - it cannot run
>> on
>>> 21.11 due to a lacking dependency. We disabled the testing on the normal
>>> 21.11 branch but I missed 21.11-staging. Sorry about the oversight. All 3
>>> tests are now disabled on 21.11-staging.
>>>
>>
>> Cool, thanks.
>>
>>> The reason the two runs happened is that we were inadvertently polling
>>> DPDK-Stable AND our jenkinsfile repo for commits, and triggering new
>> builds
>>> for commits to either repo. I have disabled polling to our Jenkinsfile
>>> repo, which should limit new builds to just commits to DPDK-Stable.
>>>
>>
>> I'm not sure that was the reason for the second test run. I did that
>> deliberately...
>>
>> - I ran the initial run on the original commit (pass)
>> - I added a new dependency/build impacting commit and saw new tests and
>> failures
>> - I wanted to understand if the new tests and failures were related to
>> the single commit allowing additional tests to run, so...
>> - I force pushed back to the original commit for a second run on the
>> original commit. There I also saw the new tests so was able to confirm
>> it was unrelated to the new commit.
>>
>> Just a thought wrt Jenkins repo, I'm not sure how often it gets updated
>> and modifies tests. If a Jenkins repo update triggers a run, then it
>> will be the only delta from the previous build, so won't that make it
>> easier to debug if there is some issue?
>>
>> Otherwise dpdk-stable and Jenkins updates will be first tested together,
>> which may make it harder to debug failures.
>>
>> I will leave it to your more better judgement the best approach wrt
>> triggering builds on jenkins updates. Either is fine with me.
>>
>> thanks,
>> Kevin.
>>
>>
>>> Best,
>>> Patrick
>>>
>>> On Thu, Mar 2, 2023 at 9:04 AM Kevin Traynor <ktraynor@redhat.com>
>> wrote:
>>>
>>>> On 02/03/2023 13:22, Lincoln Lavoie wrote:
>>>>> Hi Kevin,
>>>>>
>>>>> The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the
>>>> older
>>>>> LTS branches, because they don't include the required patches that were
>>>>> released as part of 22.11. So, you can ignore those failures.  We'll
>> make
>>>>> sure those tests are excluded from future runs on the older
>>>>> staging branches.
>>>>
>>>> ok, cool, thanks.
>>>>
>>>>>
>>>>> In terms of the two runs, I'm not sure of the cause and we'll have to
>>>> look
>>>>> into that.
>>>>>
>>>>
>>>> No problem, it's not urgent or blocking. I will keep a closer eye on the
>>>> tests ran in future and just force a re-run if necessary.
>>>>
>>>> thanks,
>>>> Kevin.
>>>>
>>>>> Cheers,
>>>>> Lincoln
>>>>>
>>>>> On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com>
>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a question about UNH CI periodic runs. I had 2x runs of CI on
>>>>>> 21.11-staging on the same commit, a few days apart.
>>>>>>
>>>>>> The issue I see is that the first test run came back all green, so I
>>>>>> assume good and I can push to 21.11 branch. However, the second run
>>>>>> comes back with additional tests that showed failures.
>>>>>>
>>>>>> So I'm wondering why there are additional tests in the second run? and
>>>>>> if/how skipped tests are being reported?
>>>>>>
>>>>>> At least with the fips tests I have seen previously so I don't think
>>>>>> they are all newly enabled tests in the days in-between.
>>>>>>
>>>>>> Details below.
>>>>>>
>>>>>> thanks,
>>>>>> Kevin.
>>>>>>
>>>>>> Initial test run:
>>>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
>>>>>>
>>>>>> Second test run:
>>>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
>>>>>>
>>>>>> Additional tests in the second run:
>>>>>> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
>>>>>> dashboard?)
>>>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest
>>>> (fail)
>>>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
>>>>>> (fail)
>>>>>> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
>>>>>> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
>>>>>> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
>>>>>> Ubuntu 20.04 ARM SVE - lpm_autotest
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>