UNH CI skipped tests

UNH CI skipped tests

[Kevin Traynor]

Hi,

I have a question about UNH CI periodic runs. I had 2x runs of CI on 
21.11-staging on the same commit, a few days apart.

The issue I see is that the first test run came back all green, so I 
assume good and I can push to 21.11 branch. However, the second run 
comes back with additional tests that showed failures.

So I'm wondering why there are additional tests in the second run? and 
if/how skipped tests are being reported?

At least with the fips tests I have seen previously so I don't think 
they are all newly enabled tests in the days in-between.

Details below.

thanks,
Kevin.

Initial test run:
https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/

Second test run:
https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/

Additional tests in the second run:
Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in dashboard?)
NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest (fail)
NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest 
(fail)
Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
Ubuntu 20.04 ARM SVE - lpm_autotest

Re: UNH CI skipped tests

[Lincoln Lavoie]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

Hi Kevin,

The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the older
LTS branches, because they don't include the required patches that were
released as part of 22.11. So, you can ignore those failures.  We'll make
sure those tests are excluded from future runs on the older
staging branches.

In terms of the two runs, I'm not sure of the cause and we'll have to look
into that.

Cheers,
Lincoln

On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com> wrote:

> Hi,
>
> I have a question about UNH CI periodic runs. I had 2x runs of CI on
> 21.11-staging on the same commit, a few days apart.
>
> The issue I see is that the first test run came back all green, so I
> assume good and I can push to 21.11 branch. However, the second run
> comes back with additional tests that showed failures.
>
> So I'm wondering why there are additional tests in the second run? and
> if/how skipped tests are being reported?
>
> At least with the fips tests I have seen previously so I don't think
> they are all newly enabled tests in the days in-between.
>
> Details below.
>
> thanks,
> Kevin.
>
> Initial test run:
> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
>
> Second test run:
> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
>
> Additional tests in the second run:
> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
> dashboard?)
> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest (fail)
> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
> (fail)
> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
> Ubuntu 20.04 ARM SVE - lpm_autotest
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 3731 bytes --]

Re: UNH CI skipped tests

[Kevin Traynor]

On 02/03/2023 13:22, Lincoln Lavoie wrote:
> Hi Kevin,
> 
> The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the older
> LTS branches, because they don't include the required patches that were
> released as part of 22.11. So, you can ignore those failures.  We'll make
> sure those tests are excluded from future runs on the older
> staging branches.

ok, cool, thanks.

> 
> In terms of the two runs, I'm not sure of the cause and we'll have to look
> into that.
> 

No problem, it's not urgent or blocking. I will keep a closer eye on the 
tests ran in future and just force a re-run if necessary.

thanks,
Kevin.

> Cheers,
> Lincoln
> 
> On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com> wrote:
> 
>> Hi,
>>
>> I have a question about UNH CI periodic runs. I had 2x runs of CI on
>> 21.11-staging on the same commit, a few days apart.
>>
>> The issue I see is that the first test run came back all green, so I
>> assume good and I can push to 21.11 branch. However, the second run
>> comes back with additional tests that showed failures.
>>
>> So I'm wondering why there are additional tests in the second run? and
>> if/how skipped tests are being reported?
>>
>> At least with the fips tests I have seen previously so I don't think
>> they are all newly enabled tests in the days in-between.
>>
>> Details below.
>>
>> thanks,
>> Kevin.
>>
>> Initial test run:
>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
>>
>> Second test run:
>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
>>
>> Additional tests in the second run:
>> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
>> dashboard?)
>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest (fail)
>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
>> (fail)
>> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
>> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
>> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
>> Ubuntu 20.04 ARM SVE - lpm_autotest
>>
>>
> 

Re: UNH CI skipped tests

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 3122 bytes --]

Hi Kevin,

The FIPS test has the same issue as the cryptodev tests - it cannot run on
21.11 due to a lacking dependency. We disabled the testing on the normal
21.11 branch but I missed 21.11-staging. Sorry about the oversight. All 3
tests are now disabled on 21.11-staging.

The reason the two runs happened is that we were inadvertently polling
DPDK-Stable AND our jenkinsfile repo for commits, and triggering new builds
for commits to either repo. I have disabled polling to our Jenkinsfile
repo, which should limit new builds to just commits to DPDK-Stable.

Best,
Patrick

On Thu, Mar 2, 2023 at 9:04 AM Kevin Traynor <ktraynor@redhat.com> wrote:

> On 02/03/2023 13:22, Lincoln Lavoie wrote:
> > Hi Kevin,
> >
> > The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the
> older
> > LTS branches, because they don't include the required patches that were
> > released as part of 22.11. So, you can ignore those failures.  We'll make
> > sure those tests are excluded from future runs on the older
> > staging branches.
>
> ok, cool, thanks.
>
> >
> > In terms of the two runs, I'm not sure of the cause and we'll have to
> look
> > into that.
> >
>
> No problem, it's not urgent or blocking. I will keep a closer eye on the
> tests ran in future and just force a re-run if necessary.
>
> thanks,
> Kevin.
>
> > Cheers,
> > Lincoln
> >
> > On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com>
> wrote:
> >
> >> Hi,
> >>
> >> I have a question about UNH CI periodic runs. I had 2x runs of CI on
> >> 21.11-staging on the same commit, a few days apart.
> >>
> >> The issue I see is that the first test run came back all green, so I
> >> assume good and I can push to 21.11 branch. However, the second run
> >> comes back with additional tests that showed failures.
> >>
> >> So I'm wondering why there are additional tests in the second run? and
> >> if/how skipped tests are being reported?
> >>
> >> At least with the fips tests I have seen previously so I don't think
> >> they are all newly enabled tests in the days in-between.
> >>
> >> Details below.
> >>
> >> thanks,
> >> Kevin.
> >>
> >> Initial test run:
> >> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
> >>
> >> Second test run:
> >> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
> >>
> >> Additional tests in the second run:
> >> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
> >> dashboard?)
> >> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest
> (fail)
> >> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
> >> (fail)
> >> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
> >> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
> >> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
> >> Ubuntu 20.04 ARM SVE - lpm_autotest
> >>
> >>
> >
>
>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 6063 bytes --]

Re: UNH CI skipped tests

[Kevin Traynor]

On 02/03/2023 18:49, Patrick Robb wrote:
> Hi Kevin,
> 

Hi Patrick,

> The FIPS test has the same issue as the cryptodev tests - it cannot run on
> 21.11 due to a lacking dependency. We disabled the testing on the normal
> 21.11 branch but I missed 21.11-staging. Sorry about the oversight. All 3
> tests are now disabled on 21.11-staging.
> 

Cool, thanks.

> The reason the two runs happened is that we were inadvertently polling
> DPDK-Stable AND our jenkinsfile repo for commits, and triggering new builds
> for commits to either repo. I have disabled polling to our Jenkinsfile
> repo, which should limit new builds to just commits to DPDK-Stable.
> 

I'm not sure that was the reason for the second test run. I did that 
deliberately...

- I ran the initial run on the original commit (pass)
- I added a new dependency/build impacting commit and saw new tests and 
failures
- I wanted to understand if the new tests and failures were related to 
the single commit allowing additional tests to run, so...
- I force pushed back to the original commit for a second run on the 
original commit. There I also saw the new tests so was able to confirm 
it was unrelated to the new commit.

Just a thought wrt Jenkins repo, I'm not sure how often it gets updated 
and modifies tests. If a Jenkins repo update triggers a run, then it 
will be the only delta from the previous build, so won't that make it 
easier to debug if there is some issue?

Otherwise dpdk-stable and Jenkins updates will be first tested together, 
which may make it harder to debug failures.

I will leave it to your more better judgement the best approach wrt 
triggering builds on jenkins updates. Either is fine with me.

thanks,
Kevin.


> Best,
> Patrick
> 
> On Thu, Mar 2, 2023 at 9:04 AM Kevin Traynor <ktraynor@redhat.com> wrote:
> 
>> On 02/03/2023 13:22, Lincoln Lavoie wrote:
>>> Hi Kevin,
>>>
>>> The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the
>> older
>>> LTS branches, because they don't include the required patches that were
>>> released as part of 22.11. So, you can ignore those failures.  We'll make
>>> sure those tests are excluded from future runs on the older
>>> staging branches.
>>
>> ok, cool, thanks.
>>
>>>
>>> In terms of the two runs, I'm not sure of the cause and we'll have to
>> look
>>> into that.
>>>
>>
>> No problem, it's not urgent or blocking. I will keep a closer eye on the
>> tests ran in future and just force a re-run if necessary.
>>
>> thanks,
>> Kevin.
>>
>>> Cheers,
>>> Lincoln
>>>
>>> On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com>
>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a question about UNH CI periodic runs. I had 2x runs of CI on
>>>> 21.11-staging on the same commit, a few days apart.
>>>>
>>>> The issue I see is that the first test run came back all green, so I
>>>> assume good and I can push to 21.11 branch. However, the second run
>>>> comes back with additional tests that showed failures.
>>>>
>>>> So I'm wondering why there are additional tests in the second run? and
>>>> if/how skipped tests are being reported?
>>>>
>>>> At least with the fips tests I have seen previously so I don't think
>>>> they are all newly enabled tests in the days in-between.
>>>>
>>>> Details below.
>>>>
>>>> thanks,
>>>> Kevin.
>>>>
>>>> Initial test run:
>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
>>>>
>>>> Second test run:
>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
>>>>
>>>> Additional tests in the second run:
>>>> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
>>>> dashboard?)
>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest
>> (fail)
>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
>>>> (fail)
>>>> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
>>>> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
>>>> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
>>>> Ubuntu 20.04 ARM SVE - lpm_autotest
>>>>
>>>>
>>>
>>
>>
> 

Re: UNH CI skipped tests

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 5465 bytes --]

Hi Kevin,

My thinking was that allowing new testruns simply due to a jenkinsfile
change might produce testrun results which yielded no value for you or
others. But, it sounds like you do not feel this way, and I agree having a
testrun kick off due to Jenkinsfile changes would provide some debugging
information to us UNH folks. So, since it sounds like it's alright with
you, I'm going to go ahead and undo my previous changes on this front.

And yes I'm sure all of those new tests popping up all of a sudden was
unexpected... Some of those were tests run on a server which was
temporarily down (ARM-Gigabyte), so once that went back online we were able
to run additional tests. And then you know the others which were failing
were never intended to run on 21.11.

Thanks,
Patrick



On Fri, Mar 3, 2023 at 4:59 AM Kevin Traynor <ktraynor@redhat.com> wrote:

> On 02/03/2023 18:49, Patrick Robb wrote:
> > Hi Kevin,
> >
>
> Hi Patrick,
>
> > The FIPS test has the same issue as the cryptodev tests - it cannot run
> on
> > 21.11 due to a lacking dependency. We disabled the testing on the normal
> > 21.11 branch but I missed 21.11-staging. Sorry about the oversight. All 3
> > tests are now disabled on 21.11-staging.
> >
>
> Cool, thanks.
>
> > The reason the two runs happened is that we were inadvertently polling
> > DPDK-Stable AND our jenkinsfile repo for commits, and triggering new
> builds
> > for commits to either repo. I have disabled polling to our Jenkinsfile
> > repo, which should limit new builds to just commits to DPDK-Stable.
> >
>
> I'm not sure that was the reason for the second test run. I did that
> deliberately...
>
> - I ran the initial run on the original commit (pass)
> - I added a new dependency/build impacting commit and saw new tests and
> failures
> - I wanted to understand if the new tests and failures were related to
> the single commit allowing additional tests to run, so...
> - I force pushed back to the original commit for a second run on the
> original commit. There I also saw the new tests so was able to confirm
> it was unrelated to the new commit.
>
> Just a thought wrt Jenkins repo, I'm not sure how often it gets updated
> and modifies tests. If a Jenkins repo update triggers a run, then it
> will be the only delta from the previous build, so won't that make it
> easier to debug if there is some issue?
>
> Otherwise dpdk-stable and Jenkins updates will be first tested together,
> which may make it harder to debug failures.
>
> I will leave it to your more better judgement the best approach wrt
> triggering builds on jenkins updates. Either is fine with me.
>
> thanks,
> Kevin.
>
>
> > Best,
> > Patrick
> >
> > On Thu, Mar 2, 2023 at 9:04 AM Kevin Traynor <ktraynor@redhat.com>
> wrote:
> >
> >> On 02/03/2023 13:22, Lincoln Lavoie wrote:
> >>> Hi Kevin,
> >>>
> >>> The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the
> >> older
> >>> LTS branches, because they don't include the required patches that were
> >>> released as part of 22.11. So, you can ignore those failures.  We'll
> make
> >>> sure those tests are excluded from future runs on the older
> >>> staging branches.
> >>
> >> ok, cool, thanks.
> >>
> >>>
> >>> In terms of the two runs, I'm not sure of the cause and we'll have to
> >> look
> >>> into that.
> >>>
> >>
> >> No problem, it's not urgent or blocking. I will keep a closer eye on the
> >> tests ran in future and just force a re-run if necessary.
> >>
> >> thanks,
> >> Kevin.
> >>
> >>> Cheers,
> >>> Lincoln
> >>>
> >>> On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com>
> >> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have a question about UNH CI periodic runs. I had 2x runs of CI on
> >>>> 21.11-staging on the same commit, a few days apart.
> >>>>
> >>>> The issue I see is that the first test run came back all green, so I
> >>>> assume good and I can push to 21.11 branch. However, the second run
> >>>> comes back with additional tests that showed failures.
> >>>>
> >>>> So I'm wondering why there are additional tests in the second run? and
> >>>> if/how skipped tests are being reported?
> >>>>
> >>>> At least with the fips tests I have seen previously so I don't think
> >>>> they are all newly enabled tests in the days in-between.
> >>>>
> >>>> Details below.
> >>>>
> >>>> thanks,
> >>>> Kevin.
> >>>>
> >>>> Initial test run:
> >>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
> >>>>
> >>>> Second test run:
> >>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
> >>>>
> >>>> Additional tests in the second run:
> >>>> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
> >>>> dashboard?)
> >>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest
> >> (fail)
> >>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
> >>>> (fail)
> >>>> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
> >>>> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
> >>>> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
> >>>> Ubuntu 20.04 ARM SVE - lpm_autotest
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>

-- 

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 9118 bytes --]

Re: UNH CI skipped tests

[Kevin Traynor]

On 03/03/2023 14:33, Patrick Robb wrote:
> Hi Kevin,
> 
> My thinking was that allowing new testruns simply due to a jenkinsfile
> change might produce testrun results which yielded no value for you or
> others. But, it sounds like you do not feel this way, and I agree having a
> testrun kick off due to Jenkinsfile changes would provide some debugging
> information to us UNH folks. So, since it sounds like it's alright with
> you, I'm going to go ahead and undo my previous changes on this front.
> 

Fine by me.

> And yes I'm sure all of those new tests popping up all of a sudden was
> unexpected... Some of those were tests run on a server which was
> temporarily down (ARM-Gigabyte), so once that went back online we were able
> to run additional tests. And then you know the others which were failing
> were never intended to run on 21.11.
> 

ok, I think that's all the differences explained.

It might be worth flagging in the dashboard if some valid tests were 
skipped for a case like this. I've no idea if that's easy to do, so not 
sure if it's worth the effort.

Thanks Lincoln and Patrick for the quick response and explanations,

Kevin.

> Thanks,
> Patrick
> 
> 
> 
> On Fri, Mar 3, 2023 at 4:59 AM Kevin Traynor <ktraynor@redhat.com> wrote:
> 
>> On 02/03/2023 18:49, Patrick Robb wrote:
>>> Hi Kevin,
>>>
>>
>> Hi Patrick,
>>
>>> The FIPS test has the same issue as the cryptodev tests - it cannot run
>> on
>>> 21.11 due to a lacking dependency. We disabled the testing on the normal
>>> 21.11 branch but I missed 21.11-staging. Sorry about the oversight. All 3
>>> tests are now disabled on 21.11-staging.
>>>
>>
>> Cool, thanks.
>>
>>> The reason the two runs happened is that we were inadvertently polling
>>> DPDK-Stable AND our jenkinsfile repo for commits, and triggering new
>> builds
>>> for commits to either repo. I have disabled polling to our Jenkinsfile
>>> repo, which should limit new builds to just commits to DPDK-Stable.
>>>
>>
>> I'm not sure that was the reason for the second test run. I did that
>> deliberately...
>>
>> - I ran the initial run on the original commit (pass)
>> - I added a new dependency/build impacting commit and saw new tests and
>> failures
>> - I wanted to understand if the new tests and failures were related to
>> the single commit allowing additional tests to run, so...
>> - I force pushed back to the original commit for a second run on the
>> original commit. There I also saw the new tests so was able to confirm
>> it was unrelated to the new commit.
>>
>> Just a thought wrt Jenkins repo, I'm not sure how often it gets updated
>> and modifies tests. If a Jenkins repo update triggers a run, then it
>> will be the only delta from the previous build, so won't that make it
>> easier to debug if there is some issue?
>>
>> Otherwise dpdk-stable and Jenkins updates will be first tested together,
>> which may make it harder to debug failures.
>>
>> I will leave it to your more better judgement the best approach wrt
>> triggering builds on jenkins updates. Either is fine with me.
>>
>> thanks,
>> Kevin.
>>
>>
>>> Best,
>>> Patrick
>>>
>>> On Thu, Mar 2, 2023 at 9:04 AM Kevin Traynor <ktraynor@redhat.com>
>> wrote:
>>>
>>>> On 02/03/2023 13:22, Lincoln Lavoie wrote:
>>>>> Hi Kevin,
>>>>>
>>>>> The FIPS and crypto (ZUC / SNOW) testing shouldn't be running on the
>>>> older
>>>>> LTS branches, because they don't include the required patches that were
>>>>> released as part of 22.11. So, you can ignore those failures.  We'll
>> make
>>>>> sure those tests are excluded from future runs on the older
>>>>> staging branches.
>>>>
>>>> ok, cool, thanks.
>>>>
>>>>>
>>>>> In terms of the two runs, I'm not sure of the cause and we'll have to
>>>> look
>>>>> into that.
>>>>>
>>>>
>>>> No problem, it's not urgent or blocking. I will keep a closer eye on the
>>>> tests ran in future and just force a re-run if necessary.
>>>>
>>>> thanks,
>>>> Kevin.
>>>>
>>>>> Cheers,
>>>>> Lincoln
>>>>>
>>>>> On Thu, Mar 2, 2023 at 5:04 AM Kevin Traynor <ktraynor@redhat.com>
>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a question about UNH CI periodic runs. I had 2x runs of CI on
>>>>>> 21.11-staging on the same commit, a few days apart.
>>>>>>
>>>>>> The issue I see is that the first test run came back all green, so I
>>>>>> assume good and I can push to 21.11 branch. However, the second run
>>>>>> comes back with additional tests that showed failures.
>>>>>>
>>>>>> So I'm wondering why there are additional tests in the second run? and
>>>>>> if/how skipped tests are being reported?
>>>>>>
>>>>>> At least with the fips tests I have seen previously so I don't think
>>>>>> they are all newly enabled tests in the days in-between.
>>>>>>
>>>>>> Details below.
>>>>>>
>>>>>> thanks,
>>>>>> Kevin.
>>>>>>
>>>>>> Initial test run:
>>>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23476/
>>>>>>
>>>>>> Second test run:
>>>>>> https://dpdkdashboard.iol.unh.edu/results/dashboard/tarballs/23560/
>>>>>>
>>>>>> Additional tests in the second run:
>>>>>> Ubuntu 20.04 VM - dpdk_fips_validation (warning, not reported in
>>>>>> dashboard?)
>>>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_zuc_autotest
>>>> (fail)
>>>>>> NA NA (Linux container host) 10000 Mbps - cryptodev_sw_snow3g_autotest
>>>>>> (fail)
>>>>>> Arm Intel XL710-QDA2 4000 Mbps - lpm_autotest, unit_tests_mbuf
>>>>>> Arm Broadcom 25000 Mbps - unit_tests_mbuf,nic_single_core_tests
>>>>>> Ubuntu 20.04 ARM GCC Cross compile - dpdk_meson_compile
>>>>>> Ubuntu 20.04 ARM SVE - lpm_autotest
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>