[dpdk-ci] What is the reliability of the community lab

[dpdk-ci] What is the reliability of the community lab

[Thomas Monjalon]

We still have some severe reliability issues with the community lab.
One under investigation, waiting for a bisect, is making the lab crashing.

Another one I would like to understand is about the ABI check.
Why the test is failing (which is normal) in Travis and GitHub
but passing in the community lab?

See this example:
https://patches.dpdk.org/project/dpdk/patch/1618283653-16510-6-git-send-email-xuemingl@nvidia.com/

Re: [dpdk-ci] [dpdklab] What is the reliability of the community lab

[Brandon Lo]

Hi Thomas,

For the ABI check failure, we were working on a few migration-related
changes that caused this issue. We have fixed the underlying cause and
the rolling migration will be completed soon. The test will be rerun
and reported back up to patchworks as well.
We are also in the process of adding more monitoring for failure
results that are not limited to only infrastructure failures, i.e.
failures caused by a test's return value will alert us in a separate
category which will allow us to validate the result.

Thanks,
Brandon

On Wed, Apr 14, 2021 at 8:10 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> We still have some severe reliability issues with the community lab.
> One under investigation, waiting for a bisect, is making the lab crashing.
>
> Another one I would like to understand is about the ABI check.
> Why the test is failing (which is normal) in Travis and GitHub
> but passing in the community lab?
>
> See this example:
> https://patches.dpdk.org/project/dpdk/patch/1618283653-16510-6-git-send-email-xuemingl@nvidia.com/
>
>


--

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

Re: [dpdk-ci] [dpdklab] What is the reliability of the community lab

[Thomas Monjalon]

15/04/2021 20:26, Brandon Lo:
> Hi Thomas,
> 
> For the ABI check failure, we were working on a few migration-related
> changes that caused this issue. We have fixed the underlying cause and
> the rolling migration will be completed soon. The test will be rerun
> and reported back up to patchworks as well.

You should avoid running tests when the platform is not ready.
It was giving false positives.

> We are also in the process of adding more monitoring for failure
> results that are not limited to only infrastructure failures, i.e.
> failures caused by a test's return value will alert us in a separate
> category which will allow us to validate the result.

The lab seems stopped currently.
There are issues for a week now.
What are the news today?


> On Wed, Apr 14, 2021 at 8:10 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > We still have some severe reliability issues with the community lab.
> > One under investigation, waiting for a bisect, is making the lab crashing.
> >
> > Another one I would like to understand is about the ABI check.
> > Why the test is failing (which is normal) in Travis and GitHub
> > but passing in the community lab?
> >
> > See this example:
> > https://patches.dpdk.org/project/dpdk/patch/1618283653-16510-6-git-send-email-xuemingl@nvidia.com/

Re: [dpdk-ci] [dpdklab] What is the reliability of the community lab

[Brandon Lo]

We have completed our rolling migration to containers, so all of our
compilation jobs are running and catching up.
Our team has also worked on getting the DTS side of things back up and
running since the DTS issue had caused the pipelines to hang a bit.

Thanks,
Brandon

On Thu, Apr 15, 2021 at 2:33 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 15/04/2021 20:26, Brandon Lo:
> > Hi Thomas,
> >
> > For the ABI check failure, we were working on a few migration-related
> > changes that caused this issue. We have fixed the underlying cause and
> > the rolling migration will be completed soon. The test will be rerun
> > and reported back up to patchworks as well.
>
> You should avoid running tests when the platform is not ready.
> It was giving false positives.
>
> > We are also in the process of adding more monitoring for failure
> > results that are not limited to only infrastructure failures, i.e.
> > failures caused by a test's return value will alert us in a separate
> > category which will allow us to validate the result.
>
> The lab seems stopped currently.
> There are issues for a week now.
> What are the news today?
>
>
> > On Wed, Apr 14, 2021 at 8:10 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > We still have some severe reliability issues with the community lab.
> > > One under investigation, waiting for a bisect, is making the lab crashing.
> > >
> > > Another one I would like to understand is about the ABI check.
> > > Why the test is failing (which is normal) in Travis and GitHub
> > > but passing in the community lab?
> > >
> > > See this example:
> > > https://patches.dpdk.org/project/dpdk/patch/1618283653-16510-6-git-send-email-xuemingl@nvidia.com/
>
>
>


--

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu


On Thu, Apr 15, 2021 at 2:33 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 15/04/2021 20:26, Brandon Lo:
> > Hi Thomas,
> >
> > For the ABI check failure, we were working on a few migration-related
> > changes that caused this issue. We have fixed the underlying cause and
> > the rolling migration will be completed soon. The test will be rerun
> > and reported back up to patchworks as well.
>
> You should avoid running tests when the platform is not ready.
> It was giving false positives.
>
> > We are also in the process of adding more monitoring for failure
> > results that are not limited to only infrastructure failures, i.e.
> > failures caused by a test's return value will alert us in a separate
> > category which will allow us to validate the result.
>
> The lab seems stopped currently.
> There are issues for a week now.
> What are the news today?
>
>
> > On Wed, Apr 14, 2021 at 8:10 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > We still have some severe reliability issues with the community lab.
> > > One under investigation, waiting for a bisect, is making the lab crashing.
> > >
> > > Another one I would like to understand is about the ABI check.
> > > Why the test is failing (which is normal) in Travis and GitHub
> > > but passing in the community lab?
> > >
> > > See this example:
> > > https://patches.dpdk.org/project/dpdk/patch/1618283653-16510-6-git-send-email-xuemingl@nvidia.com/
>
>
>


--

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu