UNH CI is offline

UNH CI is offline

[Lincoln Lavoie]

Hello All,

As discussed at the Thursday CI meeting, UNH infrastructure is now offline
for upgrades.  This means, we will not be reporting results right now,
patches will get tested once systems are restarted (but that will be a
backlog process, i.e. CI will have to catch up).

The dashboard and previous results will remain available.

During the downtime, we can not rerun tests.

We will follow up as we complete the upgrades.

Cheers,
Lincoln
-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

Re: UNH CI is offline

[Lincoln Lavoie]

Hello All,

We have completed some first steps for the major update around Jenkins.
The host's operating system has been upgraded to the latest version to pull
in security patches, etc.  Jenkins itself has also been upgraded, but is
not yet to its final desired version.  We have to update
incrementally between LTS versions, to allow Jenkins to migrate
configurations, plugins, and job data.  That process takes a little bit to
complete each time, while it processes its data and the job history.

The system is online now, and we are going to let it catch up on missed
jobs overnight.  We will be keeping a close eye on the output, but things
look stable and correct.  We don't expect Jenkins to change the actual
tests, since it just runs tests and collects results. There will be some
additional downtime, while we continue upgrading versions this week and
early next week.  If you notice anything that looks incorrect or strange,
please let us know.

That does raise a question on storage of historical results (i.e. the
results tar balls) and how long we need to keep those available. The
results (i.e. pass / fail indication) would live on in the dashboard and
database, but the actual results archives (tar balls) are stored in Jenkins
as a job.  Would limiting that storage to 4 or 6 months be acceptable to
the community?  This will reduce the future issues of upgrades and
migration, in terms of processing time required by Jenkins. A test could
always be rerun, if we needed to recreate an older set of logs/results.
I'll add this as a discussion item for the next community call.

Cheers,
Lincoln

On Mon, Dec 6, 2021 at 1:19 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:

> Hello All,
>
> As discussed at the Thursday CI meeting, UNH infrastructure is now offline
> for upgrades.  This means, we will not be reporting results right now,
> patches will get tested once systems are restarted (but that will be a
> backlog process, i.e. CI will have to catch up).
>
> The dashboard and previous results will remain available.
>
> During the downtime, we can not rerun tests.
>
> We will follow up as we complete the upgrades.
>
> Cheers,
> Lincoln
> --
> *Lincoln Lavoie*
> Principal Engineer, Broadband Technologies
> 21 Madbury Rd., Ste. 100, Durham, NH 03824
> lylavoie@iol.unh.edu
> https://www.iol.unh.edu
> +1-603-674-2755 (m)
> <https://www.iol.unh.edu>
>


-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

Re: UNH CI is offline

[Aaron Conole]

Lincoln Lavoie <lylavoie@iol.unh.edu> writes:

> Hello All,
>
> We have completed some first steps for the major update around Jenkins.
> The host's operating system has been upgraded to the latest version to pull
> in security patches, etc.  Jenkins itself has also been upgraded, but is
> not yet to its final desired version.  We have to update
> incrementally between LTS versions, to allow Jenkins to migrate
> configurations, plugins, and job data.  That process takes a little bit to
> complete each time, while it processes its data and the job history.

Makes sense, and I expected it wouldn't be a simple 5 minute process.

> The system is online now, and we are going to let it catch up on missed
> jobs overnight.  We will be keeping a close eye on the output, but things
> look stable and correct.  We don't expect Jenkins to change the actual
> tests, since it just runs tests and collects results. There will be some
> additional downtime, while we continue upgrading versions this week and
> early next week.  If you notice anything that looks incorrect or strange,
> please let us know.

Thanks.  I think right now the contribution rate is rather slow (just
because of real-life timing), so I imagine it won't be difficult to
catch up.

> That does raise a question on storage of historical results (i.e. the
> results tar balls) and how long we need to keep those available. The
> results (i.e. pass / fail indication) would live on in the dashboard and
> database, but the actual results archives (tar balls) are stored in Jenkins
> as a job.  Would limiting that storage to 4 or 6 months be acceptable to
> the community?  This will reduce the future issues of upgrades and
> migration, in terms of processing time required by Jenkins. A test could
> always be rerun, if we needed to recreate an older set of logs/results.
> I'll add this as a discussion item for the next community call.

+1 - thanks.

> Cheers,
> Lincoln
>
> On Mon, Dec 6, 2021 at 1:19 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
>
>> Hello All,
>>
>> As discussed at the Thursday CI meeting, UNH infrastructure is now offline
>> for upgrades.  This means, we will not be reporting results right now,
>> patches will get tested once systems are restarted (but that will be a
>> backlog process, i.e. CI will have to catch up).
>>
>> The dashboard and previous results will remain available.
>>
>> During the downtime, we can not rerun tests.
>>
>> We will follow up as we complete the upgrades.
>>
>> Cheers,
>> Lincoln
>> --
>> *Lincoln Lavoie*
>> Principal Engineer, Broadband Technologies
>> 21 Madbury Rd., Ste. 100, Durham, NH 03824
>> lylavoie@iol.unh.edu
>> https://www.iol.unh.edu
>> +1-603-674-2755 (m)
>> <https://www.iol.unh.edu>
>>

Re: UNH CI is offline

[Thomas Monjalon]

09/12/2021 15:33, Aaron Conole:
> Lincoln Lavoie <lylavoie@iol.unh.edu> writes:
> > That does raise a question on storage of historical results (i.e. the
> > results tar balls) and how long we need to keep those available. The
> > results (i.e. pass / fail indication) would live on in the dashboard and
> > database, but the actual results archives (tar balls) are stored in Jenkins
> > as a job.  Would limiting that storage to 4 or 6 months be acceptable to
> > the community?  This will reduce the future issues of upgrades and
> > migration, in terms of processing time required by Jenkins. A test could
> > always be rerun, if we needed to recreate an older set of logs/results.
> > I'll add this as a discussion item for the next community call.
> 
> +1 - thanks.

I think we don't need more history than current and previous releases.
So I bet 8 months is a fair amount of test history.

Re: UNH CI is offline

[Lincoln Lavoie]

Hello All,

Just a quick follow up. Jenkins upgrades have been completed and the system
is getting caught up to the patches that have come in. Please let us know
if you would like any patches rerun.  We'll also take a pass through the
history this week to look for anything that needs to be rerun as well.

Next for upgrades will be the dashboard system and our vpn / router.  The
dashboard will not impact running just, just browsing results. The vpn /
router upgrade will require us to stop the existing jobs.  We expect both
of these to be completed this week.

Cheers,
Lincoln

On Thu, Dec 9, 2021 at 11:01 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> 09/12/2021 15:33, Aaron Conole:
> > Lincoln Lavoie <lylavoie@iol.unh.edu> writes:
> > > That does raise a question on storage of historical results (i.e. the
> > > results tar balls) and how long we need to keep those available. The
> > > results (i.e. pass / fail indication) would live on in the dashboard
> and
> > > database, but the actual results archives (tar balls) are stored in
> Jenkins
> > > as a job.  Would limiting that storage to 4 or 6 months be acceptable
> to
> > > the community?  This will reduce the future issues of upgrades and
> > > migration, in terms of processing time required by Jenkins. A test
> could
> > > always be rerun, if we needed to recreate an older set of logs/results.
> > > I'll add this as a discussion item for the next community call.
> >
> > +1 - thanks.
>
> I think we don't need more history than current and previous releases.
> So I bet 8 months is a fair amount of test history.
>
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>