Re: UNH CI is offline

[Aaron Conole <aconole@redhat.com>]

Lincoln Lavoie <lylavoie@iol.unh.edu> writes:

> Hello All,
>
> We have completed some first steps for the major update around Jenkins.
> The host's operating system has been upgraded to the latest version to pull
> in security patches, etc.  Jenkins itself has also been upgraded, but is
> not yet to its final desired version.  We have to update
> incrementally between LTS versions, to allow Jenkins to migrate
> configurations, plugins, and job data.  That process takes a little bit to
> complete each time, while it processes its data and the job history.

Makes sense, and I expected it wouldn't be a simple 5 minute process.

> The system is online now, and we are going to let it catch up on missed
> jobs overnight.  We will be keeping a close eye on the output, but things
> look stable and correct.  We don't expect Jenkins to change the actual
> tests, since it just runs tests and collects results. There will be some
> additional downtime, while we continue upgrading versions this week and
> early next week.  If you notice anything that looks incorrect or strange,
> please let us know.

Thanks.  I think right now the contribution rate is rather slow (just
because of real-life timing), so I imagine it won't be difficult to
catch up.

> That does raise a question on storage of historical results (i.e. the
> results tar balls) and how long we need to keep those available. The
> results (i.e. pass / fail indication) would live on in the dashboard and
> database, but the actual results archives (tar balls) are stored in Jenkins
> as a job.  Would limiting that storage to 4 or 6 months be acceptable to
> the community?  This will reduce the future issues of upgrades and
> migration, in terms of processing time required by Jenkins. A test could
> always be rerun, if we needed to recreate an older set of logs/results.
> I'll add this as a discussion item for the next community call.

+1 - thanks.

> Cheers,
> Lincoln
>
> On Mon, Dec 6, 2021 at 1:19 PM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:
>
>> Hello All,
>>
>> As discussed at the Thursday CI meeting, UNH infrastructure is now offline
>> for upgrades.  This means, we will not be reporting results right now,
>> patches will get tested once systems are restarted (but that will be a
>> backlog process, i.e. CI will have to catch up).
>>
>> The dashboard and previous results will remain available.
>>
>> During the downtime, we can not rerun tests.
>>
>> We will follow up as we complete the upgrades.
>>
>> Cheers,
>> Lincoln
>> --
>> *Lincoln Lavoie*
>> Principal Engineer, Broadband Technologies
>> 21 Madbury Rd., Ste. 100, Durham, NH 03824
>> lylavoie@iol.unh.edu
>> https://www.iol.unh.edu
>> +1-603-674-2755 (m)
>> <https://www.iol.unh.edu>
>>