Hi Thomas,

Yes, we are adding that VM to our internal monitoring system, based on icinga2.  We are investigating what consumed the disk space, as it is just a Jenkins worker and should not have had disk space being consumed over time, as the pipelines should be either deleting temporary artifacts or moving longer lived artifacts off the workers to the headnodes (i.e. logs and such), so those items become available through the dashboard, etc.

Our internal monitoring actually utilizes Google Chat, through our company Gsuite account, so it can easily generate notifications to our phones, etc.

Cheers,
LIncoln

On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon <thomas@monjalon.net> wrote:
23/02/2021 19:04, David Liu:
> On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
> > 23/02/2021 17:21, David Liu:
> > > Hi David,
> > >
> > > This is an actual issue caused by the machine ran out of space.
> > >
> > > I had free up some spaces in the machine.
> > >
> > > Please let me know if this happens again.
> >
> > Instead of waiting for it to happen again,
> > please could you monitor the disk space of the machines?
> >
> > Yes, I will also be monitoring the pipeline result and disk space.

What does it mean? Will you make sure an alert is automatically sent
to dpdklab@iol.unh.edu when a space is below a threshold?




--
Lincoln Lavoie
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
+1-603-674-2755 (m)