From: Lincoln Lavoie <lylavoie@iol.unh.edu>
To: Thomas Monjalon <thomas@monjalon.net>
Cc: David Liu <dliu@iol.unh.edu>,
David Marchand <david.marchand@redhat.com>,
dpdklab <dpdklab@iol.unh.edu>, Aaron Conole <aconole@redhat.com>,
ci@dpdk.org, "Yigit, Ferruh" <ferruh.yigit@intel.com>
Subject: Re: [dpdk-ci] No space left on some UNH system
Date: Tue, 23 Feb 2021 13:59:27 -0500 [thread overview]
Message-ID: <CAOE1vsMW4NFNx9LmOP8-L36cfrhFaNkFXXFwQmrewyQ3N6VfBA@mail.gmail.com> (raw)
In-Reply-To: <6931268.JWQ3VreUG3@thomas>
[-- Attachment #1: Type: text/plain, Size: 1600 bytes --]
Hi Thomas,
Yes, we are adding that VM to our internal monitoring system, based on
icinga2. We are investigating what consumed the disk space, as it is just
a Jenkins worker and should not have had disk space being consumed over
time, as the pipelines should be either deleting temporary artifacts or
moving longer lived artifacts off the workers to the headnodes (i.e. logs
and such), so those items become available through the dashboard, etc.
Our internal monitoring actually utilizes Google Chat, through our company
Gsuite account, so it can easily generate notifications to our phones, etc.
Cheers,
LIncoln
On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> 23/02/2021 19:04, David Liu:
> > On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 23/02/2021 17:21, David Liu:
> > > > Hi David,
> > > >
> > > > This is an actual issue caused by the machine ran out of space.
> > > >
> > > > I had free up some spaces in the machine.
> > > >
> > > > Please let me know if this happens again.
> > >
> > > Instead of waiting for it to happen again,
> > > please could you monitor the disk space of the machines?
> > >
> > > Yes, I will also be monitoring the pipeline result and disk space.
>
> What does it mean? Will you make sure an alert is automatically sent
> to dpdklab@iol.unh.edu when a space is below a threshold?
>
>
>
--
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>
[-- Attachment #2: Type: text/html, Size: 3311 bytes --]
prev parent reply other threads:[~2021-02-23 19:02 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-23 8:31 David Marchand
2021-02-23 14:04 ` Lincoln Lavoie
2021-02-23 16:21 ` David Liu
2021-02-23 17:56 ` Thomas Monjalon
2021-02-23 18:04 ` David Liu
2021-02-23 18:06 ` Thomas Monjalon
2021-02-23 18:59 ` Lincoln Lavoie [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOE1vsMW4NFNx9LmOP8-L36cfrhFaNkFXXFwQmrewyQ3N6VfBA@mail.gmail.com \
--to=lylavoie@iol.unh.edu \
--cc=aconole@redhat.com \
--cc=ci@dpdk.org \
--cc=david.marchand@redhat.com \
--cc=dliu@iol.unh.edu \
--cc=dpdklab@iol.unh.edu \
--cc=ferruh.yigit@intel.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).