From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 94F43A034F for ; Tue, 23 Feb 2021 20:02:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 600774068C; Tue, 23 Feb 2021 20:02:00 +0100 (CET) Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) by mails.dpdk.org (Postfix) with ESMTP id A03B74068B for ; Tue, 23 Feb 2021 20:01:59 +0100 (CET) Received: by mail-ed1-f46.google.com with SMTP id s11so27081033edd.5 for ; Tue, 23 Feb 2021 11:01:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7BUAUG/ABp0bY5KXOccQcBvXela2YIoCBCrM0zhNlgM=; b=T15L+tY9v3elkIeqFRXQneZyVAdL4FDIfw6dgaKeW4i5PMsO+mV9LMzK0P0rj8wsYk 1ri4Qm1Jh74Xri0U8tIcM9ca2vo9+bktJjBDhrpBmCcegV783OlBtIHtKKpRflsUY7vk y5CLCifMEOHylGDbvZetTFwDpUYtOW59Lcvos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7BUAUG/ABp0bY5KXOccQcBvXela2YIoCBCrM0zhNlgM=; b=fDnSX1aROhgLcPkqvhcp17tuEw2FFYUEbRJPYjO3c5O7Fj6yV9Ry1w2/EvBILFi79I FtAC1/J14BXaHVFze7HEz30ybxAmQ/JCmLDsmnU4387MaU1E1XELYm52I9qzG5iK8yaP JzJvsGMWCeV2Iet/DjAQsVHS64T7HlEfSBk4idaj7naVq7QuOM2y6+M2TLE9Nuzl3puS YKZefLSsVEpXtKfb4IMRxf9IjcMfoPt9VBGJSTEJkeY8wurMUx1z+xKueRRpY+66YSxj Xy4DoXb8h69L1HbimvM8e6tWpuSzespv8jDGq7N85GXXJUmS2OQeKnFFlqK1ocnZjOoI qaCw== X-Gm-Message-State: AOAM530FZtQx+oOmvWFUA2AfL6jDlBWauDnisU1FMtrgRD4xjAW5CDGE Hhm+XjRLKvodld6lVYGHuDKjYLuW38BjwcrLbnEaWg== X-Google-Smtp-Source: ABdhPJzPhvPiGPReGX0KUPynjipce+KqkfcJmO5T9RhBZzBta/gKGZ54kxVvzFsYlkl+iOzH4HTouKNrelMxiP45fXg= X-Received: by 2002:a05:6402:1d9b:: with SMTP id dk27mr14409062edb.209.1614106919254; Tue, 23 Feb 2021 11:01:59 -0800 (PST) MIME-Version: 1.0 References: <2672243.nfMCO9gnuM@thomas> <6931268.JWQ3VreUG3@thomas> In-Reply-To: <6931268.JWQ3VreUG3@thomas> From: Lincoln Lavoie Date: Tue, 23 Feb 2021 13:59:27 -0500 Message-ID: To: Thomas Monjalon Cc: David Liu , David Marchand , dpdklab , Aaron Conole , ci@dpdk.org, "Yigit, Ferruh" Content-Type: multipart/alternative; boundary="0000000000008d549605bc058d28" Subject: Re: [dpdk-ci] No space left on some UNH system X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --0000000000008d549605bc058d28 Content-Type: text/plain; charset="UTF-8" Hi Thomas, Yes, we are adding that VM to our internal monitoring system, based on icinga2. We are investigating what consumed the disk space, as it is just a Jenkins worker and should not have had disk space being consumed over time, as the pipelines should be either deleting temporary artifacts or moving longer lived artifacts off the workers to the headnodes (i.e. logs and such), so those items become available through the dashboard, etc. Our internal monitoring actually utilizes Google Chat, through our company Gsuite account, so it can easily generate notifications to our phones, etc. Cheers, LIncoln On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon wrote: > 23/02/2021 19:04, David Liu: > > On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon > > wrote: > > > > > 23/02/2021 17:21, David Liu: > > > > Hi David, > > > > > > > > This is an actual issue caused by the machine ran out of space. > > > > > > > > I had free up some spaces in the machine. > > > > > > > > Please let me know if this happens again. > > > > > > Instead of waiting for it to happen again, > > > please could you monitor the disk space of the machines? > > > > > > Yes, I will also be monitoring the pipeline result and disk space. > > What does it mean? Will you make sure an alert is automatically sent > to dpdklab@iol.unh.edu when a space is below a threshold? > > > -- *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) --0000000000008d549605bc058d28 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi = Thomas,

Yes, we are addin= g that VM to our internal monitoring system, based on icinga2.=C2=A0 We are= investigating what consumed the disk space, as it is just a Jenkins worker= and should not have had disk space being consumed over time, as the pipeli= nes should be either deleting=C2=A0temporary artifacts or moving longer liv= ed artifacts off the workers to the headnodes=C2=A0(i.e. logs and such), so= those items become available through the dashboard, etc.

Our internal monitoring actually utilize= s Google Chat, through our company Gsuite account, so it can easily generat= e notifications to our phones, etc.

Cheers,
LIncoln

On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon <thomas@monjalon.net> wrote:
23/02/2021 19:04, David= Liu:
> On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
> > 23/02/2021 17:21, David Liu:
> > > Hi David,
> > >
> > > This is an actual issue caused by the machine ran out of spa= ce.
> > >
> > > I had free up some spaces in the machine.
> > >
> > > Please let me know if this happens again.
> >
> > Instead of waiting for it to happen again,
> > please could you monitor the disk space of the machines?
> >
> > Yes, I will also be monitoring the pipeline result and disk space= .

What does it mean? Will you make sure an alert is automatically sent
to dpdklab@iol.unh= .edu when a space is below a threshold?




--
Lincoln Lavoie
Prin= cipal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, = Durham, NH 03824
+1-603-674-= 2755 (m)

--0000000000008d549605bc058d28--