DPDK CI discussions
 help / color / mirror / Atom feed
* [dpdk-ci] No space left on some UNH system
@ 2021-02-23  8:31 David Marchand
  2021-02-23 14:04 ` Lincoln Lavoie
  0 siblings, 1 reply; 7+ messages in thread
From: David Marchand @ 2021-02-23  8:31 UTC (permalink / raw)
  To: dpdklab; +Cc: Aaron Conole, Lincoln Lavoie, ci, Thomas Monjalon, Yigit, Ferruh

Hello,

Looking at http://mails.dpdk.org/archives/test-report/2021-February/179652.html,
the logs contain "No space left" messages.

It seems fine atm but I don't know if we are just at the edge of the pit :-).
Can this be investigated so that it does not repeat?

Thanks.

"""
[3370/3375] Compiling C object
'drivers/a715181@@rte_net_mlx5@sha/meson-generated_.._rte_net_mlx5.pmd.c.o'.
[3371/3375] Compiling C object
'drivers/a715181@@rte_net_mlx5@sta/meson-generated_.._rte_net_mlx5.pmd.c.o'.
[3372/3375] Linking static target drivers/librte_net_mlx5.a.
[3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom command.
[3374/3375] Linking target drivers/librte_net_mlx5.so.21.2.
[3375/3375] Linking target buildtools/chkincs/chkincs.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 342, in do_install
    self.install_targets(d)
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 449, in install_targets
    self.do_copyfile(fname, outname)
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 257, in do_copyfile
    shutil.copyfile(from_file, to_file)
  File "/usr/lib/python3.6/shutil.py", line 122, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj
    fdst.write(buf)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 514, in run
    installer.do_install(datafilename)
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 349, in do_install
    print('Nothing to install.')
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 71, in __exit__
    append_to_log(self.lf, d)
  File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 81, in append_to_log
    lf.flush()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:
"""


-- 
David Marchand


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23  8:31 [dpdk-ci] No space left on some UNH system David Marchand
@ 2021-02-23 14:04 ` Lincoln Lavoie
  2021-02-23 16:21   ` David Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Lincoln Lavoie @ 2021-02-23 14:04 UTC (permalink / raw)
  To: David Marchand; +Cc: dpdklab, Aaron Conole, ci, Thomas Monjalon, Yigit, Ferruh

[-- Attachment #1: Type: text/plain, Size: 2682 bytes --]

Hi David,

That is on the agenda for the team to look at today.  David (from our team)
will confirm back with his findings.

Cheers,
Lincoln

On Tue, Feb 23, 2021 at 3:31 AM David Marchand <david.marchand@redhat.com>
wrote:

> Hello,
>
> Looking at
> http://mails.dpdk.org/archives/test-report/2021-February/179652.html,
> the logs contain "No space left" messages.
>
> It seems fine atm but I don't know if we are just at the edge of the pit
> :-).
> Can this be investigated so that it does not repeat?
>
> Thanks.
>
> """
> [3370/3375] Compiling C object
> 'drivers/a715181@@rte_net_mlx5@sha
> /meson-generated_.._rte_net_mlx5.pmd.c.o'.
> [3371/3375] Compiling C object
> 'drivers/a715181@@rte_net_mlx5@sta
> /meson-generated_.._rte_net_mlx5.pmd.c.o'.
> [3372/3375] Linking static target drivers/librte_net_mlx5.a.
> [3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom
> command.
> [3374/3375] Linking target drivers/librte_net_mlx5.so.21.2.
> [3375/3375] Linking target buildtools/chkincs/chkincs.
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 342, in do_install
>     self.install_targets(d)
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 449, in install_targets
>     self.do_copyfile(fname, outname)
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 257, in do_copyfile
>     shutil.copyfile(from_file, to_file)
>   File "/usr/lib/python3.6/shutil.py", line 122, in copyfile
>     copyfileobj(fsrc, fdst)
>   File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj
>     fdst.write(buf)
> OSError: [Errno 28] No space left on device
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 514, in run
>     installer.do_install(datafilename)
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 349, in do_install
>     print('Nothing to install.')
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 71, in __exit__
>     append_to_log(self.lf, d)
>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
> line 81, in append_to_log
>     lf.flush()
> OSError: [Errno 28] No space left on device
>
> During handling of the above exception, another exception occurred:
> """
>
>
> --
> David Marchand
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 4350 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23 14:04 ` Lincoln Lavoie
@ 2021-02-23 16:21   ` David Liu
  2021-02-23 17:56     ` Thomas Monjalon
  0 siblings, 1 reply; 7+ messages in thread
From: David Liu @ 2021-02-23 16:21 UTC (permalink / raw)
  To: Lincoln Lavoie
  Cc: David Marchand, dpdklab, Aaron Conole, ci, Thomas Monjalon,
	Yigit, Ferruh

[-- Attachment #1: Type: text/plain, Size: 3037 bytes --]

Hi David,

This is an actual issue caused by the machine ran out of space.

I had free up some spaces in the machine.

Please let me know if this happens again.

Best,
David


On Tue, Feb 23, 2021 at 9:07 AM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote:

> Hi David,
>
> That is on the agenda for the team to look at today.  David (from our
> team) will confirm back with his findings.
>
> Cheers,
> Lincoln
>
> On Tue, Feb 23, 2021 at 3:31 AM David Marchand <david.marchand@redhat.com>
> wrote:
>
>> Hello,
>>
>> Looking at
>> http://mails.dpdk.org/archives/test-report/2021-February/179652.html,
>> the logs contain "No space left" messages.
>>
>> It seems fine atm but I don't know if we are just at the edge of the pit
>> :-).
>> Can this be investigated so that it does not repeat?
>>
>> Thanks.
>>
>> """
>> [3370/3375] Compiling C object
>> 'drivers/a715181@@rte_net_mlx5@sha
>> /meson-generated_.._rte_net_mlx5.pmd.c.o'.
>> [3371/3375] Compiling C object
>> 'drivers/a715181@@rte_net_mlx5@sta
>> /meson-generated_.._rte_net_mlx5.pmd.c.o'.
>> [3372/3375] Linking static target drivers/librte_net_mlx5.a.
>> [3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom
>> command.
>> [3374/3375] Linking target drivers/librte_net_mlx5.so.21.2.
>> [3375/3375] Linking target buildtools/chkincs/chkincs.
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 342, in do_install
>>     self.install_targets(d)
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 449, in install_targets
>>     self.do_copyfile(fname, outname)
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 257, in do_copyfile
>>     shutil.copyfile(from_file, to_file)
>>   File "/usr/lib/python3.6/shutil.py", line 122, in copyfile
>>     copyfileobj(fsrc, fdst)
>>   File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj
>>     fdst.write(buf)
>> OSError: [Errno 28] No space left on device
>>
>> During handling of the above exception, another exception occurred:
>>
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 514, in run
>>     installer.do_install(datafilename)
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 349, in do_install
>>     print('Nothing to install.')
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 71, in __exit__
>>     append_to_log(self.lf, d)
>>   File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
>> line 81, in append_to_log
>>     lf.flush()
>> OSError: [Errno 28] No space left on device
>>
>> During handling of the above exception, another exception occurred:
>> """
>>
>>
>> --
>> David Marchand
>>
>>
>
> --
> *Lincoln Lavoie*
> Principal Engineer, Broadband Technologies
> 21 Madbury Rd., Ste. 100, Durham, NH 03824
> lylavoie@iol.unh.edu
> https://www.iol.unh.edu
> +1-603-674-2755 (m)
> <https://www.iol.unh.edu>
>

[-- Attachment #2: Type: text/html, Size: 5014 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23 16:21   ` David Liu
@ 2021-02-23 17:56     ` Thomas Monjalon
  2021-02-23 18:04       ` David Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Thomas Monjalon @ 2021-02-23 17:56 UTC (permalink / raw)
  To: Lincoln Lavoie, David Liu
  Cc: David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh

23/02/2021 17:21, David Liu:
> Hi David,
> 
> This is an actual issue caused by the machine ran out of space.
> 
> I had free up some spaces in the machine.
> 
> Please let me know if this happens again.

Instead of waiting for it to happen again,
please could you monitor the disk space of the machines?




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23 17:56     ` Thomas Monjalon
@ 2021-02-23 18:04       ` David Liu
  2021-02-23 18:06         ` Thomas Monjalon
  0 siblings, 1 reply; 7+ messages in thread
From: David Liu @ 2021-02-23 18:04 UTC (permalink / raw)
  To: Thomas Monjalon
  Cc: Lincoln Lavoie, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
wrote:

> 23/02/2021 17:21, David Liu:
> > Hi David,
> >
> > This is an actual issue caused by the machine ran out of space.
> >
> > I had free up some spaces in the machine.
> >
> > Please let me know if this happens again.
>
> Instead of waiting for it to happen again,
> please could you monitor the disk space of the machines?
>
> Yes, I will also be monitoring the pipeline result and disk space.

[-- Attachment #2: Type: text/html, Size: 863 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23 18:04       ` David Liu
@ 2021-02-23 18:06         ` Thomas Monjalon
  2021-02-23 18:59           ` Lincoln Lavoie
  0 siblings, 1 reply; 7+ messages in thread
From: Thomas Monjalon @ 2021-02-23 18:06 UTC (permalink / raw)
  To: David Liu
  Cc: Lincoln Lavoie, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh

23/02/2021 19:04, David Liu:
> On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > 23/02/2021 17:21, David Liu:
> > > Hi David,
> > >
> > > This is an actual issue caused by the machine ran out of space.
> > >
> > > I had free up some spaces in the machine.
> > >
> > > Please let me know if this happens again.
> >
> > Instead of waiting for it to happen again,
> > please could you monitor the disk space of the machines?
> >
> > Yes, I will also be monitoring the pipeline result and disk space.

What does it mean? Will you make sure an alert is automatically sent
to dpdklab@iol.unh.edu when a space is below a threshold?



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-ci] No space left on some UNH system
  2021-02-23 18:06         ` Thomas Monjalon
@ 2021-02-23 18:59           ` Lincoln Lavoie
  0 siblings, 0 replies; 7+ messages in thread
From: Lincoln Lavoie @ 2021-02-23 18:59 UTC (permalink / raw)
  To: Thomas Monjalon
  Cc: David Liu, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh

[-- Attachment #1: Type: text/plain, Size: 1600 bytes --]

Hi Thomas,

Yes, we are adding that VM to our internal monitoring system, based on
icinga2.  We are investigating what consumed the disk space, as it is just
a Jenkins worker and should not have had disk space being consumed over
time, as the pipelines should be either deleting temporary artifacts or
moving longer lived artifacts off the workers to the headnodes (i.e. logs
and such), so those items become available through the dashboard, etc.

Our internal monitoring actually utilizes Google Chat, through our company
Gsuite account, so it can easily generate notifications to our phones, etc.

Cheers,
LIncoln

On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> 23/02/2021 19:04, David Liu:
> > On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 23/02/2021 17:21, David Liu:
> > > > Hi David,
> > > >
> > > > This is an actual issue caused by the machine ran out of space.
> > > >
> > > > I had free up some spaces in the machine.
> > > >
> > > > Please let me know if this happens again.
> > >
> > > Instead of waiting for it to happen again,
> > > please could you monitor the disk space of the machines?
> > >
> > > Yes, I will also be monitoring the pipeline result and disk space.
>
> What does it mean? Will you make sure an alert is automatically sent
> to dpdklab@iol.unh.edu when a space is below a threshold?
>
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 3311 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-02-23 19:02 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-23  8:31 [dpdk-ci] No space left on some UNH system David Marchand
2021-02-23 14:04 ` Lincoln Lavoie
2021-02-23 16:21   ` David Liu
2021-02-23 17:56     ` Thomas Monjalon
2021-02-23 18:04       ` David Liu
2021-02-23 18:06         ` Thomas Monjalon
2021-02-23 18:59           ` Lincoln Lavoie

DPDK CI discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git