* [dpdk-ci] No space left on some UNH system
@ 2021-02-23 8:31 David Marchand
2021-02-23 14:04 ` Lincoln Lavoie
0 siblings, 1 reply; 7+ messages in thread
From: David Marchand @ 2021-02-23 8:31 UTC (permalink / raw)
To: dpdklab; +Cc: Aaron Conole, Lincoln Lavoie, ci, Thomas Monjalon, Yigit, Ferruh
Hello,
Looking at http://mails.dpdk.org/archives/test-report/2021-February/179652.html,
the logs contain "No space left" messages.
It seems fine atm but I don't know if we are just at the edge of the pit :-).
Can this be investigated so that it does not repeat?
Thanks.
"""
[3370/3375] Compiling C object
'drivers/a715181@@rte_net_mlx5@sha/meson-generated_.._rte_net_mlx5.pmd.c.o'.
[3371/3375] Compiling C object
'drivers/a715181@@rte_net_mlx5@sta/meson-generated_.._rte_net_mlx5.pmd.c.o'.
[3372/3375] Linking static target drivers/librte_net_mlx5.a.
[3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom command.
[3374/3375] Linking target drivers/librte_net_mlx5.so.21.2.
[3375/3375] Linking target buildtools/chkincs/chkincs.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 342, in do_install
self.install_targets(d)
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 449, in install_targets
self.do_copyfile(fname, outname)
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 257, in do_copyfile
shutil.copyfile(from_file, to_file)
File "/usr/lib/python3.6/shutil.py", line 122, in copyfile
copyfileobj(fsrc, fdst)
File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj
fdst.write(buf)
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 514, in run
installer.do_install(datafilename)
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 349, in do_install
print('Nothing to install.')
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 71, in __exit__
append_to_log(self.lf, d)
File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py",
line 81, in append_to_log
lf.flush()
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
"""
--
David Marchand
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 8:31 [dpdk-ci] No space left on some UNH system David Marchand @ 2021-02-23 14:04 ` Lincoln Lavoie 2021-02-23 16:21 ` David Liu 0 siblings, 1 reply; 7+ messages in thread From: Lincoln Lavoie @ 2021-02-23 14:04 UTC (permalink / raw) To: David Marchand; +Cc: dpdklab, Aaron Conole, ci, Thomas Monjalon, Yigit, Ferruh [-- Attachment #1: Type: text/plain, Size: 2682 bytes --] Hi David, That is on the agenda for the team to look at today. David (from our team) will confirm back with his findings. Cheers, Lincoln On Tue, Feb 23, 2021 at 3:31 AM David Marchand <david.marchand@redhat.com> wrote: > Hello, > > Looking at > http://mails.dpdk.org/archives/test-report/2021-February/179652.html, > the logs contain "No space left" messages. > > It seems fine atm but I don't know if we are just at the edge of the pit > :-). > Can this be investigated so that it does not repeat? > > Thanks. > > """ > [3370/3375] Compiling C object > 'drivers/a715181@@rte_net_mlx5@sha > /meson-generated_.._rte_net_mlx5.pmd.c.o'. > [3371/3375] Compiling C object > 'drivers/a715181@@rte_net_mlx5@sta > /meson-generated_.._rte_net_mlx5.pmd.c.o'. > [3372/3375] Linking static target drivers/librte_net_mlx5.a. > [3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom > command. > [3374/3375] Linking target drivers/librte_net_mlx5.so.21.2. > [3375/3375] Linking target buildtools/chkincs/chkincs. > Traceback (most recent call last): > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 342, in do_install > self.install_targets(d) > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 449, in install_targets > self.do_copyfile(fname, outname) > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 257, in do_copyfile > shutil.copyfile(from_file, to_file) > File "/usr/lib/python3.6/shutil.py", line 122, in copyfile > copyfileobj(fsrc, fdst) > File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj > fdst.write(buf) > OSError: [Errno 28] No space left on device > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 514, in run > installer.do_install(datafilename) > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 349, in do_install > print('Nothing to install.') > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 71, in __exit__ > append_to_log(self.lf, d) > File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", > line 81, in append_to_log > lf.flush() > OSError: [Errno 28] No space left on device > > During handling of the above exception, another exception occurred: > """ > > > -- > David Marchand > > -- *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu> [-- Attachment #2: Type: text/html, Size: 4350 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 14:04 ` Lincoln Lavoie @ 2021-02-23 16:21 ` David Liu 2021-02-23 17:56 ` Thomas Monjalon 0 siblings, 1 reply; 7+ messages in thread From: David Liu @ 2021-02-23 16:21 UTC (permalink / raw) To: Lincoln Lavoie Cc: David Marchand, dpdklab, Aaron Conole, ci, Thomas Monjalon, Yigit, Ferruh [-- Attachment #1: Type: text/plain, Size: 3037 bytes --] Hi David, This is an actual issue caused by the machine ran out of space. I had free up some spaces in the machine. Please let me know if this happens again. Best, David On Tue, Feb 23, 2021 at 9:07 AM Lincoln Lavoie <lylavoie@iol.unh.edu> wrote: > Hi David, > > That is on the agenda for the team to look at today. David (from our > team) will confirm back with his findings. > > Cheers, > Lincoln > > On Tue, Feb 23, 2021 at 3:31 AM David Marchand <david.marchand@redhat.com> > wrote: > >> Hello, >> >> Looking at >> http://mails.dpdk.org/archives/test-report/2021-February/179652.html, >> the logs contain "No space left" messages. >> >> It seems fine atm but I don't know if we are just at the edge of the pit >> :-). >> Can this be investigated so that it does not repeat? >> >> Thanks. >> >> """ >> [3370/3375] Compiling C object >> 'drivers/a715181@@rte_net_mlx5@sha >> /meson-generated_.._rte_net_mlx5.pmd.c.o'. >> [3371/3375] Compiling C object >> 'drivers/a715181@@rte_net_mlx5@sta >> /meson-generated_.._rte_net_mlx5.pmd.c.o'. >> [3372/3375] Linking static target drivers/librte_net_mlx5.a. >> [3373/3375] Generating rte_net_mlx5.sym_chk with a meson_exe.py custom >> command. >> [3374/3375] Linking target drivers/librte_net_mlx5.so.21.2. >> [3375/3375] Linking target buildtools/chkincs/chkincs. >> Traceback (most recent call last): >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 342, in do_install >> self.install_targets(d) >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 449, in install_targets >> self.do_copyfile(fname, outname) >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 257, in do_copyfile >> shutil.copyfile(from_file, to_file) >> File "/usr/lib/python3.6/shutil.py", line 122, in copyfile >> copyfileobj(fsrc, fdst) >> File "/usr/lib/python3.6/shutil.py", line 82, in copyfileobj >> fdst.write(buf) >> OSError: [Errno 28] No space left on device >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 514, in run >> installer.do_install(datafilename) >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 349, in do_install >> print('Nothing to install.') >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 71, in __exit__ >> append_to_log(self.lf, d) >> File "/usr/local/lib/python3.6/dist-packages/mesonbuild/minstall.py", >> line 81, in append_to_log >> lf.flush() >> OSError: [Errno 28] No space left on device >> >> During handling of the above exception, another exception occurred: >> """ >> >> >> -- >> David Marchand >> >> > > -- > *Lincoln Lavoie* > Principal Engineer, Broadband Technologies > 21 Madbury Rd., Ste. 100, Durham, NH 03824 > lylavoie@iol.unh.edu > https://www.iol.unh.edu > +1-603-674-2755 (m) > <https://www.iol.unh.edu> > [-- Attachment #2: Type: text/html, Size: 5014 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 16:21 ` David Liu @ 2021-02-23 17:56 ` Thomas Monjalon 2021-02-23 18:04 ` David Liu 0 siblings, 1 reply; 7+ messages in thread From: Thomas Monjalon @ 2021-02-23 17:56 UTC (permalink / raw) To: Lincoln Lavoie, David Liu Cc: David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh 23/02/2021 17:21, David Liu: > Hi David, > > This is an actual issue caused by the machine ran out of space. > > I had free up some spaces in the machine. > > Please let me know if this happens again. Instead of waiting for it to happen again, please could you monitor the disk space of the machines? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 17:56 ` Thomas Monjalon @ 2021-02-23 18:04 ` David Liu 2021-02-23 18:06 ` Thomas Monjalon 0 siblings, 1 reply; 7+ messages in thread From: David Liu @ 2021-02-23 18:04 UTC (permalink / raw) To: Thomas Monjalon Cc: Lincoln Lavoie, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh [-- Attachment #1: Type: text/plain, Size: 473 bytes --] On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net> wrote: > 23/02/2021 17:21, David Liu: > > Hi David, > > > > This is an actual issue caused by the machine ran out of space. > > > > I had free up some spaces in the machine. > > > > Please let me know if this happens again. > > Instead of waiting for it to happen again, > please could you monitor the disk space of the machines? > > Yes, I will also be monitoring the pipeline result and disk space. [-- Attachment #2: Type: text/html, Size: 863 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 18:04 ` David Liu @ 2021-02-23 18:06 ` Thomas Monjalon 2021-02-23 18:59 ` Lincoln Lavoie 0 siblings, 1 reply; 7+ messages in thread From: Thomas Monjalon @ 2021-02-23 18:06 UTC (permalink / raw) To: David Liu Cc: Lincoln Lavoie, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh 23/02/2021 19:04, David Liu: > On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net> > wrote: > > > 23/02/2021 17:21, David Liu: > > > Hi David, > > > > > > This is an actual issue caused by the machine ran out of space. > > > > > > I had free up some spaces in the machine. > > > > > > Please let me know if this happens again. > > > > Instead of waiting for it to happen again, > > please could you monitor the disk space of the machines? > > > > Yes, I will also be monitoring the pipeline result and disk space. What does it mean? Will you make sure an alert is automatically sent to dpdklab@iol.unh.edu when a space is below a threshold? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-ci] No space left on some UNH system 2021-02-23 18:06 ` Thomas Monjalon @ 2021-02-23 18:59 ` Lincoln Lavoie 0 siblings, 0 replies; 7+ messages in thread From: Lincoln Lavoie @ 2021-02-23 18:59 UTC (permalink / raw) To: Thomas Monjalon Cc: David Liu, David Marchand, dpdklab, Aaron Conole, ci, Yigit, Ferruh [-- Attachment #1: Type: text/plain, Size: 1600 bytes --] Hi Thomas, Yes, we are adding that VM to our internal monitoring system, based on icinga2. We are investigating what consumed the disk space, as it is just a Jenkins worker and should not have had disk space being consumed over time, as the pipelines should be either deleting temporary artifacts or moving longer lived artifacts off the workers to the headnodes (i.e. logs and such), so those items become available through the dashboard, etc. Our internal monitoring actually utilizes Google Chat, through our company Gsuite account, so it can easily generate notifications to our phones, etc. Cheers, LIncoln On Tue, Feb 23, 2021 at 1:06 PM Thomas Monjalon <thomas@monjalon.net> wrote: > 23/02/2021 19:04, David Liu: > > On Tue, Feb 23, 2021 at 12:56 PM Thomas Monjalon <thomas@monjalon.net> > > wrote: > > > > > 23/02/2021 17:21, David Liu: > > > > Hi David, > > > > > > > > This is an actual issue caused by the machine ran out of space. > > > > > > > > I had free up some spaces in the machine. > > > > > > > > Please let me know if this happens again. > > > > > > Instead of waiting for it to happen again, > > > please could you monitor the disk space of the machines? > > > > > > Yes, I will also be monitoring the pipeline result and disk space. > > What does it mean? Will you make sure an alert is automatically sent > to dpdklab@iol.unh.edu when a space is below a threshold? > > > -- *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu> [-- Attachment #2: Type: text/html, Size: 3311 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-02-23 19:02 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-02-23 8:31 [dpdk-ci] No space left on some UNH system David Marchand 2021-02-23 14:04 ` Lincoln Lavoie 2021-02-23 16:21 ` David Liu 2021-02-23 17:56 ` Thomas Monjalon 2021-02-23 18:04 ` David Liu 2021-02-23 18:06 ` Thomas Monjalon 2021-02-23 18:59 ` Lincoln Lavoie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).