| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 00:07 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 00:21 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 00:22 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 00:41 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 00:42 | |
| *** thecorconian1 [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection] | 01:04 | |
| paulsherwood | richard_maw: thank you very much! | 07:25 |
|---|---|---|
| *** inara [~inara@192.241.198.49] has quit [Quit: ERC Version 5.3 (IRC client for Emacs)] | 08:12 | |
| *** inara [~inara@192.241.198.49] has joined #baserock | 08:15 | |
| paulsherwood | i do wonder what has changed recently, to cause problems like this with g.b.o | 08:25 |
| richard_maw | well, there's currently two new issues 1. the git daemon ending up with a bunch of zombie git processes and 2. running out of disk space | 11:17 |
| richard_maw | there's also the old issue of the lorry controller not cleaning up zombie jobs | 11:17 |
| richard_maw | I think the zombie git processes was caused either by the change to morph to make it ls-remote before doing a build, which is probably accelerated by the number of services that automatically build new changes. | 11:19 |
| richard_maw | Maybe the git commands lost their network connection mid-operation in a place where the git-daemon wasn't expecting. | 11:19 |
| richard_maw | Another cause could be that the zombie git processes are caused by the disk space issue | 11:20 |
| richard_maw | The lorry state db includes logs of lorry commands, which while small, adds up over time, and I don't think they have a mechanism for rotating out logs that are old when space becomes a problem | 11:21 |
| pedroalvarez | looking at the journal seems like the lorry controller fails to update the state of the jobs (maybe because of the disk space issue?) | 13:33 |
| pedroalvarez | http://fpaste.org/129891/94922541/ | 13:37 |
| vmesons is now known as vmeson | 13:50 | |
| paulsherwood | if gbo has run out of diskspace, it seems strange that lc-status shows 22GiB free | 15:16 |
| paulsherwood | it also shows 'Maximum number of jobs: unlimited'. maybe change that? | 15:17 |
| pedroalvarez | the trove journal is full of this errors | 18:35 |
| pedroalvarez | http://fpaste.org/129941/40950985/ | 18:35 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 18:39 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 18:40 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 19:08 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:09 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 19:25 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:26 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 19:41 | |
| paulsherwood | so i guess /home is a different partition? | 19:43 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:45 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 19:55 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:57 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:04 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:04 | |
| pedroalvarez | yeah | 20:10 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 20:15 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:26 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:31 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:38 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 20:48 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:49 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:53 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:55 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 21:05 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:16 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 21:29 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:30 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 21:38 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:45 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 21:56 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:07 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 22:12 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:13 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 22:18 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:20 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 22:28 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:31 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 22:45 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:46 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 23:10 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:11 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 23:17 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:18 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 23:38 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:41 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 23:58 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!