IRC logs for #baserock for Sunday, 2014-08-31

*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock00:07
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]00:21
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock00:22
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]00:41
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock00:42
*** thecorconian1 [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection]01:04
paulsherwoodrichard_maw: thank you very much!07:25
*** inara [~inara@192.241.198.49] has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]08:12
*** inara [~inara@192.241.198.49] has joined #baserock08:15
paulsherwoodi do wonder what has changed recently, to cause problems like this with g.b.o08:25
richard_mawwell, there's currently two new issues 1. the git daemon ending up with a bunch of zombie git processes and 2. running out of disk space11:17
richard_mawthere's also the old issue of the lorry controller not cleaning up zombie jobs11:17
richard_mawI think the zombie git processes was caused either by the change to morph to make it ls-remote before doing a build, which is probably accelerated by the number of services that automatically build new changes.11:19
richard_mawMaybe the git commands lost their network connection mid-operation in a place where the git-daemon wasn't expecting.11:19
richard_mawAnother cause could be that the zombie git processes are caused by the disk space issue11:20
richard_mawThe lorry state db includes logs of lorry commands, which while small, adds up over time, and I don't think they have a mechanism for rotating out logs that are old when space becomes a problem11:21
pedroalvarezlooking at the journal seems like the lorry controller fails to update the state of the jobs (maybe because of the disk space issue?)13:33
pedroalvarezhttp://fpaste.org/129891/94922541/13:37
vmesons is now known as vmeson13:50
paulsherwoodif gbo has run out of diskspace, it seems strange that lc-status shows 22GiB free15:16
paulsherwoodit also shows 'Maximum number of jobs: unlimited'. maybe change that?15:17
pedroalvarezthe trove journal is full of this errors18:35
pedroalvarezhttp://fpaste.org/129941/40950985/18:35
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]18:39
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock18:40
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]19:08
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:09
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]19:25
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:26
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]19:41
paulsherwoodso i guess /home is a different partition?19:43
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:45
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]19:55
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:57
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:04
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:04
pedroalvarezyeah20:10
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]20:15
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:26
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:31
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:38
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]20:48
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:49
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:53
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:55
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]21:05
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:16
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]21:29
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:30
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]21:38
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:45
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]21:56
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:07
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]22:12
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:13
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]22:18
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:20
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]22:28
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:31
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]22:45
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:46
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]23:10
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:11
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]23:17
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:18
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]23:38
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:41
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]23:58
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!