IRC logs for #baserock for Saturday, 2014-08-30

*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 244 seconds]00:07
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock00:09
*** flatmush1 [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock00:46
*** thecorconian [~thecorcon@136.1.1.102] has quit [Remote host closed the connection]03:19
*** thecorconian [~thecorcon@136.1.1.102] has joined #baserock03:20
persiapaulsherwood: There's been a persistent issue recently with too many git-daemon processes, causing the git service to be unresponsive, which has required a restart every 2-3 days.  I suspect this is another occurance of that.07:21
*** thecorconian [~thecorcon@136.1.1.102] has quit [Remote host closed the connection]07:59
pedroalvarezShould we restart it then? 08:25
paulsherwoodyes please08:29
paulsherwooddo we know if this is a change in trove, or gitano, or something else?08:30
paulsherwoodpedroalvarez: afaict pretty much all morph operations hit this. how long does restart take?08:41
persiaPast restarts have taken only a couple minutes08:58
paulsherwoodpersia: do you have the superpowers?09:10
persiaheh, no.09:10
paulsherwoodit seems to be working again now09:11
persiaPerhaps pedroalvarez fixed it earlier or someone else noticed, or it calms down after a while as the zombies get harvested09:11
pedroalvarezRestarted 2 minutes ago 09:12
paulsherwoodtvm09:12
persiathanks09:12
pedroalvarezTook a while09:12
paulsherwoodi'm guessing most of the ls-remote calls are not actually necessary09:13
persiaDepends on the model.  If one presumes one is always network-near a trove, and that one is always working against a trove, they are necessary.09:14
persiaIf one presumes one is working locally, then checking against the local cache is more sensible.09:14
persiaPersonally, I prefer the distributed model, but it does mean more collisions and merge pain.09:15
paulsherwoodi still think my guess is right09:16
paulsherwoodeg i've already built a system, now i need to morph deploy it. what's that got to do with my trove?09:17
paulsherwoodanyways, in other news i successfully ran morph deploy --upgrade on a jetson, to get it to latest kernel09:19
persiaIf you are working in a trove-dependent model, you need to verify your trove contains the right information, and that your local caches are not out of date.09:19
persiaOtherwise your system deployment may not be reproducible.09:19
paulsherwoodmeh :)09:20
persiaIf you are working in a local model, nothing at all, and you have the burden of pushing if you want to reproduce.09:20
* persia doesn't much like troves for this amoung other reasons09:20
pedroalvarezpaulsherwood: yay for the successful upgrade!09:33
paulsherwoodpedroalvarez: yes, i'm now going to reproduce it, make a video i hope09:33
* paulsherwood likes troves. morph irritates him sometimes09:34
paulsherwoodouch, i'm starting not to like troves after all...09:45
paulsherwoodi'm seeing 'no space left on device' when trying to do git operations on g.b.o09:47
paulsherwoodhttp://fpaste.org/129751/93921251/09:48
pedroalvarezerm... we had that problem 1 or 2 days ago09:49
pedroalvarezWe thought that is was because the space left was reserved for the root user09:49
pedroalvarezWe made some space, but seems it's failing again09:50
pedroalvarezAnd worth noting that the space we made is still unused09:51
paulsherwoodinteresting. lc has enough space, but last update seems to have been 11 hours ago?09:51
paulsherwoodtoo many 'running' jobs, though09:52
pedroalvarezI see: Updated: 2014-08-30 09:51:35 UTC09:52
paulsherwoodi'm guessing maybe lots of things are swapped out and it's choked?09:52
pedroalvarezpaulsherwood: yeah, too many running jobs09:52
pedroalvarezI'm not sure about what's the best way to solve this situation09:53
paulsherwoodwhen you restarted, did you reboot the phyiscal machine?09:53
pedroalvarezit has 631 running jobs!!09:54
pedroalvarezpaulsherwood: no, just the service, as we have done in the past09:54
paulsherwoodyes - they may be 'running' but i think they'll never 'finish' :)09:54
paulsherwoodwell i personally would be tempted to reboot... but i'm not the one with keys to the machine :)09:58
pedroalvarezAfaict, rebooting causes more ghosts :/10:03
pedroalvarezGhost jobs10:03
paulsherwoodouch10:04
paulsherwoodlooking at w.b.o i can't see any guide for fiddling with lc jobs, but i'm sure there was some magic written down somewhere?10:07
persiaThere was a script, but it didn't seem to get wide adoption.10:08
persiaNeeds special powers anyway, to create a tunnel to g.b.o10:08
paulsherwoodi'm assuming pedroalvarez has the powers :)10:09
pedroalvarezI've just found a script named: exterminate-ghost-jobs. I will test it against other trove, and if it works I'll do the same in gbo10:14
paulsherwoodlol10:17
paulsherwoodthis feels like text adventure...10:17
paulsherwood'to your right, you see 631 ghost jobs. to your left, is a script called 'exterminate-ghost-jobs'10:17
paulsherwoodin other news, removing my jetson update, setting factory to default, gives me an unhappy jetson10:22
paulsherwoodthis would probably be the time to check on state of backups for gbo10:25
* paulsherwood wonders if liw-orc will notice the magic word :)10:25
persiaThe "factory" handling never quite worked for me, leading me to believe the best way to revert an upgrade was to upgrade to the prior version.10:27
persia(where "upgrade" isn't really the right semantics, because it's really "change" without the inference of improvement or enhancement that comes with "upgrade")10:28
paulsherwoodyes - this needs some work :)10:29
radiofreepaulsherwood: what's the error?10:59
radiofreei seem to remember u-boot being *insanely* picky when loading the extlinux conf from a btrfs partition10:59
tlsathere's a script for killing ghost jobs10:59
tlsabut I don't have access to it here11:00
tlsanot sure if it's in git somewhere11:00
pedroalvarezseems like the script is doing its job. I'm not sure if we are talking about the same script or not.11:33
paulsherwoodradiofree: it was invalid CRC11:39
paulsherwoodi'm taking the brute force approach and reflashing it11:39
*** inara [~inara@192.241.198.49] has quit [Ping timeout: 240 seconds]12:28
*** inara [~inara@192.241.198.49] has joined #baserock12:30
*** thecorconian [~thecorcon@eccvpn1.ford.com] has joined #baserock13:02
*** juergbi [~juerg@vserver.paldo.org] has quit [Ping timeout: 272 seconds]13:34
*** juergbi [~juerg@vserver.paldo.org] has joined #baserock13:36
*** SotK [~adamcoldr@access.ducie-dc1.codethink.co.uk] has quit [Ping timeout: 272 seconds]13:39
*** SotK [~adamcoldr@access.ducie-dc1.codethink.co.uk] has joined #baserock13:45
*** persia [quassel@ubuntu/member/persia] has quit [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.]14:16
*** persia [quassel@2400:8900::f03c:91ff:feae:3452] has joined #baserock14:17
*** persia [quassel@2400:8900::f03c:91ff:feae:3452] has quit [Changing host]14:17
*** persia [quassel@ubuntu/member/persia] has joined #baserock14:17
*** bjdooks [~ben@trinity.fluff.org] has joined #baserock14:49
*** cyndis [cyndis@2001:1bc8:1004::1] has joined #baserock14:49
*** bjdooks_ [~ben@trinity.fluff.org] has quit [Ping timeout: 260 seconds]14:54
*** cyndis_ [cyndis@lakka.kapsi.fi] has quit [Ping timeout: 260 seconds]14:54
*** rjek [~rjek@gateway/shell/pepperfish/x-xydhdwovyyidywvd] has quit [Ping timeout: 260 seconds]14:54
*** rjek [~rjek@gateway/shell/pepperfish/x-osqhaqbyiotouhzo] has joined #baserock14:57
*** flatmush1 [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]18:27
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock18:29
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]18:45
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock18:47
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]18:55
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock18:56
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]19:01
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:10
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]19:17
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:18
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds]19:26
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:28
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]19:32
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:40
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]19:49
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock19:50
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]19:58
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:01
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]20:11
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:12
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:19
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:20
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:29
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:33
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]20:39
pedroalvarezSo, this morning I managed to kill around 300 ghost-jobs, leaving the trove with less than 300, but it has now 575 'running' jobs. Something is going wrong in the lorry-controller is my guess.20:47
richard_maw:(20:49
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock20:51
richard_mawwe need to check the status of the zombie killer patch20:51
richard_mawin other news I'm currently testing a patch that makes `morph build` use the locally cached result of what the remote's branches are like20:53
richard_maw(it uses `git rev-parse $branch@{upstream}`, which it git's standard approach to this problem)20:53
richard_mawwhich also has the benefit of allowing your local name for the branch and the remote name for the branch to differ, but you're unlikely to encounter that difference in common workflows20:55
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]20:57
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:01
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]21:07
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:11
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]21:21
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:22
richard_mawpaulsherwood: I sent a patch that makes morph use the refs/remotes of git repositories in your workspace to determine whether you have unpushed branches, instead of doing a `git ls-remote`21:25
richard_mawwhich ought to reduce the number of git ls-remotes on git.baserock.org, which I think is why it's git-daemon is struggling lately21:28
richard_mawand also, it means morph won't fall over in a heap at that point when trying to build when git.baserock.org's git-daemon is having trouble21:28
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]21:34
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:37
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]21:50
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock21:51
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]22:05
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:06
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]22:15
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:43
*** thecorconian [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection]22:46
*** thecorconian [~thecorcon@eccvpn1.ford.com] has joined #baserock22:46
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]22:55
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock22:56
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]23:00
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:02
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]23:14
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:15
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]23:25
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:26
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]23:31
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:33
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds]23:37
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:37
*** thecorconian1 [~thecorcon@eccvpn1.ford.com] has joined #baserock23:38
*** thecorconian [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection]23:38
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]23:44
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock23:47
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds]23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!