| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 244 seconds] | 00:07 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 00:09 | |
| *** flatmush1 [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 00:46 | |
| *** thecorconian [~thecorcon@136.1.1.102] has quit [Remote host closed the connection] | 03:19 | |
| *** thecorconian [~thecorcon@136.1.1.102] has joined #baserock | 03:20 | |
| persia | paulsherwood: There's been a persistent issue recently with too many git-daemon processes, causing the git service to be unresponsive, which has required a restart every 2-3 days. I suspect this is another occurance of that. | 07:21 |
|---|---|---|
| *** thecorconian [~thecorcon@136.1.1.102] has quit [Remote host closed the connection] | 07:59 | |
| pedroalvarez | Should we restart it then? | 08:25 |
| paulsherwood | yes please | 08:29 |
| paulsherwood | do we know if this is a change in trove, or gitano, or something else? | 08:30 |
| paulsherwood | pedroalvarez: afaict pretty much all morph operations hit this. how long does restart take? | 08:41 |
| persia | Past restarts have taken only a couple minutes | 08:58 |
| paulsherwood | persia: do you have the superpowers? | 09:10 |
| persia | heh, no. | 09:10 |
| paulsherwood | it seems to be working again now | 09:11 |
| persia | Perhaps pedroalvarez fixed it earlier or someone else noticed, or it calms down after a while as the zombies get harvested | 09:11 |
| pedroalvarez | Restarted 2 minutes ago | 09:12 |
| paulsherwood | tvm | 09:12 |
| persia | thanks | 09:12 |
| pedroalvarez | Took a while | 09:12 |
| paulsherwood | i'm guessing most of the ls-remote calls are not actually necessary | 09:13 |
| persia | Depends on the model. If one presumes one is always network-near a trove, and that one is always working against a trove, they are necessary. | 09:14 |
| persia | If one presumes one is working locally, then checking against the local cache is more sensible. | 09:14 |
| persia | Personally, I prefer the distributed model, but it does mean more collisions and merge pain. | 09:15 |
| paulsherwood | i still think my guess is right | 09:16 |
| paulsherwood | eg i've already built a system, now i need to morph deploy it. what's that got to do with my trove? | 09:17 |
| paulsherwood | anyways, in other news i successfully ran morph deploy --upgrade on a jetson, to get it to latest kernel | 09:19 |
| persia | If you are working in a trove-dependent model, you need to verify your trove contains the right information, and that your local caches are not out of date. | 09:19 |
| persia | Otherwise your system deployment may not be reproducible. | 09:19 |
| paulsherwood | meh :) | 09:20 |
| persia | If you are working in a local model, nothing at all, and you have the burden of pushing if you want to reproduce. | 09:20 |
| * persia doesn't much like troves for this amoung other reasons | 09:20 | |
| pedroalvarez | paulsherwood: yay for the successful upgrade! | 09:33 |
| paulsherwood | pedroalvarez: yes, i'm now going to reproduce it, make a video i hope | 09:33 |
| * paulsherwood likes troves. morph irritates him sometimes | 09:34 | |
| paulsherwood | ouch, i'm starting not to like troves after all... | 09:45 |
| paulsherwood | i'm seeing 'no space left on device' when trying to do git operations on g.b.o | 09:47 |
| paulsherwood | http://fpaste.org/129751/93921251/ | 09:48 |
| pedroalvarez | erm... we had that problem 1 or 2 days ago | 09:49 |
| pedroalvarez | We thought that is was because the space left was reserved for the root user | 09:49 |
| pedroalvarez | We made some space, but seems it's failing again | 09:50 |
| pedroalvarez | And worth noting that the space we made is still unused | 09:51 |
| paulsherwood | interesting. lc has enough space, but last update seems to have been 11 hours ago? | 09:51 |
| paulsherwood | too many 'running' jobs, though | 09:52 |
| pedroalvarez | I see: Updated: 2014-08-30 09:51:35 UTC | 09:52 |
| paulsherwood | i'm guessing maybe lots of things are swapped out and it's choked? | 09:52 |
| pedroalvarez | paulsherwood: yeah, too many running jobs | 09:52 |
| pedroalvarez | I'm not sure about what's the best way to solve this situation | 09:53 |
| paulsherwood | when you restarted, did you reboot the phyiscal machine? | 09:53 |
| pedroalvarez | it has 631 running jobs!! | 09:54 |
| pedroalvarez | paulsherwood: no, just the service, as we have done in the past | 09:54 |
| paulsherwood | yes - they may be 'running' but i think they'll never 'finish' :) | 09:54 |
| paulsherwood | well i personally would be tempted to reboot... but i'm not the one with keys to the machine :) | 09:58 |
| pedroalvarez | Afaict, rebooting causes more ghosts :/ | 10:03 |
| pedroalvarez | Ghost jobs | 10:03 |
| paulsherwood | ouch | 10:04 |
| paulsherwood | looking at w.b.o i can't see any guide for fiddling with lc jobs, but i'm sure there was some magic written down somewhere? | 10:07 |
| persia | There was a script, but it didn't seem to get wide adoption. | 10:08 |
| persia | Needs special powers anyway, to create a tunnel to g.b.o | 10:08 |
| paulsherwood | i'm assuming pedroalvarez has the powers :) | 10:09 |
| pedroalvarez | I've just found a script named: exterminate-ghost-jobs. I will test it against other trove, and if it works I'll do the same in gbo | 10:14 |
| paulsherwood | lol | 10:17 |
| paulsherwood | this feels like text adventure... | 10:17 |
| paulsherwood | 'to your right, you see 631 ghost jobs. to your left, is a script called 'exterminate-ghost-jobs' | 10:17 |
| paulsherwood | in other news, removing my jetson update, setting factory to default, gives me an unhappy jetson | 10:22 |
| paulsherwood | this would probably be the time to check on state of backups for gbo | 10:25 |
| * paulsherwood wonders if liw-orc will notice the magic word :) | 10:25 | |
| persia | The "factory" handling never quite worked for me, leading me to believe the best way to revert an upgrade was to upgrade to the prior version. | 10:27 |
| persia | (where "upgrade" isn't really the right semantics, because it's really "change" without the inference of improvement or enhancement that comes with "upgrade") | 10:28 |
| paulsherwood | yes - this needs some work :) | 10:29 |
| radiofree | paulsherwood: what's the error? | 10:59 |
| radiofree | i seem to remember u-boot being *insanely* picky when loading the extlinux conf from a btrfs partition | 10:59 |
| tlsa | there's a script for killing ghost jobs | 10:59 |
| tlsa | but I don't have access to it here | 11:00 |
| tlsa | not sure if it's in git somewhere | 11:00 |
| pedroalvarez | seems like the script is doing its job. I'm not sure if we are talking about the same script or not. | 11:33 |
| paulsherwood | radiofree: it was invalid CRC | 11:39 |
| paulsherwood | i'm taking the brute force approach and reflashing it | 11:39 |
| *** inara [~inara@192.241.198.49] has quit [Ping timeout: 240 seconds] | 12:28 | |
| *** inara [~inara@192.241.198.49] has joined #baserock | 12:30 | |
| *** thecorconian [~thecorcon@eccvpn1.ford.com] has joined #baserock | 13:02 | |
| *** juergbi [~juerg@vserver.paldo.org] has quit [Ping timeout: 272 seconds] | 13:34 | |
| *** juergbi [~juerg@vserver.paldo.org] has joined #baserock | 13:36 | |
| *** SotK [~adamcoldr@access.ducie-dc1.codethink.co.uk] has quit [Ping timeout: 272 seconds] | 13:39 | |
| *** SotK [~adamcoldr@access.ducie-dc1.codethink.co.uk] has joined #baserock | 13:45 | |
| *** persia [quassel@ubuntu/member/persia] has quit [Quit: http://quassel-irc.org - Chat comfortably. Anywhere.] | 14:16 | |
| *** persia [quassel@2400:8900::f03c:91ff:feae:3452] has joined #baserock | 14:17 | |
| *** persia [quassel@2400:8900::f03c:91ff:feae:3452] has quit [Changing host] | 14:17 | |
| *** persia [quassel@ubuntu/member/persia] has joined #baserock | 14:17 | |
| *** bjdooks [~ben@trinity.fluff.org] has joined #baserock | 14:49 | |
| *** cyndis [cyndis@2001:1bc8:1004::1] has joined #baserock | 14:49 | |
| *** bjdooks_ [~ben@trinity.fluff.org] has quit [Ping timeout: 260 seconds] | 14:54 | |
| *** cyndis_ [cyndis@lakka.kapsi.fi] has quit [Ping timeout: 260 seconds] | 14:54 | |
| *** rjek [~rjek@gateway/shell/pepperfish/x-xydhdwovyyidywvd] has quit [Ping timeout: 260 seconds] | 14:54 | |
| *** rjek [~rjek@gateway/shell/pepperfish/x-osqhaqbyiotouhzo] has joined #baserock | 14:57 | |
| *** flatmush1 [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 18:27 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 18:29 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 18:45 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 18:47 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 18:55 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 18:56 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 19:01 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:10 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 19:17 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:18 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 255 seconds] | 19:26 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:28 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 19:32 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:40 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 19:49 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 19:50 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 19:58 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:01 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 20:11 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:12 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:19 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:20 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:29 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:33 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 20:39 | |
| pedroalvarez | So, this morning I managed to kill around 300 ghost-jobs, leaving the trove with less than 300, but it has now 575 'running' jobs. Something is going wrong in the lorry-controller is my guess. | 20:47 |
| richard_maw | :( | 20:49 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 20:51 | |
| richard_maw | we need to check the status of the zombie killer patch | 20:51 |
| richard_maw | in other news I'm currently testing a patch that makes `morph build` use the locally cached result of what the remote's branches are like | 20:53 |
| richard_maw | (it uses `git rev-parse $branch@{upstream}`, which it git's standard approach to this problem) | 20:53 |
| richard_maw | which also has the benefit of allowing your local name for the branch and the remote name for the branch to differ, but you're unlikely to encounter that difference in common workflows | 20:55 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 20:57 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:01 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 21:07 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:11 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds] | 21:21 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:22 | |
| richard_maw | paulsherwood: I sent a patch that makes morph use the refs/remotes of git repositories in your workspace to determine whether you have unpushed branches, instead of doing a `git ls-remote` | 21:25 |
| richard_maw | which ought to reduce the number of git ls-remotes on git.baserock.org, which I think is why it's git-daemon is struggling lately | 21:28 |
| richard_maw | and also, it means morph won't fall over in a heap at that point when trying to build when git.baserock.org's git-daemon is having trouble | 21:28 |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 21:34 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:37 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 21:50 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 21:51 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 22:05 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:06 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 22:15 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:43 | |
| *** thecorconian [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection] | 22:46 | |
| *** thecorconian [~thecorcon@eccvpn1.ford.com] has joined #baserock | 22:46 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 22:55 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 22:56 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 23:00 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:02 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 23:14 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:15 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 23:25 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:26 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 23:31 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:33 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 250 seconds] | 23:37 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:37 | |
| *** thecorconian1 [~thecorcon@eccvpn1.ford.com] has joined #baserock | 23:38 | |
| *** thecorconian [~thecorcon@eccvpn1.ford.com] has quit [Remote host closed the connection] | 23:38 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds] | 23:44 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock | 23:47 | |
| *** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 260 seconds] | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!