| *** gtristan has joined #baserock | 03:01 | |
| *** paulwaters_ has joined #baserock | 07:54 | |
| *** tiagogomes has joined #baserock | 08:23 | |
| *** tiagogomes has quit IRC | 08:26 | |
| *** tiagogomes has joined #baserock | 08:29 | |
| *** jonathanmaw has joined #baserock | 08:40 | |
| *** ssam2 has joined #baserock | 09:12 | |
| *** ChanServ sets mode: +v ssam2 | 09:12 | |
| *** CTtpollard has joined #baserock | 09:17 | |
| *** locallycompact has joined #baserock | 10:27 | |
| *** locallycompact has quit IRC | 10:47 | |
| *** locallycompact has joined #baserock | 10:47 | |
| * ssam2 still has hopes for the CI ... | 11:21 | |
| paulsherwood | context? | 11:21 |
|---|---|---|
| ssam2 | https://gitlab.com/baserock/definitions/merge_requests/63 | 11:22 |
| ssam2 | one issue now is that BuildStream seems to not be pushing artifacts to the cache | 11:22 |
| ssam2 | the branch is protected, so it should be able to do that ... | 11:22 |
| ssam2 | but e.g. in https://gitlab.com/baserock/definitions/-/jobs/37240642 it didn't seem to push anything | 11:22 |
| paulsherwood | ack | 11:23 |
| ssam2 | oh maybe it doesn't push import elements ? not sure | 11:23 |
| paulsherwood | in other news, any idea why lorry-controller hasn't been updating repos for last few days at git.baserock.org? | 11:24 |
| ssam2 | oh, the service might need restarting after the migration | 11:25 |
| ssam2 | getting: OperationalError: unable to open database file | 11:29 |
| ssam2 | from the lorry-controller webapp | 11:29 |
| paulsherwood | eek | 11:29 |
| ssam2 | ah, /home/lorry was owned by the wrong user | 11:30 |
| ssam2 | now there's some host keys issue.. which is odd | 11:31 |
| *** locallycompact has quit IRC | 11:37 | |
| * ssam2 thoroughly confused why ssh host keys would have changed given that we migrated the root disk ... maybe trove-setup regenerated them ? | 11:37 | |
| ssam2 | timestamps suggest that | 11:37 |
| ironfoot | yeah, maybe | 11:38 |
| ssam2 | some jobs seem to be succeeding now | 11:40 |
| ssam2 | quite a few failing for various reasons, i've not been watching this much though so maybe that's normal | 11:40 |
| *** locallycompact has joined #baserock | 11:40 | |
| ssam2 | buildstream definitely doesn't want to push :-( | 11:40 |
| ssam2 | maybe i broke ostree.baserock.org ... | 11:41 |
| ssam2 | no, the whole 'Checking connectivity to remote artifact cache' line just doesn't appear in the CI job | 11:44 |
| ironfoot | thanks for fixing l-c ssam2 | 11:49 |
| ironfoot | it looks like the new instance of g.b.o is performing way better than the previous one | 12:07 |
| ironfoot | 1000+ lorries updated in 30 minutes | 12:08 |
| paulsherwood | wow! :) | 12:18 |
| *** gtristan has quit IRC | 12:30 | |
| ssam2 | yeah it does seem super fast | 12:47 |
| ssam2 | journalctl still leaves something to be desired of course | 12:47 |
| ssam2 | ostree.baserock.org had an old version of buildstream installed, maybe that was the problem | 13:12 |
| ssam2 | ok, ostree.baserock.org now actually works. Hopefully that's enough to fix CI | 13:22 |
| ssam2 | although I got an exception when pushing locally, which I don't see in the CI | 13:22 |
| * ssam2 has the genius idea of sshing into the runner while it's running to see why it won't push | 13:49 | |
| ssam2 | bah: https://gitlab.com/baserock/definitions/commit/cb4317a44cca9a377c75f4f91288e1085a95c015 | 14:04 |
| benbrown_ | ssam2: hah | 14:05 |
| ssam2 | jjardon, https://gitlab.com/baserock/definitions/commit/4ace8eda2b98020e583e8f68c0758f28d3b0284d seems to break caching of gits again | 14:16 |
| ssam2 | this buildstream conversion job is cloning every git repo from git.baserock.org : https://gitlab.com/baserock/definitions/-/jobs/37276752 | 14:17 |
| ssam2 | and putting them all into /cache/ybd_base/gits | 14:17 |
| ssam2 | which will be lost as soon as the job finishes | 14:17 |
| ssam2 | oh, the `rm -rf "${YBD_base}"` line also doesn't help | 14:18 |
| ssam2 | that should be disabled for the bst-convert job | 14:18 |
| ssam2 | i'm still confused what the point of /cache is though | 14:18 |
| ssam2 | oh wait, stuff does seem to be going into /build/baserock/definitions/cache/ybd_base after all | 14:39 |
| * ssam2 continually confused by this set up | 14:39 | |
| benbrown_ | hah | 15:00 |
| benbrown_ | it's not the clearest | 15:00 |
| benbrown_ | ssam2: /cache is for caching ybd artifacts between jobs on the same runner | 15:01 |
| benbrown_ | unsure how much speed it ultimately saves, given those artifacts could be redownloaded from kbas | 15:02 |
| ssam2 | right | 15:07 |
| ssam2 | i was being confused by the fact that temporary source downloads were in /cache/ybd_base/tmp | 15:07 |
| ssam2 | but that's fine, the sources end up in /builds/baserock/definitions/cache/ybd_base eventually | 15:07 |
| ssam2 | so hopefully https://gitlab.com/baserock/definitions/-/jobs/37281584 is the last bst-convert job that will redownload all the gits | 15:08 |
| benbrown_ | I think the last issue I ended up hitting was timeout when fetching the gits cache | 15:10 |
| ssam2 | ah, that could have been due to the server migration if it was last thu/fri | 15:10 |
| *** adds68 has quit IRC | 15:11 | |
| benbrown_ | "FATAL: net/http: request canceled (Client.Timeout exceeded while reading body)" | 15:11 |
| *** adds68 has joined #baserock | 15:11 | |
| benbrown_ | that's hosted on one of the digitalocean droplets afaia | 15:12 |
| ssam2 | oh, nasty | 15:13 |
| *** gtristan has joined #baserock | 15:32 | |
| ssam2 | benbrown_, aha, I see the same thing | 15:59 |
| ssam2 | https://gitlab.com/baserock/definitions/-/jobs/37281584 | 15:59 |
| ssam2 | and think: wtf | 15:59 |
| ssam2 | seems like a glitch within digitalocean, at first glance | 16:00 |
| benbrown_ | :\ | 16:01 |
| ssam2 | the machine 138.68.132.175 still exists | 16:03 |
| ssam2 | but the manager VM can't route to it | 16:03 |
| *** adds68 has quit IRC | 16:03 | |
| ssam2 | interestingly the machine isn't listed in `docker-machine ls`, despite existing | 16:04 |
| ssam2 | and now i can ping the machine again | 16:06 |
| ssam2 | i have no idea what this could be other than digitalocean screwing up networking... | 16:06 |
| *** adds68 has joined #baserock | 16:08 | |
| *** jude_ has quit IRC | 16:15 | |
| ssam2 | benbrown_, weirdly the bst-convert job has now pulled a cache full of *buildstream* sources | 16:20 |
| ssam2 | ah, it does have one YBD git repo there | 16:20 |
| ssam2 | i guess the 'default' cache has a bunch of useless buildstream stuff in it | 16:21 |
| ssam2 | 8.1GB of buildstream stuff | 16:22 |
| ssam2 | but... having ssh'ed into the runner, I guess I could manually delete all that, and then if the job doesn't die from this timeout issue, we'll end up with the default cache having only YBD stuff in | 16:22 |
| benbrown_ | wow | 16:25 |
| benbrown_ | I guess move all the bst gits into the ybd path if they don't exist, since they both using the same directory naming logic? | 16:26 |
| ssam2 | argh, I already deleted them | 16:27 |
| ssam2 | everything I do trying to fix the CI is wrong!! | 16:27 |
| ssam2 | hmm, the .zip is still there though | 16:27 |
| benbrown_ | It's fine, given that job *should* repopulate everything. | 16:28 |
| * ssam2 breaks CI again by killing the wrong process | 16:40 | |
| ssam2 | i feel like Shrek | 16:40 |
| benbrown_ | :D | 16:42 |
| jjardon | ssam2: Im going to upgrade the ci manager machine, give me a minute | 16:42 |
| jjardon | ssam2: done | 16:44 |
| jjardon | lets see: https://gitlab.com/baserock/definitions/-/jobs/37306927 | 16:45 |
| ssam2 | ok, i'll manually move the cached gits over from buildstream to ybd again | 16:46 |
| ssam2 | done | 16:49 |
| * ssam2 now not going to touch *anything* else and let's hope this works | 16:49 | |
| paulsherwood | :) | 16:54 |
| *** ssam2 has quit IRC | 16:58 | |
| *** locallycompact has quit IRC | 17:02 | |
| *** tiagogomes has quit IRC | 17:13 | |
| *** paulwaters_ has joined #baserock | 17:14 | |
| *** jonathanmaw has quit IRC | 17:26 | |
| *** jude_ has joined #baserock | 19:23 | |
| *** jude_ has quit IRC | 20:10 | |
| *** gtristan has quit IRC | 21:18 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!