IRC logs for #baserock for Monday, 2017-10-23

*** gtristan has joined #baserock03:01
*** paulwaters_ has joined #baserock07:54
*** tiagogomes has joined #baserock08:23
*** tiagogomes has quit IRC08:26
*** tiagogomes has joined #baserock08:29
*** jonathanmaw has joined #baserock08:40
*** ssam2 has joined #baserock09:12
*** ChanServ sets mode: +v ssam209:12
*** CTtpollard has joined #baserock09:17
*** locallycompact has joined #baserock10:27
*** locallycompact has quit IRC10:47
*** locallycompact has joined #baserock10:47
* ssam2 still has hopes for the CI ...11:21
paulsherwoodcontext?11:21
ssam2https://gitlab.com/baserock/definitions/merge_requests/6311:22
ssam2one issue now is that BuildStream seems to not be pushing artifacts to the cache11:22
ssam2the branch is protected, so it should be able to do that ...11:22
ssam2but e.g. in https://gitlab.com/baserock/definitions/-/jobs/37240642 it didn't seem to push anything11:22
paulsherwoodack11:23
ssam2oh maybe it doesn't push import elements ? not sure11:23
paulsherwoodin other news, any idea why lorry-controller hasn't been updating repos for last few days at git.baserock.org?11:24
ssam2oh, the service might need restarting after the migration11:25
ssam2getting: OperationalError: unable to open database file11:29
ssam2from the lorry-controller webapp11:29
paulsherwoodeek11:29
ssam2ah, /home/lorry was owned by the wrong user11:30
ssam2now there's some host keys issue.. which is odd11:31
*** locallycompact has quit IRC11:37
* ssam2 thoroughly confused why ssh host keys would have changed given that we migrated the root disk ... maybe trove-setup regenerated them ?11:37
ssam2timestamps suggest that11:37
ironfootyeah, maybe11:38
ssam2some jobs seem to be succeeding now11:40
ssam2quite a few failing for various reasons, i've not been watching this much though so maybe that's normal11:40
*** locallycompact has joined #baserock11:40
ssam2buildstream definitely doesn't want to push :-(11:40
ssam2maybe i broke ostree.baserock.org ...11:41
ssam2no, the whole 'Checking connectivity to remote artifact cache' line just doesn't appear in the CI job11:44
ironfootthanks for fixing l-c ssam211:49
ironfootit looks like the new instance of g.b.o is performing way better than the previous one12:07
ironfoot1000+ lorries updated in 30 minutes12:08
paulsherwoodwow! :)12:18
*** gtristan has quit IRC12:30
ssam2yeah it does seem super fast12:47
ssam2journalctl still leaves something to be desired of course12:47
ssam2ostree.baserock.org had an old version of buildstream installed, maybe that was the problem13:12
ssam2ok, ostree.baserock.org now actually works. Hopefully that's enough to fix CI13:22
ssam2although I got an exception when pushing locally, which I don't see in the CI13:22
* ssam2 has the genius idea of sshing into the runner while it's running to see why it won't push13:49
ssam2bah: https://gitlab.com/baserock/definitions/commit/cb4317a44cca9a377c75f4f91288e1085a95c01514:04
benbrown_ssam2: hah14:05
ssam2jjardon, https://gitlab.com/baserock/definitions/commit/4ace8eda2b98020e583e8f68c0758f28d3b0284d seems to break caching of gits again14:16
ssam2this buildstream conversion job is cloning every git repo from git.baserock.org : https://gitlab.com/baserock/definitions/-/jobs/3727675214:17
ssam2and putting them all into /cache/ybd_base/gits14:17
ssam2which will be lost as soon as the job finishes14:17
ssam2oh, the `rm -rf "${YBD_base}"` line also doesn't help14:18
ssam2that should be disabled for the bst-convert job14:18
ssam2i'm still confused what the point of /cache is though14:18
ssam2oh wait, stuff does seem to be going into /build/baserock/definitions/cache/ybd_base after all14:39
* ssam2 continually confused by this set up14:39
benbrown_hah15:00
benbrown_it's not the clearest15:00
benbrown_ssam2: /cache is for caching ybd artifacts between jobs on the same runner15:01
benbrown_unsure how much speed it ultimately saves, given those artifacts could be redownloaded from kbas15:02
ssam2right15:07
ssam2i was being confused by the fact that temporary source downloads were in /cache/ybd_base/tmp15:07
ssam2but that's fine, the sources end up in /builds/baserock/definitions/cache/ybd_base eventually15:07
ssam2so hopefully https://gitlab.com/baserock/definitions/-/jobs/37281584 is the last bst-convert job that will redownload all the gits15:08
benbrown_I think the last issue I ended up hitting was timeout when fetching the gits cache15:10
ssam2ah, that could have been due to the server migration if it was last thu/fri15:10
*** adds68 has quit IRC15:11
benbrown_"FATAL: net/http: request canceled (Client.Timeout exceeded while reading body)"15:11
*** adds68 has joined #baserock15:11
benbrown_that's hosted on one of the digitalocean droplets afaia15:12
ssam2oh, nasty15:13
*** gtristan has joined #baserock15:32
ssam2benbrown_, aha, I see the same thing15:59
ssam2https://gitlab.com/baserock/definitions/-/jobs/3728158415:59
ssam2and think: wtf15:59
ssam2seems like a glitch within digitalocean, at first glance16:00
benbrown_:\16:01
ssam2the machine 138.68.132.175 still exists16:03
ssam2but the manager VM can't route to it16:03
*** adds68 has quit IRC16:03
ssam2interestingly the machine isn't listed in `docker-machine ls`, despite existing16:04
ssam2and now i can ping the machine again16:06
ssam2i have no idea what this could be other than digitalocean screwing up networking...16:06
*** adds68 has joined #baserock16:08
*** jude_ has quit IRC16:15
ssam2benbrown_, weirdly the bst-convert job has now pulled a cache full of *buildstream* sources16:20
ssam2ah, it does have one YBD git repo there16:20
ssam2i guess the 'default' cache has a bunch of useless buildstream stuff in it16:21
ssam28.1GB of buildstream stuff16:22
ssam2but... having ssh'ed into the runner, I guess I could manually delete all that, and then if the job doesn't die from this timeout issue, we'll end up with the default cache having only YBD stuff in16:22
benbrown_wow16:25
benbrown_I guess move all the bst gits into the ybd path if they don't exist, since they both using the same directory naming logic?16:26
ssam2argh, I already deleted them16:27
ssam2everything I do trying to fix the CI is wrong!!16:27
ssam2hmm, the .zip is still there though16:27
benbrown_It's fine, given that job *should* repopulate everything.16:28
* ssam2 breaks CI again by killing the wrong process16:40
ssam2i feel like Shrek16:40
benbrown_:D16:42
jjardonssam2: Im going to upgrade the ci manager machine, give me a minute16:42
jjardonssam2: done16:44
jjardonlets see: https://gitlab.com/baserock/definitions/-/jobs/3730692716:45
ssam2ok, i'll manually move the cached gits over from buildstream to ybd again16:46
ssam2done16:49
* ssam2 now not going to touch *anything* else and let's hope this works16:49
paulsherwood:)16:54
*** ssam2 has quit IRC16:58
*** locallycompact has quit IRC17:02
*** tiagogomes has quit IRC17:13
*** paulwaters_ has joined #baserock17:14
*** jonathanmaw has quit IRC17:26
*** jude_ has joined #baserock19:23
*** jude_ has quit IRC20:10
*** gtristan has quit IRC21:18

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!