IRC logs for #baserock for Wednesday, 2017-10-04

*** rdale has quit IRC03:05
*** benbrown_ has quit IRC04:16
*** benbrown_ has joined #baserock04:23
*** gtristan has quit IRC04:50
*** gtristan has joined #baserock05:14
*** jude_ has joined #baserock07:30
*** ctbruce has joined #baserock07:47
*** toscalix has joined #baserock07:59
*** rdale has joined #baserock08:11
*** rdale has quit IRC08:13
*** rdale has joined #baserock08:17
*** jonathanmaw has joined #baserock08:41
*** ssam2 has joined #baserock09:05
*** ChanServ sets mode: +v ssam209:05
*** gtristan has quit IRC09:32
*** gtristan has joined #baserock09:37
*** locallycompact has joined #baserock09:59
*** noisecell has joined #baserock10:05
ssam2jjardon, benbrown_: hey, how can I get access to the cache at digitalocean ?10:19
ssam2i have access to the manager machine but i don't know how to get from there to the cache10:19
ssam2and would like to debug why none of the BuildStream sources are being shared between jobs in CI10:19
jjardonlet me refresh my memory ...10:23
gitlab-br-botinfrastructure: merge request (kill-obsolete->master: Kill obsolete system definitions) #8 changed state ("opened"): https://gitlab.com/baserock/infrastructure/merge_requests/811:51
benbrown_ssam2: I don't think the cache is *at* digitalocean11:52
benbrown_it's a docker volume on the runner11:52
benbrown_which you could inspect via docker-machine ssh runner-bla11:52
ssam2ok, i'll try that11:52
ssam2we did try to set up a distributed cache though11:53
ssam2https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching11:54
ssam2as part of this investigation i will document the actual setup in baserock infrastructure.git11:54
gitlab-br-botinfrastructure: merge request (kill-obsolete->master: Kill obsolete system definitions) #8 changed state ("opened"): https://gitlab.com/baserock/infrastructure/merge_requests/814:22
*** gtristan has quit IRC14:23
gitlab-br-botinfrastructure: merge request (kill-obsolete->master: Kill obsolete system definitions) #8 changed state ("opened"): https://gitlab.com/baserock/infrastructure/merge_requests/814:25
ironfootssam2: ^ looks good, ready to merge?14:38
ironfootI can't just "approve"14:39
ssam2yeah ready i think14:39
ssam2we probably didn't configure that repo to require approvals14:39
gitlab-br-botinfrastructure: merge request (kill-obsolete->master: Kill obsolete system definitions) #8 changed state ("merged"): https://gitlab.com/baserock/infrastructure/merge_requests/814:40
ssam2i've changed the config to require approvals14:41
ironfootta!14:41
ssam2and added you, me, jjardon and benbrown_ as the approvers14:41
*** gtristan has joined #baserock14:45
*** locallycompact has quit IRC14:45
*** locallycompact has joined #baserock14:47
*** adds68 has quit IRC14:57
*** jude_ has quit IRC15:01
*** paulwaters_ has joined #baserock15:01
ssam2seems that the distributed cache is working, but all it stores is an empty directory tree15:24
ssam2 /cache/ybd_base/gits15:25
ssam2what's the correct way to tell YBD "store gits here" ?15:25
ssam2we are setting XDG_CACHE_HOME which i thought should work15:25
ssam2in fact, I do see this in the ybd output15:26
ssam2   gits=/cache/ybd_base/gits15:26
ssam2i guess it didn't build anything recently, but seems odd that it would have never downloaded any sources at all...15:27
ssam2aha15:28
ssam2as in https://gitlab.com/baserock/definitions/-/jobs/31740842, ybd is doing as told, it's just that gitlab is ignoring everything in /cache15:29
ssam2i've tried creating a `pwd`/cache symlink in the hope that we can trick it15:29
ssam2got to wait for the whole freedesktop SDK to download yet again before i'll see if that worked though15:29
*** noisecell has quit IRC15:32
* benbrown_ notes that, assuming the hack works, it will upload and pass around a .zip with everything in that path15:32
benbrown_Which, I expect, will be *huge*.15:32
ssam2yeah15:32
ssam2but it's just going from one digitalocean droplet to another, so hopefully pretty quick to transfera nyway15:33
ssam2*anyway15:33
ssam2and it's only gits, so i don't expect it to be more than 10s of GBs15:33
ssam2well, gits and ostree binaries15:33
* benbrown_ nods15:34
ssam2if the hack doesn't work, perhaps we could change the /cache volume to be at /build15:35
ssam2and then we could store everything inside the builddir without running out of space15:35
gtristancaching build dir is... well it will mean failed builds will accumulate I suspect15:54
gtristanssam2, ybd.conf sets up the directories, in a similar way to buildstream.conf (although I suspect you figured that by now)15:55
ssam2yeah i don't want to cache the whole build dir15:57
ssam2just need to satisfy gitlab-ci's seemingly arbitrary requirement that i only cache things that are within the build dir15:57
ssam2i guess it's a sanity check to stop people caching / or whatever15:57
gtristanI thought it already was successfully caching ybd's sources15:58
gtristanin which case, all you need is to configure ybd & buildstream to put their sources under the same cached subdir15:58
ssam2turns out it isn't15:58
gtristanreally ?15:58
gtristanwow15:58
ssam2look at the bottom of https://gitlab.com/baserock/definitions/-/jobs/31740842 for example15:58
ssam2its helpfully ignoring all the sources ybd downloaded because they're in /cache15:59
gtristanweird message15:59
ssam2my symlink trick seems to work though16:00
ssam2https://gitlab.com/baserock/definitions/-/jobs/3508267916:00
gtristanah right, in buildstream we cache things in cache/16:00
gtristannot /cache/16:01
gtristanseems probably a sensible limitation16:01
ssam2yeah16:01
gtristanallowing .gitlab-ci.yml to walk outside of it's cwd probably invites disaster, whenever you want to change your mounts and surrounding layout or whatever16:02
gtristan(or whenever gitlab wants to refactor how the workers work)16:02
gitlab-br-botinfrastructure: merge request (document-ci->master: Document how the Baserock runners for GitLab CI are set up) #9 changed state ("opened"): https://gitlab.com/baserock/infrastructure/merge_requests/916:09
*** ctbruce has quit IRC16:13
gitlab-br-botdefinitions: merge request (sam/fix-ci->master: Fix caching in CI) #62 changed state ("opened"): https://gitlab.com/baserock/definitions/merge_requests/6216:21
gitlab-br-botinfrastructure: merge request (document-ci->master: Document how the Baserock runners for GitLab CI are set up) #9 changed state ("merged"): https://gitlab.com/baserock/infrastructure/merge_requests/916:24
gitlab-br-botdefinitions: merge request (sam/fix-ci->master: Fix caching in CI) #62 changed state ("opened"): https://gitlab.com/baserock/definitions/merge_requests/6216:24
ironfootssam2: I found some issues in https://gitlab.com/baserock/infrastructure/merge_requests/9 but has already been merged16:29
ssam2oh sorry16:29
ssam2my badf16:30
ssam2*bad16:30
ssam2wait, this makes no sense16:30
ssam2ok, one of the commits in that branch should have been in the previous MR :_)(16:30
ssam2:-)16:30
ssam2ok yes, I see my mistake16:31
ssam2will do another MR16:31
ssam2argh, i just pushed the fix to master instead16:37
* ssam2 alters settings to make that less possible16:37
ssam2i've enabled merge request removals again, too. somehow the box was unticked even though i did tick it before16:38
ssam2s/removals/approvals/16:38
*** toscalix has quit IRC16:39
ssam2hmm, my changes still don't work16:43
* ssam2 watches as buildstream pulls the freedesktop sdk all over again16:43
*** jonathanmaw has quit IRC16:44
ssam2I guess the cache is extracted before the symlink is created16:45
gitlab-br-botdefinitions: merge request (sam/fix-ci->master: Fix caching in CI) #62 changed state ("opened"): https://gitlab.com/baserock/definitions/merge_requests/6216:53
*** paulwaters_ has joined #baserock16:54
*** paulwaters_ has quit IRC16:56
gitlab-br-botdefinitions: merge request (sam/fix-ci->master: Fix caching in CI) #62 changed state ("opened"): https://gitlab.com/baserock/definitions/merge_requests/6217:00
*** locallycompact has quit IRC17:07
*** ssam2 has quit IRC17:12

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!