IRC logs for #baserock for Thursday, 2017-10-05

*** rdale has quit IRC01:15
*** laurenceurhegyi has quit IRC02:00
*** laurenceurhegyi has joined #baserock02:01
*** gary_perkins has quit IRC02:01
*** paulsherwood has quit IRC02:02
*** paulsherwood has joined #baserock02:02
*** gary_perkins has joined #baserock02:02
*** juergbi has quit IRC02:02
*** juergbi has joined #baserock02:03
*** gtristan has quit IRC04:52
*** gtristan has joined #baserock05:26
*** jude_ has joined #baserock07:30
*** toscalix has joined #baserock07:50
*** rdale has joined #baserock08:15
*** jonathanmaw has joined #baserock08:36
*** noisecell has joined #baserock08:43
*** ssam2 has joined #baserock08:58
*** ChanServ sets mode: +v ssam208:58
*** locallycompact has joined #baserock09:01
*** jude_ has quit IRC09:05
*** locallycompact has quit IRC09:06
*** locallycompact has joined #baserock09:07
ssam2now that we have caching of sources, the cache got full almost immediately09:08
ssam2i figured that would happen eventually, but if it happens before the first CI pipeline is even finished then this will never work09:09
gtristanunless you can increase the cache size limit09:09
gtristanssam2, there *might* be a nasty evil trick to play09:09
gtristanwhich is; since you are running ybd & buildstream, you are going to have a lot of git dups, e.g. linux.git09:10
*** jude_ has joined #baserock09:10
gtristanssam2, it's quite plausible, almost 100% sure; that buildstream stores it's gits using the same uri mangling scheme09:10
ssam2hmm, yes it does09:11
ssam2that could almost half the size, so might be worth doing09:11
gtristanssam2, of course it's only nasty because; that is internals, nobody is going to guarantee that keeps working :)09:11
gtristanchances are high, that it will anyway09:11
ssam2true, but we can hopefully deprecate ybd ... soon ?09:11
ssam2now ?09:11
gtristanand that can also ultimately happen too indeed09:11
ssam2i guess we should wait for buildstream 1.0, since the current buildstream conversions will stop working after the options stuff lands09:11
gtristanWell, will it ?09:12
ssam2depends if you remove arch conditionals09:12
gtristanssam2, I'm quite sure nobody has been actively using variants09:12
gtristanoh that09:12
gtristanyes of course, that will be removed and require a change in the conversions09:12
ssam2anyway, i have a feeling the cache we're using is just storing data in the root of a docker container, rather than having a volume09:13
ssam2the actual VM has a 28GB free09:13
gtristanI wouldnt know... but ssam2 - I thought cache was defined globally in .gitlab-ci.yml, while the individual docker images can be specified on a per job basis right ?09:16
ssam2we have a separate container functioning as a distributed cache09:17
ssam2which is needed because we use on-demand runners, and so the usual kind of cache (a volume on the runner) is no use because the runners disappear when not used09:17
ssam2and their cache volumes disappear along with them09:17
ssam2https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching09:17
gtristanahh, ok I'm out of my depth :)09:18
ssam2hmm, looking at this again it's not the cache that ran out of space, it's the runner09:21
ssam2while creating the archive to upload to the cache09:21
ssam2wow seems like artifact fetching is broken in CI too09:23
ssam2everything is broken109:23
ssam2and its so much fun to debug when the runner where this build happened has already been deleted09:24
* ssam2 steps away from this timesink for now09:25
paulsherwoodbenbrown_: ^^ ?11:05
ssam2it's probably my fault, not his :-)11:05
* paulsherwood believes benbrown_ has lots of experience debugging dead runners11:05
*** benbrown_ has quit IRC11:13
*** paulsherwood has quit IRC11:13
*** benbrown_ has joined #baserock11:52
*** paulsherwood has joined #baserock11:53
*** SotK_ has joined #baserock12:41
*** SotK_ has left #baserock12:48
benbrown_ssam2, ironfoot: I think I noticed a highlight here before I disconnected13:02
benbrown_Did someone ask me anything?13:02
ironfootirclogs!!!!13:02
ironfootbenbrown_: https://irclogs.baserock.org/latest.log.html13:02
benbrown_durr13:03
benbrown_ty13:03
benbrown_ah, right13:03
benbrown_ssam2: What's going on with artifact fetching?13:05
* benbrown_ isn't sure what's actually being passed around13:05
benbrown_I assume just the conversion?13:05
benbrown_wrt huge cache, would using xz or something with greater compression that zip help?13:07
benbrown_s/that/than/13:07
benbrown_brb13:07
*** benbrown_ has quit IRC13:07
*** benbrown_ has joined #baserock13:12
* benbrown_ returns13:19
*** paulsherwood has quit IRC13:20
ssam2benbrown_, it might, i don't know if we can control that though13:37
ssam2the issue was the runner itself running out of space. i changed the /cache volume to be at /builds so maybe i screwed that up13:37
ssam2in terms of artifact fetching, it used to work so i think i just messed up the buildstream config such that it no longer works13:38
benbrown_ssam2: as in, add a manual step to xz (adds some redundancy when GitLab then tries to zip, but might help)13:46
benbrown_I have no idea what the internals of our runners look like, so I can't comment on the /builds move13:47
*** paulsherwood has joined #baserock13:56
*** paulsherwood has quit IRC13:56
*** paulsherwood has joined #baserock13:57
*** gtristan has quit IRC14:57
*** anahuelamo has quit IRC15:16
*** chrispolin has quit IRC15:16
*** laurenceurhegyi has quit IRC15:16
*** gary_perkins has quit IRC15:16
*** locallycompact has quit IRC15:16
*** locallycompact has joined #baserock15:17
*** tiagogomes has quit IRC15:17
*** tiagogomes has joined #baserock15:18
*** edcragg has quit IRC15:19
*** noisecell has quit IRC15:20
*** edcragg has joined #baserock15:21
radiofreegetting 500 on http://git.baserock.org/cgit/15:21
*** locallycompact has quit IRC15:22
ssam2weird, the server is up15:23
ssam2and less overloaded than before15:23
ssam2although still overloaded15:23
ssam2just worked for me then15:23
ssam2it's still out of RAM15:24
ssam2maybe needs the number of lorry workers reducing15:24
*** jonathanmaw has quit IRC15:26
*** tiagogomes has quit IRC15:26
*** gtristan has joined #baserock15:26
*** jonathanmaw has joined #baserock15:26
*** tiagogomes has joined #baserock15:26
*** noisecell has joined #baserock15:31
*** laurenceurhegyi has joined #baserock15:45
*** chrispolin has joined #baserock15:45
*** anahuelamo has joined #baserock15:46
*** gary_perkins has joined #baserock15:46
*** jude_ has quit IRC16:00
*** adds68 has joined #baserock16:10
*** jude_ has joined #baserock16:17
*** noisecell has quit IRC16:38
*** paulwaters_ has joined #baserock16:52
*** toscalix has quit IRC16:58
*** jonathanmaw has quit IRC17:16
*** ssam2 has quit IRC17:21
*** jude_ has quit IRC17:36
*** rdale has quit IRC18:26
*** jude_ has joined #baserock20:54
*** jude_ has quit IRC22:17

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!