| *** rdale has quit IRC | 01:15 | |
| *** laurenceurhegyi has quit IRC | 02:00 | |
| *** laurenceurhegyi has joined #baserock | 02:01 | |
| *** gary_perkins has quit IRC | 02:01 | |
| *** paulsherwood has quit IRC | 02:02 | |
| *** paulsherwood has joined #baserock | 02:02 | |
| *** gary_perkins has joined #baserock | 02:02 | |
| *** juergbi has quit IRC | 02:02 | |
| *** juergbi has joined #baserock | 02:03 | |
| *** gtristan has quit IRC | 04:52 | |
| *** gtristan has joined #baserock | 05:26 | |
| *** jude_ has joined #baserock | 07:30 | |
| *** toscalix has joined #baserock | 07:50 | |
| *** rdale has joined #baserock | 08:15 | |
| *** jonathanmaw has joined #baserock | 08:36 | |
| *** noisecell has joined #baserock | 08:43 | |
| *** ssam2 has joined #baserock | 08:58 | |
| *** ChanServ sets mode: +v ssam2 | 08:58 | |
| *** locallycompact has joined #baserock | 09:01 | |
| *** jude_ has quit IRC | 09:05 | |
| *** locallycompact has quit IRC | 09:06 | |
| *** locallycompact has joined #baserock | 09:07 | |
| ssam2 | now that we have caching of sources, the cache got full almost immediately | 09:08 |
|---|---|---|
| ssam2 | i figured that would happen eventually, but if it happens before the first CI pipeline is even finished then this will never work | 09:09 |
| gtristan | unless you can increase the cache size limit | 09:09 |
| gtristan | ssam2, there *might* be a nasty evil trick to play | 09:09 |
| gtristan | which is; since you are running ybd & buildstream, you are going to have a lot of git dups, e.g. linux.git | 09:10 |
| *** jude_ has joined #baserock | 09:10 | |
| gtristan | ssam2, it's quite plausible, almost 100% sure; that buildstream stores it's gits using the same uri mangling scheme | 09:10 |
| ssam2 | hmm, yes it does | 09:11 |
| ssam2 | that could almost half the size, so might be worth doing | 09:11 |
| gtristan | ssam2, of course it's only nasty because; that is internals, nobody is going to guarantee that keeps working :) | 09:11 |
| gtristan | chances are high, that it will anyway | 09:11 |
| ssam2 | true, but we can hopefully deprecate ybd ... soon ? | 09:11 |
| ssam2 | now ? | 09:11 |
| gtristan | and that can also ultimately happen too indeed | 09:11 |
| ssam2 | i guess we should wait for buildstream 1.0, since the current buildstream conversions will stop working after the options stuff lands | 09:11 |
| gtristan | Well, will it ? | 09:12 |
| ssam2 | depends if you remove arch conditionals | 09:12 |
| gtristan | ssam2, I'm quite sure nobody has been actively using variants | 09:12 |
| gtristan | oh that | 09:12 |
| gtristan | yes of course, that will be removed and require a change in the conversions | 09:12 |
| ssam2 | anyway, i have a feeling the cache we're using is just storing data in the root of a docker container, rather than having a volume | 09:13 |
| ssam2 | the actual VM has a 28GB free | 09:13 |
| gtristan | I wouldnt know... but ssam2 - I thought cache was defined globally in .gitlab-ci.yml, while the individual docker images can be specified on a per job basis right ? | 09:16 |
| ssam2 | we have a separate container functioning as a distributed cache | 09:17 |
| ssam2 | which is needed because we use on-demand runners, and so the usual kind of cache (a volume on the runner) is no use because the runners disappear when not used | 09:17 |
| ssam2 | and their cache volumes disappear along with them | 09:17 |
| ssam2 | https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching | 09:17 |
| gtristan | ahh, ok I'm out of my depth :) | 09:18 |
| ssam2 | hmm, looking at this again it's not the cache that ran out of space, it's the runner | 09:21 |
| ssam2 | while creating the archive to upload to the cache | 09:21 |
| ssam2 | wow seems like artifact fetching is broken in CI too | 09:23 |
| ssam2 | everything is broken1 | 09:23 |
| ssam2 | and its so much fun to debug when the runner where this build happened has already been deleted | 09:24 |
| * ssam2 steps away from this timesink for now | 09:25 | |
| paulsherwood | benbrown_: ^^ ? | 11:05 |
| ssam2 | it's probably my fault, not his :-) | 11:05 |
| * paulsherwood believes benbrown_ has lots of experience debugging dead runners | 11:05 | |
| *** benbrown_ has quit IRC | 11:13 | |
| *** paulsherwood has quit IRC | 11:13 | |
| *** benbrown_ has joined #baserock | 11:52 | |
| *** paulsherwood has joined #baserock | 11:53 | |
| *** SotK_ has joined #baserock | 12:41 | |
| *** SotK_ has left #baserock | 12:48 | |
| benbrown_ | ssam2, ironfoot: I think I noticed a highlight here before I disconnected | 13:02 |
| benbrown_ | Did someone ask me anything? | 13:02 |
| ironfoot | irclogs!!!! | 13:02 |
| ironfoot | benbrown_: https://irclogs.baserock.org/latest.log.html | 13:02 |
| benbrown_ | durr | 13:03 |
| benbrown_ | ty | 13:03 |
| benbrown_ | ah, right | 13:03 |
| benbrown_ | ssam2: What's going on with artifact fetching? | 13:05 |
| * benbrown_ isn't sure what's actually being passed around | 13:05 | |
| benbrown_ | I assume just the conversion? | 13:05 |
| benbrown_ | wrt huge cache, would using xz or something with greater compression that zip help? | 13:07 |
| benbrown_ | s/that/than/ | 13:07 |
| benbrown_ | brb | 13:07 |
| *** benbrown_ has quit IRC | 13:07 | |
| *** benbrown_ has joined #baserock | 13:12 | |
| * benbrown_ returns | 13:19 | |
| *** paulsherwood has quit IRC | 13:20 | |
| ssam2 | benbrown_, it might, i don't know if we can control that though | 13:37 |
| ssam2 | the issue was the runner itself running out of space. i changed the /cache volume to be at /builds so maybe i screwed that up | 13:37 |
| ssam2 | in terms of artifact fetching, it used to work so i think i just messed up the buildstream config such that it no longer works | 13:38 |
| benbrown_ | ssam2: as in, add a manual step to xz (adds some redundancy when GitLab then tries to zip, but might help) | 13:46 |
| benbrown_ | I have no idea what the internals of our runners look like, so I can't comment on the /builds move | 13:47 |
| *** paulsherwood has joined #baserock | 13:56 | |
| *** paulsherwood has quit IRC | 13:56 | |
| *** paulsherwood has joined #baserock | 13:57 | |
| *** gtristan has quit IRC | 14:57 | |
| *** anahuelamo has quit IRC | 15:16 | |
| *** chrispolin has quit IRC | 15:16 | |
| *** laurenceurhegyi has quit IRC | 15:16 | |
| *** gary_perkins has quit IRC | 15:16 | |
| *** locallycompact has quit IRC | 15:16 | |
| *** locallycompact has joined #baserock | 15:17 | |
| *** tiagogomes has quit IRC | 15:17 | |
| *** tiagogomes has joined #baserock | 15:18 | |
| *** edcragg has quit IRC | 15:19 | |
| *** noisecell has quit IRC | 15:20 | |
| *** edcragg has joined #baserock | 15:21 | |
| radiofree | getting 500 on http://git.baserock.org/cgit/ | 15:21 |
| *** locallycompact has quit IRC | 15:22 | |
| ssam2 | weird, the server is up | 15:23 |
| ssam2 | and less overloaded than before | 15:23 |
| ssam2 | although still overloaded | 15:23 |
| ssam2 | just worked for me then | 15:23 |
| ssam2 | it's still out of RAM | 15:24 |
| ssam2 | maybe needs the number of lorry workers reducing | 15:24 |
| *** jonathanmaw has quit IRC | 15:26 | |
| *** tiagogomes has quit IRC | 15:26 | |
| *** gtristan has joined #baserock | 15:26 | |
| *** jonathanmaw has joined #baserock | 15:26 | |
| *** tiagogomes has joined #baserock | 15:26 | |
| *** noisecell has joined #baserock | 15:31 | |
| *** laurenceurhegyi has joined #baserock | 15:45 | |
| *** chrispolin has joined #baserock | 15:45 | |
| *** anahuelamo has joined #baserock | 15:46 | |
| *** gary_perkins has joined #baserock | 15:46 | |
| *** jude_ has quit IRC | 16:00 | |
| *** adds68 has joined #baserock | 16:10 | |
| *** jude_ has joined #baserock | 16:17 | |
| *** noisecell has quit IRC | 16:38 | |
| *** paulwaters_ has joined #baserock | 16:52 | |
| *** toscalix has quit IRC | 16:58 | |
| *** jonathanmaw has quit IRC | 17:16 | |
| *** ssam2 has quit IRC | 17:21 | |
| *** jude_ has quit IRC | 17:36 | |
| *** rdale has quit IRC | 18:26 | |
| *** jude_ has joined #baserock | 20:54 | |
| *** jude_ has quit IRC | 22:17 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!