| *** narispo has joined #buildstream | 00:53 | |
| *** narispo has quit IRC | 07:25 | |
| *** narispo has joined #buildstream | 07:25 | |
| *** narispo has quit IRC | 08:01 | |
| *** narispo has joined #buildstream | 08:03 | |
| *** narispo has quit IRC | 08:06 | |
| *** narispo has joined #buildstream | 08:07 | |
| *** traveltissues has joined #buildstream | 09:01 | |
| *** phildawson-ct has joined #buildstream | 09:26 | |
| *** ikerperez has joined #buildstream | 09:31 | |
| *** benschubert has joined #buildstream | 09:31 | |
| *** tme5 has joined #buildstream | 09:33 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1826 (juerg/buildbox-signals->master: _sandboxbuildboxrun.py: Fix signal handling) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1826 | 09:34 |
|---|---|---|
| gitlab-br-bot | juergbi merged MR !1826 (juerg/buildbox-signals->master: _sandboxbuildboxrun.py: Fix signal handling) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/1826 | 09:44 |
| *** santi has joined #buildstream | 09:49 | |
| *** jonathanmaw has joined #buildstream | 10:23 | |
| WSalmon | benschubert, git-lfs is a nice example as we have done a load to help support it so i have half a idea about it, but i am no expert. basicly if you dont use the git_tag plugin and use the core git plugin then you get diffrent result when cloneing git-lfs repos depending on wheather or not the system git has gitlfs installed. but this comes from the way git manges hooks, if system git has hooks installed then they may alter what happenes, very | 10:41 |
| WSalmon | popular hooks like gitlfs, (git-review, are commit rather than checkout but are one by default) are on by default. I think we have plenty on our plate for bst2 but i think not using system tools for sources is a big step towards reproducibility for bst3. | 10:41 |
| WSalmon | *if you try to use gitlfs with master the diffrence is that it brakes rather than you getting the objects but you still get diffrent behavoiur | 10:42 |
| WSalmon | /master/core | 10:43 |
| benschubert | WSalmon: > if system git has hooks installed then they may alter what happenes, very | 10:44 |
| benschubert | Seems like the fix here is to provide a config that disables the hooks we don't want forcefully? | 10:44 |
| WSalmon | if the hook allows for that then great but i dont know how well that would scale across a variety of plug ins, it also sounds like a lot of maintenance conpaired to the one off cost of setting it up. i dont think its a battle for any time soon but i think its something to fix if long turm/ heterogeneous run environments reproducibility is important | 10:56 |
| juergbi | WSalmon: the remote asset API might help here as well. for reproducibility you also need a local copy of all sources as you can't expect all sources to remain available upstream indefinitely | 11:20 |
| juergbi | if you keep those in non-expiring CAS with a remote asset server as frontend, the `git` tool is not necessary on the client side | 11:20 |
| juergbi | this is not supported yet but there is a recent ML thread about it | 11:21 |
| *** cs-shadow has joined #buildstream | 11:26 | |
| WSalmon | i think a lorry is a good idea from baserock, but that brakes if your on some corprate system that mandates that you use a newer local source. also source cache is a really good part of this story | 11:41 |
| WSalmon | good point juergbi | 11:42 |
| coldtom | if an artifact pull fails for somehow missing blobs, do people think that the entire build should fail? imo bst should continue and build that component locally (of course, bst artifact pull should fail though) | 11:46 |
| cs-shadow | coldtom: I agree. If the server is missing blobs, then the client should behave as if the artifact was not cached on the server, i.e. falling back to local builds if posible | 11:52 |
| *** phil has quit IRC | 12:33 | |
| juergbi | fallback is not trivial as build-only dependencies of that element may not be in the pipeline | 12:37 |
| *** delli3 has joined #buildstream | 12:52 | |
| benschubert | I think the reason why this happens is because some local artifacts are not checked correctly for completeness, because otherwise, we wouldn't be marking the thing as 'cached' and it would then have its build dependencies included, right? | 13:29 |
| WSalmon | juergbi, is that right? normally bst tries to pull the target element, if it is not locally cached and then works back from there. i think the issues is that if when it checks the report says it is there but then there is a failur mid fetch then you get a hard crash rather than it being treated as if the artifct server had just said its not there? | 13:39 |
| *** toscalix has joined #buildstream | 13:52 | |
| *** traveltissues has quit IRC | 13:53 | |
| *** traveltissues has joined #buildstream | 13:54 | |
| *** toscalix has quit IRC | 13:54 | |
| *** flatmush has quit IRC | 13:55 | |
| *** traveltissues has quit IRC | 13:57 | |
| coldtom | juergbi: i'd only want this fallback in a build pipeline, i don't think it'd make sense anywhere else, surely a `bst build` pipeline contains the build deps of everything in it by necessity? | 14:00 |
| *** santi has quit IRC | 14:00 | |
| *** santi has joined #buildstream | 14:00 | |
| benschubert | coldtom: not necessarily, because you might have things cached at the start | 14:01 |
| coldtom | but if i have something cached locally, then surely its build dependencies are irrelevant in this case | 14:03 |
| juergbi | right, regular pull in a build pipeline doesn't have an issue with fallback | 14:19 |
| juergbi | there are two fallback-related issues, though | 14:19 |
| juergbi | one is we can't fallback to building if the artifact is considered locally cached during session initialization. however, this issue should normally not arise because we block required blobs from being expired by casd | 14:21 |
| juergbi | there is still an open bug with regards to logs/metadata files, though, iirc. shouldn't be hard to fix | 14:21 |
| juergbi | the second issue is with remote execution when the user doesn't ask for all blobs to be present in the local cache. remote CAS servers don't provide absolute guarantees with regards to when blobs are deleted | 14:22 |
| juergbi | and thus it's possible that an artifact is available in a remote CAS at the beginning of a session but becomes unavailable as part of a build | 14:23 |
| juergbi | in that case the whole pipeline should likely be retried (still reusing cached artifacts, of course). if it happens frequently, the CAS server is too aggressive in expiry or doesn't have enough disk space | 14:24 |
| *** toscalix has joined #buildstream | 14:54 | |
| *** santi has quit IRC | 15:05 | |
| *** santi has joined #buildstream | 15:05 | |
| *** traveltissues has joined #buildstream | 15:08 | |
| *** santi has quit IRC | 15:53 | |
| *** santi has joined #buildstream | 15:55 | |
| *** phildawson-ct has quit IRC | 16:25 | |
| *** phildawson has joined #buildstream | 16:33 | |
| *** phildawson_ has joined #buildstream | 16:37 | |
| *** phildawson has quit IRC | 16:38 | |
| *** phildawson_ has quit IRC | 16:41 | |
| *** phildawson has joined #buildstream | 16:41 | |
| *** toscalix has quit IRC | 16:46 | |
| *** toscalix has joined #buildstream | 16:58 | |
| *** toscalix has quit IRC | 17:14 | |
| *** narispo has quit IRC | 17:18 | |
| *** phildawson has quit IRC | 17:25 | |
| *** tme5 has quit IRC | 17:30 | |
| *** narispo has joined #buildstream | 17:35 | |
| *** traveltissues has quit IRC | 17:52 | |
| *** flatmush has joined #buildstream | 17:57 | |
| *** santi has quit IRC | 18:05 | |
| *** rdale has quit IRC | 18:14 | |
| *** jonathanmaw has quit IRC | 18:20 | |
| *** slaf_ has joined #buildstream | 18:34 | |
| *** slaf_ has joined #buildstream | 18:35 | |
| *** slaf has quit IRC | 18:36 | |
| *** slaf_ is now known as slaf | 18:36 | |
| *** narispo has quit IRC | 18:53 | |
| *** narispo has joined #buildstream | 18:53 | |
| *** traveltissues has joined #buildstream | 19:40 | |
| *** benschubert has quit IRC | 20:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!