| *** semanticdesign has quit IRC | 01:07 | |
| *** semanticdesign_ has quit IRC | 01:07 | |
| *** tristan has joined #buildstream | 06:06 | |
| *** jude has joined #buildstream | 07:45 | |
| *** jonathanmaw has joined #buildstream | 08:34 | |
| *** anahuelamo has quit IRC | 08:44 | |
| *** anahuelamo has joined #buildstream | 08:44 | |
| *** anahuelamo has quit IRC | 08:46 | |
| *** anahuelamo has joined #buildstream | 08:46 | |
| *** bochecha has joined #buildstream | 08:47 | |
| *** jude has quit IRC | 08:58 | |
| *** jude has joined #buildstream | 09:14 | |
| gitlab-br-bot | buildstream: issue #119 ("Cryptic error when using .bsts with arches") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/119 | 09:15 |
|---|---|---|
| *** ssam2 has joined #buildstream | 09:16 | |
| *** jude has quit IRC | 09:21 | |
| *** jude has joined #buildstream | 09:21 | |
| *** tlater has joined #buildstream | 09:24 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: _yaml.py: Fixes #119 - Custom implementation of ChainMap) https://gitlab.com/BuildStream/buildstream/commit/6f820111052a85b1051bb7bf3aed5475a0470e21 | 09:25 |
| gitlab-br-bot | buildstream: issue #119 ("Cryptic error when using .bsts with arches") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/119 | 09:25 |
| tristan | jonathanmaw, you might be interested to know what that error was; it ended up being much more complicated than I thought :) | 09:25 |
| jonathanmaw | tristan: oh, what was it? | 09:26 |
| tristan | the link above tells: https://gitlab.com/BuildStream/buildstream/commit/6f820111052a85b1051bb7bf3aed5475a0470e21 | 09:26 |
| tristan | That patch could be a bit neater and cleaner; and should probably derive from MutableMapping, and do the recursive copying bits below on it's own | 09:28 |
| tristan | But meh, it does the trick for now, and needed a quick fix | 09:28 |
| tristan | feel free to improve on it if anyone likes :D | 09:28 |
| tristan | mutable copy-on-write dictionaries are pretty neat :) | 09:28 |
| *** tiagogomes has quit IRC | 09:34 | |
| *** tiagogomes has joined #buildstream | 09:34 | |
| gitlab-br-bot | push on buildstream@74-prevent-artifacts-from-containing-files-in-buildstream-build (by Tristan Maat): 1 commit (last: Add warnings when staging to /buildstream/build) https://gitlab.com/BuildStream/buildstream/commit/266aa6032d507bc97c1bf1da5fc3148f0c69198e | 09:36 |
| gitlab-br-bot | buildstream: merge request (74-prevent-artifacts-from-containing-files-in-buildstream-build->master: Add warnings when staging to /buildstream/build) #93 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/93 | 09:36 |
| *** tiagogomes has quit IRC | 09:38 | |
| *** tiagogomes has joined #buildstream | 09:39 | |
| *** tiagogomes has quit IRC | 09:42 | |
| *** tiagogomes has joined #buildstream | 09:42 | |
| *** tiagogomes has quit IRC | 09:44 | |
| *** tiagogomes has joined #buildstream | 09:45 | |
| *** tiagogomes has quit IRC | 09:46 | |
| *** tiagogomes has joined #buildstream | 09:46 | |
| *** tiagogomes has quit IRC | 09:49 | |
| *** tiagogomes has joined #buildstream | 09:49 | |
| tlater | tristan: Would you mind having a look at !106 (the ctrl-c one) again? I'd really like to get that one out of the way. | 09:49 |
| *** anahuelamo has quit IRC | 09:55 | |
| *** anahuelamo has joined #buildstream | 09:55 | |
| *** givascu has joined #buildstream | 10:04 | |
| *** tiagogomes has quit IRC | 10:07 | |
| *** tiagogomes has joined #buildstream | 10:07 | |
| *** jude has quit IRC | 10:07 | |
| *** jude has joined #buildstream | 10:07 | |
| *** tiagogomes_ has joined #buildstream | 10:11 | |
| *** tiagogomes has left #buildstream | 10:11 | |
| *** jonathanmaw has quit IRC | 10:11 | |
| *** jonathanmaw has joined #buildstream | 10:11 | |
| *** laurenceurhegyi has joined #buildstream | 10:13 | |
| tristan | tlater, yeah... | 10:24 |
| tristan | tlater, me too, but... huge loop ? | 10:24 |
| tristan | ugh... | 10:24 |
| tlater | tristan: I really have no idea how to work around that otherwise, since sometimes C-c shouldn't make us stop waiting | 10:25 |
| tlater | I can factor it out into another function | 10:25 |
| tlater | But it will still be a huge loop :/ | 10:25 |
| tristan | also, what's with the addition of artifactcache.py:fetch_remote_refs() ? | 10:26 |
| tlater | Ah, that seems to be a regression of some sort; when artifact caches are configured the linux platform can't run because buildstream looks for that function | 10:27 |
| tlater | It's in a separate commit, though I can set up an independent issue for that. | 10:27 |
| tristan | I see, I always look at the "changes" and missed that | 10:27 |
| tristan | if it's needed and in a separate commit, no need to split it out | 10:27 |
| tristan | tlater, why is process.communicate() removed ? | 10:28 |
| tlater | tristan: I thought the idea was to use os.waitpid for that now instead? | 10:28 |
| tristan | tlater, I thought we wanted to cause least disruption to the subprocess module as possible, and only afterwards step in and check the exit status with waitpid() ? | 10:29 |
| tristan | And wondered if it's indeed plausible, i.e. in that context do we still have the exit status around ? | 10:30 |
| tlater | Yeah, I assumed we wouldn't, but I never actually tried | 10:30 |
| * tlater will do that | 10:30 | |
| tristan | currently subprocess I believe makes a reasonable assumption, but at the same time, I dont normally see C code making that assumption | 10:31 |
| tristan | I.e. that a posix compliant system will report a wrapped negative return code (turning -1 into 127) | 10:32 |
| tristan | and reserving < 0 returns for crashes | 10:32 |
| tristan | tlater, that would explain why < 0 was perhaps reasonable | 10:32 |
| tristan | and if it's that much less code, maybe we should run with that and a detailed comment | 10:32 |
| *** bochecha_ has joined #buildstream | 10:33 | |
| tlater | tristan: Unfortunately I think the loop is actually required for either approach | 10:33 |
| *** adds68 has quit IRC | 10:33 | |
| *** adds68 has joined #buildstream | 10:33 | |
| tristan | Ahhh I see | 10:33 |
| tlater | There would be one less 'try' for the bwrap process, perhaps | 10:33 |
| tristan | tlater, so basically, the loop handles the case that if there is a keyboard interrupt, it might not mean the end of the process ? | 10:34 |
| tlater | Yep | 10:34 |
| tristan | I.e. bash handling it | 10:34 |
| tristan | Ok | 10:34 |
| tristan | That makes sense to me now | 10:34 |
| tristan | however, I suppose in that case, communicate has to go ? | 10:34 |
| *** bochecha has quit IRC | 10:35 | |
| *** bochecha_ is now known as bochecha | 10:35 | |
| tlater | tristan: It could be either, the loop could be around communicate instead. | 10:35 |
| tlater | If os.waitpid still gets the status *after* the process ends, that is | 10:35 |
| tristan | tlater, because communicate will block; are we allowed to re-enter communicate ? | 10:35 |
| tlater | tristan: communicate is also interrupted by a KeyboardInterrupt, so I assume yes | 10:36 |
| tristan | so confusing | 10:36 |
| tristan | OK - so os.waitpid() blocks indefinitely; and this does not interfere with stdin/stdout | 10:37 |
| tristan | i.e. not using communicate() | 10:37 |
| tristan | tlater, lets close the book on this with the patch as is, seeing I now understand why the loop is needed | 10:37 |
| tristan | If | 10:38 |
| tristan | tlater, if only we are sure about safeness of omitting communicate | 10:38 |
| tlater | tristan: I believe it is - although I haven't tried it in non-interactive mode | 10:39 |
| tristan | tlater, you might test this with say... echo "foo" | bst shell element.bst -- cat | 10:39 |
| tristan | or such | 10:39 |
| tlater | tristan: There were some comments about signal handling on stackoverflow about using os.waitpid | 10:40 |
| tristan | Any links ? | 10:40 |
| tristan | Or just a random vague thing I dont really want to go searching for ? | 10:41 |
| tlater | Probably not worth searching for, I don't have a link anymore :/ | 10:41 |
| ssam2 | seems i have broken the docs build | 10:41 |
| ssam2 | https://gitlab.com/BuildStream/buildstream/-/jobs/36752179 | 10:41 |
| tlater | tristan: Cat receives 'foo', is that the expected behaviour? | 10:44 |
| tristan | tlater, in interactive shells yeah we want stdio/stdout connected to invoking terminal | 10:46 |
| * tlater wonders *why* it is connected, but hey | 10:46 | |
| tristan | tlater, I was wondering if communicate() was needed, seeing as it is documented to interfere with stdio in some way | 10:46 |
| *** tiagogomes has joined #buildstream | 10:47 | |
| tristan | in previous s/stdio/stdin | 10:47 |
| tristan | tlater, naturally it is inherited, we pass it explicitly sys.stdin/sys.stdout etc | 10:47 |
| *** tiagogomes_ has quit IRC | 10:47 | |
| tristan | tlater, and we share the session - in a scheduled task context, stdin is devnull I think, and stdout/stderr is the log file | 10:47 |
| tlater | tristan: Is it ever a pipe? | 10:48 |
| tristan | tlater, yes, when subprocess special macros are used | 10:48 |
| tlater | It's possible that communicate() needs to interfere in that case. | 10:48 |
| tristan | but afaics, not with the sandbox | 10:49 |
| tristan | Maybe communicate is *only* for that purpose | 10:49 |
| tristan | tlater, ok so... can you just; build something and ensure that it's log file has something in it ? | 10:49 |
| tristan | something that comes from the sandbox that is, not our custom things we add to that | 10:50 |
| tlater | tristan: Sure. I'd also like to try pausing a build just to make sure signal handling still works | 10:50 |
| tristan | should be fine, but yes please go ahead :) | 10:50 |
| tristan | ssam2, I dont know, that looks fraking weird :-/ | 10:53 |
| ssam2 | yeah, good old pip | 10:53 |
| ssam2 | I guess it's the `-e` flag | 10:53 |
| ssam2 | which we don't need right? that's only useful when hacking locally | 10:53 |
| ssam2 | yeah, it's "editable" mode | 10:54 |
| tristan | ssam2, anyway we're not ensuring that for docs anyway, so we dont need the --no-index ? | 10:54 |
| tristan | we're already downloading stuff | 10:54 |
| ssam2 | sure | 10:54 |
| ssam2 | i'll remove both | 10:54 |
| tristan | And it's already been ensured to be provided by previous tests (i.e. it wont trigger downloads without --no-index anyway) | 10:54 |
| tristan | the sphinx stuff will of course download | 10:55 |
| tristan | but not pip install . | 10:55 |
| tlater | tristan: Heh, looking at the logs, it seems that the exit code isn't preserved this way... | 10:58 |
| gitlab-br-bot | push on buildstream@sam/fix-docs (by Sam Thursfield): 1 commit (last: .gitlab-ci.yml: Fix documentation build) https://gitlab.com/BuildStream/buildstream/commit/2d5b24d8fc3c2aab5520f48100221b5a45c6e7a9 | 11:09 |
| gitlab-br-bot | buildstream: merge request (sam/fix-docs->master: .gitlab-ci.yml: Fix documentation build) #116 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/116 | 11:09 |
| gitlab-br-bot | push on buildstream@sam/fix-docs (by Sam Thursfield): 1 commit (last: .gitlab-ci.yml: Fix documentation build) https://gitlab.com/BuildStream/buildstream/commit/66458761fbd3791065dd37305bd385f3eeee5c1e | 11:11 |
| gitlab-br-bot | buildstream: merge request (sam/fix-docs->master: .gitlab-ci.yml: Fix documentation build) #116 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/116 | 11:11 |
| *** adds68 has quit IRC | 11:16 | |
| *** adds68 has joined #buildstream | 11:16 | |
| *** laurenceurhegyi is now known as ltu | 11:21 | |
| jonathanmaw | O frabjous day, calloh callay! I've figured out what was going wrong with trying to generate sphinx documentation - I wasn't seeing my changes get through to the documentation because my PYTHONPATH was wrong, so it fell back on the old, installed version | 12:01 |
| *** xjuan has joined #buildstream | 12:48 | |
| gitlab-br-bot | push on buildstream@98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shell (by Tristan Maat): 4 commits (last: .gitlab-ci.yml: Issue #100 - Avoid installing setup.py test_requires) https://gitlab.com/BuildStream/buildstream/commit/a243a3335b7c113a49b1d5f0daa2c11efe2299bd | 13:04 |
| gitlab-br-bot | buildstream: merge request (98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shell->master: Fix keyboardinterrupts caused by subprocesses) #106 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/106 | 13:04 |
| tristan | tlater, gah, sorry I totally dropped off there and missed this | 13:26 |
| tlater | tristan: nw, I've already found a fix for it | 13:26 |
| tlater | Writing to log files also seems to work fine | 13:26 |
| tristan | tlater, ok lets run with this | 13:27 |
| tristan | It's damn hard problem space, and it's possible that observing subprocess communicate 'returncode' for -1 is "correct" for posix compliant platforms | 13:28 |
| tristan | but eh; WIFEXITED and WEXITSTATUS seems to me a bit safer | 13:29 |
| * tlater disliked the python documentation calling the WIFEXITSTATUS 'meaningless' | 13:30 | |
| tlater | Assuming it could just end up as 0, and then cause issues, I figured it would be safer to return a known falsey value | 13:30 |
| gitlab-br-bot | push on buildstream@remove-arches (by Tristan Van Berkom): 13 commits (last: .gitlab-ci.yml: Use `pip3 install --no-index` when installing BuildStream) https://gitlab.com/BuildStream/buildstream/commit/fd91491a7619964e01d1fc8dede1c07cb68a433f | 13:31 |
| gitlab-br-bot | buildstream: issue #120 ("Remove 'arches' conditionals") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/120 | 13:43 |
| gitlab-br-bot | buildstream: issue #121 ("Remove pre-/post- command prefix support") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/121 | 13:45 |
| gitlab-br-bot | buildstream: merge request (remove-arches->master: WIP: Remove arches) #117 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/117 | 13:46 |
| gitlab-br-bot | buildstream: merge request (98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shell->master: Fix keyboardinterrupts caused by subprocesses) #106 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/106 | 13:52 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch 98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shell | 13:52 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: Fix keyboardinterrupts caused by subprocesses) https://gitlab.com/BuildStream/buildstream/commit/23c1e284ea748942e7067454aae2b279a1ab0e68 | 13:52 |
| tlater | \o/ | 13:54 |
| gitlab-br-bot | push on buildstream@74-prevent-artifacts-from-containing-files-in-buildstream-build (by Tristan Van Berkom): 5 commits (last: .gitlab-ci.yml: Issue #100 - Avoid installing setup.py test_requires) https://gitlab.com/BuildStream/buildstream/commit/a243a3335b7c113a49b1d5f0daa2c11efe2299bd | 13:55 |
| gitlab-br-bot | buildstream: merge request (74-prevent-artifacts-from-containing-files-in-buildstream-build->master: Add warnings when staging to /buildstream/build) #93 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/93 | 13:55 |
| tristan | tlater, ok so I was just about to merge the other... but you should look | 13:55 |
| tristan | tlater, not sure how to link to it, but regarding MR 93... element.py has a completely useless diff | 13:56 |
| tristan | removes an empty line, imports pathlib (wat?!) and imports SourceError... and that's it | 13:57 |
| tlater | Ah, yeah, presumably that's leftover from removing the SourceError dance | 13:57 |
| tristan | source.py... changes the line where tempfile is imported ? | 13:57 |
| tristan | weird | 13:57 |
| * tlater missed these changes | 13:58 | |
| tristan | tlater, in case you were not aware... introducing _only_ diffs that are strictly related to what is being done, is very much a goal | 13:58 |
| tristan | I mean really, you might easily not be aware :) | 13:58 |
| tristan | The reason for this, is to increase the value of `git annotate` | 13:58 |
| tristan | and `git blame` | 13:59 |
| tristan | over time | 13:59 |
| tlater | I'll keep better track of that, I tend to do so but didn't prioritize it - wasn't aware there was an actual reason | 13:59 |
| tristan | tlater, good, then I'm telling you something useful :) | 14:00 |
| tlater | :) | 14:00 |
| tristan | tlater, for instance; look at a line of code you're not sure what the hell it's doing there - and look at the file with `git annotate` | 14:00 |
| tristan | then, you should be always able to trace back line-of-code -> commit-which-introduces -> Ideally linked bug report from commit message | 14:01 |
| tristan | And get the whole story behind any line of code | 14:01 |
| tristan | Which reminds me, we're going to have to start linking issue numbers from commit messages | 14:02 |
| tristan | Ideally subject line runs as: Topic: Issue #number - Short Description | 14:03 |
| tristan | We've been working around that and tiptoeing because of that gitlab issue that annoyingly goes and closes stuff | 14:03 |
| tristan | But lets change that, it's not doing us good, lets try to refer to issues as "Issue #number" | 14:04 |
| tristan | and only "Fixes #number" or "Closes #number", if it's like a really damn obvious thing | 14:04 |
| tristan | but anyway, point is to not be *too* strict, but keep our provenance in tact | 14:05 |
| * tristan doesnt care *that* much about subject line format | 14:05 | |
| tristan | But, I should start rejecting patches based on failing to link issue number from commit (and also making sure I practice that as well) | 14:06 |
| gitlab-br-bot | push on buildstream@74-prevent-artifacts-from-containing-files-in-buildstream-build (by Tristan Maat): 1 commit (last: Add warnings when staging to /buildstream/build) https://gitlab.com/BuildStream/buildstream/commit/10847a66dc0748696829d5fd18ad17043c2c6878 | 14:07 |
| gitlab-br-bot | buildstream: merge request (74-prevent-artifacts-from-containing-files-in-buildstream-build->master: Add warnings when staging to /buildstream/build) #93 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/93 | 14:07 |
| ssam2 | something going quite wrong with the integration_unix test in gitlab CI ... | 14:07 |
| ssam2 | e.g. https://gitlab.com/BuildStream/buildstream/-/jobs/36843167 | 14:07 |
| ssam2 | it just times out after an hour | 14:07 |
| ssam2 | but it looked like it had finished so... weird | 14:08 |
| tlater | Humm... I think it just happened to time out while pushing the cache | 14:08 |
| * tlater has had that issue occasionally | 14:08 | |
| tristan | yeah that sounds likely | 14:08 |
| ssam2 | seems the timeout can be increased | 14:08 |
| tristan | That stuff needs optimization | 14:08 |
| ssam2 | i'll try doubling it for now | 14:08 |
| tlater | tristan: We could start using "Does unspeakable things to #number" in protest | 14:08 |
| tristan | ssam2, can it, with gitlab.com runners ? | 14:08 |
| ssam2 | yeah | 14:09 |
| ssam2 | https://gitlab.com/BuildStream/buildstream/settings/ci_cd | 14:09 |
| tristan | cool | 14:09 |
| tristan | lets increase that number | 14:09 |
| ssam2 | i've set it to 120 (was 60) | 14:09 |
| tristan | good | 14:09 |
| tristan | still we need to optimized that... some of it is still using that old crack I cooked up | 14:09 |
| tlater | \o/ No more unnecessary timeouts :) | 14:09 |
| tristan | before I understood how the layout of a runtime worked | 14:10 |
| tristan | that and, we'll be able to leverage the tiny runtimes too | 14:10 |
| ssam2 | yeah... although still a few blockers to Baserock producing those | 14:12 |
| ssam2 | mainly the Baserock CI is still broken due to cache getting full | 14:12 |
| tristan | So - I didnt get around to converting baserock from 'arches' | 14:13 |
| tristan | I did it for GNOME conversions | 14:13 |
| tristan | And, I really need to focus right now on GNOME, time is tight | 14:14 |
| tristan | Can someone else take care of baserock ? the conversion related bits should be ridiculously easy to change, at least for arches | 14:14 |
| tristan | For the pre-/post- stuff, might need a bit of work in the script itself | 14:14 |
| tristan | just CI for baserock being broken is annoying, but we should not be blocking on this I think, it's good time for brute force | 14:15 |
| ssam2 | i can put it on my list; it will be low priority though | 14:15 |
| tristan | sigh | 14:15 |
| ssam2 | do you need it urgently for something ? | 14:16 |
| tristan | Ok - I dont know but... those baserockers are not fixing their CI - and it feels a bit wrong to block our progress... people who dont care ? | 14:16 |
| tristan | ssam2, It's a blocker for us to remove arches support, currently | 14:16 |
| tristan | which is a blocker for 1.0 | 14:16 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 3 commits (last: .gitlab-ci.yml: Issue #100 - Avoid installing setup.py test_requires) https://gitlab.com/BuildStream/buildstream/commit/a243a3335b7c113a49b1d5f0daa2c11efe2299bd | 14:16 |
| tristan | So, maybe we should just not block ? | 14:16 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 14:16 |
| ssam2 | feel free to break the BuildStream conversions of Baserock definitions | 14:17 |
| ssam2 | I never declared them to be anything more than a proof of concept | 14:17 |
| ssam2 | the minimal runtime part is the only interesting thing that might come out of it... but that could equally be done as part of adds68's project | 14:17 |
| tristan | ssam2, myeah... *you* did not... I made them work a long time ago, and ensuring that worked is kind of very important to our project | 14:17 |
| ssam2 | i think more people cared about baserock back then | 14:18 |
| tristan | ssam2, there are projects who couldnt care less about baserock, but need the YBD conversions to work | 14:19 |
| tristan | and baserock is the only place to ensure that works | 14:19 |
| ssam2 | hmm, true | 14:19 |
| tristan | afaik | 14:19 |
| ssam2 | ok, well i imagine ltu can find someone to hand that task off to | 14:20 |
| * tristan hopes :) | 14:20 | |
| ssam2 | i mean, i'm happy to do it, just still wrestling feature proposals which i imagine to be more important | 14:21 |
| tristan | to be honest; *I'm* not personally in a hurry to roll out "1.0" | 14:21 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/aab260eec99d7467266e0288aaba995bdb58e274 | 14:21 |
| tristan | but it would be nice to have it out soon, and also to drop the deprecated stuff soon | 14:21 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 14:21 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/bdd507f231e4f4b584d16e9275ff2c5bace443b9 | 14:26 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 14:26 |
| gitlab-br-bot | push on buildstream@remove-pre-post-commands (by Tristan Van Berkom): 3 commits (last: buildelement.py: Issue #121 - Remove traces of pre-/post- commands) https://gitlab.com/BuildStream/buildstream/commit/604c1f76a22da9e9b144eba412028fdfae1c9e85 | 14:26 |
| gitlab-br-bot | push on buildstream@remove-pre-post-commands (by Tristan Van Berkom): 5 commits (last: Fix keyboardinterrupts caused by subprocesses) https://gitlab.com/BuildStream/buildstream/commit/23c1e284ea748942e7067454aae2b279a1ab0e68 | 14:27 |
| gitlab-br-bot | buildstream: merge request (remove-pre-post-commands->master: WIP: Remove pre post commands) #118 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/118 | 14:28 |
| gitlab-br-bot | push on buildstream@remove-arches (by Tristan Van Berkom): 9 commits (last: Fix keyboardinterrupts caused by subprocesses) https://gitlab.com/BuildStream/buildstream/commit/23c1e284ea748942e7067454aae2b279a1ab0e68 | 14:28 |
| gitlab-br-bot | buildstream: merge request (remove-arches->master: WIP: Remove arches) #117 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/117 | 14:29 |
| tristan | Ok, ducks all in a row | 14:29 |
| tristan | perfectly aligned for this golf club :D | 14:29 |
| * tlater apologizes, I'm quite literally testing the CI and doesn't want to install a local copy of gitlab. | 14:30 | |
| tristan | Oh ? | 14:30 |
| tristan | I dont get it... | 14:30 |
| tristan | but, leaving closing coffee shop right now | 14:30 |
| tristan | tlater, I marked the latest MR from you to merge after CI | 14:31 |
| tlater | \o/ | 14:31 |
| tristan | Ah I guess you are talking about running in docker as regular user | 14:31 |
| tlater | Yup | 14:31 |
| tristan | mhm, I think some people around you might know about installing gitlab | 14:31 |
| tristan | might help | 14:31 |
| ssam2 | you don't need to install gitlab, just docker | 14:31 |
| tristan | not sure, also might be a lot of stuff | 14:31 |
| tlater | ssam2: I think this is gitlab-runner specific | 14:32 |
| tristan | I'll try to see if I cant find more important stuff for you too | 14:32 |
| * tristan is sure there is a ton of stuff to do | 14:32 | |
| tlater | \o/ | 14:32 |
| ssam2 | tlater, ah ok | 14:32 |
| tlater | Either way, o/ tristan | 14:32 |
| ssam2 | or perhaps, specific to the gitlab CI shared runners? | 14:33 |
| tlater | ssam2: Unlikely, but there's a chance I suppose. | 14:33 |
| ssam2 | note that you can attach custom runners to gitlab.com, so at worst you'd have to set up your own runner | 14:33 |
| ssam2 | which is not so hard: https://docs.gitlab.com/runner/ | 14:34 |
| tlater | I've done it before, I'm just not sure it's worth it | 14:34 |
| ssam2 | you might be right :-) | 14:34 |
| tlater | Doesn't bring me much closer to solving the issue, and takes an hour or two | 14:34 |
| tlater | Although spamming #buildstream with commits isn't neat :/ | 14:34 |
| ssam2 | you could fork a personal buildstream repo and push them there ? | 14:35 |
| tlater | Ah, see, that's a solution :) | 14:35 |
| tlater | ta ssam2 | 14:35 |
| *** tristan has quit IRC | 14:35 | |
| *** tristan has joined #buildstream | 14:37 | |
| ssam2 | tristan, any thoughts on allowing checkout of artifacts by cache key ? | 14:40 |
| ssam2 | its a feature request i have on my list, seems fairly unobtrusive and easy to implement | 14:40 |
| tristan | ssam2, So we have to spec that out and make an issue I think | 14:40 |
| tristan | ssam2, but I feel like it's related to something else, which is a bit deeper | 14:41 |
| ssam2 | right. i'm not entire sure what the use case is for it | 14:41 |
| tristan | thats also a point | 14:41 |
| tristan | But | 14:41 |
| tristan | ssam2, maybe we need overall more thought on this | 14:42 |
| ssam2 | that's what i'm doing, FWIW | 14:42 |
| tristan | ssam2, one of the things, which people have recently disagreed with me about; and I should lend credibility to... | 14:42 |
| ssam2 | trying to think provenance from the ground up and get some actual use cases together | 14:42 |
| tristan | is that I have never considered cache key stability to be a goal | 14:42 |
| ssam2 | right | 14:43 |
| tristan | However, it's clear that in *any* case, cache keys remain stable for a given stable release | 14:43 |
| tristan | because they can only break with features | 14:43 |
| ssam2 | i feel they should be an implementation detail. i also feel they should remain stable whenever possible though just to avoid rebuilds | 14:43 |
| tristan | The main use case I can see of checkout by cache key, is to allow buildstream to still work for checkout out artifacts which it no longer understands the cache key of | 14:44 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Add warnings when staging to /buildstream/build) https://gitlab.com/BuildStream/buildstream/commit/10847a66dc0748696829d5fd18ad17043c2c6878 | 14:44 |
| ssam2 | i hadn't considered that case | 14:44 |
| gitlab-br-bot | buildstream: issue #74 ("Prevent artifacts from containing files in `/buildstream/build`") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/74 | 14:44 |
| gitlab-br-bot | buildstream: merge request (74-prevent-artifacts-from-containing-files-in-buildstream-build->master: Add warnings when staging to /buildstream/build) #93 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/93 | 14:44 |
| ssam2 | buildstream should be able to rebuild the artifact with the new cache key, though | 14:45 |
| tristan | ssam2, it also implies, that artifacts should encode the cache keys of their dependencies; and we should be internally changing some logic - so that we can walk the cache keys to stage and check something out | 14:45 |
| ssam2 | so at best its an optimization | 14:45 |
| tristan | Well, it depends on what cache keys and artifacts eventually become really | 14:45 |
| tristan | If it ends up being a store of sorts, which seems desirable by some, then it's surely important | 14:46 |
| tristan | Whether "same cache key can be reproduced in 10 years with bleeding edge buildstream" is in any way important, I'm really unsure of | 14:46 |
| tristan | There should be other ways to assert reproducibility of builds from the exact same buildstream project | 14:47 |
| tristan | than looking at keys and thinking they are important | 14:47 |
| tristan | So, this probably needs some additional real thought before going anywhere, I guess | 14:47 |
| tristan | ssam2, along the same lines, if we want cache keys to become *even more stable*, then we might consider splitting them - so that there is a portion that the user sees that is more stable than the full key | 14:52 |
| tristan | i.e. parts of the cache key which relate to extensions of the artifact format (like relying on a manifest being encoded as metadata), effects real cache key - meh I dont know if I like this line of thinking anyway | 14:52 |
| ssam2 | seems a bit awkward | 14:53 |
| tristan | yeah, thats in response to a user oriented argument that "users want to see the same cache key and think that that means reproducible" | 14:53 |
| tristan | So, I dont really like that idea either | 14:53 |
| persia | I don't think "reproducible" needs to mean "sees the same cache key", but it would be nice if an update that happened several years later could reuse previously built artifacts, to reduce integration testing time. | 14:54 |
| persia | That said, how it can know these artifacts are safe to use becomes an interesting question. | 14:54 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 3 commits (last: Fix keyboardinterrupts caused by subprocesses) https://gitlab.com/BuildStream/buildstream/commit/23c1e284ea748942e7067454aae2b279a1ab0e68 | 14:55 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 14:55 |
| tlater | Gah, wrong remote | 14:55 |
| tristan | persia, we can never know, but built in crypto as a first class citizen might just do it | 14:55 |
| tristan | persia, I mean; without that; we just have to assume that what was cached under that key, was actually created with buildstream and arrives safely at the destination, and no mitm occured | 14:56 |
| persia | tristan: I'm willing to presume trusted curation of the server and uploads that are key authenticated, encrypted (with checksum), and logged is sufficient. ssh transport gives most of that. I'd be happier with more explicit certification, but that doesn't solve the "find an artifact that was produced by this element in this context" issue. | 14:59 |
| persia | s/presume trusted/presume that trusted/ | 14:59 |
| tristan | no it doesnt | 15:00 |
| tristan | persia, the question rather will be: "Is that really an issue" | 15:01 |
| persia | Hrm? If it is interesting to reuse artifacts previoulsy generated, where the span between generation and update is long, then it is important to be able to determine whether the server has any such artifacts. | 15:05 |
| persia | I think it would be nice if an update that happened several years later could reuse previously built artifacts. If this is to be supported, it becomes necessary to be able to determine if those artifacts are on the server. | 15:05 |
| tristan | It is interesting to reuse artifacts as much as possible | 15:06 |
| persia | Whether anyone certified the artifacts or not is entirely orthogonal to the issue I am raising. | 15:06 |
| tristan | However it is uninteresting to bind the codebase to forever use the same keys | 15:06 |
| persia | Yes. | 15:06 |
| tristan | Also, it will be impossible later, when buildstream evolves with new requirements on artifact data, to use old artifacts | 15:06 |
| tristan | So, it is simply a matter of expiration, and desiring to keep that low | 15:07 |
| persia | One potential way to address it is to have key generation be versioned, such that the codebase can have several key generation algorithms, always posting with the current version, but when reviewing old artifacts, comprehending other algorithms may have been in use. | 15:07 |
| tristan | Yes, it's desirable to keep that expiration *low* | 15:07 |
| tristan | But that is a huge difference from "stable" | 15:07 |
| tristan | persia, Yes, I hate that idea with a passion | 15:07 |
| tristan | persia, please never say it again :D | 15:07 |
| persia | How else would it be possible to determine if a server has a decade old artifact? | 15:08 |
| *** jude has quit IRC | 15:08 | |
| tristan | Ah I read half way and stopped sorry, you are just saying to raise some assertion ? | 15:08 |
| tristan | Oh no | 15:08 |
| tristan | You are adding the evil "comprehending other algorithms may have been in use" bit to it to | 15:08 |
| tristan | *too | 15:08 |
| tristan | persia, I really think that is entirely irrational | 15:09 |
| persia | So, I don't believe it possible to make a forward-looking statement that the algorithm won't change. | 15:09 |
| tristan | possible, and irrational | 15:09 |
| persia | As a result, if there is any interest in supporting ancient artifacts, there needs to be some way to determine whether a given artifact was produced by a given project state. | 15:09 |
| persia | Heh, yes, I don't believe it sane to make such a forward-looking statement :) | 15:10 |
| tristan | persia, supporting ancient artifacts for us can mean "Ability to check one out, recursively, from the artifact cache" | 15:10 |
| tristan | And it can stop right there. | 15:10 |
| persia | Expand on that? | 15:10 |
| tristan | I mean, we can easily run `bst pull element:cache-key` | 15:11 |
| tristan | and `bst checkout element:cache-key` | 15:11 |
| tristan | for an artifact built 10 years ago | 15:11 |
| tristan | That only requires we can lookup artifacts and dependencies recursively, by their key | 15:11 |
| persia | Where "cache-key" in those examples is the actual key? | 15:11 |
| persia | How do I determine that key? | 15:12 |
| persia | (using BuildStream 17.38) | 15:12 |
| tristan | Where cache-key in those is the actual key that it was built with | 15:12 |
| tristan | There is absolutely no reason you have to determine what that key is | 15:12 |
| tristan | You have save it from a build you deployed 10 years ago | 15:12 |
| tristan | *saved | 15:12 |
| tristan | And now you want to check it out again and look at it, no worry | 15:12 |
| persia | OK. Once I check those out, will buildstream use them when generating a new image? | 15:13 |
| tristan | No way :) | 15:13 |
| tristan | You can still reuse the image deployment artifact you used 10 years ago, though | 15:14 |
| ltu | ok, reading scroll-back, so let me check i understand the task correctly here: update the BaseRock defs to bst conversion, so that it converts from arches, and then we can remove arches support from BuildStream. | 15:14 |
| tristan | If you want to keep it in the store for later retrieval | 15:14 |
| tristan | ltu, that and pre-/post- commands | 15:14 |
| persia | And BuildStream 17.38 will consume them when they were built with 1.2? | 15:14 |
| tristan | ltu, issue #120 and #121 | 15:15 |
| tristan | persia, it can be able to pull them and check them out | 15:15 |
| tristan | in that sense of "consuming" | 15:15 |
| tristan | It can help you get the data you want from the distant artifact store, onto your harddisk | 15:16 |
| ltu | tristan, ok | 15:16 |
| persia | tristan: That doesn't help my use case. | 15:17 |
| tristan | persia, I posit that it's not really a use case | 15:17 |
| tristan | and that is really worth some thought | 15:18 |
| persia | I have a system that takes 3 days to build (don't ask, annoying). I want to update it to use the newly released utility library because of a zero-day exploit. I don't have time to rebuild everything. | 15:18 |
| persia | How isn't it a use case? | 15:18 |
| persia | I'm willing to narrow it, so that I only care about being able to upgrade BuildStream without having to rebuild my entire system. | 15:18 |
| tristan | Ok, you have the build log of what you built it with last | 15:18 |
| tristan | So use the buildstream version you used then | 15:18 |
| persia | Except that is ancient, and doesn't run on my current system, because the OS it was built against stopped existing. | 15:19 |
| persia | Essentially, if I have to use old BuildStream to do it, I will never upgrade. | 15:19 |
| persia | And if I never upgrade, then I'm behind, and my managers will tell me to switch to a differnt build system. | 15:20 |
| tristan | Ok, then whenever you upgrade buildstream to a new stable release line (minor point changed), make sure you have a recent build right away | 15:20 |
| tristan | if you want to use recent buildstream and have cached artifacts | 15:20 |
| persia | In some fields, that requires months of validation for regulatory reasons, and costs insane amounts of money. Can I do it a different way? | 15:20 |
| persia | Or, is there a way to prove the results are identical computationally, so I don't have to go through revalidation? | 15:21 |
| *** semanticdesign_ has joined #buildstream | 15:22 | |
| *** semanticdesign has joined #buildstream | 15:22 | |
| tristan | persia, I am quite certain that any attempt to continue supporting "old cache key algorithms" (which, btw, are declared on a *per element* basis, there is no ONE artifact version) - is ludicris | 15:22 |
| tristan | Probably even, with the correct spelling of ludicris | 15:22 |
| *** jude has joined #buildstream | 15:22 | |
| persia | You keep saying that, but I'd prefer you helped me solve my problem. | 15:22 |
| persia | I don't know if the solution requires features you hate, but if it doesn't, I'd like to find that method, or I'll keep wanting the feature you hate. | 15:23 |
| tristan | persia, ok - achieve bit-for-bit reproducibility on the files portion of your artifacts, then. | 15:23 |
| tristan | Avoid regulatory validation of what you can assert is exactly the same | 15:23 |
| persia | I'm willing to assume that either I can generate reproducible builds or this is impossible. | 15:23 |
| tristan | The world is pretty close to reproducible builds as it stands I think | 15:24 |
| tristan | So it shouldnt be that far out | 15:24 |
| tristan | to achieve | 15:24 |
| persia | So, given different cache keys (where I know both from logs), I would be able to extract some form of the artifact that was identical for the comparison? | 15:24 |
| tristan | The files | 15:24 |
| ssam2 | another possibility for this is to have a tool outside buildstream that 'migrates' your existing artifacts to use the new cache keys | 15:24 |
| tristan | I.e. the actual content is what you care about, not the build logs, or the metadata that buildstream wants to programatically read | 15:25 |
| persia | That doesn't work, because different filesystems can cause a set of files to differ even if the file content remains identical. | 15:25 |
| ssam2 | basically knows both cache key algorithms, looks for artifacts under algorithm 1, and commits them into the cache under algorithm 2 | 15:25 |
| persia | ssam2: That would solve my use case, yes. When I upgrade, I would run the tool against the server. | 15:25 |
| tristan | persia, nah, that can certainly be worked around | 15:25 |
| persia | tristan: How? | 15:25 |
| tristan | persia, export to tarball and compare that, for instance | 15:25 |
| tristan | export to tarball is not heavy duty really, and we probably need that functionality at least internally to communicate with fallback platform clients | 15:26 |
| tristan | persia, that is basically by the same logic that you can safely GPG sign the content, given that buildstream at least knows what it is *supposed* to be | 15:27 |
| tristan | even if your FS doesnt support it, it can be verified in some way | 15:27 |
| tristan | (i.e. we're talking permissions and extended attribute support, etc) | 15:27 |
| * persia knows of 4 different tar algorithms, so hopes we can be more specific | 15:28 | |
| persia | tristan: Would ssam2's solution work for you? One-time migration tools for each algorithm change? | 15:29 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/8901330663390c5295e95b1a162bfbb7220bf6df | 15:29 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 15:29 |
| tristan | persia, if we generate the tar using buildstream / python, different algos shouldnt come into question | 15:31 |
| tristan | persia, also `bst diff` is an option, although I'm expecting that your regulatory people wont want to trust it | 15:31 |
| persia | Except python has a) switched default tar algorithms in the past and b) still uses host algorithms for some operating systems | 15:32 |
| tristan | still `bst diff` can be very nice to have to check reproducibility | 15:32 |
| persia | `bst diff` solves my problem, if the code can be audited to be shown to be correct, and it actually checks the content carefully. | 15:32 |
| tristan | persia, checking if something is bit-for-bit reproduced should *really* not be a problem | 15:32 |
| tristan | I mean, we're splitting hairs on the how to export and yada yada | 15:33 |
| tristan | the data is there and must be recorded anyway | 15:33 |
| persia | Importantly, I need to be able to check when *changing cache-key algorithm* | 15:33 |
| tristan | so when changing to a new stable line of buildstream, you can take the list of artifacts / cache keys that you care about, create a new build and diff the artifacts with the old ones | 15:35 |
| tristan | that shouldnt be a problem | 15:35 |
| tristan | migration scripts, also plausible, less delicious but plausible | 15:35 |
| tristan | still a bit weird considering we can't truly control or know what that migration script is going to be at all if the project uses any elements we dont control | 15:36 |
| tristan | and we dont want to control all elements in use | 15:36 |
| persia | The combination of the two would be a perfect match to my use case, and nicely avoids the BuildStream codebase needing to have all the historical algorithms. | 15:36 |
| persia | What do you mean "control"? | 15:36 |
| tristan | I mean; elements are written by project authors | 15:36 |
| tristan | If those bump their artifact revisions for any reason | 15:37 |
| persia | I presume the migration script would only work if the admin had (read) access to all the projects stored on the cache. | 15:37 |
| tristan | We dont know how | 15:37 |
| tristan | remember, there is no ONE artifact version, there is one for the core, and one for each element | 15:37 |
| persia | Oh, right. For that sort of thing, I'm happy with implementors being on their own. Nice to have docs, or some plugin mechanism where the migration script can consume old and new versions of such elements. | 15:37 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/5c5cd5e845ba8cb00d0099e8b9c3c3d465c91513 | 16:25 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 16:25 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/28830191ccaf910d5743a0826d31adb8157e00ba | 16:27 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 16:27 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Drop root privileges for some tests) https://gitlab.com/BuildStream/buildstream/commit/2d89293351ab5e58a9741991adb9b470928a856a | 16:42 |
| gitlab-br-bot | buildstream: merge request (102-run-ci-as-non-root-user->master: Resolve "Run CI as non-root user") #104 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/104 | 16:42 |
| *** tlater has quit IRC | 17:00 | |
| *** jonathanmaw has quit IRC | 17:02 | |
| *** anahuelamo has quit IRC | 17:15 | |
| *** anahuelamo has joined #buildstream | 17:17 | |
| *** bochecha has quit IRC | 17:24 | |
| *** jjardon[m] has quit IRC | 17:42 | |
| *** ptomato[m] has quit IRC | 17:42 | |
| *** mrmcq2u[m] has quit IRC | 17:42 | |
| *** mattiasb has quit IRC | 17:42 | |
| *** waltervargas[m] has quit IRC | 17:42 | |
| *** cgmcintyre[m] has quit IRC | 17:42 | |
| *** ptomato[m] has joined #buildstream | 18:16 | |
| *** jjardon[m] has joined #buildstream | 18:16 | |
| *** mattiasb has joined #buildstream | 18:17 | |
| *** waltervargas[m] has joined #buildstream | 19:03 | |
| *** bochecha has joined #buildstream | 19:06 | |
| *** bochecha has quit IRC | 19:11 | |
| *** bochecha has joined #buildstream | 19:11 | |
| *** xjuan has quit IRC | 19:19 | |
| *** ssam2 has quit IRC | 19:31 | |
| *** cgmcintyre[m] has joined #buildstream | 21:02 | |
| *** givascu has quit IRC | 21:04 | |
| *** mrmcq2u[m] has joined #buildstream | 21:09 | |
| *** semanticdesign has quit IRC | 22:40 | |
| *** semanticdesign_ has quit IRC | 22:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!