| *** tristan has quit IRC | 06:40 | |
| *** tristan has joined #buildstream | 07:04 | |
| tristan | <tristan> jjardon[m], jf you change build-essential in baserock, you have to manually change the gnu-toolchain used in the conversion too | 07:04 |
|---|---|---|
| tristan | <tristan> but yeah, Sam has been focusing a lot on this so it's much better to base your work on his, than on the initial conversion I did months ago | 07:04 |
| tristan | <tristan> of course his toolchain stack is much more up to date | 07:04 |
| * tristan repeats after ensuring he really has a connection :) | 07:04 | |
| *** ChanServ sets mode: +o tristan | 07:05 | |
| jjardon[m] | Tristan thanks, but I didn't change buuld-essential, libffi is in core | 08:19 |
| jjardon[m] | Now I'm hitting this : https://gitlab.com/baserock/definitions/-/jobs/22760527 seems an issue in buildstream; did you see it before? | 08:20 |
| juergbi | that might be my fault, let me take a look | 08:26 |
| juergbi | yes, missing variable initialization. i had this but apparently removed this on rebasing or so | 08:27 |
| juergbi | that's regarding the push failure, which should not be considered fatal | 08:29 |
| *** tlater has joined #buildstream | 08:31 | |
| *** jonathanmaw has joined #buildstream | 08:32 | |
| gitlab-br-bot | push on buildstream@master (by Jürg Billeter): 1 commit (last: _artifactcache/pushreceive.py: Fix exception handling) https://gitlab.com/BuildStream/buildstream/commit/b5861b148bbe4f1e71128955d709c6b28f2394e5 | 08:34 |
| jjardon[m] | Thanks juergbi ! I will try again later | 08:35 |
| jonathanmaw | tristan: hrm, it looks like I can't get any useful information out by using strace. Mainly because it breaks fuse somehow. | 08:48 |
| jonathanmaw | I'll try and dig up the fuse error again. | 08:48 |
| jonathanmaw | tristan: said error is https://pastebin.com/tazk2zkN | 08:58 |
| jonathanmaw | and looking at the strace log, that seems to be happening when it fusermounts "." | 08:59 |
| jonathanmaw | full syscall is mount("SafeHardlinkOps", ".", "fuse", MS_NOSUID|MS_NODEV, "fd=3,rootmode=40000,user_id=1000"...) = -1 EPERM (Operation not permitted) | 09:00 |
| *** ssam2 has joined #buildstream | 09:06 | |
| jonathanmaw | so I'm going to look at making tests for dpkg-build and dpkg-deploy elements | 09:08 |
| jonathanmaw | which looks like it would be best served by creating a "test" debian source | 09:10 |
| tlater | Have you had time to look at my merge request yet, tristan? | 09:10 |
| tristan | jonathanmaw, I was going to suggest the same, if you're hitting walls then that's a better use of your time | 09:11 |
| tristan | jjardon[m], ah, ok that is strange, it must have changed and something about -dumpmachine doesnt work for it (was building few months ago), anyway looks like you got a fix from sam nevertheless :) | 09:12 |
| tristan | tlater, sigh... unfortunately I spent literally all weekend trying to solve a problem which I still have not solved :'( | 09:13 |
| tristan | Apparently my variant resolution code is borked | 09:14 |
| tristan | one of those moments you sort of wish you had a strong maths background | 09:14 |
| tlater | tristan: I'll write some more tests for that MR for now then :) | 09:14 |
| tristan | I have a working solution, but the complexity is exponential, and takes minutes to result small pipelines of only ~50 elements (with around 15 variants in play) | 09:15 |
| tristan | so I havent committed that | 09:15 |
| tristan | and trying to figure out how to solve this properly | 09:15 |
| tristan | tlater, ok so I wonder about checking _tainted() state | 09:37 |
| tristan | Currently we look over everything and check all dependencies tainted state | 09:37 |
| tristan | tlater, I feel like this would be stronger if, we just knew if a dependency of the artifact was tainted by looking at the artifact data | 09:38 |
| tristan | rather than checking all dependencies at this point | 09:38 |
| tlater | That would involve figuring out which artifacts are dependencies | 09:39 |
| tristan | It seems equal at this point, but it may be more defensive and future proof if we could A.) encode that into the artifact (the fact that it was built against a workspaced dependency) and B.) Just resolve tainted as 'workspaced or dependency_workspaced' from the artifact | 09:39 |
| tristan | tlater, right that's not what I meant | 09:40 |
| tristan | tlater, I mean encoding that at artifact creation time | 09:40 |
| tristan | we will probably want to encode dependencies (and maybe their keys) into artifact data for other reasons, but that is rather orthogonal | 09:40 |
| tlater | Ah, I understand | 09:41 |
| tlater | So instead of just checkin whether a dependency is currently tainted, when the metadata is stored check if a dependency is tainted. | 09:42 |
| tristan | tlater, or workspaced rather | 09:45 |
| tristan | still resolve tainted from whether the artifact was workspaced, or any dependency was workspaced | 09:46 |
| tristan | tlater, actually provenance is even better if the other attribute is a list of which dependencies which it was built against, were workspaced | 09:46 |
| tlater | That would have extend to all possible reasons for tainting, though. | 09:47 |
| tristan | I think the philosophy here is: A.) Encode the facts of what happened in the artifact and B.) Resolve whether that means the artifact was tainted at runtime | 09:48 |
| tristan | tlater, so indeed, any other things in the future would have other attributes to indicate them in the artifact | 09:48 |
| tristan | whoa | 10:03 |
| tristan | Ok I think I have a sub-par solution which may just save the day | 10:03 |
| tristan | I'm going to need a real, better solution, I think... I still might be able to trick it | 10:05 |
| tristan | but it's passing all the tests (including some recent additions), and passing my expectations for the GNOME modulesets | 10:05 |
| tristan | tlater, ok so... I think after that change, lets just run with it | 10:15 |
| tristan | I have a bunch of code to land now | 10:15 |
| tristan | but if you can push a new branch soonish, I'd rather rebase against you; there will be a couple of conflicts I anticipate | 10:16 |
| tlater | Alright. I'll probably have this done an hour, maybe two after the meeting, build times make things take a while. | 10:17 |
| tlater | Looks like this might just be working already... | 10:21 |
| *** jonathanmaw_ has joined #buildstream | 10:30 | |
| *** jonathanmaw has quit IRC | 10:31 | |
| *** jonathanmaw_ has quit IRC | 11:28 | |
| *** jonathanmaw has joined #buildstream | 11:28 | |
| jjardon[m] | juergbi: Im getting a different error now: https://gitlab.com/baserock/definitions/-/jobs/22812938 | 11:49 |
| tristan | whoa, that looks unlikely | 12:03 |
| juergbi | that's a really large xz binary... | 12:10 |
| juergbi | ah, no, that's just message size | 12:10 |
| juergbi | 64k, hm | 12:10 |
| gitlab-br-bot | push on buildstream@workspaces (by Tristan Maat): 3 commits (last: element.py: Encode workspaced dependencies in metadata) https://gitlab.com/BuildStream/buildstream/commit/4a38386b4d5dc740c68eaf5e5fd3f7e4f59e003c | 12:16 |
| gitlab-br-bot | buildstream: merge request (workspaces->master: Workspaces) #50 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/50 | 12:16 |
| tlater | Alright, that should be it. I've written integration tests for this, will update those ASAP too - permission issues, as usual. | 12:17 |
| tristan | tlater, great | 12:17 |
| tristan | tlater, I'll give it a quick lookover, merge and rebase on it | 12:17 |
| tristan | I dont expect issues | 12:17 |
| tlater | Alright, I'll go for lunch real quick :) | 12:25 |
| juergbi | tristan: do we need to keep supporting remote artifact servers that don't have a summary file or is it ok to hard require this (to reduce the number of code paths)? | 12:43 |
| ssam2 | we only have 2 servers so far :-) | 12:45 |
| ssam2 | i'm happy to update ostree.baserock.org | 12:45 |
| ssam2 | or is this going to require a bleeding edge ostree ? | 12:45 |
| juergbi | no ostree update should be required. for the minimal version it would just be a cron job that runs ostree summary -u to update the summary file | 12:58 |
| juergbi | the question is more about whether we're ok with requiring servers to have this in place than compatibility with existing servers | 12:58 |
| juergbi | i would say yes but wanted to check before i rip out some code paths ;) | 12:59 |
| ssam2 | we already require them to have buildstream installed, i don't think a cron job is much extra effort | 13:00 |
| ssam2 | buildstream could ship suitable systemd unit files, even | 13:00 |
| ssam2 | is this going to mean we can only download artifacts we need ? :-) | 13:01 |
| juergbi | that's already the case anyway. it will allow us to avoid trying to download artifacts that don't exist | 13:04 |
| juergbi | already the case for the most part, that is | 13:05 |
| juergbi | we can generate the build plan after checking available artifacts in the summary file. this means that some artifacts that are only build dependencies of already built artifacts can be skipped altogether | 13:05 |
| juergbi | will be nice to skip downloading bootstrap in the first session | 13:06 |
| tristan | juergbi, hard require, reduce codepaths, please | 13:08 |
| juergbi | gladly | 13:09 |
| tristan | juergbi, I admit that I dont like the cron job, but other approaches are buggy :-/ | 13:09 |
| tristan | i.e., we could lock it in `bst-artifact-receive` | 13:09 |
| tristan | but if we ever fail to exit gracefully, it's locked for life | 13:09 |
| juergbi | we could also have bst-artifact-receive trigger a systemd job | 13:10 |
| tristan | also not sure how long it would take to update summary on huge server | 13:10 |
| juergbi | where systemd ensures it completes | 13:10 |
| juergbi | and is never run multiple times at once | 13:10 |
| tristan | then we require systemd | 13:10 |
| tristan | which... is sort of the same reason I dont like the cron | 13:10 |
| tristan | moving parts | 13:10 |
| juergbi | sure, it would be great to not require anything external but well.. | 13:11 |
| tristan | anyway | 13:12 |
| juergbi | the good thing is that we could improve this later on without anything else being affected | 13:12 |
| juergbi | so i think we should just go with cron for now and look into improvements later | 13:12 |
| tristan | juergbi, this detail will have to go into `doc/sources/artifacts.rst` | 13:12 |
| juergbi | yes | 13:12 |
| tristan | I also agree the API contract for artifact servers should be less severe than, things like project format | 13:13 |
| * tristan has working project variants now | 13:14 | |
| tristan | pretty nice feature actually... project is loaded first with floating variants; then project is loaded; element variants are resolved (element variants can have different requests for a project variant), after that, agreement of loaded elements is found for the desired project variant | 13:15 |
| tristan | Then that decided project variant is used to composite project config with the selected variant | 13:16 |
| tristan | All of this before Element instances are created | 13:16 |
| tristan | And before project inheritance happens | 13:16 |
| * tristan will fix test cases which will surely be broken by this change (need to adapt project/_loader loading APIs in there) | 13:17 | |
| juergbi | :) | 13:21 |
| * tristan opens a workspace | 14:04 | |
| tlater | Has everything gone up in flames yet? | 14:05 |
| tristan | haha, colorful `bst workspace --list` | 14:05 |
| tlater | :) | 14:05 |
| * tristan holds breath... | 14:07 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 20 commits (last: _loader.py: Fixed variant priority in variant resolution) https://gitlab.com/BuildStream/buildstream/commit/fb69e5546256a513fc903a393fe448d9f4f4383b | 14:07 |
| gitlab-br-bot | buildstream: merge request (workspaces->master: Workspaces) #50 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/50 | 14:07 |
| tlater | \o/ | 14:07 |
| jjardon[m] | juergbi: do you want me to file a bug report somewhere? | 14:13 |
| *** tristan has quit IRC | 14:40 | |
| *** jude has quit IRC | 15:07 | |
| juergbi | jjardon[m]: is this with gnome7 or another server? | 15:09 |
| juergbi | it looks like we're hitting a protocol limitation, 64 kB for the info message with all the refs | 15:10 |
| juergbi | buildstream produces a lot of refs, so that's actually not surprising. i guess Dan didn't expect that when designing the protocol | 15:11 |
| juergbi | we could widen the size field but it would probably be better to avoid sending all refs in the first place | 15:12 |
| *** jude has joined #buildstream | 15:20 | |
| jjardon[m] | juergbi: its the server ssam2 setup for baserock, let me check | 15:22 |
| jjardon[m] | https://ostree.baserock.org/ | 15:22 |
| juergbi | ta. yes, i suspect we're already hitting that limit. need to fix this in the protocol | 15:25 |
| juergbi | we should file a bug at https://gitlab.com/BuildStream/buildstream/issues to track this | 15:26 |
| gitlab-br-bot | buildstream: issue #47 (""OverflowError: int too big to convert" when pushing artifacts") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/47 | 15:44 |
| tlater | Hm. I'm unable to remove cache directories in my tests, for some reason | 15:44 |
| tlater | `rm -rf ${XDG_CACHE_HOME:-~/.cache}/buildstream/artifacts` doesn't seem to have any effect. | 15:45 |
| ssam2 | is it using bash ? | 15:45 |
| tlater | Yup | 15:46 |
| tlater | The strange thing is that it just works fine outside the test script | 15:47 |
| ssam2 | try changing the `rm -rf` to an `echo` ? without context, the path being wrong is the only thing I can think of | 15:50 |
| tlater | Echo displays the path I expect... | 15:50 |
| ssam2 | oh right | 15:50 |
| tlater | It *is* quoted, by the way. | 15:52 |
| tlater | Ah, looks like ~ isn't expanded in that variable | 15:54 |
| tlater | That took me 2 hours -.- | 15:54 |
| ssam2 | shell \o/ | 15:54 |
| *** jude has quit IRC | 16:07 | |
| *** jude has joined #buildstream | 16:32 | |
| tlater | Why does the CI take 20+ minutes all of the sudden :/ | 16:44 |
| *** tlater has quit IRC | 16:53 | |
| *** tristan has joined #buildstream | 16:56 | |
| *** jonathanmaw has quit IRC | 17:02 | |
| *** jude has quit IRC | 18:16 | |
| *** ssam2 has quit IRC | 18:23 | |
| gitlab-br-bot | buildstream: merge request (pedro/docker-bzr->master: Add support for bzr sources in Docker) #58 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/58 | 22:57 |
| gitlab-br-bot | buildstream: merge request (pedro/docker-bzr->master: Add support for bzr sources in Docker) #58 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/58 | 23:03 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!