| *** tristan has quit IRC | 03:35 | |
| *** tristan has joined #buildstream | 04:06 | |
| *** ChanServ sets mode: +o tristan | 04:36 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: utils.py: Improved glob() function) https://gitlab.com/BuildStream/buildstream/commit/f2d890e7494e06fd835b6f63a5c1e3a2dc1018bd | 07:08 |
|---|---|---|
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: utils.py: Use proper reStructuredText notes in API docs) https://gitlab.com/BuildStream/buildstream/commit/be7db99e767a66b02e87686874ffcc1c678efaa6 | 07:26 |
| *** jonathanmaw has joined #buildstream | 07:42 | |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Change the default behaviour of `bst track` to --deps=none) https://gitlab.com/BuildStream/buildstream/commit/e6441c354d8972775795e333ab1211c6ec1ff970 | 07:43 |
| gitlab-br-bot | buildstream: merge request (sam/track-default-none->master: Change the default behaviour of `bst track` to --deps=none) #44 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/44 | 07:44 |
| gitlab-br-bot | buildstream: issue #43 ("Enhancement: `bst track --deps=none` should be default behaviour") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/43 | 07:45 |
| *** tlater has joined #buildstream | 08:12 | |
| tristan | tlater, so I've merged the initial buildstream-tests stuff... | 08:21 |
| tristan | tlater, it would be good to have a single shell script which tests can source to do stuff like assert permissions of files in a checkout, or whatever kind of code sharing can be done with those shell scripts | 08:21 |
| tristan | tlater, also, I would like buildstream-tests *itself* to also have a .gitlab-ci.yml, doing the sort of backwards of what the buildstream one will do | 08:22 |
| tristan | so for instance, every time we add some test or modify some test, we can have that push automatically run against buildstream master | 08:22 |
| tlater | Alright, should add CI on the buildstream-tests repo before I finish the MR for the buildstream repo CI? | 08:24 |
| tristan | tlater, either way is fine | 08:25 |
| tristan | tlater, note though, today I have finally changed over the split rules to be globs | 08:25 |
| tlater | I've kept tests that work with split rules outside my MR so far, because I couldn't get them to work. | 08:26 |
| tristan | the supported syntax is basic glob plus bash's 'globstar', it's documented here: https://buildstream.gitlab.io/buildstream/buildstream.utils.html#buildstream.utils.glob | 08:26 |
| tristan | Ok | 08:26 |
| tlater | Oh, so this doesn't require the .* anymore? | 08:27 |
| gitlab-br-bot | buildstream: merge request (tristan/test_tests->master: WIP: Add integration tests) #40 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/40 | 08:27 |
| tristan | tlater, correct, they are no longer regex | 08:28 |
| tristan | (although internally they become regex, I just feel that regex is not a user friendly API for split rules) | 08:28 |
| tlater | Hmm. I was a bit afraid of this. The tests seem to take very long :/ | 08:35 |
| tristan | Yeah, they might do that | 08:35 |
| tristan | tlater, ok so, there is something else | 08:36 |
| tristan | tlater, I think that we should change our general approach to the scripts | 08:36 |
| tlater | Ok | 08:36 |
| tristan | But I wonder how much we can get done dynamically in .gitlab-ci.yml :) | 08:37 |
| tristan | that can be fun | 08:37 |
| tlater | So you want to run the individual tests from inside the gitlab-ci file? | 08:37 |
| tristan | tlater, so basically the yaml lets us define some stages and a bunch of tests which need to be run in a given stage | 08:37 |
| tristan | yeah | 08:37 |
| tristan | that will get us a better fail-early experience | 08:38 |
| tristan | and also complete faster I think | 08:38 |
| tlater | I guess that can work. We can quite easily source shell scripts from inside that. | 08:38 |
| tlater | And just build a generic library which makes this go down to a few commands. | 08:38 |
| tlater | Perhaps there's already something out there that does this, too. | 08:38 |
| tristan | We can easily source shell scripts, but... can we use data in the repo to compose the structure of the tests ? | 08:38 |
| tristan | I guess it's just a minor issue but it would be nice if we can | 08:39 |
| tlater | What do you mean? | 08:39 |
| tristan | tlater, basically, I would like to see a CI pipeline where say, we run ./setup.py test first, and then we run every integration test in parallel | 08:39 |
| tlater | And the integration tests boil down to .bst definitions? | 08:39 |
| tristan | But... ideally at least; I dont want the definitions for each test to be hard coded into the .gitlab-ci.yml | 08:40 |
| tristan | tlater, not because I want something so ultra fancy, but because it would be nice to maintain this in one place (in buildstream-tests), instead of having to duplicate that also in the buildstream .gitlab-ci.yml | 08:41 |
| tristan | i.e. it just boils down to "where does the list of tests go ?" | 08:41 |
| tlater | Yeah, that's definitely possible. I can write tests that just need .bst elements to be defined in some directory in the test repo. | 08:41 |
| tlater | Oh, wait, but how would these be run in parallel then? | 08:43 |
| tristan | tlater, ok look at this for example: https://gitlab.com/baserock/ybd/pipelines/9168503 | 08:43 |
| tristan | there is a nice CI pipeline | 08:43 |
| tristan | the Build "stage"... does 5 builds in parallel | 08:43 |
| tristan | looking at this sample: https://gitlab.com/baserock/ybd/blob/master/.gitlab-ci.yml | 08:44 |
| tristan | the different pipeline parts (left to right) are defined as "stages" | 08:44 |
| tristan | line 11 | 08:44 |
| tlater | Yeah, but the individual tests in build need to be defined individually. | 08:44 |
| tristan | tlater, that's what I'm wondering if we can do dynamically :D | 08:45 |
| tlater | I'll read into gitlab's documentation, but I kind of doubt it. | 08:45 |
| gitlab-br-bot | buildstream: merge request (artifact-storage->master: WIP: Artifact storage enhancements) #45 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45 | 08:48 |
| tristan | tlater, yeah I'm asking in #gitlab right now, but also doubt it | 08:49 |
| juergbi | tristan: i'm not sure about what metadata to store / in what format. i've implemented it as storing the dict that is also used to generate the cache key (as yaml file) for now. you might have had something else in mind, though | 08:51 |
| tristan | juergbi, that sounds perfect; yes everything in yaml; the most important afaics is the public data; but lets record also at least the cache key | 08:52 |
| tristan | it will be interesting in the next phase, because we will want to store both cache keys in there | 08:52 |
| tristan | Also, I guess any interesting provenance information can be fine | 08:52 |
| tristan | but not required really | 08:53 |
| juergbi | tristan: it looks like this at the moment: https://pastebin.com/ssyAte4e | 08:53 |
| tristan | juergbi, so I dont think the dict that is get_unique_key() is really correct, if we're using it directly | 08:54 |
| tristan | because we will want to read into that; binding the two together makes a very strange implied requirement on get_unique_key() | 08:55 |
| juergbi | the public data part would actually have to be guaranteed stable | 08:57 |
| juergbi | do we need/want stability for all metadata we save or only selected parts? | 08:57 |
| tristan | good point | 08:57 |
| tristan | juergbi, in which case, we should also record the major/minor buildstream version | 08:58 |
| tristan | just in case | 08:58 |
| juergbi | right | 08:58 |
| tristan | So there could be an informative part meant only for humans to observe | 08:59 |
| tristan | but lets not care about that immediately | 08:59 |
| tristan | lets assume that everything that goes in there is something that buildstream is going to read back | 08:59 |
| juergbi | right now we don't use the element's own public data in the cache key at all (only public data of dependencies). however, if we store the element's own public data in the artifact cache, we also need to use it for cache key generation (to ensure we trigger rebuild if it changes) | 08:59 |
| juergbi | ok, so we write only what we really need right now | 08:59 |
| tristan | I have been thinking of that yeah, actually we very selectively use only parts of the public data of dependencies | 09:00 |
| tristan | depending on the element type | 09:00 |
| tristan | So anyway regarding cache key calculations, the fact that artifact builds can modify public data changes things significantly | 09:01 |
| tlater | Oh, wait, gitlab CI supports caching? | 09:01 |
| tristan | tlater, I have no idea :) | 09:01 |
| tlater | I'll have to try this | 09:01 |
| juergbi | tristan the cache key would be calculated based on statically declarred public data. dynamic public data should not matter for the cache key | 09:02 |
| tristan | tlater, I think it helps a lot though yes, for instance you can have your sources already cached at least | 09:02 |
| tlater | Yeah, that's pretty big | 09:02 |
| tristan | juergbi, that is a fair point, although it means that the pipeline/engine has to see these as two separate things | 09:02 |
| juergbi | yes, but we can't get around that | 09:03 |
| juergbi | if the element is not built/cached yet, it's impossible to calculate cache key using dynamic public data | 09:03 |
| tristan | hmmm | 09:03 |
| tristan | fair point | 09:03 |
| juergbi | either we can somehow make sure that cache key is always generated before we read back dynamic public data | 09:04 |
| juergbi | or we store them both at the same time, aware of the differences | 09:04 |
| tristan | cache keys are generated along the way, only in the case of TrackQueue being enabled | 09:04 |
| tristan | it's nice to know them before building | 09:04 |
| tristan | although it *can* technically be delayed further, would rather not | 09:05 |
| tristan | I guess that is fine | 09:05 |
| juergbi | it would change the whole pipeline approach | 09:05 |
| tlater | Nothing in the gitlab documentation suggests we can push jobs from within the shell, unfortunately. | 09:06 |
| tristan | yeah I really dont want that... I mean "I guess that is fine, that cache keys are calculated based only on static data" | 09:06 |
| tristan | juergbi, maybe we can simplify the plugin facing API by stating that: "Whenever you read public data, you get only statically defined public data, except during the assemble() methods, when you get the effective public data" ? | 09:07 |
| tristan | So the two exist, but plugins dont really have to care | 09:07 |
| tristan | When they do get_unique_key(), they get static data; when they are doing a build, they get the effective data | 09:07 |
| juergbi | hm, same method returning different values depending on from where / when it's called? | 09:08 |
| tristan | hehehe | 09:08 |
| tristan | yes I did just suggest that | 09:08 |
| juergbi | doesn't sound very intuitive | 09:08 |
| tristan | I'm throwing it out there, not sure it's a great idea | 09:08 |
| tristan | but having the two concepts separate is also undesirable | 09:09 |
| juergbi | hm, do plugins actually need the static data of dependencies? | 09:10 |
| juergbi | isn't it only Element._get_cache_key() that needs it? | 09:10 |
| tristan | ok so | 09:13 |
| tristan | The BuildElement wants to know if any of it's dependencies have changed integration-commands, a build element will do something different if they have changed, so cache key is different | 09:14 |
| tristan | but if a BuildElement's *own* integration commands change, it does not effect the build... *HOWEVER*, this changes now | 09:15 |
| tristan | interesting | 09:15 |
| *** ssam2 has joined #buildstream | 09:15 | |
| juergbi | well, an element's dynamic public data should never change as long as its cache key stays the same | 09:15 |
| juergbi | as the same sources / static data should generate the same dynamic public data | 09:15 |
| tristan | juergbi, Also... a compose element wants to know about the split rules that are effective when composing/splitting, the rationale there is that if glibc changes split-rules, only that compose element at the end of the pipeline needs to be rebuilt | 09:16 |
| juergbi | isn't this also handled properly by just using the cache key (with static public data included)? | 09:16 |
| tristan | I dont think so... here is a case that I'm worried about | 09:17 |
| tristan | User A builds glibc | 09:17 |
| tristan | glibc then changes the split-rules, which does not effect it's own cache key | 09:18 |
| tristan | User B downloads the older, cached artifact of glibc | 09:18 |
| tristan | User B composes a system, but now has outdated public data they loaded from the downloaded artifact | 09:18 |
| tristan | which they use to split and compose the system ? | 09:19 |
| juergbi | split-rules would be in static public data of glibc, wouldn't it? | 09:19 |
| juergbi | or do you mean dynamic split rules? | 09:19 |
| tristan | I mean the dynamic one | 09:19 |
| juergbi | ok, in that case, they can only change if glibc sources or commands change, right? | 09:19 |
| tristan | because it's sensible that an element should only ever rely on the dynamic one at assemble() time | 09:19 |
| tristan | Maybe we bite the bullet and just say that all static public data is included in an artifact cache key | 09:20 |
| juergbi | didin't i already mention this? | 09:20 |
| tristan | juergbi, also because; especially if we are going to upload it to an artifact share, we want the recorded metadata to effect what was uploaded | 09:21 |
| juergbi | static public data needs to be included | 09:21 |
| tristan | it currently is not | 09:21 |
| tristan | juergbi, however it means that we lose that optimization | 09:21 |
| tristan | of being able to modify split rules in the lower stack without rebuilding | 09:21 |
| juergbi | right, we would lose that | 09:22 |
| juergbi | i don't see this as something extremely frequent, though | 09:22 |
| juergbi | i.e., rather less frequent than source updates | 09:22 |
| tristan | Okay. | 09:22 |
| tristan | Lets do that then | 09:22 |
| tristan | tlater, so people in #gitlab have confirmed that no such beast exists | 09:25 |
| tlater | Aww | 09:26 |
| tristan | so dont worry about it, we'll just have to live with it | 09:26 |
| tristan | they did point out this feature: https://docs.gitlab.com/ee/ci/yaml/#anchors | 09:26 |
| tristan | which might be helpful to make more brief yaml, but doesnt really do anything dynamic | 09:26 |
| tlater | Yeah. I'd like to stay away from defining every test inside the CI, there are just too many for this. | 09:27 |
| tristan | that usage of a '<<' key in a dict is interesting though | 09:27 |
| tristan | it's inspiring that we might be able to use that for doing tricky stuff in our own yaml dict compositing | 09:27 |
| tristan | (like explicitly clearing out values) | 09:28 |
| tlater | I managed to wrongly check for git's unshared permissions in the script... I'll have to figure out what exactly it keeps before I can complete the MR for the CI. | 09:30 |
| tlater | So it turns out any files in the gitlab runner have permissions set to 777 | 10:09 |
| tlater | The only way to handle this sensibly is to completely stop checking permissions | 10:09 |
| jonathanmaw | tristan: hmm, I'm getting an error when trying to search for an element during configure. Am I right in thinking that information's not available at that point? | 10:13 |
| jonathanmaw | for that matter, it also seems like self.info doesn't work in configure. Is there a kind of logging that is available during that stage? | 10:14 |
| ssam2 | tlater, that sounds weird, a filesystem shouldn't just lose all permissions | 10:16 |
| ssam2 | tlater, unless it's FAT or something | 10:16 |
| tlater | No, it's git. Git doesn't preserve permissions. | 10:16 |
| tlater | I thought it preserved at least the executable bit, but it doesn't seem to. | 10:17 |
| ssam2 | it should store exec bit | 10:17 |
| tlater | Unless it actually is FAT | 10:17 |
| ssam2 | but yeah I guess it doesn't store others | 10:17 |
| tlater | I suppose we'll have to live with just checking diffs | 10:18 |
| tristan | git certainly does store permissions | 10:25 |
| tristan | tlater, are you using the docker image with the fresh ostree ? | 10:25 |
| tristan | jonathanmaw, try launching with `bst --debug` ? | 10:26 |
| tlater | I'm using ssam2's docker image | 10:26 |
| tristan | jonathanmaw, I think we have messages disabled during that stage though | 10:26 |
| tlater | The problem isn't bst | 10:27 |
| tristan | jonathanmaw, and no I dont expect the pipeline to be searchable at that stage | 10:27 |
| tristan | hmm | 10:27 |
| jonathanmaw | tristan: yep, --debug made the messages show | 10:27 |
| tristan | git certainly stores permissions though, your recent MRs were changing 0644 -> 0755 if you remember | 10:27 |
| tlater | Oh, nvm | 10:27 |
| tristan | jonathanmaw, so yeah thats a bit... I dont know what to do with that | 10:28 |
| tlater | Yeah, that's true. Strangely it doesn't propagate into the container/image. | 10:28 |
| tristan | jonathanmaw, we could allow it, but it will mess with the status of loading the pipeline | 10:28 |
| tlater | It can't be FAT either because otherwise I wouldn't get permission issues | 10:28 |
| tlater | They'd just all be ignored | 10:28 |
| tristan | jonathanmaw, maybe better to just indicate that in the documentation for Plugin.configure() ? | 10:29 |
| jonathanmaw | tristan: I'm fine with it not showing up unless you're using --debug, so documenting it seems the best approach. | 10:30 |
| * jonathanmaw goes to lunch | 10:32 | |
| jonathanmaw | I'll submit a merge request to document Plugin.configure() when I get back. | 10:33 |
| tlater | ssam2: Is there a way we can set up the cache to be inside the checkout directory? | 10:36 |
| tlater | GitLab CI allows creating a cache as long as it's in the checkout dir | 10:36 |
| ssam2 | i'm not sure | 10:36 |
| ssam2 | jjardon[m] might know | 10:37 |
| tristan | tlater, so you know you can easily set the sources directory with the config file right ? | 10:43 |
| tristan | tlater, for BuildStream, just in case you were wondering about that | 10:44 |
| tlater | Oh, right | 10:44 |
| tlater | That's definitely better | 10:44 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 4 commits (last: utils.py: Rewording some docstrings for new glob() api) https://gitlab.com/BuildStream/buildstream/commit/499bacc276497fa12b71a778cdaf15d83e8efdd2 | 10:45 |
| tlater | Ah, no, the issue was that that caused the out of space issue ssam2 had earlier, didn't it? | 10:46 |
| tristan | tlater, alternatively, note that everything defaults to XDG_CACHE_HOME https://buildstream.gitlab.io/buildstream/config.html#config | 10:47 |
| tristan | tlater, the issue before was that it was building on an undesirable partition, indeed | 10:47 |
| tristan | I suspect it was a tmpfs or smth | 10:47 |
| tlater | But the project directory should not be in that partition | 10:48 |
| tlater | I'll have to test this... | 10:48 |
| tristan | I dont know where the project directory or what should be where :) | 10:48 |
| tlater | I might be able to speed up the tests to like 4x the speed x) | 10:49 |
| tristan | ok some docs improvements appeared: https://buildstream.gitlab.io/buildstream/ | 10:55 |
| tristan | tlater, notice "Public Data" section | 10:55 |
| tristan | which is there to cover any core public data types in the "bst" domain | 10:56 |
| tristan | I think plugins which have their own special data, and no core APIs, should probably use different domains | 10:56 |
| tristan | not sure | 10:56 |
| tlater | Where is the public data section? | 10:59 |
| tristan | in the first area | 11:00 |
| tristan | after "The BuildStream Format" | 11:00 |
| * tristan adds a link to utils.glob() as that should be useful from split-rules description | 11:01 | |
| ssam2 | i find the docs a bit hard to navigate at times. I think the issue is that none of the links under "Using BuildStream" end up in the table of contents | 11:01 |
| tlater | Ah, sorry, I had the page cached... | 11:02 |
| ssam2 | there's lots of useful info in the "Installing BuildStream", "Invoking BuildStream" etc. pages, but they can only be reached by scrolling halfway down the front page | 11:02 |
| tlater | It's hard to get back to the front page, too. | 11:03 |
| tlater | Great, that explains split rules. I think I attempted this format at some point, so that may actually be a bug, but I'll have to check. | 11:04 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Enhanced documentation for split-rules public data) https://gitlab.com/BuildStream/buildstream/commit/426bbcbc744be8097a8324d7ca111336e8427801 | 11:08 |
| tristan | tlater, right but we changed from regex to glob now, so that *might* have been the issue | 11:08 |
| tristan | it could of course be a bug :) | 11:08 |
| tristan | ssam2, maybe I have a taller browser than you do as I see that at first, but we should probably have a page dedicated to being an introduction and reduce the top part to just a short paragraph | 11:09 |
| ssam2 | yeah that would help | 11:10 |
| tristan | there is supposed to be something on the left you can navigate with, but it was difficult to get it to work with sphinx | 11:12 |
| tristan | I ended up giving up trying to enhance that | 11:12 |
| tlater | I can't get the tests to succeed on the ci server. The build environments are completely different for no clear reason :( | 11:12 |
| tristan | second time now I got this error: https://bpaste.net/show/d4e3bd8922b4 | 11:19 |
| tristan | gonna be tricky | 11:20 |
| tristan | But doesnt make sense to me | 11:20 |
| tristan | it's very rare that it happens, and we unmount by sending SIGTERM to a child process that runs fuse in the foreground | 11:21 |
| tristan | that process is in this loop: https://github.com/libfuse/libfuse/blob/master/lib/helper.c#L257 | 11:22 |
| tristan | doing fuse_loop() with signal handlers installed, which cleanly exit the loop and unmount on SIGTERM :-S | 11:23 |
| * tlater Goes for lunch while waiting for a build | 11:23 | |
| tristan | so even, when we wait/join that subprocess, the mount should be synchronously unmounted | 11:23 |
| tristan | I think maybe my problem is that I've been using fork() without execve() | 11:31 |
| tristan | (multiprocessing) | 11:31 |
| tristan | But its annoying to call out through the shell to start a process with... just a python function again | 11:31 |
| tristan | ah, it could be a bug in the FuseOperations itself ! if it crashes, this might happen | 11:36 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: _fuse/mount.py: Check exit status of fuse mount process) https://gitlab.com/BuildStream/buildstream/commit/c5101a06fdc49d358924245b0ad4da1ec94da26d | 11:54 |
| gitlab-br-bot | buildstream: merge request (tristan/test_tests->master: WIP: Add integration tests) #40 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/40 | 11:54 |
| tlater | The cache seems to be working :) | 12:21 |
| gitlab-br-bot | buildstream: merge request (jonathan/plugin-doc->master: plugin.py: Document that logging is normally suppressed during configure) #46 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/46 | 12:28 |
| *** tristan has quit IRC | 12:29 | |
| gitlab-br-bot | buildstream: merge request (tristan/test_tests->master: WIP: Add integration tests) #40 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/40 | 13:30 |
| tlater | I'll create a MR with all passing tests for now, just so that the CI can be up and running soon. | 13:38 |
| tlater | Caching/weird failures still need debugging... | 13:38 |
| gitlab-br-bot | buildstream: merge request (tristan/test_tests->master: WIP: Add integration tests) #40 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/40 | 13:40 |
| jonathanmaw | hrm, it seems that the problem might not have been that I was searching for depends in configure | 13:51 |
| jonathanmaw | since it's failing in assemble, too | 13:52 |
| ssam2 | it seems that an element gets rebuild if there's a change to one of its runtime dependencies | 13:52 |
| ssam2 | which is illogical in theory, because the build should turn out exactly the same | 13:52 |
| ssam2 | but I guess changing runtime dependencies changes the cache key | 13:52 |
| juergbi | ssam2: that shouldn't be the case | 14:04 |
| juergbi | cache key is generated based on build dependencies | 14:04 |
| ssam2 | hmm | 14:05 |
| ssam2 | i could be wrong, but will keep an eye on it | 14:05 |
| juergbi | dependencies = [e._get_cache_key() for e in self._direct_deps(Scope.BUILD)] | 14:05 |
| ssam2 | that seems pretty conclusive | 14:06 |
| *** violeta has joined #buildstream | 14:06 | |
| juergbi | there could be bugs in dependency calculation but i would have expected this to work properly | 14:06 |
| ssam2 | i just removed a runtime dependency from an element, rebuilt it, and it is being rebuilt from scratch | 14:07 |
| ssam2 | so maybe there is a bug | 14:07 |
| ssam2 | yes, the cache key is different | 14:08 |
| juergbi | and it was only a direct runtime dependency that was removed, no other changes? | 14:10 |
| ssam2 | yeah | 14:11 |
| juergbi | e.g., removing a runtime dependency of a build dependency will trigger a cache key change | 14:11 |
| ssam2 | I have buildstream.bst which builds BuildStream | 14:11 |
| ssam2 | and then it as Git as a runtime dependency | 14:11 |
| ssam2 | *it has Git | 14:11 |
| ssam2 | removing Git as a runtime dependency changes the cache key | 14:11 |
| juergbi | ssam2: what kind of element is buildstream.bst? | 14:12 |
| ssam2 | distutils | 14:12 |
| ssam2 | might be nice having a `bst show --format '%{cache-key-unencoded}'` or something to investigate cases like this | 14:13 |
| juergbi | ok, i don't see why this happened right away | 14:13 |
| juergbi | agreed | 14:14 |
| *** hergertme has quit IRC | 14:15 | |
| *** hergertme has joined #buildstream | 14:17 | |
| *** violeta has quit IRC | 14:21 | |
| jonathanmaw | hrm, after rebasing against the latest master, my dpkg-build elements aren't working | 14:34 |
| jonathanmaw | because rst2man is now a broken symlink | 14:35 |
| jonathanmaw | "/usr/bin/rst2man: broken symbolic link to ../../../../../../../../../home/jonathanmaw/.cache/buildstream/build/debian-python-ea2ze8p8/root/etc/alternatives/rst2man" | 14:35 |
| jonathanmaw | I'm also unable to track down why my dpkg-deploy elements no longer have any Scope.BUILD dependencies | 14:38 |
| jonathanmaw | but I think that's a separate matter | 14:39 |
| ssam2 | oh dear, has it been a while since you rebased? | 14:39 |
| ssam2 | could be the symlink patches that we've put in the past couple of weeks | 14:39 |
| jonathanmaw | ssam2: I think so, yes | 14:39 |
| ssam2 | i haven't had a chance to dig into that yet, apparently it also broke deployments | 14:40 |
| ssam2 | might be best to revert the symlink patches for now if its blocking you | 14:40 |
| ssam2 | ah, I think the issue might be the 'integration' entry of the cache key | 14:43 |
| ssam2 | (for my rebuilds issue) | 14:43 |
| jonathanmaw | ssam2: can confirm, rebasing my changes onto an earlier commit removed the symlink problem | 14:44 |
| ssam2 | ok | 14:44 |
| ssam2 | juergbi: https://pastebin.com/Hn8C1ekr | 14:44 |
| ssam2 | juergbi: seems that it includes the name of every dependency in the 'integration' section, which probably doesn't make sense | 14:44 |
| juergbi | ssam2: that should only cover build dependencies and their runtime dependencies, though, not direct runtime dependencies | 14:45 |
| ssam2 | ah ok | 14:45 |
| ssam2 | you're right | 14:45 |
| ssam2 | let me try a proper diff then | 14:46 |
| ssam2 | seems the difference is in the 'dependencies' element | 14:46 |
| ssam2 | looks like each runtime dependency is included in that list | 14:47 |
| juergbi | that's odd | 14:47 |
| ssam2 | looks like that's hope element._direct_deps() is intended to work | 14:48 |
| ssam2 | *how | 14:48 |
| ssam2 | no wait | 14:48 |
| juergbi | scope should be limited to build deps | 14:48 |
| ssam2 | yeah, it should be | 14:48 |
| ssam2 | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L584 | 14:50 |
| ssam2 | surely that should be: `elif scope == Scope.BUILD` ?? | 14:50 |
| juergbi | eh, yes, most likely | 14:50 |
| ssam2 | i think we got it then :-) | 14:50 |
| juergbi | :) good catch | 14:50 |
| juergbi | it's correct for dependencies() which includes indirect deps, but wrong in _direct_deps() | 14:51 |
| gitlab-br-bot | push on buildstream@master (by Sam Thursfield): 1 commit (last: element.py: Fix behaviour of Element._direct_deps(Scope.BUILD)) https://gitlab.com/BuildStream/buildstream/commit/b194179d422b65ce8eaa599901b0362840584adb | 14:58 |
| gitlab-br-bot | buildstream: merge request (jonathan/plugin-doc->master: plugin.py: Document that logging is normally suppressed during configure) #46 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/46 | 15:00 |
| *** jonathanmaw has quit IRC | 15:02 | |
| ssam2 | going to have to start looking at cleaning up the BuildStream artifact cache before long | 15:13 |
| ssam2 | hmm, actually I have to rebuild anyway because I just changed how cache keys work | 15:13 |
| ssam2 | for now I will cheat and just delete the whole thing :-) | 15:13 |
| tlater | ^ That solves a lot of issues ;P | 15:17 |
| *** tristan has joined #buildstream | 15:27 | |
| *** ChanServ sets mode: +o tristan | 15:27 | |
| tristan | ssam2, reading the logs, yeah that really should not be happening | 15:31 |
| * tristan reads more logs and sees there is some kind of conclusion... | 15:31 | |
| tristan | ah, good [pc]atch :) | 15:32 |
| ssam2 | :-) | 15:47 |
| ssam2 | tlater, I think I've figured out the glibc issue you were having in the source bundle | 16:49 |
| tlater | Oh, what was it? | 16:49 |
| ssam2 | somehow /dev/null ends up in the source bundle, but it's not a real /dev/null | 16:49 |
| ssam2 | so mounting a proper devtmpfs at /dev should fix | 16:49 |
| ssam2 | just testing that out now (but setting up nohup to run it so that I can go home :-) | 16:49 |
| tlater | Why does /dev/null end up in the bundle? x) | 16:50 |
| ssam2 | not sure | 16:50 |
| tlater | Perhaps fhs-dirs? | 16:50 |
| ssam2 | the contents are the output of `mkdir --version` | 16:50 |
| ssam2 | so I guess something is doing `mkdir --version > /dev/null` for whatever reason | 16:50 |
| tlater | Yeah, makes sense | 16:50 |
| tlater | Probably testing whether it runs, since that should return 0 | 16:51 |
| ssam2 | yeah | 16:51 |
| ssam2 | definitely something a bit weird going on, but I'll try ignoring it for now | 16:51 |
| tlater | The permissions were a gitlab issue btw | 16:51 |
| ssam2 | aha | 16:51 |
| ssam2 | hmm actually it's not the bundle that contains /dev/null, it's the stage2 sysroot | 16:53 |
| ssam2 | so something weird happens during bootstrap | 16:53 |
| ssam2 | which shouldn't happen, as bst should be mounting device nodes in the staging area... | 16:53 |
| ssam2 | anyway, glibc is compiling now, let's see what breaks over the weekend :-) | 16:53 |
| tlater | Probably the very next package ;) | 16:54 |
| ssam2 | ha, you were right :-( | 17:05 |
| ssam2 | got as far as zlib | 17:06 |
| tlater | :( | 17:06 |
| ssam2 | oh well, lots to do on monday | 17:06 |
| ssam2 | have a good weekend | 17:06 |
| tlater | You too :) | 17:06 |
| *** ssam2 has quit IRC | 17:06 | |
| *** tlater has quit IRC | 17:08 | |
| *** tiagogomes has quit IRC | 17:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!