| *** narispo has quit IRC | 06:39 | |
| *** narispo has joined #buildstream | 06:39 | |
| gitlab-br-bot | aevri opened (was WIP) MR !1674 (aevri/fuse_mount_private->master: _fuse/mount: make mount() and unmount() private) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1674 | 08:50 |
|---|---|---|
| gitlab-br-bot | juergbi approved MR !1673 (aevri/testutils_artifactshare->master: tests/artifactshare: safer cleanup_on_sigterm use) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1673 | 08:52 |
| *** benschubert has joined #buildstream | 09:24 | |
| *** santi has joined #buildstream | 09:48 | |
| *** jonathanmaw has joined #buildstream | 09:57 | |
| *** tiagogomes has quit IRC | 10:00 | |
| *** phil has joined #buildstream | 10:00 | |
| *** jonathanmaw_ has joined #buildstream | 10:00 | |
| *** jonathanmaw has quit IRC | 10:01 | |
| *** akvilebirgelyte has joined #buildstream | 10:03 | |
| *** jonathanmaw_ has quit IRC | 10:05 | |
| *** jonathanmaw has joined #buildstream | 10:05 | |
| *** traveltissues has joined #buildstream | 10:09 | |
| *** tiagogomes has joined #buildstream | 10:16 | |
| gitlab-br-bot | traveltissues approved MR !1674 (aevri/fuse_mount_private->master: _fuse/mount: make mount() and unmount() private) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1674 | 10:17 |
| *** lachlan has joined #buildstream | 10:29 | |
| *** lachlan has quit IRC | 10:43 | |
| *** lachlan has joined #buildstream | 10:45 | |
| gitlab-br-bot | marge-bot123 merged MR !1674 (aevri/fuse_mount_private->master: _fuse/mount: make mount() and unmount() private) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1674 | 10:47 |
| *** narispo has quit IRC | 10:49 | |
| *** narispo has joined #buildstream | 10:49 | |
| *** narispo has joined #buildstream | 10:50 | |
| *** narispo has quit IRC | 10:50 | |
| *** bochecha has joined #buildstream | 10:50 | |
| *** narispo has joined #buildstream | 10:50 | |
| gitlab-br-bot | traveltissues opened (was WIP) MR !1651 (traveltissues/1161->master: extend source api and remove private use from workspace plugin) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1651 | 10:51 |
| milloni | does buildstream have a cache of cloned repositories? | 10:52 |
| tpollard | do you mean locally, or hosted as a project? | 10:53 |
| *** bochecha has quit IRC | 11:08 | |
| *** lachlan has quit IRC | 11:14 | |
| *** narispo has quit IRC | 11:25 | |
| *** narispo has joined #buildstream | 11:25 | |
| *** narispo has quit IRC | 11:29 | |
| *** lachlan has joined #buildstream | 11:33 | |
| *** narispo has joined #buildstream | 11:36 | |
| * tlater[m] wonders if we're too far down to ever start using black for formatting | 11:37 | |
| *** bochecha has joined #buildstream | 11:37 | |
| tlater[m] | I've *really* gotten used to hitting my "format this for me please" button whenever I write a long line of code because of rustfmt :( | 11:38 |
| *** lachlan has quit IRC | 11:39 | |
| Kinnison | hehe | 11:39 |
| benschubert | I wouldn';t be against. Would be a mess for the history for a time though | 11:39 |
| Kinnison | Rebase the entire history of the project, blacking every commit | 11:40 |
| * Kinnison hides | 11:40 | |
| * tlater[m] wonders how long that'd take | 11:40 | |
| benschubert | Kinnison: I would not be against, but blood might flow | 11:42 |
| tlater[m] | benschubert: Think I should start a ML thread discussing whether we can even think about it? I imagine it'd be a problem for supporting bst 1.4, since it would make backporting even harder. | 11:42 |
| * tlater[m] wasn't serious, but sort of likes the idea | 11:42 | |
| *** tpollard has quit IRC | 11:49 | |
| *** tpollard has joined #buildstream | 11:58 | |
| *** dtf has joined #buildstream | 12:03 | |
| tlater[m] | Should `bst show --format '%{full-key}'` respect non-strict mode settings? | 12:07 |
| * tlater[m] notices that a couple of tests fall over if we don't set strict cache keys in non-strict pipelines, even if this shouldn't be a problem | 12:08 | |
| *** lachlan has joined #buildstream | 12:12 | |
| *** bochecha has quit IRC | 12:23 | |
| juergbi | tlater[m]: I'd say all commands should respect that option. however, strong keys may not always be available in non-strict mode | 12:26 |
| tlater[m] | juergbi: That's my point. Currently `bst show --format '%{full-key}'` will print a strong key | 12:26 |
| juergbi | the main purpose of strict cache keys in non-strict mode is for pulling | 12:26 |
| tlater[m] | Why don't we print a weak key if we're in non-strict mode? | 12:26 |
| juergbi | tlater[m]: that's correct. strong keys also exist in non-strict mode. strong != strict (even if the terminology may be confusing) | 12:27 |
| tlater[m] | Yes, however, if there is no strong key, instead of printing the weak key in non-strict mode we will print ?????????????????????/ | 12:27 |
| juergbi | iirc, strict cache keys are used in non-strict mode also for key formatting in the frontend (bold if the key matches the strict key, non-bold otherwise) | 12:27 |
| juergbi | tlater[m]: yes, that's correct | 12:27 |
| juergbi | if we can't know the strong key, we can't know it.. | 12:28 |
| juergbi | we shouldn't print the weak key just because we don't know the strong key | 12:28 |
| tlater[m] | juergbi: Is there a reason for this? | 12:28 |
| juergbi | (unless we would clearly mark it as not being a strong key, but that may be confusing) | 12:28 |
| juergbi | a weak key can refer to multiple artifacts built with different dependency versions | 12:29 |
| *** lachlan has quit IRC | 12:29 | |
| juergbi | so the user should never accidentally use a weak key as that could lead to confusing results | 12:29 |
| * tlater[m] thinks it would make sense to print a weak key if we're asked to print a key in non-strict mode and there is no strong key | 12:29 | |
| tlater[m] | Since the user knows that they're using non-strict mode | 12:29 |
| juergbi | blindly mixing doesn't make sense to me, imo | 12:29 |
| juergbi | we could have a weak-key format if we don't have that yet | 12:30 |
| juergbi | and then users could use that | 12:30 |
| juergbi | but just printing a 'random' key is bad UX, imo | 12:30 |
| tlater[m] | Hrm | 12:30 |
| juergbi | I mean, what would the user do with the weak key? | 12:30 |
| tlater[m] | I suppose I'll just start calling __update_non_strict_cache_key in get_cache_key instead | 12:30 |
| juergbi | if the artifact is available, there will be a strong cache key | 12:31 |
| juergbi | and the user can't really do anything with a weak key to a non-existing artifact, right? | 12:31 |
| tlater[m] | I believe they could plug it into pull? | 12:32 |
| juergbi | I don't think that would work | 12:32 |
| juergbi | and why would the user want to? | 12:32 |
| juergbi | element name would work and would properly first attempt to get the artifact with the strict key => better result | 12:33 |
| tlater[m] | Yeah, I suppose | 12:33 |
| gitlab-br-bot | traveltissues opened issue #1186 (Sources advertising no prestage key and not needing previous sources should skip tracking) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1186 | 12:39 |
| *** lachlan has joined #buildstream | 12:42 | |
| traveltissues | are there any further comments for !1651? | 12:49 |
| gitlab-br-bot | MR !1651: extend source api and remove private use from workspace plugin https://gitlab.com/BuildStream/buildstream/merge_requests/1651 | 12:49 |
| gitlab-br-bot | traveltissues opened issue #1187 (Add test for incorreclty nested directory staging) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1187 | 12:57 |
| *** akvilebirgelyte has quit IRC | 13:08 | |
| *** lachlan has quit IRC | 13:12 | |
| *** akvilebirgelyte has joined #buildstream | 13:12 | |
| *** santi has quit IRC | 13:19 | |
| *** santi has joined #buildstream | 13:31 | |
| gitlab-br-bot | marge-bot123 closed issue #1161 (Follow-up from "Defer workspace source commits") on buildstream https://gitlab.com/BuildStream/buildstream/issues/1161 | 13:37 |
| gitlab-br-bot | marge-bot123 merged MR !1651 (traveltissues/1161->master: extend source api and remove private use from workspace plugin) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1651 | 13:37 |
| gitlab-br-bot | aevri opened (was WIP) MR !1676 (aevri/enable_spawn_ci_5a->master: spawn tests: enable most non-integration) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1676 | 14:19 |
| *** lachlan has joined #buildstream | 14:20 | |
| *** lachlan has quit IRC | 14:42 | |
| gitlab-br-bot | traveltissues opened MR !1677 (traveltissues/1184->master: buildgrid-compose.yml: reduce log level to WARN) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1677 | 14:43 |
| *** lachlan has joined #buildstream | 14:49 | |
| gitlab-br-bot | jonathanmaw opened MR !1678 (jonathan/win32_ci->master: Add native windows CI configuration) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1678 | 14:53 |
| milloni | [10:52:54] <milloni> does buildstream have a cache of cloned repositories? | 14:56 |
| milloni | [10:53:47] <tpollard> do you mean locally, or hosted as a project? | 14:56 |
| milloni | locally, say im running a buildstream build, does it cache all cloned repositories? | 14:56 |
| milloni | and can i push that to a remote cash? | 14:56 |
| tpollard | buildstream master caches sources, and can push them to a remote cache | 14:57 |
| milloni | (this isnt that useful since i would aim for all mirrors to be in a nearby network, presumably the same network where the remote cache lives, but ...) | 14:57 |
| milloni | oh nice | 14:57 |
| milloni | thanks | 14:57 |
| tpollard | milloni: https://docs.buildstream.build/master/using_config.html#config-sources | 14:58 |
| tlater[m] | Does anyone know why we set a strict cache key in non strict builds if assembly is scheduled? https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/element.py#L3174 | 14:58 |
| tlater[m] | It reads like it just so happens that if assembly is scheduled, all other requirements for a strict cache key existing are met, so we might as well | 15:00 |
| tlater[m] | But I'd like to know what state actually causes this to toggle, instead of blanket setting it when assembly is scheduled. | 15:00 |
| juergbi | tlater[m]: it's not setting the strict cache key, it's setting the strong cache key | 15:00 |
| juergbi | strong cache key is crucial in both strict and non-strict build plans | 15:01 |
| tlater[m] | Sorry, yes, that's what I mean | 15:01 |
| juergbi | e.g., after an element is built in non-strict mode, the strong cache key is needed to add it to the cache | 15:01 |
| tlater[m] | juergbi: This happens *before* the element is built, though | 15:02 |
| tlater[m] | And I'm running into a failing test case where no build should ever run that seems to rely on the element being scheduled to build | 15:03 |
| juergbi | when assembly is scheduled it can be calculated | 15:03 |
| juergbi | we could potentially have a separate trigger | 15:03 |
| milloni | out of intereset when's the next release of buildstream? | 15:03 |
| tlater[m] | juergbi: Why can it be calculated at that point, but not earlier? | 15:03 |
| juergbi | as it guarantees that all build dependencies have their strong cache keys | 15:04 |
| tlater[m] | milloni: TBA - There are no plans for a 2.0 release atm, though I'd like to see one over the next couple of months. | 15:04 |
| tlater[m] | juergbi: Should we not then set the strong cache key once all build dependencies have their strong cache keys? | 15:05 |
| milloni | thanks | 15:05 |
| juergbi | tlater[m]: that should be possible, yes. but we need to make sure to not perform that calculation as long as pull is pending or if the artifact is already cached | 15:06 |
| juergbi | tlater[m]: actually, not sure whether it'd be a good idea to decouple it from assemble | 15:07 |
| tlater[m] | juergbi: why not? | 15:07 |
| *** lachlan has quit IRC | 15:07 | |
| juergbi | in non-strict mode elements may have a set of possible strong cache keys. the one we calculate in that `elif` only makes sense for the current build environment | 15:09 |
| juergbi | so I can't think of a situation where that calculated strong cache key would have any meaning if we're not building | 15:09 |
| tlater[m] | juergbi: It apparently has meaning when we're using a nested unfetched junction | 15:10 |
| juergbi | hm, how are junctions relevant here? | 15:11 |
| juergbi | they aren't in the build graph | 15:11 |
| juergbi | and there are no junction artifacts | 15:11 |
| tlater[m] | juergbi: Junctions stage their dependencies into a directory named after the element's strong cache key | 15:11 |
| juergbi | junctions don't have dependencies | 15:11 |
| juergbi | they are not part of the dependency graph | 15:11 |
| tlater[m] | juergbi: https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_loader/loader.py#L662 | 15:12 |
| tlater[m] | I might be misunderstanding what's going on there | 15:13 |
| tlater[m] | But for some reason they need a cache key at that point | 15:13 |
| tlater[m] | And said cache key currently relies on __schedule_assemble() being invoked on a nested junction | 15:13 |
| * tlater[m] is trying to find out why, and how to avoid that | 15:13 | |
| juergbi | right, it's a pretty special case, can hopefully get rid of the need for that cache anyway | 15:13 |
| *** lachlan has joined #buildstream | 15:13 | |
| tlater[m] | How would we? | 15:14 |
| juergbi | tlater[m]: one thing that we theoretically could do is to say self.__cache_key = self.__weak_cache_key for elements that don't have dependencies | 15:14 |
| juergbi | junction cache: I'd like to replace it with direct CAS access (probably via CASBasedDirectory) | 15:15 |
| tlater[m] | Right | 15:15 |
| tlater[m] | juergbi: In that case, surely I can just use the weak cache key for that invocation? | 15:15 |
| juergbi | yes, that might be the simplest approach | 15:15 |
| tlater[m] | I.e., use KeyStrength.WEAK instead of trying to get the strong key | 15:15 |
| tlater[m] | juergbi: since they don't have dependencies that's guaranteed to be unique at the granularity it needs to be? | 15:16 |
| juergbi | yes, it's even guaranteed to be identical to the strict cache key | 15:17 |
| tlater[m] | In that case I'm not too worried about aliasing the cache key to the weak key in general for now | 15:18 |
| * tlater[m] isn't sure that's a good idea | 15:18 | |
| *** lachlan has quit IRC | 15:18 | |
| coldtom | hi, is there a way i can force bst to prune my cache? i.e. I didn't have a quota set but i don't want to lose recent artifacts and want to reduce my cache size | 15:28 |
| coldtom | (this is in 1.4.1) | 15:28 |
| tlater[m] | coldtom: You can reduce the cache size and run a build of something you already have, I suppose. | 15:30 |
| tlater[m] | (Or something small, it might be too smart for that :D) | 15:30 |
| coldtom | it was indeed too smart for that :P | 15:31 |
| tlater[m] | Damn you BuildStream and your smart cache resolution! | 15:32 |
| *** santi has quit IRC | 15:43 | |
| coldtom | hmmm it turns out that my cas cache was not actually so big, but the `build/` directory of the cache was | 15:45 |
| coldtom | is that build trees? | 15:45 |
| Kinnison | build/ under buildstream isn't needed IIRC | 15:46 |
| Kinnison | it's failed build trees etc I think | 15:46 |
| coldtom | that would explain why it's so huge, i've had a lot of failed builds | 15:47 |
| tpollard | in 1.4 anyhow :) | 15:47 |
| tlater[m] | coldtom: Kinnison is right, yes :) | 15:47 |
| * Kinnison savours this moment. 'tis not often I hear that | 15:48 | |
| tlater[m] | In 1.99 we've finally got our build trees in the cache as well, and buildstream will clean them up for you ;) | 15:48 |
| tpollard | and not always cache them (it does in 1.4) | 15:48 |
| * coldtom is excited for bst 2 | 15:49 | |
| tpollard | or did that make it into 1.4.... I can't remember at this point | 15:49 |
| *** lachlan has joined #buildstream | 15:51 | |
| coldtom | hah cache size before removing build/: 112G, cache size after: 5.8G | 15:52 |
| Kinnison | much better | 15:53 |
| *** lachlan has quit IRC | 15:56 | |
| *** santi has joined #buildstream | 15:58 | |
| *** lachlan has joined #buildstream | 15:59 | |
| *** lachlan has quit IRC | 16:28 | |
| *** phildawson_ has joined #buildstream | 16:42 | |
| *** phil has quit IRC | 16:43 | |
| *** lachlan has joined #buildstream | 16:44 | |
| *** lachlan has quit IRC | 17:06 | |
| *** narispo has quit IRC | 17:22 | |
| *** lachlan has joined #buildstream | 17:26 | |
| *** santi has quit IRC | 17:30 | |
| gitlab-br-bot | traveltissues approved MR !1676 (aevri/enable_spawn_ci_5a->master: spawn tests: enable most non-integration) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1676 | 17:31 |
| *** tiagogomes has quit IRC | 18:01 | |
| *** traveltissues has quit IRC | 18:02 | |
| *** jonathanmaw has quit IRC | 18:08 | |
| *** ikerperez has quit IRC | 18:19 | |
| *** adds68 has quit IRC | 18:19 | |
| *** qinusty has quit IRC | 18:19 | |
| *** adds68 has joined #buildstream | 18:20 | |
| *** phildawson_ has quit IRC | 19:04 | |
| *** lachlan has quit IRC | 20:44 | |
| *** qinusty has joined #buildstream | 21:29 | |
| *** narispo has joined #buildstream | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!