| *** nimish2711_ has joined #buildstream | 00:17 | |
| *** nimish2711 has quit IRC | 00:18 | |
| *** nimish2711_ is now known as nimish2711 | 00:18 | |
| *** swick has quit IRC | 01:08 | |
| *** nimish2711 has quit IRC | 02:04 | |
| *** nimish2711 has joined #buildstream | 02:06 | |
| *** nimish2711 has quit IRC | 02:21 | |
| *** nimish2711 has joined #buildstream | 02:30 | |
| *** nimish2711 has quit IRC | 02:58 | |
| *** swick has joined #buildstream | 04:43 | |
| *** tristan has joined #buildstream | 06:30 | |
| *** mohan43u has joined #buildstream | 06:40 | |
| *** alatiera has joined #buildstream | 07:12 | |
| *** user_ has joined #buildstream | 07:20 | |
| *** alatiera_ has joined #buildstream | 07:30 | |
| *** alatiera has quit IRC | 07:30 | |
| *** alatiera_ is now known as alatiera | 07:31 | |
| *** user_ has quit IRC | 07:32 | |
| *** toscalix has joined #buildstream | 09:05 | |
| *** toscalix has quit IRC | 09:06 | |
| *** toscalix has joined #buildstream | 09:06 | |
| *** rdale has joined #buildstream | 09:28 | |
| gitlab-br-bot | juergbi approved MR !1239 (mablanch/629-remote-execution-test->master: Add remote execution tests to the CI pipeline) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1239 | 09:36 |
|---|---|---|
| *** tpollard has joined #buildstream | 09:40 | |
| *** raoul has joined #buildstream | 09:48 | |
| *** jonathanmaw has joined #buildstream | 09:59 | |
| benschubert | Is there a way of completely disabling the cache cleanup mechanism of BuildStream? | 10:03 |
| benschubert | Like, I'd rather have it crash than start cleaning | 10:03 |
| *** ChanServ sets mode: +o tristan | 10:06 | |
| tristan | benschubert, for hacking/testing purposes ? | 10:06 |
| tristan | benschubert, that can be pretty easy to comment out I think | 10:06 |
| benschubert | tristan: because it's quicker to remove the whole cache manually, in order to not clog my CI for 30h+ :) | 10:07 |
| tristan | Hmmm | 10:07 |
| benschubert | gah, that would be less ideal | 10:07 |
| tristan | No I thought you were trying to look at the bug which is that we have stack traces when we hit the limits | 10:08 |
| tristan | for which commenting out code might be a decent approach :) | 10:08 |
| benschubert | ah right, that's something else that I should end up looking at x') | 10:08 |
| benschubert | But no here is rather that in its current state the cache cleanup is putting too much strain on my CI | 10:09 |
| tristan | Aborting on reaching the cache limit might be a decent option ? I only wonder if it is going to really be relevant after improving cache cleanup | 10:09 |
| benschubert | after improving, definitely (?) not, before that, it seems a pretty decent stop gap solution | 10:09 |
| tristan | I guess the question is... if we have performant cleanups, would that option still be relevant ? | 10:10 |
| benschubert | but then it's a specific use case | 10:10 |
| benschubert | probably not | 10:10 |
| tristan | It would be preferable to avoid adding API surface that we don't want to keep around, but of course, I sympathize with the situation | 10:11 |
| Kinnison | benschubert: does the cache help your CI? | 10:11 |
| benschubert | I guess I'll trigger the cache cleanup with smaller limit in the meantime :) | 10:11 |
| benschubert | Kinnison: it does reduce the time by approximately 50% yes, but I can just cleanup earlier | 10:12 |
| tristan | So for this kind of stop-gap thing, I wonder if environment variables might be a decent approach | 10:12 |
| tristan | We don't expose any env vars in public API, and it might send better messaging (to ourselves) that "this stop-gap hack can be removed when foo is implemented" | 10:13 |
| tristan | That is, *if* we really judge that we need a stop gap | 10:14 |
| benschubert | tristan: I have a workaround now, I think it might be better concentrate on the root cause instead :) | 10:14 |
| tristan | I certainly agree | 10:14 |
| gitlab-br-bot | tristanvb merged MR !1256 (tristan/backport-update-state-changes-1.2->bst-1.2: Tristan/backport update state changes 1.2) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1256 | 10:22 |
| gitlab-br-bot | tristanvb closed issue #919 ('bst build <elem>' does not assemble all required elements in some circumstances) on buildstream https://gitlab.com/BuildStream/buildstream/issues/919 | 10:23 |
| *** toscalix has quit IRC | 10:26 | |
| *** toscalix has joined #buildstream | 10:27 | |
| benschubert | tristan: oh by the way if you have time: https://gitlab.com/BuildStream/buildstream/merge_requests/1253 was that what you had in mind? | 10:35 |
| benschubert | for the double lookup in the plugin table | 10:35 |
| tristan | benschubert, Looks quite perfect to me | 10:36 |
| benschubert | tristan: then to marge it goes! thanks! | 10:36 |
| tristan | benschubert, Honestly, I dont think running buildstream with `python -O` is a valid way to run BuildStream anyway, but the extra line doesnt hurt | 10:37 |
| tristan | As we discussed, we'd probably need a bunch more changes for assertion disabling to be ummm, safer | 10:37 |
| benschubert | tristan: I agree, however, it's common practice in some production environments to do it, so I'd rather be safe than sorry :) | 10:37 |
| tristan | Right, we probably need some flagship issue to conform to that practice | 10:38 |
| tristan | I'll file one now | 10:38 |
| tristan | benschubert, I suppose that can be changed somehow from a system integration perspective ? | 10:39 |
| tristan | i.e., for us... people launch `bst` as a CLI tool, we dont really have ways to launch it with python | 10:39 |
| *** lachlan has joined #buildstream | 10:40 | |
| benschubert | tristan: would be possible, but very hacky, and could break in many cases, so I guess no :) (Basically, entrypoints can be shell scripts, but then you need a LOT of work to template them correctly to handle all edge cases on how python is installed (RHEL/Windows/venv/conda/etc) | 10:41 |
| tristan | I'll open an issue anyway just to say that there is one | 10:42 |
| tristan | Even if we never change that | 10:42 |
| gitlab-br-bot | tristanvb opened issue #971 (BuildStream does behave correctly with assertions disabled) on buildstream https://gitlab.com/BuildStream/buildstream/issues/971 | 10:51 |
| raoul | Tristan, juergbi, are you happy with !1214? Keen to finally get it over the line | 10:51 |
| gitlab-br-bot | MR !1214: Remote source cache https://gitlab.com/BuildStream/buildstream/merge_requests/1214 | 10:51 |
| tristan | There, filed #971, just to say that we know about it - I'm really ambivalent as to whether it gets done or not :) | 10:52 |
| tristan | raoul, I think juergbi reviewed the bulk of it, I have only responded to a few questions | 10:53 |
| tristan | I am happy if juergbi is happy with it, if you want me to look at the whole thing, that'll take more time :) | 10:53 |
| juergbi | I expect it to be ready to be merged | 10:54 |
| raoul | Should I hand it over to marge or do you want another quick look over? | 10:54 |
| tristan | One question - Does this have src-pull in a separate queue from fetch ? | 10:55 |
| tristan | I suspect that it doesnt, and I don't mind that much | 10:55 |
| juergbi | no, src-push is separate, src-pull is part of fetch | 10:55 |
| tristan | However I do feel that splitting those up would be desirable | 10:55 |
| tristan | I dont think that it should block the patch, though | 10:55 |
| *** lachlan has quit IRC | 10:56 | |
| raoul | As in seperate queues for pulling from a remote source cache and fetching from the original source? | 10:56 |
| juergbi | hm, doesn't this burden users with complexity that they normally don't care about? | 10:56 |
| tristan | Just, feel that it would be desirable to treat these as separate operations, this would probably make scheduler logic more powerful and easier to reason about | 10:56 |
| tristan | I.e., the question of whether to fetch or pull, under what circumstance, I dont think it is in the domain of the queue itself, which should "just do a thing" | 10:57 |
| tristan | juergbi, users ? | 10:57 |
| juergbi | separate queue would normally be exposed to the UI | 10:57 |
| *** nimish2711 has joined #buildstream | 10:57 | |
| tristan | That is true, but it would be more consistent and honest about what is happening | 10:58 |
| tristan | On the flipside, I don't think it's something users should have to care about | 10:58 |
| tristan | That said, I dont have very strong feelings about this | 10:59 |
| juergbi | yes, not sure | 10:59 |
| juergbi | I think it should be consistent with CLI commands | 10:59 |
| juergbi | i.e., we should split the queues only if we also want to split 'bst source pull' from 'bst source fetch' | 11:00 |
| tristan | Which is why I only wanted to raise this as a side note, I think the architecture is more consistent with them separate | 11:00 |
| tristan | juergbi, I dont really agree with that honestly | 11:00 |
| gitlab-br-bot | marge-bot123 merged MR !1225 (juerg/partial-cas->master: Initial support for partial local CAS) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1225 | 11:00 |
| tristan | I think the user asks "Do this thing" and BuildStream does a number of operations it needs to do | 11:00 |
| tristan | juergbi, part of that is driven by project/user configuration (is there an artifact/source cache to interact with ?) | 11:01 |
| tristan | part of it is driven by what activity the user asked for | 11:01 |
| tristan | I wouldn't look for symmetry between CLI commands and queues | 11:01 |
| juergbi | I think it could be confusing to use the same word 'fetch' for two different meanings | 11:01 |
| *** lachlan has joined #buildstream | 11:02 | |
| tristan | That I can agree with :) | 11:02 |
| juergbi | so if the fetch queue only fetches from the original source | 11:02 |
| juergbi | 'bst source fetch' shouldn't pull from the source cache | 11:02 |
| juergbi | anyway, as you wrote, we shouldn't block the MR on this | 11:03 |
| tristan | No no | 11:03 |
| juergbi | so let's marge it | 11:03 |
| tristan | Marge it baby ! | 11:04 |
| tristan | As an aside, there is certainly a big difference in what you get from a source cache and an actual fetch, and a user may very well want to distinguish | 11:05 |
| * tpollard would like Patty & Selma bots | 11:05 | |
| tristan | Having a fully fetched repo means you can switch revisions (as long as it's not a tarball, but its a VCS repo), and have the confidence of being able to do so without blocking on network | 11:05 |
| tristan | (and of course workspacability) | 11:05 |
| raoul | I kinda now see the fetch stage as getting the sources into the local source cache, which to do it should be checking configure remote source caches too so I'm not sure about splitting it up, though I can definitely see points about making the logic easier to reason about | 11:06 |
| juergbi | yes, it certainly makes sense to think about this aspect some more | 11:08 |
| *** tristan changes topic to "BuildStream 1.2.5 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://mail.gnome.org/mailman/listinfo/buildstream-list | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmaps" | 11:38 | |
| gitlab-br-bot | marge-bot123 merged MR !1253 (bschubert/fix-double-lookup->master: plugin.py: Don't make a double lookup in the plugin table to get one) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1253 | 11:45 |
| gitlab-br-bot | marge-bot123 closed issue #440 (Should we implement a (remote) CAS-based SourceCache?) on buildstream https://gitlab.com/BuildStream/buildstream/issues/440 | 12:29 |
| gitlab-br-bot | marge-bot123 merged MR !1214 (raoul/440-source-cache-remotes->master: Remote source cache) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1214 | 12:29 |
| *** lachlan has quit IRC | 12:31 | |
| *** lachlan has joined #buildstream | 12:31 | |
| *** tristan has quit IRC | 12:35 | |
| *** nimish2711 has quit IRC | 13:00 | |
| *** tristan has joined #buildstream | 13:03 | |
| *** nimish2711 has joined #buildstream | 13:05 | |
| gitlab-br-bot | juergbi merged MR !1239 (mablanch/629-remote-execution-test->master: Add remote execution tests to the CI pipeline) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1239 | 13:29 |
| Kinnison | Thank you to those who have taken the time to review !1257 as James and I inch it toward non-WIP | 13:38 |
| gitlab-br-bot | MR !1257: WIP: YAML New World Order https://gitlab.com/BuildStream/buildstream/merge_requests/1257 | 13:38 |
| gitlab-br-bot | tpollard opened (was WIP) MR !1252 (tpollard/945->master: Add initial TestArtifact() abstraction class to testutils) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1252 | 13:55 |
| *** phildawson_ has joined #buildstream | 14:02 | |
| *** phil has quit IRC | 14:03 | |
| gitlab-br-bot | juergbi opened (was WIP) MR !1232 (juerg/partial-cas-remote->master: Initial partial CAS support for remote execution) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1232 | 14:07 |
| *** phildawson_ has quit IRC | 14:40 | |
| *** phildawson_ has joined #buildstream | 14:40 | |
| gokcennurlu | has it ever been discussed why we can't build junctions/why do they always fail? | 14:40 |
| juergbi | gokcennurlu: do you mean why `bst build somejunction.bst` is not supported? | 14:43 |
| gokcennurlu | yes | 14:43 |
| juergbi | junctions point to a project, not to a single element | 14:44 |
| juergbi | until recently there was no concept at all of building a whole project | 14:44 |
| gokcennurlu | yep, OK, I guess I expected to be a no-op. | 14:44 |
| gokcennurlu | instead of hard failing | 14:44 |
| gokcennurlu | so the reason I am asking is that; I maintain two small integration repositories and one junctions to the other | 14:44 |
| gokcennurlu | I was trying to do something like `bst source track *.bst && bst build *.bst`, simple bash script for a simple CI | 14:46 |
| juergbi | with the recently landed default target support, you can simply drop the target argument | 14:47 |
| juergbi | the default is to track/build all elements, automatically excluding junctions | 14:47 |
| juergbi | (for the build where it's not supported) | 14:47 |
| juergbi | and this will also work with subdirectories, unlike shell expansion | 14:48 |
| gokcennurlu | would that auto-fetch the junctions if there were not fetched yet? | 14:48 |
| juergbi | auto-fetch is the default for junctions, iirc | 14:48 |
| juergbi | at least for bst build | 14:49 |
| jennis | Just did a quick test in doc/examples/junctions (`bst build`), auto-fetch is default :) | 14:50 |
| gokcennurlu | thank you a lot Jürg and James :) | 14:50 |
| jennis | Ahhh, but the source track probably requires the fetch | 14:51 |
| gokcennurlu | James sorry, I meant `fetch` there ! it should be ok :) | 14:52 |
| jennis | ahh nice, ok | 14:53 |
| Kinnison | juergbi: If you could have a quick look at the two threads you opened on !1257 I'd appreciate input so I can try and unWIP it later | 14:58 |
| gitlab-br-bot | MR !1257: WIP: YAML New World Order https://gitlab.com/BuildStream/buildstream/merge_requests/1257 | 14:58 |
| Kinnison | raoul: ^^ | 14:58 |
| gitlab-br-bot | adds68 opened issue #972 (Integrate BuildStream into an IDE) on buildstream https://gitlab.com/BuildStream/buildstream/issues/972 | 14:59 |
| raoul | Was just looking :) | 14:59 |
| juergbi | Kinnison: not quite sure what further to add. in one you replied that you haven't reached a decision yet and in the other I've already replied again | 14:59 |
| Kinnison | juergbi: aah I see you've replied :) | 15:00 |
| * Kinnison will split the cache removal out | 15:00 | |
| *** rdale has quit IRC | 15:50 | |
| *** rdale has joined #buildstream | 15:56 | |
| *** nimish2711 has quit IRC | 16:17 | |
| *** nimish2711 has joined #buildstream | 16:21 | |
| tpollard | Is there plans to add benchmarking feedback to MR CI? | 16:23 |
| tpollard | I'm in the realm of update_state and I know it's a problem point | 16:24 |
| *** lachlan has quit IRC | 16:39 | |
| *** toscalix has quit IRC | 16:44 | |
| *** lachlan has joined #buildstream | 16:45 | |
| gitlab-br-bot | LaurenceUrhegyi closed issue #629 (Remote-execution testing (technical debt)) on buildstream https://gitlab.com/BuildStream/buildstream/issues/629 | 16:49 |
| *** lachlan has quit IRC | 16:52 | |
| *** lachlan has joined #buildstream | 16:55 | |
| benschubert | tpollard: we would like to, however, we need to find a right balance to not slow down too much the rest. Would you have some ideas? | 16:56 |
| benschubert | Also, what are you looking at around _update_state()? | 16:57 |
| tpollard | benschubert: if the benchmark can run as a seperate job in parallel with the rest of the runners and is of a similar timescale, then it could be beneficial? | 16:58 |
| laurence | there is a hacky way for devs to test their branch against master for performance already, tpollard, as jennis mailed about | 16:58 |
| tpollard | benschubert: some of the follow up for !955 | 16:58 |
| laurence | of course, it's not in the CI | 16:58 |
| gitlab-br-bot | MR !955: WIP: bst fmt: Add basic functionality https://gitlab.com/BuildStream/buildstream/merge_requests/955 | 16:58 |
| tpollard | sorry, I mean the issue https://gitlab.com/BuildStream/buildstream/issues/955 | 16:58 |
| laurence | and it's not guaranteed same environment, and doesn't force you to push your results for public analysis | 16:59 |
| tpollard | tpollard: yep I'm aware thanks for mentioning it though | 16:59 |
| benschubert | tpollard: if we could run it in a reasonable time scale (like a test job) it would be awesome yep! However, the results would be kind-of flaky. Another Idea we were thinking about would be to have Marge run it before merging and fail if it's too slow unless told otherwise, but again, that's hacky | 17:01 |
| benschubert | tpollard: ok, you might want to sync with jonathanmaw , he's working in splitting up _upadte_state() and doing finner grain updates, to make sure you don't clash too much | 17:02 |
| WSalmon | juergbi, tristan i have been looking at utils.safe_copy/copy_files and it seems to do everything that the local plugins staging function is doing, i am guesing that the code removed from https://gitlab.com/BuildStream/buildstream/commit/d76af9df32c6164a2e1e61861dd53f27a4fa358c is just code that used to be needed but isn't now. given that the pipe line passes, https://gitlab.com/BuildStream/buildstream/pipelines/53516652 i am thinking that ether | 17:11 |
| WSalmon | the code is not needed or we need a new test. | 17:11 |
| WSalmon | is this right? | 17:12 |
| juergbi | WSalmon: the utils functions don't do chmod, afaict | 17:16 |
| juergbi | however, as CAS doesn't store permissions, the use of source cache might have made this redundant | 17:16 |
| gitlab-br-bot | willsalmon opened MR !1260 (willsalmon/simpletest->master: Refactor local source to use the shinny new code in utils) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1260 | 17:17 |
| juergbi | the source-determinism test should catch missing chmod | 17:17 |
| juergbi | maybe raoul could also take a quick look and confirm this is no longer required | 17:19 |
| juergbi | or maybe we still need it for bst source checkout or something else that doesn't use the source cache? | 17:19 |
| juergbi | (in which case a test would make sense) | 17:19 |
| WSalmon | the safe_copy that copy_files usesw has `shutil.copystat` | 17:20 |
| juergbi | yes but the code that you removed doesn't copy permissions, it sets the same permission bits for everything | 17:22 |
| WSalmon | oh, ok, so are the directories getting missed? is that the only diffrence? | 17:24 |
| *** lachlan has quit IRC | 17:27 | |
| juergbi | WSalmon: also regular files are not set to 0644/0755 by safe_copy, if I'm not mistaken | 17:38 |
| juergbi | when staged via source cache, the CAS code will handle that | 17:38 |
| juergbi | however, I think there is still a code path (not for building) where CAS isn't/can't be used | 17:39 |
| juergbi | so it might still cause an issue there | 17:39 |
| jonathanmaw | juergbi: when you have time, can you have a look at my analysis of what happens with Element._update_state() in https://gitlab.com/BuildStream/buildstream/issues/902#note_153368417 and https://gitlab.com/BuildStream/buildstream/issues/902#note_153971383 ? I've been poking around and puzzling out how it all fits together, but there are bits that I'm probably misunderstanding | 17:41 |
| juergbi | will take a look | 17:41 |
| jonathanmaw | thanks, juergbi | 17:41 |
| *** nimish2711 has quit IRC | 17:44 | |
| *** lachlan has joined #buildstream | 17:44 | |
| *** lachlan has quit IRC | 17:49 | |
| WSalmon | juergbi, so i think a workspace from a local source would not have its permissions set correctly, so i was thinking we need a issue saying we should have a test for that case and some doc's for plug in authers explaining better there responsibility hear? or do those docs exist and i have missed them? | 17:52 |
| *** lachlan has joined #buildstream | 17:59 | |
| *** raoul has quit IRC | 18:10 | |
| *** lachlan has quit IRC | 19:04 | |
| *** lachlan has joined #buildstream | 19:14 | |
| *** lachlan has quit IRC | 19:18 | |
| gitlab-br-bot | jennis opened (was WIP) MR !1257 (shared/yaml-rework->master: YAML New World Order) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1257 | 19:18 |
| *** lachlan has joined #buildstream | 19:27 | |
| *** lachlan has quit IRC | 20:00 | |
| *** jonathanmaw has quit IRC | 20:51 | |
| *** alatiera_ has joined #buildstream | 22:08 | |
| *** alatiera has quit IRC | 22:08 | |
| *** alatiera_ is now known as alatiera | 22:09 | |
| *** alatiera_ has joined #buildstream | 23:00 | |
| *** alatiera has quit IRC | 23:00 | |
| *** alatiera_ is now known as alatiera | 23:00 | |
| *** alatiera has quit IRC | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!