IRC logs for #buildstream for Thursday, 2019-06-27

*** mohan43u has quit IRC04:04
*** mohan43u has joined #buildstream04:04
*** tristan has quit IRC05:45
*** tristan has joined #buildstream05:50
juergbihi tristan, hm, I think we always said that BuildStream should be able to use more scalable CAS severs than bst-artifact-server. it seems like you're saying the opposite in your latest comment is saying, or am I misreading that?06:59
juergbioops, s/is saying//06:59
*** ChanServ sets mode: +o tristan07:00
tristanjuergbi, I've never had that understanding for the artifact cache07:00
juergbiok, but I think it's needed07:01
tristanjuergbi, I've been told that CAS is not going to understand refs, as such we *need* our own custom thing07:01
juergbiyes but not instead of CAS but in addition to CAS07:01
tristanI've never heard of any plans to bake something in between07:01
tristanjuergbi, if there are such plans, then feel free to say so :)07:01
juergbihttps://gitlab.com/BuildStream/buildstream/issues/104107:01
tristanHonestly that design doesnt bother me07:01
juergbiin my mind the plan is to keep bst-artifact-server supporting both, our artifact service protocol and standard CAS07:02
tristanjuergbi, What that conversation looks like is a user and a developer and nobody in between - jjardon gets told that "You can use a small service in addition to an abstract CAS implementation", leads to jjardon telling his friends, then we get random folks in the channel complaining that they don't understand how to do it07:03
juergbiI wrote that this will be possible when #1041 is implemented, and that should be correct07:03
gitlab-br-botIssue #1041: Support seperate end points for artifact caches https://gitlab.com/BuildStream/buildstream/issues/104107:03
tristanYes I know07:03
juergbinothing else should be needed for that07:03
tristanOh really ?07:04
tristanSorry I didn't know that part at all07:04
juergbiit would simply not use the CAS part of bst-artifact-server07:04
juergbiwe could also consider supporting bst-artifact-server to proxy CAS requests but that's not strictly needed for that07:05
tristanjuergbi, That sounds very sensible, especially in cases where you want to use the same CAS for remote execution as you do for storing artifacts07:05
juergbiexactly07:05
tristanI wonder how expiry is going to work there07:05
juergbiexpiry is anyway blob-based already on the server side07:05
juergbithere should probably be some logic in bst-artifact-server to expire refs when external CAS server is used07:06
juergbiI don't think it's a blocker, though07:06
juergbialthough we probably need a bst-artifact-server CLI option to disable internal CAS but that should be trivial07:07
juergbii.e., the ideal solution will require a bit more work than just #1041 but it should already be usable before that07:07
tristanI guess we already handle gracefully the cases where you download an artifact which turns out to be partial half way through the download ?07:09
tristanBecause of course I guess... third party CAS implementations expire objects "however they please"07:09
tristanjuergbi, I edited my misinformed comment above07:12
tristanabove well... in the gitlab issue #105007:13
gitlab-br-botIssue #1050: Not possible to use generic servers implementing the remote-apis as remote cache https://gitlab.com/BuildStream/buildstream/issues/105007:13
juergbipartial remote artifacts: yes, I think that's already handled in a reasonable way07:13
tristanI suppose the server just has to change it's expiry technique to also be object based... and then it could do nightly cleanup jobs which eliminate incomplete refs07:14
tristanas a brute force approach07:14
juergbiyes, periodic check with CAS server is a possibility, although it's a bit tricky as if the artifact server called FindMissingBlobs() on the CAS server, the CAS server will likely update the timestamps of all these blobs07:16
juergbian alternative is simple time-based expiry on the artifact server, independent of what CAS is doing07:17
juergbias CAS blob-based expiry typically deletes top-level directory objects at the same time or before its descendants, there is no huge overhead when a client has to check CAS whether the objects for an artifact are still there07:20
juergbi(it will typically fail very quickly)07:20
juergbiso artifact cache expiry for refs doesn't have to be very sophisticated. just need to make sure it doesn't collect cruft forever07:20
juergbidisk space should not be a real issue for refs as long as there is some expiry07:21
*** toscalix has joined #buildstream07:21
*** tristan has quit IRC07:28
*** bochecha has joined #buildstream07:47
*** tristan has joined #buildstream07:49
*** alexandrufazakas has joined #buildstream08:18
benschubertjuergbi: I just read the cas discussion. Does that mean that even with a BuildGrid RE farm + BuildBox CAS server, we'll still need the bst-artifact-server, or am I mixing some stuff?08:20
juergbibenschubert: to have an actual artifact server, yes, you still need that08:21
juergbiit's not a prerequisite for remote execution, however, relying only on RE action cache would be quite a bit less efficient than having a real artifact server08:22
benschubert'actual artifact server' -> a server from which I can pull artifacts with BuildStream corrrect?08:22
juergbiyes, i.e., only bst-artifact-server right now, although there could be other implementations08:22
benschubertare there any such plans for alternate implementations already existing?08:23
juergbino, as the artifact server part (without CAS) should not be too resource intensive, the simple bst-artifact-server will likely be sufficient even for larger deployments08:24
juergbithat said, we'll have to wait and see whether we hit bottlenecks with the simple approach after all08:25
juergbithe artifact service protocol is very simple, though, so other implementations wouldn't require a lot of work (besides the actual scalable storage part if the filesystem approach is considered insufficient)08:26
juergbi(depending on the requirements for expiry but relatively simple approaches should work for that)08:26
*** raoul has joined #buildstream08:27
benschubertjuergbi: I see, but the current one doesn't support a multi-node setup right? I could not have 4 nodes behind a proxy and expect them to work correctly?08:30
juergbibenschubert: it's filesystem-based. I think with the CAS part disabled and a suitable network filesystem, it might just work08:31
benschubertI see, thanks for the clarification08:31
juergbishould only require regular POSIX atomic rename08:32
juergbino in-place modification after rename08:32
benschubertandw hat is stored exactly on the FS?08:32
juergbithe artifact protos for each buildstream cache key08:33
benschubertok thanks!08:33
juergbiartifact protos contain the CAS digests for the artifact files, logs and some other metadata08:33
juergbiwould probably also be fairly simple to write backends for arbitrary key value databases but there are no plans for that at the moment08:35
*** tpollard has quit IRC08:38
*** tpollard has joined #buildstream08:39
juergbitristan: do you have an opinion on these new config keys? https://gitlab.com/BuildStream/buildstream/merge_requests/1402#note_18241892208:44
juergbitwo new config keys for this doesn't seem ideal. not sure what alternative would be better, though.08:45
*** jonathanmaw has joined #buildstream09:07
*** phildawson has joined #buildstream09:08
alexandrufazakasThe `-C` option is not available on the stable version of bst, right?09:11
*** ChanServ sets mode: +o tristan09:12
tristanalexandrufazakas, yes it has always been there09:12
tpollardit should be in 1.2 I think alexandrufazakas09:12
tristanjuergbi, I probably do... this looks like it will take quite a context switch... downloading blobs in which context etc... lemme try to decrypt this...09:14
tristanOk I remember this long ago conversation09:15
tristanjuergbi, Essentially I mentioned that the expectation when you build is to have the result, and that it should be the default09:15
alexandrufazakastristan, tpollard: Err, am I doing something wrong here then?09:15
tristanremote or local execution shouldnt matter, default behaviors are default behaviors09:15
tristanalexandrufazakas, probably ? ;-)09:15
alexandrufazakashttps://pastebin.com/CaHQvK0h09:16
tpollardalexandrufazakas: if you do bst --help, you should see it under the main options09:16
alexandrufazakasI forgot to drop that, sorry heh09:16
juergbitristan: yes, that's the case in master and the MR souldn't change that. targets + runtime deps are in local cache by default09:16
alexandrufazakasohh09:16
tpollardalexandrufazakas: it's a main option, so goes before the init command09:16
alexandrufazakasit's bst -C dir init09:16
tristanjuergbi, right, I'm just recalling the conversation... so this is related to that...09:16
alexandrufazakasnot bst init _C dir09:16
alexandrufazakasYep09:16
tristanalexandrufazakas, :)09:16
juergbitristan: exactly, the MR is about making this configurable09:16
alexandrufazakastpollard: Thanks09:16
alexandrufazakasSorry for pinging you guys with that :/09:16
tristanalexandrufazakas, no worries !09:17
tpollardnp :)09:17
tristanjuergbi, So withing thinking from too deeply in the patch... what are the behaviors we would *like* ?09:17
gitlab-br-botaevri opened (was WIP) MR !1408 (aevri/smallerjobs2->master: _scheduler: don't pass whole queue to child job) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/140809:18
tristanjuergbi, I think that clearly we are talking about presence of the target and it's Scope.RUN dependencies09:18
tristanjuergbi, if one wanted to download *all* dependencies, then this would be a separate object that is certainly not remote-execution specific09:18
juergbitristan: afaict, the main option we want is to allow builds without guaranteeing any actual artifact files in the local cache09:19
tristanright, I think you suggested in the ticket that we might want configurability for all, but I think that is in the wrong place09:19
juergbiwell, for local builds, --deps all would suffice for this09:19
juergbiwhile it isn't right now for RE09:19
tristanSo enter consistency questions09:20
tristanWhy have separate "remote execution" specific configurations for this ?09:20
juergbithere is a difference between local builds and remote builds in that for local builds we have to completely download all build dependencies for all builds we have to perform09:21
juergbiwhile with remote execution there is no such requirement, independent of --deps09:21
tristanjuergbi, Perhaps it's better to have a configuration of (A) "Whether we want to ensure that build results are present in the local cache" (default I think)... (B) Whether we only care that build results end up in configured artifact cache servers (implied by A)... (C) Whether we want to ensure that all dependencies including build-of-build-of-build end up in the local cache09:22
tristanAnd not have it remote-execution related, just have remote-execution honor the setting ?09:22
juergbimaybe we can indeed go in that direction09:23
juergbiexisting option --all could be used for (C) even for remote execution09:23
tristanFurther, at first sight... I am quite strongly against any configuration with the wording "file blob" or "blob" in it's name09:23
tristanThose are words about low level things that BuildStream developers know about, they are not a part of the knowledge base of a user09:23
juergbiindeed, we need a good word for this. maybe simply artifact files or so09:24
tristanWhether we get the artifacts locally is the right word yeah, these things are artifacts09:24
juergbito be precise, artifact metadata and directories will always be pulled09:24
juergbithey are required even for remote builds09:25
juergbihowever, we might want to hide that detail from the user as well09:25
tristanjuergbi, Then we need to decide (A) Does the user have to be burdened with the knowledge that having the metadata is any different from having the artifacts ? ... and (B) If not... then we should ensure that `bst artifact show` commands and the like, when operating on the local cache, don't yield results which will later fail on other artifact commands like `bst artifact checkout`09:26
tristanjuergbi, right, essentially what you said with a lot more words :)09:26
tristanI rather tend to favor hiding that metadata from the user09:27
juergbiI'm not sure yet. for someone working in a pure remote execution environment it would be useful to have bst artifact show etc. still work09:27
tristanuntil a day comes that we have real justification to present that to the user, in which case we have to put thought into making it a coherent experience09:27
juergbiand bst artifact checkout implicitly pulling files09:27
tristanif I `list artifactcts in the local cache`, I certainly expect checking them out to succeed without any internet activity09:28
tristanon any artifact which was listed09:28
juergbiby default, yes09:28
juergbihowever, there might be environments where a user wants to fully rely on remote caching09:28
tristanif checking out artifacts implicitly pulls, that is a different topic I think anyway09:28
tristannice to implicitly pull yes, but expectations of scripts running in possibly intentionally isolated environments should be respected too09:29
tristanI might have some automation which does something to "all artifacts which match a given glob pattern"09:29
juergbiyes, we should definitely keep allowing the current behavior, and that should very likely remain the default09:30
tristanjuergbi, Yes exactly09:30
tristanboth should be allowed09:30
tristannow back to the more precise topic09:30
tristanIf --all in `bst build` is a global non-remote-execution-specific option, I doubt that "ensure I have the artifacts locally" should be remote execution specific09:31
tristanThose appear to be quite the same setting09:31
tristanAlso, --all has other implications09:31
juergbiyes, I think we can simplify things quite a bit09:31
tristanjuergbi, --all not only ensures that the results are downloaded for build-of-build dependencies, but it ensures they are on the artifact cache server09:31
tristanfor instance, what if we want to ensure the artifact cache server has build results fresh and ready to download for the bootstrapping phase, but we don't care about having those locally09:32
tristanit seems like a valid usecase for --all as well09:32
juergbiyes, but if we have one new option for local caching, the combination possibilities with --all might suffice09:33
tristanthat does sound sensible09:33
alexandrufazakasDo all the directories created for testing have a `cache` subdirectory?09:33
juergbitristan: I can't imagine the need for more fine-grained options at the moment, so I think we should do this09:34
tristanjuergbi, we probably want to take the `--all` command line option and craft a user config option for it, and put this new option of "ensure-local-artifacts" or whatever it will be named, in the same group in the user config09:34
juergbitristan: one aspect I'm not sure anymore is what exactly that new config option should be09:34
juergbi--all as user config option could make sense but I think we can consider this separate09:35
tristanjuergbi, I am thinking that it is "Whether the user wants to be sure that the build result is present in the local cache after a build", and it is default True09:35
tristanjuergbi, It is always true anyway in local build scenarios, but if remote execution is enabled, it will change the behavior09:36
juergbiright, but keeping the previous part of our discussion in mind, should it also mean 'bst artifact show' will show even partial artifacts09:36
tristanjuergbi, but what the user is expressing with this option is what they want to happen09:36
tristanjuergbi, lets tackle that discussion separately - there are multiple definitions of partial artifacts too09:37
tristanThe answer might be yes if the artifact is there but is missing it's build tree09:37
juergbiit increases the risk of having to rename the option later if we don't discuss this yet, though09:38
* tristan goes to fetch his clean laundry...09:38
tristanHmmm09:38
juergbibut maybe a separate discussion is indeed warranted and we might not want to block this MR on that09:38
tristanjuergbi, ok well... think on it a bit if you have any ideas... I'll be back in 1509:39
juergbiok :)09:39
tristanI think it's a good start to go the way we discussed though09:39
juergbiyes, I think we're definitely going in the right direction09:40
gitlab-br-botmarge-bot123 merged MR !1427 (juerg/source-checkout->master: Fetch sources as needed for bst source checkout) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/142709:41
tlater[m]alexandrufazakas: Not 100% on this, but you may find this useful: https://docs.pytest.org/en/latest/parametrize.html09:45
*** raoul has quit IRC09:48
alexandrufazakastlater: I'll have a look, thanks09:49
tlater[m]Is there any way to run the overnight tests early for a specific branch?09:57
* tlater[m] wants to try aarch64 because he changed the base image09:58
tristanjuergbi, I had some thoughts while folding my clothes09:58
tristanjuergbi, I think definitely `bst artifact show` should never show artifacts for which it only has metadata, and if the *data* locally is partial, it should be possible to express that09:59
tristanjuergbi, Secondly, I think `bst artifact show` shows the local cache by default09:59
tristanand if `bst artifact --remote URL show` or similar is used... then it *only* shows artifacts on *that* remote, not a combination of local + remote10:00
tristanjuergbi, most importantly to bear in mind I think... is that when you write `bst artifact show` and check your local cache, you are basically guaranteed that the artifact will *still* be there in your next line in the script10:00
tristanunless you have called bst artifact delete or modified your local cache in some way10:01
tristanjuergbi, When you type `bst artifact show` for a *remote* URL, it's really just speculation as to whether it will exist one second later or not10:01
tristanjuergbi, essentially this is why we stopped printing "downloadable" state or checking what can be downloaded in advance10:02
tristanjuergbi, does this generally make sense ?10:02
juergbiif 'bst artifact show' calls FindMissingBlobs on the remote, the CAS server is expected to retain those blobs for a reasonable amount of time10:02
tristanreasonable is not exactly specified10:03
tristanjuergbi, I think that is all fine and good from an internal remote execution standpoint, and I see why10:03
tristanjuergbi, Do you think that with this new API then... we should revive the initial query to the artifact server and "downloadable" state in `bst show` and `bst artifact show` ?10:04
juergbiI imagine some people will work in environments where they can rely on artifact/CAS server implementation that provides these guarantees10:04
tristanthat is also another valid perspective I guess10:05
tristanthe initial query can help guarantee that required dependencies will exist whilst building10:05
tristanjuergbi, "reasonable" time frame is not a guarantee though... it implies that "When all is working well, it wont fail", while also implying that "anything built on top of this should expect things to sometimes fail"10:06
juergbiyes but even the local cache has expiry, so there is also no absolute guarantee10:07
juergbirobust scripts will require support for retry10:07
tristanjuergbi, As long as there is not parallel instances there is an absolute guarantee yes10:07
tristanas long as you didnt write `bst artifact delete` or launch another build in between10:07
juergbieven bst pull in between can trigger expiry10:08
tristanAnd even if there are parallel local instances, scripts which setup their own artifact cache directory can have this guarantee10:08
juergbiso you have the guarantee really only if you don't do anything between those lines10:08
tristanexactly10:08
juergbisure and scripts will always be able to do that10:08
juergbibut I don't think this means that we shouldn't provide the option to work in more opportunistic ways10:08
juergbias in some environments that will work very well, at least as I imagine10:09
juergbiwould probably make sense to confirm this with deployment plans of real users10:09
tristanI don't understand exactly what you mean by opportunistic, or your use of the word :-/10:09
juergbiI mean without absolute guarantees10:09
tristanjuergbi, What I want is certainty and knowing what I am dealing with10:10
juergbii.e., relying on reasonable time frame being sufficient10:10
juergbiby default that will be the case10:10
tristanif I want to know what might be there for an undetermined by "probably long enough" amount of time, I will query the remote explicitly right ?10:10
tristanIf I query the local cache, I get what is right now there10:10
juergbia user shouldn't have to specify --remote every time10:10
tristanThere I certainly disagree10:11
juergbiif they are in an environment where they can rely on the server being available10:11
tristanThe artifacts which arent there just arent there10:11
juergbimaybe their environment actually has a never expiring CAS?10:11
juergbithat's not impossible10:11
tristanthey might be in a remote, they are unobtainium until pulled10:11
tristanNow, you might want a semantic to query "the project remotes" for `bst artifact show`10:12
tristanbut I really don't think that should be default10:12
juergbiI'm talking more generally, not specifically about 'bst artifact show'10:12
juergbimaybe that should be local only10:12
juergbiand not project-specific10:13
tristanBut what other commands are there really ?10:13
juergbibut at least for everything project-specific that would make sense10:13
tristanI mean... for everything else, there is a clear definition of what will be done with remotes right ?10:13
juergbiand if a global not-project specific remote is configured, it probably makes sense for 'bst artifact show' as well10:14
tristanin user config10:14
juergbiyes10:14
tristanI still disagree that that should be default10:14
juergbiI'm not saying this should be the default10:14
tristan<tristan> Now, you might want a semantic to query "the project remotes" for `bst artifact show`10:14
tristanEssentially that10:14
juergbiI simply think that we should have an option for this in user config10:15
tristanthe ones defined by normal recursive project definitions and user config10:15
tristanAh, yeah you wont get any argument from me on that of course10:15
juergbiok, was probably some misunderstanding10:15
juergbiwith regards to the pending MR, my question is whether this should be the same user config option10:15
tristanI think mostly, if there is a CLI option, it should be in the user config10:16
tristanwhile the opposite is not always true10:16
tristan(there are current exceptions to this, but I think it's the right way generally)10:16
tristananything configurable should be in the conf file (with a few obvious exceptions).... while the CLI is more selective about showing optionalities10:17
tristanjuergbi, back to the original discussion ? still a question ?10:17
juergbi<juergbi> with regards to the pending MR, my question is whether this should be the same user config option10:18
juergbiI think this question still stands10:18
tristanWell lets thought experiment... I thought what we had was 2 options: (A) Whether we want to ensure everything is built (currently --all)... (B) Whether we want to ensure that build results exist locally after running a build10:18
juergbiyes, if we count --all, we definitely want at least two options10:19
juergbimy question is whether (B) should also control bst artifact show with regards to metadata-only local artifacts10:19
tristanI don't think those are the same10:20
tristanUnless you can think of something a user could express which really accurately covers both cases, I think they are different things10:20
juergbii.e., should it essentially be a option to use a workflow designed for a reliable remote server instead of isolated operation10:21
tristanWhat you expect to materialize locally is different from what you expect to be querying10:21
tristanI see where you are going10:21
juergbisomething like transparent-remote-caching (may not be the best name but might help with the meaning)10:21
tristanI feel like it's an attractive idea, but with all the moving pieces I'm reluctant to say with any certainty that that would suffice10:22
juergbiI can definitely understand that10:23
juergbimaybe it's better to just focus on the local caching for build aspect10:23
juergbiand reconsider the generic option in the future10:23
juergbi(possibly deprecating the other option)10:23
tristanwas going to say similar10:23
tristanWe could analyze workflows after those workflows materialize, learn from that and possibly add more global/easy switches10:24
juergbiyes, adapting based on experience is probably best10:24
juergbiworst case we have to deprecate an option. not the end of the world10:25
tristanjuergbi, it's probably also better anyway to start with this separate approach code-wise10:26
tristanjuergbi, i.e. already have resolved Context.foo switches which enable very specific modes of operation10:26
tristaninstead of having code segments trying to know how to behave because of some global switch that that code segment thinks is smart enough to reason about10:27
alexandrufazakastlater: Pushed my changes to !1430, I think all looks alright right now10:27
gitlab-br-botMR !1430: Add bst init argument https://gitlab.com/BuildStream/buildstream/merge_requests/143010:27
tristankeep that reasoning in one place10:27
alexandrufazakasOr works, rather10:27
juergbithat can go either way. too fine grained switches can also be an issue when an operation could be influenced by multiple contradicting switches10:27
tlater[m]alexandrufazakas: I'll take a look10:27
tristanjuergbi, fair, but I hope we don't get there :)10:28
alexandrufazakastlater: Sweet, thanks10:28
tristanjuergbi, in this case at least we're talking about the behavior of very separate activities10:29
tlater[m]alexandrufazakas: Almost there. I think I spotted another bug that you'll have to write a test case for ;)10:41
alexandrufazakastlater: yay10:42
alexandrufazakasBtw, to run the linter I should get `detox`, right?10:42
* tlater[m] has never heard of detox personally10:42
tlater[m]Running the linter is possible using `tox -e linting` or somesuch10:43
tlater[m]Running the linter is possible using `tox -e lint` actually somesuch10:43
alexandrufazakasOh10:43
* tlater[m] isn't great at editing stuff in non-emacs apparently10:43
alexandrufazakasThat's for testing and linting in parallel I think?10:43
alexandrufazakashttps://gitlab.com/BuildStream/buildstream/blob/master/CONTRIBUTING.rst#L150910:43
alexandrufazakastlater: Yeah, I'll do that, thank you10:43
tlater[m]alexandrufazakas: No, it will only run pylint10:43
tlater[m]And pycodestyle, actually, probably mypy in the future (:10:44
tlater[m]alexandrufazakas: I've seen you use vi... If you're going to be programming for a little longer, I'd suggest installing https://github.com/prabirshrestha/vim-lsp and the python language server10:46
tristantlater[m], alexandrufazakas, linting and testing in parallel *is* possible when you have detox installed10:46
alexandrufazakastlater: Alright, i'll do that. Thank you :)10:46
tristanyou can lint + test py[35,36,37] all in parallel10:47
tristanalexandrufazakas, however as tlater[m] says, that's not how we use it... we just lint separately from tests in gitlab CI10:47
alexandrufazakastristan: Yep,I'll just use `tox -e`10:47
tristanbut... it worked last time I checked, when I ensured that tmp directories were properly namespaced and fixed it to work10:47
alexandrufazakas`tox -e lint`, rather10:47
tristanalexandrufazakas, yeah, and `tox -e py{your favorit version}`10:48
tristan:)10:48
alexandrufazakasTrue10:48
* tristan has all three and used to run `detox .`10:48
tlater[m]alexandrufazakas: Also, if you get it installed, vim-lsp will run the linter for you and show errors as you edit :)10:48
alexandrufazakasI only have .5 right now so I'm only using that, I should probably look for all of them10:48
tristanbut it's been a while since I did that10:48
tristanalexandrufazakas, yeah you actually don't have to specify, it will just test for each discovered python version on your system (that we support)10:49
alexandrufazakasAh that makes sense, ye10:49
alexandrufazakastlater: Thanks for the review, I'll start working on them in a second :)10:51
*** lachlan has joined #buildstream10:51
alexandrufazakasI was considering renaming that argument btw, I just wasn't sure what. Naming things is so cumbersome :/10:51
tlater[m]Haha, agreed10:51
tlater[m]Tell me if you can figure out how to use that "recommendation" feature10:51
tlater[m]It looked cute, so I wrote actual recommendation code for once10:51
tlater[m]You're supposed to be able to apply it directly somehow10:52
alexandrufazakastlater: I'll have a look at it10:52
alexandrufazakastlater: Not sure how to install those vim thingies however, how should I `Plug 'prabirshrestha/async.vim'`10:53
tlater[m]alexandrufazakas: That is for you to find out, I've never used vim? in any capacity, haha. Just recommending a nicer workflow with the tools you have :)10:54
alexandrufazakasRight10:55
alexandrufazakasI'm sure google might help10:55
*** lachlan has quit IRC10:57
tlater[m]Could I get some review for https://gitlab.com/BuildStream/buildstream/merge_requests/1431/diffs?11:02
alexandrufazakastlater: I'm not so sure about that change btw, the one you proposed on gitlab11:06
alexandrufazakasBefore that if, then we need to set the directory t= _main_options['directory'], right?11:06
alexandrufazakasThen, if target_directory: join()11:06
alexandrufazakasDoes it make sense?11:07
*** lachlan has joined #buildstream11:07
tlater[m]Ah, yes11:07
tlater[m]Sorry, my bad11:07
alexandrufazakasAlright11:08
alexandrufazakastlater: regarding this https://gitlab.com/BuildStream/buildstream/merge_requests/1430#note_18601771511:16
alexandrufazakasI'd need to create a temporary directory, so the test can be ran multiple times?11:16
alexandrufazakasSince we'd initialize a project in the home directory of the user11:16
tlater[m]Remember, pylint does that for you. Don't use the exact arguments I gave you there, those are just an example11:16
tlater[m]The point is that the second argument must be an absolute directory11:17
alexandrufazakasSo you wouldn't mind if we did it the other way around11:17
alexandrufazakase.g. -C into /home/alex/foo11:17
alexandrufazakasand init in tmpdir11:17
alexandrufazakasAlright11:17
alexandrufazakas:D11:17
tlater[m](Spoiler: You'll find that your code will create a path like `/home/alex/foo/tmp/directory`, though you wanted it to create `/tmp/directory`)11:18
tlater[m]I think anyway, it's possible os.path.join() is smarter about this than I think.11:18
gitlab-br-botBenjaminSchubert merged MR !1432 (bschubert/node-copy->bschubert/new-node-api: _yaml: Remove 'node_copy' and add 'Node.copy()') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/143211:27
tristanis this what this "add new argument to bst init" is all about ?11:36
tristanadding another redundant -C like argument ?11:36
tristantlater[m], why not just wontfix and close https://gitlab.com/BuildStream/buildstream/issues/702 ?11:37
tlater[m]tristan: Is it redundant? Can you create a new project with bst -C init /home/projects/new_directory?11:41
tristanofcourse11:41
tpollardwell with the dir before init11:42
tpollardsubtly different11:42
tristantpollard, exactly11:43
tristanthis would be just adding yet another redundant way to specify the project directory in `bst init`, all commands take `-C` as the project directory already11:43
tlater[m]Doesn't https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_frontend/cli.py#L220 imply the directory must exist already?11:43
tristanwe're just adding surface to accomplish nothing... not very much of a fan of "more than one way to achieve the same goal" at all11:43
tristantlater[m], nope11:43
tristantlater[m], it will be created11:44
tristantlater[m], try it, I just did, works fine11:44
tlater[m]Right, in that case I agree with you11:44
*** raoul has joined #buildstream11:44
tlater[m]And I assume that #702 was created before -C or somesuch.11:44
gitlab-br-botIssue #702: Make `bst init` take an argument https://gitlab.com/BuildStream/buildstream/issues/70211:44
tristantlater[m], it was created maybe because you didn't know11:45
tristantlater[m], -C/--directory existed since the beginning11:45
tlater[m]Possible, too11:45
tpollardI think there's reasons to also have init take a directory11:45
tristanWe do have one other redundant thing which Ed wanted11:46
tristannot really a fan but whatever11:46
tristanthat is `bst help foo` exists, while `bst foo --help` is perfectly fine and does exactly the same11:46
tlater[m]tpollard: I'll agree that it's a bit more obvious what it does if it's an argument, and we'd be behaving more like git.11:46
tlater[m]But it is redundant11:47
tristanNot sure why we have to mirror git, adding an extra way to do something makes us more complicated in general, not less11:47
tristan-C is basically chosen because it works with very many unixy things11:48
tlater[m]Eh, more complicated code-wise, but an entirely new user will know the pattern from using git11:48
tristanlike tar and make11:48
tlater[m]Git also has -C, and I like it (:11:48
tristantlater[m], I mean more complicated user wise11:48
tristantlater[m], that came later11:48
tristanI had no idea git accepts that11:48
tristanbut it's fine and nice that it does too11:48
tristanit's both more complicated code-wise, AND user wise because now we have TWO user stories of how a thing can work11:49
tristanwhich means one person can say "do it like this", another can say "do it like that", they can both be right... and now we've just crowded user's knowledge base with extra useless information11:50
tristanthat is basically true for every case that there is more than one way to do the same thing11:50
tlater[m]The difference is subtle in that -C sets buildstream's cwd. What would you think about disallowing -C into a nonexistent dir?11:50
tristanThat is not true11:50
tristan-C does not by any means set buildstreams CWD11:50
tristanit informs BuildStream of the project directory to operate on11:51
tlater[m]That's what it means in other unixy tools11:51
tristanThen argue that we change it from -C/--directory to something else ? I don't know, it seems like a good fit to me anyway11:51
tlater[m]-Change :)11:52
tristanI don't see any reason why BuildStream should ever modify it's CWD11:52
tlater[m]It doesn't have to explicitly do so, since it already behaves as if11:52
tristanit does not really11:52
tristanactually, it really does not11:53
tlater[m]It's useful in things such as make, and if we're already recycling unixy CI sticking to their stuff makes it a bit easier to get used to.11:53
tristanwhen I type `make -C directory` I expect to run the build declared by that directory11:53
tristanI dont expect it to internally change it's CWD11:53
SotKI think allowing the git-like syntax is sensible, there is a lot to be said for mirroring patterns from widely used tools for helping users to feel familiar with your tool11:54
tlater[m]tristan: But that's what it does11:54
tristanIf I expected that of BuildStream, then I would expect `bst --log-file ./output.log -C /path/to/project build` to create a log file in the project directory11:54
tlater[m]Gnu make anyway does11:54
SotKI wouldn't guess that -C would do that since I've not encountered it much, but the git style is very familiar11:54
tlater[m]make -C changes cwd specifically so recursive make things work11:55
tristantlater[m], it's rather an important distinction that it has nothing to do with CWD... brings back the painful discussions about how "element names are paths", but they are clearly not11:55
tristantlater[m], sure it might do so as an implementation detail11:55
tlater[m]In that case -C is a bit of a misnomer in bst11:56
tristanwell, it's even documented to do exactly that11:56
tlater[m]tristan: yep11:56
tristantlater[m], but altogether appropriate and familiar, misnomer only if you want to be that pedantic11:56
tlater[m]You started the pedantry with cwd vs project being important, haha11:57
tristanWell okay, fair point :)11:58
tristanStill there is a difference11:58
tlater[m]My point is; we could behave a bit more like git/make and make people's lives a bit easier if we add that argument11:58
tristanThat said, `bst init` is just about the most useless command we could have every added *at all*, and I would also favor removing it altogether11:59
coldtompersonally i would expect `bst init [directory]` to work11:59
tlater[m]tristan: Given that only init will work on a nonexistent dir11:59
tlater[m]Should we make -C prohibit those and make init have an argument?12:00
tristanany -C argument will always fail if it refers to an invalid project directory, of course `bst init` must accept it because it's purpose is to *create* the project directory in the first place12:00
tristanbut again, my vote would really be to just nuke `bst init` altogether and have people create project.conf12:01
SotKthat sounds like it would make starting new projects more painful than necessary to me12:02
tristanSotK, have you started a new project ? project.conf is essentially only a mandatory project name12:03
tristanand a desirable but optional format version12:03
tlater[m]SotK unlike git you don't really create projects that often12:03
tlater[m]There also really aren't that many obvious defaults12:03
tristanHonestly, I cannot imagine any scenario where I would create a project and not want to modify the project.conf by hand *anyway*12:03
tlater[m]tristan: maybe make init more useful by printing some - if not all - the default settings, so users know them without going to the docs?12:04
* tristan really only added this because of complaints that "bst commands would fail in a directory without a project.conf"12:05
tristantlater[m], that's what it does by default in the interactive experience right ?12:05
tristanholds your hand while you enter some things12:05
tlater[m]Oh, right12:05
SotKonly a couple of times and not for a while, but in general I personally prefer an obvious command to do initial config setup than having to find the docs and work out what I need12:06
tlater[m]I forgot it was interactive12:06
tristanyeah the interactive part is probably the most useful part of it12:06
tlater[m]Yeah, I think we could perhaps make the command more useful to get people started12:06
SotKI've not looked at the docs in a while so maybe they make it super trivial to understand anyway, I just remember appreciating the existence of bst init when I was trying out buildstream a while ago12:06
tristanhttps://docs.buildstream.build/tutorial/first-project.html12:07
tristanif you started there well... yeah it does choose the version12:07
tristanbut the version needs to be fixed as per a previous proposal which didnt go through yet as it has to do with the inevitable step of making BuildStream 2 parallel installable12:08
tristani.e. the version should really be the BuildStream version, and not an internal version12:08
tristanyou should be able to just say "2.2" if you want to depend on features added in or before 2.212:08
tristanthe micro managed format-version is only useful for those relying on development snapshots in between stable releases12:09
tristan(and even then it's not immensely useful as features are allowed to change in advance of being released, even more so in the current long dev cycle)12:09
tlater[m]In any case, looks like init is more useful to newcomers than we thought...12:10
tlater[m]tristan: Oddly, you've managed to convince me that the current behavior is definitely not like other tools and that this MR *should* go through, haha12:12
gitlab-br-botaevri opened (was WIP) MR !1433 (aevri/cascache_nits->master: cascache.py: pick some nits) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/143312:13
gitlab-br-bottlater opened (was WIP) MR !1413 (tlater/freedesktop->master: Use a freedesktop-sdk tar as a base image) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/141312:15
coldtomeven without the interactivity of `bst init` i think that having a simple command to create a boilerplate project is useful12:18
coldtomi definitely have appreciated it when creating projects12:18
tristantlater[m], sure, I honestly don't care about `bst init`12:19
tristantlater[m], it just irks me a bit that we adding more surface code to accomplish something we can already accomplish... and having to handle edge cases in addition to that12:21
tristani.e. actually spending time on it :)12:21
tristantlater[m], I think that if you want to go ahead with this though, you would bail out if both `-C` AND a project directory is specified in `bst init`12:22
tlater[m]I was surprised the issue was seeing any attention in the first place.12:22
tristantlater[m], don't mislead people to believe that "we are switching to this directory first and then creating your project inside of it"12:22
tlater[m]tristan: Yep, that's probably better behavior in context. I frankly believed -C did that just by association. I'd also want to disallow using nonexistent directories with -C, since that makes no sense, and that way we avoid having multiple user stories.12:24
* tlater[m] comments on the MR12:25
tristanyou would prefer to break existing invocations of `bst init` ?12:25
tristantlater[m], that means that instead of both -C and the directory being conflicting... -C cannot be used *at all* in `bst init` right ?12:25
tlater[m]tristan: Doesn't 2.0 basically break any idea of backwards compatible scripting anyway?12:25
tristantlater[m], remember that the condition for `-C` is not only that it is an existing directory12:26
tristanit must be *a valid project directory*, or else we bail out early anyway12:26
tristantlater[m], I'm just pointing it out12:26
tristantlater[m], sure you can break it... but be conscious that you are breaking it :)12:26
tristanthat means necessary NEWS entry as well12:27
tlater[m]Yep, these are good notes12:28
tristantlater[m], to clarify on the above... all bst commands will bail out immediately if the project directory is missing a project.conf12:28
tristan(or doesnt exist of course)12:28
tlater[m]And bst init will not even allow you to -C12:28
tristanright, then you would bail out with "Attempted to specify project directory when creating a new project" detail="Use bst pony --foo-bar instead"12:29
tristansomething like that12:29
tlater[m]Yup. I think this makes sense, it's more comparable to similar tools than what we had previously and doesn't pretend we're doing something we don't.12:30
tristanwell, what we currently have is (A) a global way of setting the project directory ... and (B) a project initializer which assumes you mean to initialize the project in the directory you specified12:30
tristanwhat we have now is not nonsense12:31
tlater[m]No, it's not, it's just unfamiliar compared to other tools, so I want to change it.12:31
tristanyes I know12:31
tristanok sorry for whining my head off about this insignificant little detail :)12:32
tlater[m]tristan: Don't worry, good thing you said something, I only realized -C works this way when you did :)12:32
tristanI also think that we should take the user experience of `make` as a more appropriate comparison in general... it's a lot more similar to how you use BuildStream than something like git :)12:34
gitlab-br-botBenjaminSchubert merged MR !1424 (bschubert/node-api-nodel->bschubert/new-node-api: _yaml: Remove 'node_del' and support `del mapping[key]`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/142412:34
tristanessentially all the stuff you put in a make project is stuff that you author by hand, it is not a store, it is all input for the tooling12:34
tlater[m]tristan: That's why I sort of agree with you that we shouldn't have an init ;p12:35
tristanyeah, but people like it, especially the hand holding is nice... not sure really how valuable it is12:36
tristantlater[m], if we fixed "version:" (and perhaps removed "format-version") to work more intuitively, it would be rendered almost pointless12:36
tristanknowing that secret number is kind of what bst init does best12:37
* tristan goes out to dinner12:37
tlater[m]o/12:37
*** phildawson_ has joined #buildstream13:13
*** phildawson has quit IRC13:14
*** phildawson_ has quit IRC13:15
*** phildawson has joined #buildstream13:16
gitlab-br-botmarge-bot123 merged MR !1433 (aevri/cascache_nits->master: cascache.py: pick some nits) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/143313:17
*** bochecha has quit IRC13:20
benschuberttristan: I need your input on an API question: should 'node_find_target' be something public or just for us?13:33
gitlab-br-botBenjaminSchubert opened MR !1434 (bschubert/node-find-target->bschubert/new-node-api: Replace 'node_find_target' by 'MappingNode.find') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/143413:56
*** lachlan has quit IRC14:25
tpollardhad never seen this feature of gitlab before... https://gitlab.com/BuildStream/buildstream/merge_requests/1429#note_18609478914:44
tpollardnifty14:46
benschuberttpollard: it's brand new :D14:47
gitlab-br-botraoul.hidalgocharman opened MR !1435 (raoul/1038-source-cache-proto->master: Proto based source cache service) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/143514:48
*** lachlan has joined #buildstream14:49
tpollardbenschubert: :)14:50
*** lachlan has quit IRC15:15
benschubertI've got a question about provenance of nodes. Currently, when we override an entry, if the new value is not already a node from somewhere else, we set the node's provenance to be the one we just overwrote. I think this is wrong, as we mask real nodes with synthetic ones. I would therefore be in favor of keeping the synthetic provenance. Would someone see a disadvantage to that? tristan, jennis, juergbi ?15:21
juergbithat's not needed for writeback?15:22
juergbiI'm not the YAML expert, others can probably better answer this15:23
jennisI agree with your concern and generally think that we shouldn't overwrite15:24
benschubertjuergbi: that's a very good point, I need to check what would happen15:24
jennisHowever, I'm worried that we'd get an exception which just gives complete garbage provenance15:25
benschubertif that happened, it would be a programming error on our end anyways :/15:28
jennisTrue15:29
jennisI don't see any obvious disadvantages15:30
*** lachlan has joined #buildstream15:54
raouljuergbi, tristan, what was the decision for !1402? If I've understood the scrollback correctly: use --all to decide whether to fetch run time or all dependencies (and expose this as a user config option) and then another option as to whether to fetch artifact files at all?15:54
gitlab-br-botMR !1402: Configuration option for disabling blob fetching with RE https://gitlab.com/BuildStream/buildstream/merge_requests/140215:54
juergbiraoul: yes, does this make sense to you as well?15:55
raoulor rather --all would decide what to build, and thus would fetch those dependencies15:55
raoulyeah I think so15:55
juergbito be clear, the behavior of --all in master is not quite sufficient. also should 'require artifact files' for all elements in that case15:57
juergbi(instead of only for targets and their runtime deps)15:57
juergbiand imo, the user config option for --all could be done in a separate MR. but fine to include in the same MR, of course, if it's simple enough15:57
raoulso --all should override the other option to fetch artifact files?15:59
raoulMight make sense to do it in the same MR, should be small enough a change15:59
*** toscalix has quit IRC16:24
*** raoul has quit IRC16:33
*** alexandrufazakas has quit IRC16:46
*** lachlan has quit IRC16:55
*** phildawson has quit IRC18:10
*** jonathanmaw has quit IRC18:12
*** benschubert has quit IRC20:02
*** cs-shadow has quit IRC20:40
*** slaf has quit IRC21:50
*** slaf has joined #buildstream22:07
*** slaf has joined #buildstream22:07
*** slaf has joined #buildstream22:07
*** slaf has joined #buildstream22:07
*** slaf has joined #buildstream22:08
*** slaf has joined #buildstream22:08
*** slaf has joined #buildstream22:08
*** slaf has joined #buildstream22:08
*** slaf has joined #buildstream22:09
*** slaf has joined #buildstream22:09
*** slaf has joined #buildstream22:09
*** slaf has joined #buildstream22:09
*** slaf has joined #buildstream22:10
*** slaf has joined #buildstream22:10
*** slaf has joined #buildstream22:10
*** slaf has joined #buildstream22:10
*** slaf has joined #buildstream22:11

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!