IRC logs for #buildstream for Monday, 2020-05-25

*** aminbegood has quit IRC01:23
*** aminbegood has joined #buildstream01:25
*** aminbegood has quit IRC01:44
*** aminbegood has joined #buildstream01:54
*** aminbegood has quit IRC02:07
*** aminbegood has joined #buildstream02:43
*** aminbegood has quit IRC02:55
*** tristan has quit IRC03:07
*** aminbegood has joined #buildstream03:11
*** aminbegood has quit IRC03:25
*** aminbegood has joined #buildstream04:24
*** aminbegood has quit IRC04:35
*** aminbegood has joined #buildstream04:52
*** aminbegood has quit IRC05:09
*** aminbegood has joined #buildstream05:12
*** aminbegood has quit IRC05:31
*** aminbegood has joined #buildstream05:33
*** aminbegood has quit IRC05:48
*** tristan has joined #buildstream06:10
*** aminbegood has joined #buildstream07:17
*** aminbegood has quit IRC07:33
*** ChanServ sets mode: +o tristan07:58
tristanLooking more at junction configuration, I find the documentation and naming around `cache-junction-elements` and `ignore-juction-remotes` confusing07:58
tristanMoreover, I think there is some confusion or lack of clarity around how those options interact with the `target` option08:01
tristanWe say only "Note that this option cannot be used in conjunction with sources.", and in preflight errors, we only bail out with an error if sources, options or path is specified, but not when the artifact pull/push options are specified (but will be ignored in the current implementation)08:03
tristanAre we allowed to specify `None` as a default value for `MappingNode.get_bool()` ?08:29
tristanAh I think this doesnt matter, we can still use `in` to check if a parameter was specified explicitly or not.08:31
tristanjuergbi, I've noticed that some people are oh holiday today (including gitlab-br-bot apparently)...09:21
tristanif you are around and have some spare cycles, you could look at https://gitlab.com/BuildStream/buildstream/-/merge_requests/193709:21
* tristan doing some house cleaning while two significant junction branches are still hanging in the balance09:22
* juergbi is around09:22
juergbiwill take a look09:23
juergbishould probably give others a chance to comment as well before a merge, though09:23
tristanSure, should I raise this on the list you think ?09:24
tristanIf so, I will combine with a proposal to also rename `cache-junction-elements` and `ignore-juction-remotes`09:24
tristanthat could make sense09:24
tristanLooking at those, it looks like both options are "whether to push" and "whether to pull" from remotes defined by that junction, when dealing with elements from that junction09:25
tristanIt looks like an opportunity to much simplify the expression, maybe have a dictionary with a "pull" and "push" boolean, and better structured docs around that - but I will look deeper into this to verify that I've understood these new options correctly first09:26
tristanI'd like to understand what the impacts of these options are in the context of deeply nested pipelines and hopefully clarify this story09:27
* tristan starts by pinging cs-shadow on the MR as this was his feature after all09:29
juergbitristan: it's not about push vs. pull. it's about whether this project's remotes should cache elements from the subproject and whether remotes configured in subproject's project.conf should be used or ignored09:32
juergbiI'd expect it to be push/pull in both cases (although remote defined in project.conf is most likely pull-only as push typically requires authentication)09:33
juergbiclarifying the name/purpose sounds like a good idea to me, though09:34
tristanAhhh09:39
tristanjuergbi, I feel like, we should be able to observe these configurations for a linked junction (or a "targetted" junction)09:42
tristanSeems like09:42
tristanbut I think we're not, looking at loader code, we just parse the junction and discard it right away if it's a link to another one09:42
juergbihm, shouldn't that be handled by the target junction?09:46
juergbithe question is how to handle it for nested junctions, I suppose09:46
juergbinaively I would expect these flags to apply recursively09:47
juergbidon't know whether that might not be enough configurability09:47
valentindtristan, Because junctions elements have a partial configuration, I need to make variables defined using missing variables allowed. We can either make this case only for junction. Or we could transform the resolution of variables on demand for all elements (which remove the preprocess and replace it by a cache).09:49
tristanjuergbi, I'm thinking honestly; maybe this should be removed from junctions and setup in the project globally, perhaps in a new "session" group defining behaviors which only apply to the toplevel session09:52
tristanjuergbi, because I think the junction level configuration doesnt achieve the (probable) intent and pain point driving this feature09:52
juergbitristan: hm, so you wouldn't be able to specify it per subproject anymore?09:52
tristanI'll try to find a trace of this on the ML09:53
tristanjuergbi, Basically I suspect that the driving principal here is "I'm building something with my project, and I want to be sure that I am in control of all my artifacts"09:53
tristanI want to ensure that I have my own copy in my own artifact server, etc09:53
tristanWith junction level configuration, we don't achieve this09:54
tristanI wonder what were the motivations/arguments for having fine grained control09:54
juergbisome subprojects might be trusted09:54
juergbiI don't remember the details OTOH09:54
* tristan finds https://mail.gnome.org/archives/buildstream-list/2019-June/msg00049.html09:57
tristanvia https://gitlab.com/BuildStream/buildstream/-/merge_requests/140309:58
tristanlooks like there was a lot of confusion around this and this was a revert of something else09:58
tristanvalentind, I think your comment is about solving include behavior across junction boundaries10:00
valentindNo.10:01
valentindA project includes some variables through a junction in project.conf.10:01
tristanOk10:02
valentindproject.conf defines variables depending on those variables.10:02
tristanYup10:02
valentindThen just loading a junction element, we do not have all the definitions. The variables on top level cannot be fully resolved.10:02
tristanYes, there is a possibility of circular dependencies10:03
tristanof variables10:03
valentindIt is not even circular.10:03
valentindWell kind of.10:03
valentindYou could need the junction to load the variables.10:03
tristanIf I need variables from the junction to define the variables I use to configure that same junction, it's a circular dependency10:04
tristanIt would be good to have an error for that I think ?10:04
valentindBut we already said that junctions never have full configuration even if there is no circular dependencies.10:04
tristanUsually the way we use junctions, these cases don't occur10:05
valentindYes, but we could force evaluating all variables later for non junction element.s10:05
tristanvalentind, I'm having trouble parsing "junctions never have full configuration"10:05
valentindjunction use the project first pass configuration.10:05
valentindjunctions10:06
valentindSo it is the project.conf with only local includes.10:06
tristanRight okay10:06
valentindSo it is partial.10:06
tristanBut the project across that junction is fully loaded10:06
tristanwhen it comes to including from that junction right ?10:07
valentindEven in a case where it could have loaded another through junction fine. But that would make behavior quite complex.10:07
valentindYes.10:07
valentindLater.10:07
valentindJunction elements have partial configuration. But all projects have full configuration loaded when we load other elements.10:07
tristanI see the circular dependency issue, and now I think I understand your original comment better10:08
valentindSo if you say "variables: foo: "foo_%{bar}"10:08
valentindAnd then you have "(@): junction.bst:variables.yml"10:09
valentindWhere variables.yml defines bar.10:09
tristanWe still have error cases to report regardless of which way it's implemented, those need to report something useful10:09
valentindtristan, Yes but we could delay checking for all variables for other elements after loading has been done.10:09
tristanEvery element resolves their variables in their own time10:10
tristanVariables are not a project wide thing10:10
valentindOK, I should check in master. But in 1.5, the variables are resolved when element is loaded.10:10
valentindSo that means quite early.10:10
tristanFor each element yes, well10:11
valentindSeems the same in master.10:11
tristanvalentind, They are resolved when the Element is instantiated10:11
tristannow when the MetaElement is loaded as I understand10:11
tristans/now/not10:11
valentindSure.10:12
tristanThat is also quite late, depending on how you look at it :D10:12
tristanheh10:12
tristanvalentind, to be honest, I do like the latter of your two suggested approaches better than the former (at first sight)10:13
tristanKeeping a cache and resolving on demand seems like it will perform well and be more maintainable than special casing junction load behavior10:14
valentindI have to try, but yes, I think it might possible to make code  that is very simple and still fast.10:14
valentindI will give a try.10:15
tristanvalentind, I think it would be good to consult benschubert on this if possible, as he's also been playing very recently around variables (completely removing the Element.get_variable() APIs)10:15
valentindNoted.10:15
tristanBut essentially I think that it's a good approach, and I think that with the nice nodes and provenance we have in master, we should be able to produce good errors10:16
tristanThe root of all of this is #40110:32
tristanWith which I pretty much agree with jjardon's points10:32
tristanI can see from https://mail.gnome.org/archives/buildstream-list/2019-June/msg00049.html, james tried to bake in the best of all worlds10:33
tristanBut I think that in the end, it was overlooked that the final solution fails to give the toplevel project full control10:34
tristanI'm still having a hard time tracking down why so much configurability was required, though10:34
tristanjuergbi, maybe the junction configurations work if for instance, we consider that they apply to subprojects recursively ?10:36
tristanI guess "trusted caches" is important10:36
juergbiyes, recursive is what I'd expect intuitively10:36
tristanAh, it's not what I'd expect intuitively :D10:37
tristanLooking at the docs and config, I expect that each junction expresses what it wants from it's subproject10:37
tristanAnd then when you throw it all together... "Something happens"10:37
tristanAlso, it looks like while the "default is False" for these options, I feel like setting it explicitly to False is different from leaving it default (i.e. "no matter what, behave like False")10:39
tristanLets see, recursive seems to fix things except for linked/targetted junctions10:39
juergbitristan: well, better than recursive might be transitive, although they are fairly close here10:43
juergbialthough it can depend on the perspective10:44
juergbiif you disconnect the project's remotes from the subproject's remotes, all subsubproject remotes should also be disconnected, imo10:44
juergbithat's for the ignore-... option10:44
tristanjuergbi, I feel like setting either to True should do so recursively10:45
juergbiit might be the simplest approach10:45
tristanI.e. I want to be sure that I always have a separate copy of the artifact in my server10:46
tristanOf course, if a parent project said ignore-junction-remotes and I (the subproject) says cache-junction-elements for my subprojects, the parent project wins10:47
tristanI mean I say "of course", but it's by no means outlined or described anywhere10:47
tristanspecified being the word I'm looking for :)10:48
tristanjuergbi, Basically... any setting of True for either of those settings is recursive and overrides the whole subtree... but then, what about False ?10:49
tristanAlso, when I say true at the toplevel, I mean cache everything built recursively in *my* project, it's not the same as setting True to each junction recursively10:50
tristanthat would mean "try to cache everything in every project"10:50
juergbiyes10:50
juergbifor False I don't think it should be recursive10:50
juergbiTrue is currently the override10:51
juergbiand that makes sense to be recursive to let the top-level project be in control10:51
tristanRight, so in cases that I build a project which does not override, but builds a project which does say that... that subproject's True override is still in effect ?10:51
juergbifor ignore, I would definitely think so10:52
juergbifor cache, it would be cached as part of that middle project, right?10:52
tristanAnd for cache, it ends up meaning you will just get failed push attempts ?10:52
juergbifailed push?10:52
tristanAssuming the middle project said "cache-junction-elements: True" when declaring the sub-sub-project10:53
tristanThen when the toplevel (which did not set cache-... to True), builds... it probably doesnt have credentials for that middle project cache server10:54
tristanBut it will try to push everything for that subproject and subsubproject to the middle project's cache server10:54
tristanOr maybe that is not a failed push10:54
tristanit will just do it if it was able to push at initialization time10:54
tristanThat doesn't seem to sound perfectly sensible :-S10:55
tristanFor ignore, I would definitely think so also10:55
tristanFor cache, it's unclear10:55
juergbiI wouldn't expect project.conf to actually specify push remotes due to the authorization issue10:56
juergbiin which case nothing would happen10:56
tristanGood point10:56
juergbiimo, cache-junction elements in a junction in the middle project should definitely not suddenly result in elements being pushed to the top-level remote10:57
tristanno10:59
tristanI don't think so either10:59
tristanHowever this makes me curious11:00
tristanWhere to the artifacts go if not to the toplevel project ?11:00
tristanI mean, in the normal scenario, the toplevel project uses other subprojects but controls their own cache, you'd think we'd want to push to the toplevel always by default, especially if mid-level elements needed rebuilding and couldn't be obtained from a mid-project-level cache ?11:01
*** aminbegood has joined #buildstream11:01
tristanjuergbi, I guess I feel like the decision should be "cache-junction-elements: never | when-built | always"11:02
tristanwhere "when-built" is the natural default11:03
juergbithat requires some extra state, which I'd rather avoid11:03
tristanThe current default is to always rebuild elements from a subproject if that subproject's cache didn't have the exact artifact you want11:04
juergbielements from all (sub)projects will always be cached locally, of course11:04
juergbiit's only about artifact servers11:04
tristanWhich, is a bit bad especially given that (A) The subproject might not define a public artifact cache for downloads, and (B) You can override subproject junction configurations to bring them inline11:04
tristanYeah, but artifact servers are important heh11:05
juergbiand we shouldn't forget, we always said the preferred setup is to have artifact server configured in the user config11:05
juergbiwhich also makes sense regarding credentials11:05
tristanYes11:05
juergbiand with user config it should still work well for all subprojects11:05
juergbi(depending on whether the servers in user config are project-specific or not)11:05
tristanOk, are we allowed to wildcard all subprojects in user config and do what we want ?11:05
juergbino, I don't think we have anything like that right now11:06
juergbijust global user and per-project user config11:06
juergbiin practice I'd expect many users to just use a global one11:06
tristanThat's problematic for me, as I do want to push the concept that "You don't need to know about subprojects"11:06
tristanuntill you really do need to of course11:06
tristanjuergbi, Oh, global is good enough11:07
*** mohan43u has quit IRC11:07
tristanI sort of meant that by wildcarding, not glob patterns or anything weirdly fancy like that11:07
juergbiyes, the only issue I see is if people work with two (or more) completely different project hierarchies where they want to use different servers11:08
tristanhttps://docs.buildstream.build/master/using_config.html#artifact-server11:08
tristanindeed, that's practical and allows the working group to say "Ok lets use our cache"11:09
juergbibtw: I'd expect CI to be the main (or even only) artifact pusher in many projects11:09
tristanjuergbi, I think that we can (with backwards compatibility) add a 'recursive: True' option to the per-project user configuration11:10
tristanHowever11:10
tristanI do sort of wonder, if I've specified a cache for a given project name in my user config, don't I mean "All subprojects of these when I build that project as a toplevel" ?11:11
tristanjuergbi, I do agree with that yes11:11
tristanI think CI being the pusher is a bit orthogonal though, the setup/config needs to be done somewhere; I'm just trying to make sure we have the right configuration surface to properly capture subprojects of dependencies which we don't want to constantly know about as a project11:12
juergbiwould it make sense for those server lists to be combined (and I think we already have duplicate detection)11:13
tristanI.e. the "I want to use all my own infra for caching, and I want to be able upgrade my junctions regularly and still capture everything in my infra"11:13
juergbie.g, if I have a push server that I use for fd-sdk, I probably want to use that when building fd-sdk on its own but also when building gnome-build-meta or similar11:14
juergbi(for the fd-sdk part)11:14
juergbicombinatorial explosion *sigh*11:15
juergbitristan: CI: right, but CI could handle this with a global user config and would probably want to ignore all project.conf remote definitions11:15
tristanHonestly, I really mostly agree with jjardon's comment on #401 in this regard - when you are owner of a project: you decide if you want to trust third party upstream artifact servers for pulling from, and you probably don't have credentials to push to those11:16
tristanSo what you do is you first try to pull from your project, and fallback to pulling from the upstream junctioned project, in the worst case: you build11:16
tristanBut you always end up pushing to your *own* artifact cache server11:17
juergbiwho is the owner of a project?11:17
tristanA group/organization11:17
juergbiuser of a project might not match maintainer of the corresponding project.conf11:17
juergbiso maybe the project maintainer trusts some server but the user doesn't11:17
tristanThe ones who collaborate to create that project, and who decide to advertize a public cache server to pull from in project.conf if they want to11:17
juergbia user should be able to completely avoid project.conf remotes11:18
juergbi(without editing the project)11:18
tristanI think that makes sense too yes11:18
juergbiI think there was a proposal somewhere about a CLI / user config option for this11:18
juergbii.e., to specify whether to use project.conf remotes at all or not11:18
tristanI feel like there is a minimal amount of flexibility we need to provide, and we've got a lot, really a lot of confusing configuration API surface to reflect it11:19
juergbiyes, feels too complex to me as well11:19
tristanuser config, project.conf, and now junction11:20
tristanthere must be a way to cut this all in half and still support the use cases that are important :-S11:20
juergbiwould things be a bit simpler if we completely disallowed push remotes in project.conf?11:21
juergbibecause I don't think that makes really sense anyway11:21
*** aminbegood has quit IRC11:21
juergbi(given the required authentication)11:22
juergbitheoretically some company-internal (not publicly accessible) server could allow unauthorized push but even that seems odd11:22
tristanIt does seem odd, but I'm not entirely sure11:24
tristanI mean I'm not entirely sure that it saves us from the complexity11:24
*** aminbegood has joined #buildstream11:26
tristanjuergbi, it would, but I think it's also questionable that projects and junctions get to decide these things11:27
tristanjuergbi, what might save us the complexity is to say "Projects can publish their download urls if they like" and user config decides the rest11:27
juergbii.e., let user config decide for which projects the project.conf remotes are used?11:30
*** tristan has quit IRC11:31
juergbiand maybe also have a global user config option to decide the default?11:31
*** tristan has joined #buildstream11:35
*** ChanServ sets mode: +o tristan11:35
tristanjuergbi, errrrm, gotta run soon what's my last line before getting cut off ?11:36
tristaneh, even the IRC logs are having trouble today11:37
tristanall the bots on vacation11:37
tristanhttps://irclogs.baserock.org/%23baserock.2020-05-25.log.html11:37
tristanI'll just repaste for now11:37
tristan<tristan> juergbi, it would, but I think it's also questionable that projects and junctions get to decide these things11:37
tristan<tristan> juergbi, what might save us the complexity is to say "Projects can publish their download urls if they like" and user config decides the rest11:37
tristan<tristan> juergbi, I have to run soon, have an appointment, I'll chew on this more11:37
tristan<tristan> Maybe for the junction decisions, it would make more sense to (A) Have the per-project user config things be recursive, and (B) Allow user config to also make these decisions11:37
tristan<tristan> This would at least centralize all this configuration to one place, plus hints from project.conf (pull urls)11:37
tristan<tristan> And it would eliminate the whole complexity of analyzing and negotiating project preferences11:37
tristan<tristan> Having one place where a decision is made, and a good default behavior11:37
tristan<tristan> To bring something randomly into context... the reason we originally added push url lists to project configuration was the output of a workgroup session at GUADEC in Manchester11:37
tristan<tristan> It was in consideration of "long standing branches of buildstream projects"11:37
tristan<tristan> We had the idea that, when creating the branch, we might have an alternative push url and use 2 pull urls11:37
tristan<tristan> so that development artifacts from the work branch would go to a separate cache11:37
tristan<tristan> and avoid polluting the mainline cache with those artifacts11:37
juergbi<juergbi> i.e., let user config decide for which projects the project.conf remotes are used?11:39
juergbi<juergbi> and maybe also have a global user config option to decide the default?11:39
*** aminbegood has quit IRC11:44
tristanjuergbi, myeah gotta chew on this a bit11:45
tristanhave to think about allowing users to have good control without requiring constant knowledge of all subprojects in play11:46
* tristan is off to dinner11:46
*** tristan has quit IRC11:49
*** aminbegood has joined #buildstream12:10
*** aminbegood has quit IRC12:28
douglaswinshipI have a merge request for pi-3b-plus-bsp https://gitlab.com/celduin/bsps/pi-3b-plus-bsp/-/merge_requests/415:12
douglaswinship... to resolve issue https://gitlab.com/celduin/bsps/pi-3b-plus-bsp/-/issues/115:13
douglaswinshipBut I'm not sure how we would test that?15:13
douglaswinshipI'm guessing we use the example app, for testing that sort of thing?15:15
douglaswinshipI'm not familiar with the CI on that project though. The lines at the end of .gilab-ci.yml seem to indicate that it pushes the built artifact(s) to a cache server, at celduin-cache.codethink.co.uk15:17
douglaswinship(https://gitlab.com/celduin/bsps/example-app/-/blob/master/.gitlab-ci.yml#L97)15:17
douglaswinshipIs that something I ought to have access to, and is that where I'd look to see if my test gives the results I was hoping for?15:18
*** jude has joined #buildstream15:18
douglaswinshipAlso... I'm in the wrong chat window15:19
douglaswinshipSorry!15:19
douglaswinshipIgnore everything I just said.15:19
*** jude has quit IRC15:23
*** benschubert has joined #buildstream17:05
*** tristan has joined #buildstream17:07
*** cphang has joined #buildstream17:50
cphangHi! Is there any appetite for having development versions of BuildStream packaged as flatpaks?18:17
juergbicphang: does BuildStream even work as a flatpak right now (requiring FUSE and [nested] bubblewrap)?19:09
juergbi(for local execution)19:09
cphangI think valentind  has done some work with 1.4, would 1.9x have additional requirements/19:10
juergbishouldn't require anything fundamentally different. more recent version of libfuse and buildbox binaries but those shouldn't be an issue20:21
*** tristan has quit IRC20:38
*** tristan has joined #buildstream20:38
jjardoncphang: yes! https://gitlab.com/BuildStream/buildstream/-/issues/54821:51
*** cphang has quit IRC22:22
*** benschubert has quit IRC22:54
*** lantw44 has quit IRC23:23
*** Trevinho has quit IRC23:23
*** lantw44 has joined #buildstream23:23
*** Trevinho has joined #buildstream23:23

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!