IRC logs for #buildstream for Tuesday, 2020-05-05

*** narispo has quit IRC01:41
*** mohan43u has joined #buildstream02:04
*** aminbegood has joined #buildstream02:59
*** aminbegood has quit IRC03:13
*** aminbegood has joined #buildstream03:14
*** aminbegood has quit IRC03:28
*** tristan has quit IRC05:30
*** tristan has joined #buildstream05:40
*** ChanServ sets mode: +o tristan05:40
juergbiWSalmon: that it can't connect to that artifact server. the "all addresses" part is simply how the gRPC error message is formulated because gRPC allows specifying multiple addresses to try, iirc05:41
tristanjuergbi, related to the junction jungle, how about this format: https://paste.centos.org/view/db75734e06:39
tristanthat is the direction of the config you wanted right ?06:39
* tristan also wants to rename 'target' -> 'inherit' or 'inherits' or 'inherit-config'06:42
tristan'target' sounds like the opposite of what it's doing to me06:42
juergbitristan: yes, this is the right direction. however, also need to keep in mind how the syntax would be for the junctions we don't want to override despite potential conflict06:45
juergbitristan: 'target' terminology matches symlink terminology but maybe it is not that clear because we don't actually mention this junction being a link anywhere06:46
tristanI get the order of `ln` arguments backwards... every, single, time06:48
juergbiin some way it would actually make sense to use something like `kind: link` instead06:48
juergbiyes, `ln` order is infamous. I remember it by thinking about it as the same order as `cp`06:49
juergbihowever, `target` of a symlink seems clear to me06:49
tristanWell, renaming target is less important, although I do feel like target is something that you shoot at, squashing/smashing whatever was there with the thing that you shot at it06:50
juergbiit's like thinking of a link as an arrow, pointing towards its target06:50
juergbiyou follow the arrow when resolving names06:51
tristanFrom the perspective of the loader code I can see how one might think of it as a link06:51
tristanFrom the perspective of the user, I feel like "I want this junction's configuration to be inherited from that junction"06:51
tristanI.e. I feel that I still have a junction06:52
tristaneven though I've overridden it from the other project06:52
juergbiI like reusing well-known filesystem concepts such as symlinks instead of inventing our own terminology06:52
juergbithat should require less buildstream-specific knowledge, imo06:52
tristanI think by that logic, `link: local-junction.bst:junctioned-junction.bst` makes more sense06:54
tristani.e. there is nothing expressing that there is a link, target by itself does not imply this06:54
juergbithat sounds reasonable to me06:54
tristanI won't pull the trigger on that right now anyway06:55
tristanit's easy enough to rename, I do want to rename it but lets leave it for a bit and gather input06:55
tristanfor the other thing... you say "junctions we don't want to override despite potential conflict"06:56
tristanFor this, I think we'll want a separate whitelist06:56
tristanBut I'm trying to keep these concepts separate for now, hard to wrap my head around both at the same time06:57
tristanI'd like to get the overrides in place with test cases, and then add test cases for the whitelisting of allowed duplicate projects and support separately06:57
tristan(possibly on the same branch, though)06:58
tristanjuergbi, something like this: https://paste.centos.org/view/b1528097 ?07:04
tristanjuergbi, "duplicates" might not be a good word, but I think a separate list of junction elements (either local or in subprojects) would serve that purpose07:04
juergbitristan: a bit confused by 'the bar.bst project'07:05
juergbiwhy is there a .bst in the project name?07:05
tristanInteresting07:06
juergbithe way I thought about it is by saying that certain junctions in subprojects are 'private'07:06
tristanI didn't mean it this way, I rather meant "the project referred to by the local bar.bst junction"07:06
tristanDo you think this whitelist should use project names ?07:06
juergbidepending on whether it references a junction or a project name07:07
tristanI think we need to reference junction names not project names07:07
tristanotherwise we leave room for new instances of junctions to pop up referring to the same project, and not explicitly whitelisting the duplication07:08
juergbiit might be useful to come up with a few realistic use cases and corresponding project/junction structures07:08
juergbithinking about it in foo/bar doesn't realy make it clear what is needed, imo07:08
tristanI think the point is that, if a subproject suddenly starts to point to another project that is also a subproject, the parent project must be forced to make a decision07:09
tristanwhether they are going to override something or duplicate elements07:10
tristanor "have separately loaded elements from the same project with different configuration"07:10
juergbiyes, agreed (unless the subproject explicitly marks that new junction as being private as per idea on ML)07:11
tristanThat's what I don't get ;-)07:11
tristanI think I reread the ML discussion today but didn't really grasp that concept07:12
juergbimy example use case was the statically linked application project that uses a junction for the (static) libraries07:12
tristanSo you're saying, instead of forcing the parent project to know that they are duplicating elements from the subproject, the subproject can say "I'm doing this under the hood static junction and it's separate from everything else"07:13
tristanI think this is error prone07:13
tristanBecause the subproject can very, very easily propagate runtime dependencies forward from private junctions07:14
tristanjuergbi, I completely agree that having the subproject be able to abstract that away and hide it's junction is a more desirable API07:14
juergbimaybe we can detect such cases without significant overhead?07:15
tristanPossibly07:15
tristanWe probably do a hand full of dependency walks already, with some caching we could get that in07:15
tristanAt the same time, I'm not sure "private" is the right word for this07:16
tristanI think I like "static" more07:16
juergbiit could make sense to also discuss private non-junction elements in this regard07:16
tristanReason is: We don't currently have any guards as to what is considered private or not, and leave that to the user to invent conventions like using underscores or such to denote privateness07:16
juergbistatic library is just an example, it doesn't have to be about static linking07:16
tristanstatic data07:17
juergbito me that's more confusing07:17
juergbibut anyway, I think that discussion is not a blocker for the first junction branch07:18
juergbiso should probably defer it07:18
tristanI mean; what is the perceived danger here; to me I think the danger is that I might runtime depend on the same thing twice07:18
tristanunknowningly07:18
tristanbut there may be other dangers07:18
juergbibtw: overlap checks might catch these issues as well07:19
juergbi(much later, though)07:19
tristanI was thinking that too, but still it would be a more confusing error to sort out07:19
juergbiyes, we definitely want to catch this as early as possible07:19
juergbitristan: if it's in contrast to 'runtime', maybe it should be called a 'build junction'07:20
juergbilike 'build dependency'07:20
*** jude has joined #buildstream07:22
tristanjuergbi, 'isolated: True' ?07:23
tristanmaybe ?07:23
tristanAnyway I agree we should punt this part of the change and revisit it07:24
tristanI think that there are some interesting corner cases, like for example cross-junction re-use of a 'build junction' or 'isolated junction'07:24
juergbiyes. that's why I think a general discussion about 'private elements' may be warranted07:25
* tristan thinks of those gnu utils which are submoduled to a bunch of low level gnu projects, in the context of BuildStream we might want to reuse an isolated junction in multiple projects07:25
tristanin that sense, it's not 'private' if I can also inherit it's configuration across a junction and reuse it at multiple levels (which still seems interesting)07:26
*** benschubert has joined #buildstream07:26
juergbihm, yes, I wasn't considering that use case07:26
tristanhowever it does remain 'isolated' in the sense that its dependencies cannot transiently leak out to a parent project07:26
*** seanborg has joined #buildstream07:44
tristanheh, junctions dont call node.validate_keys() even07:45
tristanWhy does junction have a 'path' option ??07:47
tristanThis doesn't seem to make any sense07:47
tristanMaybe this comes from way back in the days that we did not properly police local resources referred to by buildstream projects, and freedesktop-sdk was two projects in the same repository, making references to files at "../foo/"07:51
juergbitristan: e.g. a git repository could have multiple projects in subdirectories07:58
juergbi(maybe we should handle such things as part of the source but I think we can't right now)07:59
tristanhttps://gitlab.com/BuildStream/buildstream/-/issues/130008:01
tristanjuergbi, I filed #1300 for the sake of discussing this, I think I captured all the reasons why it was added08:01
tristanI could be wrong though08:01
tristanindeed, there is no reason to add a redundant configuration to the `junction` when this has always been handled by `sources` (I think we should not have extra, redundant configuration surfaces to achieve the same thing)08:02
juergbitristan: `directory` places sources in a subdirectory, it's the inverse08:02
tristanDoes it ?08:03
juergbiyes08:03
tristanWe don't have anything for that in sources ?08:03
juergbinot afair08:03
tristanHmmmm08:03
juergbibut we probably should, if we can find a good way08:03
tristanOk that foils my theory of why it was added08:03
tristanWe do it differently for different sources08:03
tristanI.e. some archive like sources support it (tar/zip)08:04
juergbiright08:04
juergbiI have been thinking recently that I might prefer a tree of sources instead of a list08:06
juergbie.g., clearly specifying which sources `patch` applies to08:07
juergbiand allowing other source transforms such as a 'subdir' filter in a clean and efficient way08:07
juergbinot sure whether it's worth it overall08:08
juergbiI think it would be cleaner, although the syntax could be considered a bit more complex from the user point of view08:08
tristanI think that we can express pretty much all of that with what we have, I'm not sure that a tree would be more readable08:09
juergbiwell, the .bst currently doesn't make clear which sources actually transform other sources08:09
juergbithe source plugins declare this but this is not visible in the .bst08:10
tristanI don't think that's the right story though08:10
tristanI mean, there is no such thing as a "source transform", that is just an artifact of an old mailing list discussion which keeps popping up as a term08:11
tristanFrom a user perspective, sources are a list of things where the order is significant, and sources are allowed to access what was there before them and do things with that as input08:11
juergbiI guess it's a bit like imperative vs. declarative. different perspectives08:13
juergbifor caching (and possibly parallel operations) it's important that we internally know what can affect what08:15
juergbihow we represent this to the user is a separate decision, of course08:15
tristanI think, if there is no performance impact of just having BST_REQUIRES_PREVIOUS_SOURCES_FETCH/TRACK/STAGE set to True across the board, we should just remove all that API and do it unconditionally08:15
tristan(but I expect that there is a performance hit to that)08:15
juergbithere is08:15
juergbiwell, there may be, it depends08:16
tristanRight, ok but there is definitely potential for optimization which we could lose08:16
juergbiyep08:16
tristanI'm not sure about STAGE though08:16
tristanmaybe it makes a difference for remote execution cases ?08:16
juergbiand I generally prefer keeping the mapping from user declaration to internal logic as simple as possible. not only because this simplifies code but I tend to also prefer this as a user08:17
juergbithe difference is relevant since using a CAS-based source cache08:17
juergbiindependent of local or remote execution08:17
juergbiwe want to skip the overhead of staging of previous sources when we don't need to08:18
tristanAs a user, I appreciated knowing as little as possible08:18
tristanWhen I get burdened with any knowledge I don't need about how something is implemented, I get annoyed08:18
juergbiI agree with that, if the abstraction is really helpful08:19
juergbihowever, if it's just a different perspective, it actually makes things more difficult when debugging issues, e.g.08:19
juergbias you then need to understand both, and how they are mapped08:19
tristanI think you can't have it in absolutes though; i.e. there will also be a bit of both08:20
juergbisure, it's not always clear cut08:21
tristanFor sources, I think that most of the time, there is only one or a few08:21
tristanit is very rare to have an element with a very complicated preparation of the source tree08:21
tristana dictionary / tree-ish representation would be more useful if complicated source tree preparation was common08:22
juergbiyes, I'm not sure right now what the best representation would be08:22
*** phildawson has joined #buildstream08:22
tristanjuergbi, if sources had "extension points" (subdirectories) where other sources could be applied to, that might provide more opportunity for parallelism08:23
juergbijust mentioned it as I think for the 'only take subdir' case, a tree-ish representation might be clearer to the user08:23
* tristan thinks that's what you may have been getting at08:23
tristanbut only in the rare case of complex source tree construction08:24
juergbiindeed, in most cases sources are very simple08:25
*** tpollard has joined #buildstream08:26
tristancache-junction-elements and cache-junction-remotes, these seem a bit high tech; I wonder why we have to spell out the word "junction" here when it's a configuration on a junction already08:27
* tristan wants to go on a renaming spree hehe08:28
tristanI should make a list and post to the ML08:28
*** santi has joined #buildstream08:37
scottclarkeI'm getting missing blob errors when testing buildstream with buildbarn for remote execution, the problem would appear to be on the client side and I'm wondering is there any chance that there may be a bug where buildstream can tell the server to execute before the uploads of blobs have finished?09:20
cphangjuergbi scottclarke has updated https://gitlab.com/celduin/infrastructure/celduin-infra/-/issues/120 since we last spoke on Friday09:21
cphangscottclarke you're also seeing it occur where actions would fail due to missing blobs, and then succeed on retrying the action associated with the element?09:22
scottclarkeyes I was just about to point that out and add it to the issue as well09:23
*** lachlan has joined #buildstream09:23
cphangthanks scottclarke09:24
juergbiscottclarke: I don't see how this could happen. neither buildstream nor buildbox-casd currently use async gRPC09:26
juergbiand we check the response from casd for errors before moving on to call Execute09:27
*** tristan has quit IRC09:32
*** lachlan has quit IRC09:34
juergbiscottclarke: I've noticed that this hits a code path that doesn't use batch upload. I'll see whether I can fix that. however, this should only be a performance concern, not a functional one09:43
scottclarkethanks for the info juergbi09:44
juergbi(implicit upload as part of FindMissingBlobs)09:45
juergbiwe also do appear to have test cases to cover that case in buildbox-casd09:46
juergbiscottclarke: would it be possible to temporarily completely disable any expiry/purging on the server side to check whether the issue might be too quick expiry?09:47
*** lachlan has joined #buildstream09:48
scottclarkeIt may be possible, I'll ask around and see if I can find a way to try that09:49
*** lachlan has quit IRC09:55
*** cs-shadow has joined #buildstream10:35
*** lachlan has joined #buildstream10:53
*** lachlan has quit IRC10:57
*** lachlan has joined #buildstream11:23
*** lachlan has quit IRC11:51
robjhwhat was that thing called that bst-artifact-cache uses that isnt http but totally is http?12:52
robjhgRPC, thanks history!12:53
juergbirobjh: gRPC12:53
robjhgRPC, thanks juergbi :)12:53
*** lachlan has joined #buildstream13:27
*** lachlan has quit IRC13:30
*** lachlan has joined #buildstream13:44
*** tristan has joined #buildstream13:55
*** lachlan has quit IRC14:03
*** lachlan has joined #buildstream14:21
* WSalmon presumes that everyone will have a opinion for the value of https://gitlab.com/BuildStream/buildstream/-/merge_requests/1897/diffs#1704d735db78736236d4e19c7fc79281ed02a1db_166_166 but can we pick one soon as we need this for freedesktop-sdk at master14:34
WSalmonalso i presume we have decided to keep the tar at master rather than in the plugins?14:36
valentindjuergbi, This one is straight forward, I think we can merge it: https://gitlab.com/BuildStream/buildstream/-/merge_requests/189714:56
valentindOr tristan, if you are online.14:56
*** lachlan has quit IRC14:58
WSalmonIts a bit late for tristan to be up..14:59
juergbiWSalmon: you need to fix the format14:59
juergbitox -e format14:59
WSalmondone15:00
WSalmoni forget how nit picky black is but also fair15:01
WSalmonjuergbi,15:01
juergbivalentind, WSalmon: the difference between bst 1.4 and master in this regard is still a mystery, right?15:02
WSalmoni did not investigate 1.x but i can back port for consistency if wanted.15:03
WSalmonThe internet sugested that this was a common problem with using urllib tho15:04
WSalmonapparently its default agent is commonely black listed15:04
juergbiyes, the change seems reasonable to me in any case15:05
juergbijust a bit odd that apparently it is or was working on bst 1.4 where we don't set the user agent15:05
WSalmondont know why bst1.x dosent suffer the same tho15:05
juergbimaybe that's caching-related after all15:05
valentindjuergbi, It is also an issue on 1.4 I think.15:07
coldtomit is an issue on 1.415:07
valentindOnly gitlab has changed of behavior.15:07
WSalmonoh ok15:07
valentindIt worked before.15:07
WSalmonthen we should back port it?15:07
valentindBut gitlab broke it.15:07
valentindWSalmon, Yes please.15:07
juergbiok, thanks for the info15:08
*** lachlan has joined #buildstream15:12
*** lachlan has quit IRC15:16
*** lachlan has joined #buildstream15:31
*** seanborg has quit IRC15:31
*** lachlan has quit IRC15:34
*** seanborg has joined #buildstream15:36
*** phoenix has joined #buildstream15:39
WSalmonI think this is my first back port, im not sure i have got the eticate right https://gitlab.com/BuildStream/buildstream/-/merge_requests/189816:11
*** seanborg has quit IRC16:15
*** lachlan has joined #buildstream16:21
juergbiWSalmon: please note that marge should still be working from her home office ;)16:23
WSalmonjuergbi, sorry i had some bad luck with her and notice she wasnt being used all the time atm, i will use her next time16:25
juergbino worries, it's only a problem if there are multiple MRs that get merged concurrently16:33
*** lachlan has quit IRC16:42
valentindI have noticed (never noticed before) that checkout --tar uses the user of bst. Does bst2 do the same? Or it correctly keep the users from the sanbox?16:48
juergbivalentind: uid/gid are currently not stored in artifacts. bst master sets uid/gid to 016:52
juergbiI fixed this a few weeks back16:53
valentindjuergbi, I am making a work-around using getfacl/getfattr setfacl/setfattr to keep attributes across artifacts. But I was disappointed to see that I cannot checkout.16:55
juergbivalentind: where do you store that info?16:57
valentindIn the artifact.16:57
juergbiout of band file?16:57
valentindBut I restore it as integration  command.16:57
juergbiok, I see16:57
valentindI do not modify bst.16:57
juergbibut uid/gid can't be changed in the sandbox16:58
juergbias we don't use subuid/subgid right now16:58
juergbiso I'm not sure how this is related to the tar export16:58
juergbiif it's acceptable for a project to support subuid/subgid support on the host (at least for the final target element), it might not be that difficult to properly fix #38 now but it will require changes in buildbox and buildstream17:02
juergbiand it should better wait until I've updated the code for the latest node property changes, which I can hopefully do within a couple of weeks or so17:02
valentindis not the uid of files 0 in the sandbox?17:03
juergbiyes, by default it's 0 and you can also configure it in the project/element in bst master17:03
valentindSo setuid works with what I have done. But if it is not root in the tar, then it is a bit useless.17:03
juergbihowever, we don't support multiple uids right now in a single sandbox17:03
valentindI want to see in the tar exactly what I see in the sandbox.17:03
*** narispo has joined #buildstream17:04
valentindThere is so much more than uid. I still want the rest to work. But if the uid is wrong, then It is not good. I do not mind that I cannot have other uid than 0. I just want to have it in the tar.17:04
juergbithat seems reasonable. so we should use the configured uid/gid of the particular project/element when generating a tar file17:04
*** narispo has quit IRC17:04
*** narispo has joined #buildstream17:04
valentindI think the output tar should have the same as it is in the sandbox.17:05
juergbiiirc, in bst 1.x uid is always 017:05
juergbiso it would probably be an easy fix for bst 1.x17:05
juergbicould try a backport of my change in master but I suspect the code might be quite different, so may not be worth it17:05
valentindjuergbi, do not worry.17:07
valentindIt is not an emergency.17:07
valentindBut yes, if you plan to fix things around it, that is nice.17:07
valentindSpecially it would be nice that bst 2 properly capture the sandbox permissions and users in the tar.17:07
valentindI can work around that in bst 1.17:08
juergbivalentind: can you please open an issue so that we don't forget about it?17:08
valentindSure.17:08
*** tpollard has quit IRC17:09
juergbifixing uid/gid to match configured sandbox uid/gid is probably simple enough17:09
juergbithe rest will require #38 to be fixed before we can worry about the tar part17:09
*** santi has quit IRC17:36
*** pointswaves has joined #buildstream17:41
*** jude has quit IRC17:54
*** lachlan has joined #buildstream18:02
*** pointswaves has quit IRC18:09
*** lachlan has quit IRC18:14
*** aminbegood has joined #buildstream18:16
*** aminbegood has quit IRC18:21
*** phoenix has quit IRC19:49
*** benschubert has quit IRC19:54
*** cs-shadow has quit IRC20:44
*** aminbegood has joined #buildstream21:02
*** aminbegood has quit IRC21:47
*** mohan43u has quit IRC23:09
*** lachlan has joined #buildstream23:10
*** lachlan has quit IRC23:10
*** mohan43u has joined #buildstream23:11
*** lachlan has joined #buildstream23:14
*** lachlan has quit IRC23:17
*** aminbegood has joined #buildstream23:32
*** aminbegood has quit IRC23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!