*** narispo has quit IRC | 01:41 | |
*** mohan43u has joined #buildstream | 02:04 | |
*** aminbegood has joined #buildstream | 02:59 | |
*** aminbegood has quit IRC | 03:13 | |
*** aminbegood has joined #buildstream | 03:14 | |
*** aminbegood has quit IRC | 03:28 | |
*** tristan has quit IRC | 05:30 | |
*** tristan has joined #buildstream | 05:40 | |
*** ChanServ sets mode: +o tristan | 05:40 | |
juergbi | WSalmon: that it can't connect to that artifact server. the "all addresses" part is simply how the gRPC error message is formulated because gRPC allows specifying multiple addresses to try, iirc | 05:41 |
---|---|---|
tristan | juergbi, related to the junction jungle, how about this format: https://paste.centos.org/view/db75734e | 06:39 |
tristan | that is the direction of the config you wanted right ? | 06:39 |
* tristan also wants to rename 'target' -> 'inherit' or 'inherits' or 'inherit-config' | 06:42 | |
tristan | 'target' sounds like the opposite of what it's doing to me | 06:42 |
juergbi | tristan: yes, this is the right direction. however, also need to keep in mind how the syntax would be for the junctions we don't want to override despite potential conflict | 06:45 |
juergbi | tristan: 'target' terminology matches symlink terminology but maybe it is not that clear because we don't actually mention this junction being a link anywhere | 06:46 |
tristan | I get the order of `ln` arguments backwards... every, single, time | 06:48 |
juergbi | in some way it would actually make sense to use something like `kind: link` instead | 06:48 |
juergbi | yes, `ln` order is infamous. I remember it by thinking about it as the same order as `cp` | 06:49 |
juergbi | however, `target` of a symlink seems clear to me | 06:49 |
tristan | Well, renaming target is less important, although I do feel like target is something that you shoot at, squashing/smashing whatever was there with the thing that you shot at it | 06:50 |
juergbi | it's like thinking of a link as an arrow, pointing towards its target | 06:50 |
juergbi | you follow the arrow when resolving names | 06:51 |
tristan | From the perspective of the loader code I can see how one might think of it as a link | 06:51 |
tristan | From the perspective of the user, I feel like "I want this junction's configuration to be inherited from that junction" | 06:51 |
tristan | I.e. I feel that I still have a junction | 06:52 |
tristan | even though I've overridden it from the other project | 06:52 |
juergbi | I like reusing well-known filesystem concepts such as symlinks instead of inventing our own terminology | 06:52 |
juergbi | that should require less buildstream-specific knowledge, imo | 06:52 |
tristan | I think by that logic, `link: local-junction.bst:junctioned-junction.bst` makes more sense | 06:54 |
tristan | i.e. there is nothing expressing that there is a link, target by itself does not imply this | 06:54 |
juergbi | that sounds reasonable to me | 06:54 |
tristan | I won't pull the trigger on that right now anyway | 06:55 |
tristan | it's easy enough to rename, I do want to rename it but lets leave it for a bit and gather input | 06:55 |
tristan | for the other thing... you say "junctions we don't want to override despite potential conflict" | 06:56 |
tristan | For this, I think we'll want a separate whitelist | 06:56 |
tristan | But I'm trying to keep these concepts separate for now, hard to wrap my head around both at the same time | 06:57 |
tristan | I'd like to get the overrides in place with test cases, and then add test cases for the whitelisting of allowed duplicate projects and support separately | 06:57 |
tristan | (possibly on the same branch, though) | 06:58 |
tristan | juergbi, something like this: https://paste.centos.org/view/b1528097 ? | 07:04 |
tristan | juergbi, "duplicates" might not be a good word, but I think a separate list of junction elements (either local or in subprojects) would serve that purpose | 07:04 |
juergbi | tristan: a bit confused by 'the bar.bst project' | 07:05 |
juergbi | why is there a .bst in the project name? | 07:05 |
tristan | Interesting | 07:06 |
juergbi | the way I thought about it is by saying that certain junctions in subprojects are 'private' | 07:06 |
tristan | I didn't mean it this way, I rather meant "the project referred to by the local bar.bst junction" | 07:06 |
tristan | Do you think this whitelist should use project names ? | 07:06 |
juergbi | depending on whether it references a junction or a project name | 07:07 |
tristan | I think we need to reference junction names not project names | 07:07 |
tristan | otherwise we leave room for new instances of junctions to pop up referring to the same project, and not explicitly whitelisting the duplication | 07:08 |
juergbi | it might be useful to come up with a few realistic use cases and corresponding project/junction structures | 07:08 |
juergbi | thinking about it in foo/bar doesn't realy make it clear what is needed, imo | 07:08 |
tristan | I think the point is that, if a subproject suddenly starts to point to another project that is also a subproject, the parent project must be forced to make a decision | 07:09 |
tristan | whether they are going to override something or duplicate elements | 07:10 |
tristan | or "have separately loaded elements from the same project with different configuration" | 07:10 |
juergbi | yes, agreed (unless the subproject explicitly marks that new junction as being private as per idea on ML) | 07:11 |
tristan | That's what I don't get ;-) | 07:11 |
tristan | I think I reread the ML discussion today but didn't really grasp that concept | 07:12 |
juergbi | my example use case was the statically linked application project that uses a junction for the (static) libraries | 07:12 |
tristan | So you're saying, instead of forcing the parent project to know that they are duplicating elements from the subproject, the subproject can say "I'm doing this under the hood static junction and it's separate from everything else" | 07:13 |
tristan | I think this is error prone | 07:13 |
tristan | Because the subproject can very, very easily propagate runtime dependencies forward from private junctions | 07:14 |
tristan | juergbi, I completely agree that having the subproject be able to abstract that away and hide it's junction is a more desirable API | 07:14 |
juergbi | maybe we can detect such cases without significant overhead? | 07:15 |
tristan | Possibly | 07:15 |
tristan | We probably do a hand full of dependency walks already, with some caching we could get that in | 07:15 |
tristan | At the same time, I'm not sure "private" is the right word for this | 07:16 |
tristan | I think I like "static" more | 07:16 |
juergbi | it could make sense to also discuss private non-junction elements in this regard | 07:16 |
tristan | Reason is: We don't currently have any guards as to what is considered private or not, and leave that to the user to invent conventions like using underscores or such to denote privateness | 07:16 |
juergbi | static library is just an example, it doesn't have to be about static linking | 07:16 |
tristan | static data | 07:17 |
juergbi | to me that's more confusing | 07:17 |
juergbi | but anyway, I think that discussion is not a blocker for the first junction branch | 07:18 |
juergbi | so should probably defer it | 07:18 |
tristan | I mean; what is the perceived danger here; to me I think the danger is that I might runtime depend on the same thing twice | 07:18 |
tristan | unknowningly | 07:18 |
tristan | but there may be other dangers | 07:18 |
juergbi | btw: overlap checks might catch these issues as well | 07:19 |
juergbi | (much later, though) | 07:19 |
tristan | I was thinking that too, but still it would be a more confusing error to sort out | 07:19 |
juergbi | yes, we definitely want to catch this as early as possible | 07:19 |
juergbi | tristan: if it's in contrast to 'runtime', maybe it should be called a 'build junction' | 07:20 |
juergbi | like 'build dependency' | 07:20 |
*** jude has joined #buildstream | 07:22 | |
tristan | juergbi, 'isolated: True' ? | 07:23 |
tristan | maybe ? | 07:23 |
tristan | Anyway I agree we should punt this part of the change and revisit it | 07:24 |
tristan | I think that there are some interesting corner cases, like for example cross-junction re-use of a 'build junction' or 'isolated junction' | 07:24 |
juergbi | yes. that's why I think a general discussion about 'private elements' may be warranted | 07:25 |
* tristan thinks of those gnu utils which are submoduled to a bunch of low level gnu projects, in the context of BuildStream we might want to reuse an isolated junction in multiple projects | 07:25 | |
tristan | in that sense, it's not 'private' if I can also inherit it's configuration across a junction and reuse it at multiple levels (which still seems interesting) | 07:26 |
*** benschubert has joined #buildstream | 07:26 | |
juergbi | hm, yes, I wasn't considering that use case | 07:26 |
tristan | however it does remain 'isolated' in the sense that its dependencies cannot transiently leak out to a parent project | 07:26 |
*** seanborg has joined #buildstream | 07:44 | |
tristan | heh, junctions dont call node.validate_keys() even | 07:45 |
tristan | Why does junction have a 'path' option ?? | 07:47 |
tristan | This doesn't seem to make any sense | 07:47 |
tristan | Maybe this comes from way back in the days that we did not properly police local resources referred to by buildstream projects, and freedesktop-sdk was two projects in the same repository, making references to files at "../foo/" | 07:51 |
juergbi | tristan: e.g. a git repository could have multiple projects in subdirectories | 07:58 |
juergbi | (maybe we should handle such things as part of the source but I think we can't right now) | 07:59 |
tristan | https://gitlab.com/BuildStream/buildstream/-/issues/1300 | 08:01 |
tristan | juergbi, I filed #1300 for the sake of discussing this, I think I captured all the reasons why it was added | 08:01 |
tristan | I could be wrong though | 08:01 |
tristan | indeed, there is no reason to add a redundant configuration to the `junction` when this has always been handled by `sources` (I think we should not have extra, redundant configuration surfaces to achieve the same thing) | 08:02 |
juergbi | tristan: `directory` places sources in a subdirectory, it's the inverse | 08:02 |
tristan | Does it ? | 08:03 |
juergbi | yes | 08:03 |
tristan | We don't have anything for that in sources ? | 08:03 |
juergbi | not afair | 08:03 |
tristan | Hmmmm | 08:03 |
juergbi | but we probably should, if we can find a good way | 08:03 |
tristan | Ok that foils my theory of why it was added | 08:03 |
tristan | We do it differently for different sources | 08:03 |
tristan | I.e. some archive like sources support it (tar/zip) | 08:04 |
juergbi | right | 08:04 |
juergbi | I have been thinking recently that I might prefer a tree of sources instead of a list | 08:06 |
juergbi | e.g., clearly specifying which sources `patch` applies to | 08:07 |
juergbi | and allowing other source transforms such as a 'subdir' filter in a clean and efficient way | 08:07 |
juergbi | not sure whether it's worth it overall | 08:08 |
juergbi | I think it would be cleaner, although the syntax could be considered a bit more complex from the user point of view | 08:08 |
tristan | I think that we can express pretty much all of that with what we have, I'm not sure that a tree would be more readable | 08:09 |
juergbi | well, the .bst currently doesn't make clear which sources actually transform other sources | 08:09 |
juergbi | the source plugins declare this but this is not visible in the .bst | 08:10 |
tristan | I don't think that's the right story though | 08:10 |
tristan | I mean, there is no such thing as a "source transform", that is just an artifact of an old mailing list discussion which keeps popping up as a term | 08:11 |
tristan | From a user perspective, sources are a list of things where the order is significant, and sources are allowed to access what was there before them and do things with that as input | 08:11 |
juergbi | I guess it's a bit like imperative vs. declarative. different perspectives | 08:13 |
juergbi | for caching (and possibly parallel operations) it's important that we internally know what can affect what | 08:15 |
juergbi | how we represent this to the user is a separate decision, of course | 08:15 |
tristan | I think, if there is no performance impact of just having BST_REQUIRES_PREVIOUS_SOURCES_FETCH/TRACK/STAGE set to True across the board, we should just remove all that API and do it unconditionally | 08:15 |
tristan | (but I expect that there is a performance hit to that) | 08:15 |
juergbi | there is | 08:15 |
juergbi | well, there may be, it depends | 08:16 |
tristan | Right, ok but there is definitely potential for optimization which we could lose | 08:16 |
juergbi | yep | 08:16 |
tristan | I'm not sure about STAGE though | 08:16 |
tristan | maybe it makes a difference for remote execution cases ? | 08:16 |
juergbi | and I generally prefer keeping the mapping from user declaration to internal logic as simple as possible. not only because this simplifies code but I tend to also prefer this as a user | 08:17 |
juergbi | the difference is relevant since using a CAS-based source cache | 08:17 |
juergbi | independent of local or remote execution | 08:17 |
juergbi | we want to skip the overhead of staging of previous sources when we don't need to | 08:18 |
tristan | As a user, I appreciated knowing as little as possible | 08:18 |
tristan | When I get burdened with any knowledge I don't need about how something is implemented, I get annoyed | 08:18 |
juergbi | I agree with that, if the abstraction is really helpful | 08:19 |
juergbi | however, if it's just a different perspective, it actually makes things more difficult when debugging issues, e.g. | 08:19 |
juergbi | as you then need to understand both, and how they are mapped | 08:19 |
tristan | I think you can't have it in absolutes though; i.e. there will also be a bit of both | 08:20 |
juergbi | sure, it's not always clear cut | 08:21 |
tristan | For sources, I think that most of the time, there is only one or a few | 08:21 |
tristan | it is very rare to have an element with a very complicated preparation of the source tree | 08:21 |
tristan | a dictionary / tree-ish representation would be more useful if complicated source tree preparation was common | 08:22 |
juergbi | yes, I'm not sure right now what the best representation would be | 08:22 |
*** phildawson has joined #buildstream | 08:22 | |
tristan | juergbi, if sources had "extension points" (subdirectories) where other sources could be applied to, that might provide more opportunity for parallelism | 08:23 |
juergbi | just mentioned it as I think for the 'only take subdir' case, a tree-ish representation might be clearer to the user | 08:23 |
* tristan thinks that's what you may have been getting at | 08:23 | |
tristan | but only in the rare case of complex source tree construction | 08:24 |
juergbi | indeed, in most cases sources are very simple | 08:25 |
*** tpollard has joined #buildstream | 08:26 | |
tristan | cache-junction-elements and cache-junction-remotes, these seem a bit high tech; I wonder why we have to spell out the word "junction" here when it's a configuration on a junction already | 08:27 |
* tristan wants to go on a renaming spree hehe | 08:28 | |
tristan | I should make a list and post to the ML | 08:28 |
*** santi has joined #buildstream | 08:37 | |
scottclarke | I'm getting missing blob errors when testing buildstream with buildbarn for remote execution, the problem would appear to be on the client side and I'm wondering is there any chance that there may be a bug where buildstream can tell the server to execute before the uploads of blobs have finished? | 09:20 |
cphang | juergbi scottclarke has updated https://gitlab.com/celduin/infrastructure/celduin-infra/-/issues/120 since we last spoke on Friday | 09:21 |
cphang | scottclarke you're also seeing it occur where actions would fail due to missing blobs, and then succeed on retrying the action associated with the element? | 09:22 |
scottclarke | yes I was just about to point that out and add it to the issue as well | 09:23 |
*** lachlan has joined #buildstream | 09:23 | |
cphang | thanks scottclarke | 09:24 |
juergbi | scottclarke: I don't see how this could happen. neither buildstream nor buildbox-casd currently use async gRPC | 09:26 |
juergbi | and we check the response from casd for errors before moving on to call Execute | 09:27 |
*** tristan has quit IRC | 09:32 | |
*** lachlan has quit IRC | 09:34 | |
juergbi | scottclarke: I've noticed that this hits a code path that doesn't use batch upload. I'll see whether I can fix that. however, this should only be a performance concern, not a functional one | 09:43 |
scottclarke | thanks for the info juergbi | 09:44 |
juergbi | (implicit upload as part of FindMissingBlobs) | 09:45 |
juergbi | we also do appear to have test cases to cover that case in buildbox-casd | 09:46 |
juergbi | scottclarke: would it be possible to temporarily completely disable any expiry/purging on the server side to check whether the issue might be too quick expiry? | 09:47 |
*** lachlan has joined #buildstream | 09:48 | |
scottclarke | It may be possible, I'll ask around and see if I can find a way to try that | 09:49 |
*** lachlan has quit IRC | 09:55 | |
*** cs-shadow has joined #buildstream | 10:35 | |
*** lachlan has joined #buildstream | 10:53 | |
*** lachlan has quit IRC | 10:57 | |
*** lachlan has joined #buildstream | 11:23 | |
*** lachlan has quit IRC | 11:51 | |
robjh | what was that thing called that bst-artifact-cache uses that isnt http but totally is http? | 12:52 |
robjh | gRPC, thanks history! | 12:53 |
juergbi | robjh: gRPC | 12:53 |
robjh | gRPC, thanks juergbi :) | 12:53 |
*** lachlan has joined #buildstream | 13:27 | |
*** lachlan has quit IRC | 13:30 | |
*** lachlan has joined #buildstream | 13:44 | |
*** tristan has joined #buildstream | 13:55 | |
*** lachlan has quit IRC | 14:03 | |
*** lachlan has joined #buildstream | 14:21 | |
* WSalmon presumes that everyone will have a opinion for the value of https://gitlab.com/BuildStream/buildstream/-/merge_requests/1897/diffs#1704d735db78736236d4e19c7fc79281ed02a1db_166_166 but can we pick one soon as we need this for freedesktop-sdk at master | 14:34 | |
WSalmon | also i presume we have decided to keep the tar at master rather than in the plugins? | 14:36 |
valentind | juergbi, This one is straight forward, I think we can merge it: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1897 | 14:56 |
valentind | Or tristan, if you are online. | 14:56 |
*** lachlan has quit IRC | 14:58 | |
WSalmon | Its a bit late for tristan to be up.. | 14:59 |
juergbi | WSalmon: you need to fix the format | 14:59 |
juergbi | tox -e format | 14:59 |
WSalmon | done | 15:00 |
WSalmon | i forget how nit picky black is but also fair | 15:01 |
WSalmon | juergbi, | 15:01 |
juergbi | valentind, WSalmon: the difference between bst 1.4 and master in this regard is still a mystery, right? | 15:02 |
WSalmon | i did not investigate 1.x but i can back port for consistency if wanted. | 15:03 |
WSalmon | The internet sugested that this was a common problem with using urllib tho | 15:04 |
WSalmon | apparently its default agent is commonely black listed | 15:04 |
juergbi | yes, the change seems reasonable to me in any case | 15:05 |
juergbi | just a bit odd that apparently it is or was working on bst 1.4 where we don't set the user agent | 15:05 |
WSalmon | dont know why bst1.x dosent suffer the same tho | 15:05 |
juergbi | maybe that's caching-related after all | 15:05 |
valentind | juergbi, It is also an issue on 1.4 I think. | 15:07 |
coldtom | it is an issue on 1.4 | 15:07 |
valentind | Only gitlab has changed of behavior. | 15:07 |
WSalmon | oh ok | 15:07 |
valentind | It worked before. | 15:07 |
WSalmon | then we should back port it? | 15:07 |
valentind | But gitlab broke it. | 15:07 |
valentind | WSalmon, Yes please. | 15:07 |
juergbi | ok, thanks for the info | 15:08 |
*** lachlan has joined #buildstream | 15:12 | |
*** lachlan has quit IRC | 15:16 | |
*** lachlan has joined #buildstream | 15:31 | |
*** seanborg has quit IRC | 15:31 | |
*** lachlan has quit IRC | 15:34 | |
*** seanborg has joined #buildstream | 15:36 | |
*** phoenix has joined #buildstream | 15:39 | |
WSalmon | I think this is my first back port, im not sure i have got the eticate right https://gitlab.com/BuildStream/buildstream/-/merge_requests/1898 | 16:11 |
*** seanborg has quit IRC | 16:15 | |
*** lachlan has joined #buildstream | 16:21 | |
juergbi | WSalmon: please note that marge should still be working from her home office ;) | 16:23 |
WSalmon | juergbi, sorry i had some bad luck with her and notice she wasnt being used all the time atm, i will use her next time | 16:25 |
juergbi | no worries, it's only a problem if there are multiple MRs that get merged concurrently | 16:33 |
*** lachlan has quit IRC | 16:42 | |
valentind | I have noticed (never noticed before) that checkout --tar uses the user of bst. Does bst2 do the same? Or it correctly keep the users from the sanbox? | 16:48 |
juergbi | valentind: uid/gid are currently not stored in artifacts. bst master sets uid/gid to 0 | 16:52 |
juergbi | I fixed this a few weeks back | 16:53 |
valentind | juergbi, I am making a work-around using getfacl/getfattr setfacl/setfattr to keep attributes across artifacts. But I was disappointed to see that I cannot checkout. | 16:55 |
juergbi | valentind: where do you store that info? | 16:57 |
valentind | In the artifact. | 16:57 |
juergbi | out of band file? | 16:57 |
valentind | But I restore it as integration command. | 16:57 |
juergbi | ok, I see | 16:57 |
valentind | I do not modify bst. | 16:57 |
juergbi | but uid/gid can't be changed in the sandbox | 16:58 |
juergbi | as we don't use subuid/subgid right now | 16:58 |
juergbi | so I'm not sure how this is related to the tar export | 16:58 |
juergbi | if it's acceptable for a project to support subuid/subgid support on the host (at least for the final target element), it might not be that difficult to properly fix #38 now but it will require changes in buildbox and buildstream | 17:02 |
juergbi | and it should better wait until I've updated the code for the latest node property changes, which I can hopefully do within a couple of weeks or so | 17:02 |
valentind | is not the uid of files 0 in the sandbox? | 17:03 |
juergbi | yes, by default it's 0 and you can also configure it in the project/element in bst master | 17:03 |
valentind | So setuid works with what I have done. But if it is not root in the tar, then it is a bit useless. | 17:03 |
juergbi | however, we don't support multiple uids right now in a single sandbox | 17:03 |
valentind | I want to see in the tar exactly what I see in the sandbox. | 17:03 |
*** narispo has joined #buildstream | 17:04 | |
valentind | There is so much more than uid. I still want the rest to work. But if the uid is wrong, then It is not good. I do not mind that I cannot have other uid than 0. I just want to have it in the tar. | 17:04 |
juergbi | that seems reasonable. so we should use the configured uid/gid of the particular project/element when generating a tar file | 17:04 |
*** narispo has quit IRC | 17:04 | |
*** narispo has joined #buildstream | 17:04 | |
valentind | I think the output tar should have the same as it is in the sandbox. | 17:05 |
juergbi | iirc, in bst 1.x uid is always 0 | 17:05 |
juergbi | so it would probably be an easy fix for bst 1.x | 17:05 |
juergbi | could try a backport of my change in master but I suspect the code might be quite different, so may not be worth it | 17:05 |
valentind | juergbi, do not worry. | 17:07 |
valentind | It is not an emergency. | 17:07 |
valentind | But yes, if you plan to fix things around it, that is nice. | 17:07 |
valentind | Specially it would be nice that bst 2 properly capture the sandbox permissions and users in the tar. | 17:07 |
valentind | I can work around that in bst 1. | 17:08 |
juergbi | valentind: can you please open an issue so that we don't forget about it? | 17:08 |
valentind | Sure. | 17:08 |
*** tpollard has quit IRC | 17:09 | |
juergbi | fixing uid/gid to match configured sandbox uid/gid is probably simple enough | 17:09 |
juergbi | the rest will require #38 to be fixed before we can worry about the tar part | 17:09 |
*** santi has quit IRC | 17:36 | |
*** pointswaves has joined #buildstream | 17:41 | |
*** jude has quit IRC | 17:54 | |
*** lachlan has joined #buildstream | 18:02 | |
*** pointswaves has quit IRC | 18:09 | |
*** lachlan has quit IRC | 18:14 | |
*** aminbegood has joined #buildstream | 18:16 | |
*** aminbegood has quit IRC | 18:21 | |
*** phoenix has quit IRC | 19:49 | |
*** benschubert has quit IRC | 19:54 | |
*** cs-shadow has quit IRC | 20:44 | |
*** aminbegood has joined #buildstream | 21:02 | |
*** aminbegood has quit IRC | 21:47 | |
*** mohan43u has quit IRC | 23:09 | |
*** lachlan has joined #buildstream | 23:10 | |
*** lachlan has quit IRC | 23:10 | |
*** mohan43u has joined #buildstream | 23:11 | |
*** lachlan has joined #buildstream | 23:14 | |
*** lachlan has quit IRC | 23:17 | |
*** aminbegood has joined #buildstream | 23:32 | |
*** aminbegood has quit IRC | 23:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!