| *** tristan has joined #buildstream | 05:41 | |
| *** jude has joined #buildstream | 07:44 | |
| tristan | dude ! | 07:52 |
|---|---|---|
| *** jonathanmaw has joined #buildstream | 08:03 | |
| *** juergbi has quit IRC | 08:20 | |
| *** juergbi has joined #buildstream | 08:23 | |
| *** adds68 has quit IRC | 08:59 | |
| *** adds68 has joined #buildstream | 08:59 | |
| *** ssam2 has joined #buildstream | 08:59 | |
| paulsherwood | tristan: ? | 09:04 |
| tristan | paulsherwood, it's an insider, but jude didnt notice anyway | 09:07 |
| jude | tristan, still on jude the dude? :) | 09:28 |
| tristan | jude, of course I wouldnt let you down :) | 09:30 |
| tristan | jude, I havent seen you around here much, welcome to the channel ! | 09:30 |
| paulsherwood | ah, i get it now | 09:31 |
| * paulsherwood is slow | 09:31 | |
| jude | paulsherwood, you need to catch up :P | 09:34 |
| jude | tristan, i have been around, though trying to keep an eye on a multiple channels. Hence why you have not heard from me in a while :) | 09:36 |
| tristan | jude, I guess I have to create a really frustrating bug for you to complain about ! | 09:36 |
| *** jude has quit IRC | 09:39 | |
| *** tlater has joined #buildstream | 09:42 | |
| *** jude has joined #buildstream | 09:52 | |
| tlater | Hm, it seems that ostree will refuse to commit WO directories, and since it manages the permissions of its files I don't think caching WO directories is possible | 09:59 |
| tlater | I can get it to throw an error at least | 09:59 |
| tristan | tlater, that is why I was raising the other, deeper issue the other day | 10:00 |
| tristan | i.e. https://gitlab.com/BuildStream/buildstream/issues/38 | 10:01 |
| tlater | tristan: Yup, that would solve this issue. | 10:02 |
| tristan | tlater, in any case lets not get carried away; that will have to be treated separately; and we're going to have to make a better statement about the weird corner cases and what is supported | 10:02 |
| tlater | Currently it looks like RO support is quite easy to get (and implemented), and WO support is impossible until #38 is solved | 10:03 |
| tristan | but yeah, there seems to be no way to handle write only directories without that, because it must remain forbidden at all times to modify artifacts in the cache | 10:03 |
| tristan | So, we are not allowed to temporarily chmod a directory in order to read it's contents, because that is in the cache | 10:04 |
| tristan | besides the fact that ostree wouldnt give it to us anyway; I think it's impossible to commit | 10:04 |
| tristan | but again not sure about that | 10:04 |
| tlater | I don't really see a way it would be. I'm assuming just throwing a pretty error is alright for now, possibly solving it later with a neat way to manage permissions? | 10:05 |
| tristan | So this is what currently works: https://bpaste.net/show/91a60a2ca588 | 10:09 |
| tristan | And I'm wondering, while taking a pause, what would be the best... certainly a less featureful initial landing of conditionals is acceptable, but what can we get right off the bat | 10:10 |
| tristan | ssam2, you expressed a need for a sort of 'else' clause | 10:11 |
| ssam2 | yeah | 10:13 |
| tristan | for instance, a list of conditionals is easy to handle: https://bpaste.net/show/3f96f3ecdc7a | 10:13 |
| ssam2 | ah, that sounds probably fine | 10:13 |
| ssam2 | i'm just thinking of lists of architecture-specific things really | 10:13 |
| tristan | But then things need to get annoyingly wordy if we want a sort of if, elif, elif, else semantic | 10:13 |
| ssam2 | for example, this: https://gitlab.com/baserock/definitions/blob/master/project.conf#L42 | 10:14 |
| tristan | right, and in that case; else is only really a requirement in the case that you are casing a free form string | 10:14 |
| ssam2 | indeed... which I hope to not be doing :-) | 10:14 |
| tristan | But if we disallow freeform strings from the get go, the problem doesnt present itself right ? | 10:14 |
| tristan | right | 10:14 |
| ssam2 | you might want to leave some thing open ended | 10:15 |
| tristan | yeah | 10:15 |
| ssam2 | like, "do this on architecture a, b, or c: else do this on any other architecture" | 10:15 |
| tristan | Ah | 10:15 |
| tristan | Right | 10:15 |
| ssam2 | it would be annoying to have to list every possible thing in the 'else' clause there | 10:16 |
| tristan | That is easy if every conditional is allowed to have an else value | 10:16 |
| tristan | so (?): condition, value, else | 10:16 |
| ssam2 | yeah | 10:16 |
| tristan | I also tried something weird, maybe I should try it again | 10:17 |
| tristan | ssam2, https://bpaste.net/show/eca846344631 | 10:18 |
| tristan | haha | 10:18 |
| tristan | I think it must be a no go, though | 10:18 |
| tristan | there has to be something wrong with it | 10:18 |
| ssam2 | what's weird about that ? | 10:19 |
| ssam2 | that pony isn't defined anywhere ? | 10:19 |
| tristan | Well, somehow, it manages to get parsed, unquoted | 10:19 |
| tristan | Already, I find this strange | 10:19 |
| tristan | An else value is tricky, how would we do it, I guess... with an else key in the same dictionary ? | 10:20 |
| tristan | So... "any key that is not 'else', is a conditional" | 10:21 |
| tristan | And the conditional must have only one key ? | 10:21 |
| tristan | besides a potential else ? | 10:21 |
| tristan | So I guess, no options named 'else' for you | 10:21 |
| tristan | (not horrible) | 10:21 |
| tristan | it's certainly more pretty | 10:21 |
| ssam2 | yeah, that's a common syntax limitation :-) | 10:21 |
| ssam2 | so this is an alternative to having "condition, value, else, ..." ? | 10:22 |
| tristan | ssam2, only in the interest of being less wordy and more pretty yeah | 10:23 |
| ssam2 | I guess the only issue is the extensibility, like if we want to add a new keyword in future besides 'else', it wouldn't be possible because people might have used that as a variable name | 10:23 |
| tristan | but it also make extensibility screwy | 10:23 |
| tristan | exactly | 10:23 |
| ssam2 | but, it looks much nicer! | 10:23 |
| ssam2 | hmm | 10:23 |
| ssam2 | how would the condition/value dict look with an else condition? | 10:24 |
| ssam2 | it seems like the body of the 'else' would be at a different level of indent, which is especially not nice | 10:24 |
| tristan | https://bpaste.net/show/43cf1b48b9a3 | 10:25 |
| tristan | ssam2, actually it looks great | 10:25 |
| tristan | because it's the same dict as the condition itself | 10:25 |
| tristan | of course, the same can be written like this, which is more confusing: https://bpaste.net/show/bdb49e750f32 | 10:26 |
| tristan | because the symbol is expanded and treated as a truthy/falsy value, it tends not to jump out in your face as much | 10:27 |
| tristan | as it would if it were 'conditional: pony' | 10:28 |
| ssam2 | right, yeah that is nice | 10:29 |
| tristan | hrrm | 10:29 |
| tristan | What potential ways would be extend branching | 10:29 |
| ssam2 | I guess 'else' here is just shorthand for "not the-other-thing" | 10:29 |
| tristan | yeah | 10:30 |
| ssam2 | you could do away with it and just have "pony" and "not pony" | 10:30 |
| tristan | ssam2, but it will suck as soon as your conditional is more complicated | 10:30 |
| tristan | Maybe conditionals are never more complicated anyway | 10:31 |
| ssam2 | yeah | 10:31 |
| ssam2 | i mean, we do want to discourage complex conditionals | 10:31 |
| ssam2 | but on the other hand, if we don't add 'else' now we couldn't add it later | 10:32 |
| ssam2 | i can't think of any other keywords we'd add, for what it's worth | 10:32 |
| tristan | I could see 'element.bst in debug_elements or global_debug' being a thing | 10:32 |
| ssam2 | true | 10:33 |
| ssam2 | https://bpaste.net/show/bdb49e750f32 does look really nice ... | 10:33 |
| ssam2 | it gets my vote | 10:33 |
| persia | Another way to do a similar thing is to drop simple conditionals, and only permit swtich/case like statements, with a "default". | 10:33 |
| persia | When there is only one entry in the switch/case, it ends up acting like a conditional. When there are more, it ends up behaving differently. | 10:34 |
| persia | And "default" for default values is semantically safe in most situations. | 10:34 |
| tristan | persia, I keep thinking that I want switches too | 10:34 |
| tristan | how wordy is it going to be ? | 10:34 |
| * persia tries something | 10:34 | |
| tristan | And will I get my fallthroughs :) | 10:34 |
| tristan | ssam2, on the other hand, extensibility could potentially be added via other (?) symbols | 10:36 |
| tristan | a switch could be a different thing | 10:36 |
| tristan | Also I'm not sure that we want a switch as much as we want an if, elif, else | 10:37 |
| persia | https://bpaste.net/show/c89edc62ebc9 ish? | 10:37 |
| tristan | with zebra and default as other list elements | 10:38 |
| persia | Well, zebry, but yes. | 10:38 |
| tristan | heh | 10:39 |
| * tristan likes zebries :) | 10:40 | |
| tristan | persia, unfortunately this is a bit more restrictive | 10:40 |
| tristan | though | 10:40 |
| tristan | persia, i.e.; conditionals happen at the dictionary level, any dictionary can contain only one (?) | 10:40 |
| tristan | persia, the reason for having a list acceptable there, is to allow the user to have multiple conditionals, and let the user decide in which order they are applied | 10:41 |
| persia | So https://bpaste.net/show/43cf1b48b9a3 was never a possibility? | 10:41 |
| tristan | persia, it was, however what's happening is the user specified 2 conditionals and applied them in their decided order | 10:41 |
| tristan | ohhh | 10:42 |
| tristan | persia, right, *that* was never possible no | 10:42 |
| tristan | I.e. it's just a sample and I was writing it out in different forms, but that's not parseable | 10:42 |
| persia | https://bpaste.net/show/8e0305c6b493 is closer to what I meant then, now that I understand that. | 10:43 |
| tristan | persia, except that dictionaries are unordered | 10:44 |
| tristan | there is no guarantee that in the second conditional, I can evaluate horsy before zebry | 10:44 |
| persia | Right. https://bpaste.net/show/bdb49e750f32 is best then | 10:45 |
| tristan | hmmmm, perhaps with evil hacks there could be | 10:45 |
| tristan | but, it would be leveraging things outside of yaml spec for sure heh | 10:45 |
| persia | We'd definitely have to move "default" to the end, which parsing doesn't give for free, which is annoying. | 10:46 |
| persia | But isn't the same true for "else" | 10:46 |
| tristan | We cannot control the user placing else after the conditional, we just suspect users wont write: else { foo } if (condition) {bar} | 10:47 |
| tristan | i.e. else would be explicitly extracted from the dictionary | 10:47 |
| tristan | whereas the expression is extracted as "the only key in the dictionary that is not 'else' " | 10:47 |
| tristan | cannot, being a strong word here... we can actually use ruamel's knowledge of column/line to deduce it | 10:49 |
| tristan | but eh.... | 10:49 |
| tristan | getting more and more on the weird side ;-) | 10:49 |
| persia | Ah, you wanted rich fall-through. Yeah, you can't have that. | 10:50 |
| persia | You can have cases, but only if they are fully discriminatory (my example was bad, beacause quaggas randomly become zebras or horses) | 10:52 |
| persia | and, since we can't trust users to always be fully discriminatory, I assert simple if is the correct choice. | 10:52 |
| persia | (as the alternate may cause non-deterministic behaviour if someone has a quagga) | 10:53 |
| tristan | I know | 10:54 |
| tristan | So here... | 10:54 |
| tristan | (?) is initially defined as a list of conditionals | 10:54 |
| tristan | Where a conditional is a single key dictionary, who's value is a dictionary to be composited to the dictionary holding (?) | 10:54 |
| tristan | I.e., we disallow the case of a single conditional not in a list | 10:54 |
| tristan | ssam2, if you want to write a single conditional, you have to write it as a list with one element | 10:54 |
| ssam2 | i think the two extra characters involved wouldn't pose a problem | 10:55 |
| tristan | ssam2, which is no more typing than otherwise (it's ' ' vs '- ') | 10:55 |
| ssam2 | yeah | 10:55 |
| tristan | ssam2, no two extra characters... looking at https://bpaste.net/show/43cf1b48b9a3 | 10:55 |
| tristan | So | 10:55 |
| tristan | That leaves the "parse the value of (?) as a dictionary instead of a list" door open for future extensibility | 10:56 |
| ssam2 | yeah | 10:56 |
| ssam2 | and effectively allows 'elseif' type things | 10:56 |
| tristan | Well, not sure about that | 10:57 |
| tristan | ssam2, I.e. we can extend it to allow 'else if' things, with some extra typing, you mean ? | 10:57 |
| tristan | that would certainly be true | 10:57 |
| tristan | in the simplistic form, you have a list of ifs and that's it; it's up to you to say 'not A and B' in the second conditional | 10:58 |
| ssam2 | yeah, that's what i was thinking | 10:58 |
| tristan | Right | 10:58 |
| ssam2 | but since you can only have a single `(?):` key in a dict, it's more flexibility than the non-list approach | 10:58 |
| tristan | we could have (?) and a dictionary with 'exclusive:' and the exclusive is an ordered list of if/elif/elif... | 10:59 |
| tristan | but we can leave that out for now | 10:59 |
| ssam2 | yeah | 10:59 |
| tristan | (also note that say, a trailing 'True' expression at the end would effectively be a 'default') | 11:00 |
| persia | How does "trailing" help if we cannot guarantee an order when parsing? | 11:05 |
| tristan | persia, we can guarantee the order of lists | 11:05 |
| tristan | not dictionaries | 11:05 |
| tristan | only lists are ordered | 11:05 |
| persia | Then all is good :) | 11:05 |
| gitlab-br-bot | push on buildstream@81-non-empty-read-only-directories-not-handled-during-bst-build-and-others (by Tristan Maat): 1 commit (last: Ensure that artifact file permissions are set in the right order) https://gitlab.com/BuildStream/buildstream/commit/23e1e11708621030d928215966795bccd78e3124 | 11:08 |
| gitlab-br-bot | buildstream: merge request (81-non-empty-read-only-directories-not-handled-during-bst-build-and-others->master: Ensure that artifact file permissions are set in the right order) #100 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/100 | 11:08 |
| gitlab-br-bot | buildstream: merge request (81-non-empty-read-only-directories-not-handled-during-bst-build-and-others->master: Ensure that artifact file permissions are set in the right order) #100 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/100 | 11:12 |
| gitlab-br-bot | push on buildstream@81-non-empty-read-only-directories-not-handled-during-bst-build-and-others (by Tristan Maat): 1 commit (last: Ensure that artifact file permissions are set in the right order) https://gitlab.com/BuildStream/buildstream/commit/95158527b47f5ae9963c7bbe04d0ae4d87d29ab2 | 11:12 |
| ssam2 | i need some way of having an increasing counter in the Docker image tags | 11:14 |
| ssam2 | i'm thinking the easiest approach is $(git rev-list --count master) | 11:14 |
| ssam2 | it's guaranteed to increase on every change and requires no faff, the only problem is it can jump by more than 1 | 11:15 |
| ssam2 | and maybe we'll need something cleverer once we start having stable branches of buildstream | 11:15 |
| tristan | Is a date stamp not good enough ? | 11:17 |
| tristan | I mean, this is something we'll be explicitly referring to in the .gitlab-ci.yml right ? | 11:18 |
| tristan | any reason it even has to be a number ? | 11:18 |
| ssam2 | the problem with date stamp is, what if we do 8 builds in one day | 11:18 |
| ssam2 | there will also be a "latest" tag | 11:18 |
| tristan | all of which resulted in a docker rebuild ? | 11:18 |
| tristan | so we committed the dockerfile 8 times ? | 11:18 |
| ssam2 | it can happen | 11:19 |
| ssam2 | i'm trying to ensure we have some kind of persistent label for each build | 11:19 |
| ssam2 | so you can depend on a specific version if you want and know it won't change under your feet | 11:19 |
| tristan | abbreviated git sha ? | 11:19 |
| ssam2 | possible, but then it's hard to know which is the latest | 11:19 |
| ssam2 | docker tooling could help there, but i'm not sure it does in practice | 11:19 |
| tristan | Latest doesnt make sense really with multiple branches | 11:20 |
| tristan | regardless of naming | 11:20 |
| ssam2 | true, it's kind of docker convention to have it though | 11:20 |
| ssam2 | but you're right, 'master-latest' and '1.0-latest' would make more sense once we have stable branches | 11:20 |
| tristan | ssam2, I guess it would be desirable also if it is possible to have `bst-here` derive the appropriate docker tag to use ? | 11:21 |
| tristan | not sure if that's possible, though | 11:22 |
| ssam2 | we could allow the user to choose | 11:22 |
| tristan | probably we need to update .gitlab-ci.yml & the bst-here script (if that's what we're really going to end up calling it), at the same time | 11:22 |
| ssam2 | if we had a 'latest-stable' tag then bst-here could just pull that | 11:23 |
| tristan | Maybe we want to allow the user to choose, but I'm pretty sure we dont want to force them :) | 11:23 |
| ssam2 | and it would automatically use the latest stable release | 11:23 |
| tristan | wouldnt be exactly correct, though; I would prefer to have to update it manually | 11:24 |
| tristan | at the same time as .gitlab-ci.yml | 11:24 |
| ssam2 | but we control the latest-stable tag | 11:25 |
| tristan | but we dont control when users upgrade buildstream | 11:25 |
| ssam2 | we do if they're using the docker image | 11:25 |
| tristan | And ideally the dockerfile with the buildstream version you have, is the one which defines the image you use | 11:25 |
| tristan | Do we ? | 11:25 |
| ssam2 | you just said you don't want to force them to choose which version to use, but now you want them to control when they upgrade | 11:25 |
| tristan | I didnt know that | 11:25 |
| gitlab-br-bot | push on buildstream@non-sandbox-builds (by Tristan Maat): 1 commit (last: Add %{script} format to `buildstream show`) https://gitlab.com/BuildStream/buildstream/commit/30d9012c528ebc42c2438cac89e3c065e3c63ef6 | 11:25 |
| ssam2 | the docker image contains buildstream, so they use whatever version is in there | 11:26 |
| tristan | ssam2, I was presuming that the docker image was something supporting of the user's installation/checkout of buildstream | 11:26 |
| ssam2 | aha | 11:26 |
| tristan | Not that that is very important to me, though | 11:26 |
| ssam2 | it could work that way, but i think people who choose the docker approach are exactly the sort to thing "I don't care what version i have as long as it works" | 11:27 |
| ssam2 | *think | 11:27 |
| tristan | I guess so long as the script to run it remains simple and rarely changes; we only need to care about that script working with any docker image | 11:27 |
| ssam2 | yeah; and the script is indeed simple | 11:27 |
| tristan | sounds fine to me; maybe when we document this approach to using buildstream, we want to start with "download this script" | 11:28 |
| tristan | instead of 'clone buildstream.git' | 11:28 |
| ssam2 | ah yeah | 11:29 |
| ssam2 | we could link to the latest git version of the script | 11:29 |
| ssam2 | and then the workflow pretty much becomes "get the script, run the script" | 11:29 |
| tristan | Will we have bash completions ? | 11:29 |
| ssam2 | there's two ways to do this I guess | 11:30 |
| tristan | As I recall, the bst-here thing was a run once and enter build streamable environment | 11:30 |
| ssam2 | yeah | 11:30 |
| ssam2 | that seems the better way | 11:30 |
| tristan | Can we have that be a subshell which can be exited with exit and ^D ? | 11:31 |
| ssam2 | yeah | 11:31 |
| ssam2 | we can offer both approaches, but the 'wrap every bst command in bst-here' approach will be a bit slower, and will not have completions | 11:31 |
| tristan | awesome | 11:31 |
| ssam2 | but yeah, docker has a 'interactive' mode that lets you run a shell as normal | 11:31 |
| tristan | Right, I dont really think we need to care about that; but doing something in the environment to make it apparent would be good | 11:31 |
| tristan | ssam2, what I dont like is having to remember anything custom for exiting the env | 11:32 |
| ssam2 | ah right. with docker it's a real subprocess, it's not like virtualenv | 11:32 |
| tristan | like virtualenv, it has some kind of function you have to call to exit the env, which of course, I forget the name of | 11:32 |
| gitlab-br-bot | push on buildstream@options-refactor (by Tristan Van Berkom): 20 commits (last: doc/source/index.rst: Fix documentation to link the correct Sandbox module) https://gitlab.com/BuildStream/buildstream/commit/11c77a67006b34983d9af813560d5d9f6920c8cc | 12:04 |
| persia | a concern about "I don't care what version I have as long as it works": isn't there a significant chance that one's artifact cache only functions with a specific version of BuildStream? Does this mean that folk following master will have an incentive to freeze on a random version (perhaps a release)? | 12:10 |
| ssam2 | i'm assuming that tristan's plan is to maintain a stable branch of buildstream that doesn't break compatibility of things like the artifact cache | 12:12 |
| ssam2 | thus I expect folk to only follow master if they know the risks | 12:12 |
| ssam2 | and to have `bst-here` default to following stable | 12:12 |
| ssam2 | i guess moving from 1.0 to 2.0 or whatever can break stuff, not sure how we'd handle that | 12:13 |
| ssam2 | maybe have bst-here-1.0, bst-here-2.0, etc. ? | 12:13 |
| tristan | ssam2, that makes sense yes, no bumping of artifact version in stable branches | 12:14 |
| tristan | Moving from one stable branch to another should not *break* things; however a bump in the artifact version is not "breaking things" | 12:15 |
| ssam2 | all true | 12:16 |
| tristan | anyway, by virtue of regular stable branch policy (i.e. only bugfixes land here), artifact cache keys shouldnt ever change there | 12:16 |
| * paulsherwood would hope that latest buildstream can support all old artifact versions, not just latest artifact version | 12:16 | |
| tristan | Also, we still need a way to checkout artifacts via metadata | 12:16 |
| tristan | paulsherwood, I dont think there is any sense in trying to support every algo that was ever used; to *re-derive* old artifact cache keys | 12:17 |
| tristan | that sounds absurdly expensive | 12:17 |
| tristan | However being able to deal with existing artifacts which dont match the current scheme, is totally possible | 12:18 |
| paulsherwood | tristan: we're talking about a few if statements, in a few hundred lines of python. how can that be 'absurdly expensive'? | 12:19 |
| tristan | I think we're talking about 10 years of accumulative changes in various parts of a code base | 12:19 |
| tristan | Including policy for plugins which are not included in that code base | 12:19 |
| paulsherwood | for cache-key calculation? | 12:20 |
| tristan | Yes | 12:20 |
| tristan | this is of course, entirely orthogonal to format backwards compatibility | 12:20 |
| paulsherwood | interesting. how many versions have you bumped so far? | 12:21 |
| tristan | The cache key algo has change quite a few times | 12:21 |
| tristan | That or, reason to change it | 12:21 |
| tristan | However I'm holing off on bumping actual numbers pre 1.0 | 12:21 |
| tristan | (I would if there was an absolute need, but seems to not be), also; the versioning support for this landed after a few of the changes which would require bumps | 12:22 |
| paulsherwood | so, for reproducing an old artifact (say in 10 years time) i guess you're saying use the old version of bst from that time? | 12:22 |
| tristan | addition of metadata which buildstream needs to read back from an artifact, is one reason which absolutely requires a bump for our intermediate pre-stable dev purposes | 12:23 |
| tristan | But last that happened, it was before we had the versioning | 12:23 |
| tristan | Certainly | 12:24 |
| paulsherwood | let's make sure that is documented, then, and that bootstrapping old builstream versions is locked down... | 12:26 |
| paulsherwood | it would be a shame, in 10 years, to find that we could no longer bootstrap today's bst because (say) that version of ostree no longer works (for whatever reason) | 12:27 |
| persia | Configuring CI to reliably build systems with each released BuildStream is probably wise. Those branches may not get many changes, but it helps to know if a point is reached where it is very hard to run the old version. | 12:27 |
| persia | And if the old version is hard to run, then it makes sense to apply effort to make sure it *can* be used, to support folk who actually need multidecade support. | 12:27 |
| tristan | Well | 12:28 |
| tristan | Not that that CI would be a bad thing | 12:28 |
| tristan | But I think we need to consider what "multidecade support" means | 12:28 |
| tristan | Considering that a cache is so far by definition a cache, and not a store, I'm not really convinced that the key we use for that is what matters most | 12:29 |
| tristan | Checksumming and comparing the output data for a truly reproducible build, sounds more interesting | 12:29 |
| tristan | but then, that itself could change with artifact bumps (especially if you consider the metadata encoded into an artifact as a part of that checksum) | 12:30 |
| persia | tristan: If we can be absolutely confident that the same source generates the same result with different buildstream versions, then I agree. | 12:30 |
| persia | But if we're changing the artifact versions, I'm fairly confident we're no longer producing the same thing. | 12:31 |
| tristan | It's a generally unsolved problem, but rather an almost solved one by now | 12:31 |
| persia | I'm willing to assume every upstream is reproducible, for the sake of this discussion. | 12:31 |
| persia | My issue is that *if* we have a different version of buildstream, we are no longer generating the same artifact. | 12:31 |
| persia | Therefore, how could we possibly know we reproduced our result? | 12:32 |
| tristan | In which case our biggest blocker to that I suppose is https://gitlab.com/BuildStream/buildstream/issues/38 | 12:32 |
| persia | And why use BuildStream, whern other tools preserve the reproducibility of the source? | 12:32 |
| tristan | persia, preserving a key used for the purpose of caching the results, does not necessarily mean that the results are not reproducible from the same source | 12:33 |
| tristan | i.e. those are not necessarily related | 12:33 |
| persia | tristan: How can I tell? | 12:34 |
| tristan | persia, compare the actual output ? | 12:34 |
| persia | If I build X, and I get Y, and then I build X and I get Z, why should I believe Y == Z? | 12:34 |
| persia | If the artifacts are identical, I should be able to reuse them from cache. If they are not, I cannot. If I cannot, I would assume they are not identical. | 12:34 |
| tristan | persia, you are trying to state that the folder name (cache key) under which buildstream stores the result, *is* the result | 12:34 |
| tristan | persia, I'm contesting that statement | 12:35 |
| paulsherwood | we call it a cache-key for historic reasons, but it's not just used for caching... it's a way to identify an artifact | 12:35 |
| persia | I7m suggesting that if we organise the structure based on an element we expect to change, we're confusing the users for no good reason. If we expect identical results, let's use a key that is based on that identity, so users don't need to rebuild their cache because a release happened. | 12:36 |
| *** bochecha has joined #buildstream | 12:36 | |
| persia | I don't really care about "cache key", only about the artifact. | 12:36 |
| persia | And if the artifact remains identical, I don't understand why we have to change the URI for the artifact. | 12:36 |
| paulsherwood | right... but the 'cache key' is an identifier for the artifact. and multiple identifiers may actually lead to the exact same set of bits | 12:37 |
| tristan | persia, then consider this for instance: https://gitlab.com/BuildStream/buildstream/issues/82 | 12:37 |
| tristan | persia, if we start to store a manifest in order to optimize buildstream's performance, that manifest is not technically part of the build output; but it still requires a cache key bump | 12:39 |
| persia | tristan: does "cache key" perhaps conflate identity of the artifact and identity of the API to interact with the artifact? | 12:39 |
| persia | If so, are there any reasons that we could not take an existing cache, and run tooling against it to generate the new cache keys for the cached artifacts and add some hardlinks? | 12:40 |
| tristan | persia, I rather think splitting the key in two is more sensible no ? | 12:46 |
| persia | tristan: I concur :) | 12:46 |
| persia | A key, used for sorting things in the filesystem, linked to the identity of the bits in the artifact, and another key, used for API versioning. | 12:47 |
| tristan | currently, as far as element configuration go, it is possible to add new features to your format (element configuration) while remaining backwards compatible with the unique keys (contributes to cache key) | 12:47 |
| persia | The extra advantage of that is that one can do richer negotiation and capabilities-based interaction with an API version key. | 12:47 |
| tristan | and that's certainly recommended | 12:48 |
| tristan | but writing plugins is not fully documented yet | 12:48 |
| gitlab-br-bot | buildstream: issue #102 ("Run CI as non-root user") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/102 | 13:31 |
| tristan | tlater, thanks :) | 13:31 |
| tlater | :) | 13:32 |
| tlater | tristan: As an aside, I assumed the `--format '%{script}'` output was supposed to be piped into a shell to execute builds outside of the sandbox - but this won't work if the correct sources aren't in /buildstream/build. | 13:33 |
| tlater | Am I misunderstanding the purpose of that feature? | 13:34 |
| tristan | tlater, my thoughts are this; the useful part of this feature is just to show the project author what will be run for a build element | 13:35 |
| tristan | beyond this; any mangling of it should be achieved by allowing some pre-emptive changes to variables before formatting | 13:36 |
| tristan | That said, /buildstream/build is rarely to never expanded in scripts | 13:36 |
| tristan | it will however want to install in /buildstream/install | 13:36 |
| tristan | Hmmm | 13:36 |
| tristan | Gah, my usefulness just flew out the window :-/ | 13:36 |
| tristan | Now I realize the script will generate something... without the configure-commands:, build-commands: etc | 13:37 |
| tristan | hmmm, maybe it's still useful | 13:37 |
| tlater | tristan: It's useful for what you want, it just doesn't exactly do what we intended it to do :) | 13:38 |
| tristan | I intended it to show something useful to a project author who wants to know whats going on | 13:38 |
| tristan | with full knowledge that someone will think of abusing it for something pointless | 13:39 |
| persia | tlater: I think of it as more useful as a means to get an overview of what is going to happen, so I can run a *single command* by copy&paste in my terminal to go through the next step of my source whilst I'm working on it, without needing to wait for a full sandbox-create, build, report cycle. | 13:56 |
| persia | Once things build properly and some local test runs, I can commit, and then call buildstream normally. | 13:57 |
| tristan | Could be nice in conjunction with `bst shell --build <element.bst>` | 13:58 |
| tlater | Hm, alright, that makes sense. I had another look at the scripts too, and it seems that the $SRCDIR can just be set to something, so this does work as a build script if need-be. | 13:58 |
| tristan | fool around with editing bst file and running `bst show --format %{script}` in a separate terminal | 13:58 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:01 |
| gitlab-br-bot | push on buildstream@options-refactor (by Tristan Van Berkom): 5 commits (last: context.py: Now expects cli_options in the constructor) https://gitlab.com/BuildStream/buildstream/commit/d811c729ac7550b114413fd257d7b76b98b99cf1 | 14:01 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:02 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:02 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:02 |
| tlater | Alright, !102 is waiting for a review. I think it might need a bit more documentation, but I'm not sure where to put that. | 14:03 |
| tristan | tlater, for now the `bst show` formats are unfortunately only documented in `bst show --help` | 14:05 |
| tristan | but it's a bad situation | 14:05 |
| tristan | especially because of the formatting bugs with generating man pages and html documentation with the click docstrings | 14:05 |
| tlater | tristan: Yeah, I added it to that, but I don't exactly want to put a wall of text explaining environment variables there. | 14:05 |
| tristan | tlater, omg no - we're not going to encourage abuse of that by specifically documenting the weird stuff you can do with it | 14:06 |
| tlater | Heh, alright, then this should be fine :) | 14:07 |
| tristan | tlater, can I see what a sample output looks like ? | 14:07 |
| tlater | tristan: It's not the neatest because of bash being weird with subshells, but I'll add that output to the issue | 14:07 |
| tlater | I'll also rebase, didn't realize I was behind again | 14:08 |
| tristan | :-/ | 14:10 |
| tristan | Ah right, each command is expected to be it's own shell | 14:11 |
| tristan | that is unfortunate indeed | 14:11 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:11 |
| tlater | It's worse than that, since chaining multiple commands in a subshell can sometimes make bash miss the second ')' - so we need a newline before we close subshells | 14:12 |
| tristan | bochecha, hmmm, I'll have to dig in deeper to that `bst shell` issue, fwiw interactive shells in user namespace is non-trivial and "doing it properly" might be our only recourse | 14:14 |
| gitlab-br-bot | push on buildstream@non-sandbox-builds (by Tristan Maat): 4 commits (last: sandbox.py: Fixup placement of main Sandbox docstring) https://gitlab.com/BuildStream/buildstream/commit/03a646e708d3559b3e0f06c0103b76ef8290872e | 14:14 |
| gitlab-br-bot | buildstream: merge request (non-sandbox-builds->master: Add %{script} format to `buildstream show`) #102 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/102 | 14:14 |
| tristan | where I think "doing it properly" involves something with an additional helper process delivering data to/from the console in the user namespace | 14:14 |
| * tristan has been up to his ears in this last minute project options turnaround... and needs to reproduce a build to debug issue #98 | 14:15 | |
| tristan | jjardon[m], fwiw... there is a whole "thing" going on presently, and for the last week, about frustration with yocto and inability to properly generate a correctly targetted native compiler | 14:20 |
| tristan | jjardon[m], as the leader of the replacing yocto initiative, might be nice if you hang out in #flatpak :D | 14:20 |
| *** tlater has quit IRC | 14:21 | |
| jjardon[m] | mmm, am I not there already? | 14:22 |
| tristan | nope | 14:22 |
| tristan | unless you changed your nick :) | 14:22 |
| jjardon[m] | lets fix that :) | 14:22 |
| *** tlater has joined #buildstream | 14:25 | |
| gitlab-br-bot | buildstream: Tristan Maat created branch 102-run-ci-as-non-root-user | 14:33 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/93b3d42bb49d8b70d22708aadd902256481e98a6 | 14:44 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/2ec492dd187e979a31ee4ada352d0b62b4027cbb | 14:47 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/d7d980abef62e735785ae493a8566c9ffc869a7e | 14:49 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/4a7901c4a911cd15351ec1a673557c90bce08331 | 14:55 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/bbd7730f73eb278555ac3e3d1c6f62347e451c33 | 14:59 |
| *** tristan has quit IRC | 14:59 | |
| tlater | ssam2: Is there something in the docker image that prevents me from switching users? https://gitlab.com/BuildStream/buildstream/-/jobs/34931116 | 15:23 |
| tlater | su Also doesn't appear to work | 15:23 |
| ssam2 | there are no users, just root | 15:26 |
| ssam2 | that's the Docker way | 15:26 |
| ssam2 | you could add users if you need one for something | 15:26 |
| benbrown | That's what tlater has done in that pipeline ;) | 15:27 |
| ssam2 | oh right | 15:27 |
| ssam2 | it's hard to debug without seeing what `whoami` actually outputted | 15:28 |
| tlater | ssam2: It's root, I made sure it was in another run | 15:28 |
| ssam2 | hmm ok | 15:28 |
| tlater | No strange newlines or anything | 15:28 |
| * tlater would try locally, but debian refuses to install docker | 15:29 | |
| ssam2 | weird | 15:29 |
| ssam2 | you could set up a VM on http://nebula.ducie.codethink.co.uk/ with an OS that is capable of running Docker, perhaps | 15:29 |
| benbrown | Perhaps each command is run as the runner's configured user (which is likely root by default)? | 15:30 |
| tlater | Good point, benbrown | 15:30 |
| tlater | Although usually changes to the environment work, such as cd | 15:30 |
| * tlater attempts | 15:30 | |
| ssam2 | ah it maybe that `sudo -s` fails because you have no stdin | 15:30 |
| tlater | ssam2: That wouldn't explain why su fails | 15:31 |
| ssam2 | no it wouldn't | 15:31 |
| ssam2 | it's also possible that there's some capabilities or user namespaces thing preventing any change of UID | 15:31 |
| benbrown | tlater: Aye, I'm just guessing :) | 15:31 |
| ssam2 | although given that bwrap works inside the container, it can't be too much of an issue | 15:32 |
| ssam2 | hmm, but do we do nested user namespaces with bwrap? | 15:32 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/08116bb9996cfeae0f8c8ee716e5c62a219d8dd8 | 15:32 |
| ssam2 | i think it's worth spending time reproducing this locally rather than theorising, anyway | 15:32 |
| tlater | ssam2: Reproducing might require running a docker-runner instance | 15:33 |
| tlater | Might be fastest doing it on the CI | 15:33 |
| ssam2 | worth trying it in a local docker container | 15:33 |
| benbrown | can't reproduce with just docker | 15:33 |
| benbrown | fyi | 15:33 |
| ssam2 | ah, right | 15:33 |
| tlater | ta benbrown :_ | 15:33 |
| tlater | *:) | 15:33 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/be07c45b349aed8ed106f890bb2cbd59e0a49a0b | 15:39 |
| tlater | benbrown was right, it's probably executed by the specific user somehow :) I was pretty sure it's just a docker thing, but apparently not. | 15:40 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/9f3fd6f1e23d14eeaa80fbd594eab7a8440dd726 | 15:43 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/cf5b73ebcd2e7d21aa101ea0aeebf58bf1b783ca | 15:46 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/7b0ee298566cd370ce3d969f2a36a7a8af531220 | 15:52 |
| *** tristan has joined #buildstream | 16:03 | |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/daafae01fb64f3e1a849b3ab07163d07aed20796 | 16:05 |
| gitlab-br-bot | push on buildstream@102-run-ci-as-non-root-user (by Tristan Maat): 1 commit (last: TEST COMMIT) https://gitlab.com/BuildStream/buildstream/commit/1af988506ecc24e28ef9645b37ed081b7f5f66e5 | 16:15 |
| *** ssam2 has quit IRC | 16:24 | |
| *** ssam2 has joined #buildstream | 16:27 | |
| gitlab-br-bot | push on buildstream@81-non-empty-read-only-directories-not-handled-during-bst-build-and-others (by Tristan Maat): 1 commit (last: Ensure that artifact file permissions are set in the right order) https://gitlab.com/BuildStream/buildstream/commit/ba10db7e98f948e14abf9bab3232a8ad466b7636 | 16:29 |
| gitlab-br-bot | buildstream: merge request (81-non-empty-read-only-directories-not-handled-during-bst-build-and-others->master: Ensure that artifact file permissions are set in the right order) #100 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/100 | 16:29 |
| *** jonathanmaw has left #buildstream | 16:40 | |
| *** jonathanmaw has joined #buildstream | 16:40 | |
| *** xjuan has joined #buildstream | 16:58 | |
| *** jonathanmaw has quit IRC | 17:02 | |
| *** ssam2 has quit IRC | 17:02 | |
| *** tlater has quit IRC | 17:17 | |
| *** anahuelamo has quit IRC | 17:18 | |
| *** jude has quit IRC | 18:06 | |
| *** jude has joined #buildstream | 19:08 | |
| *** jude has quit IRC | 21:37 | |
| *** xjuan has quit IRC | 23:14 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!