| *** laurenceurhegyi has joined #buildstream | 00:29 | |
| gitlab-br-bot | push on buildstream@format-version (by Tristan Van Berkom): 7 commits (last: element.py: Changing artifact version API) https://gitlab.com/BuildStream/buildstream/commit/2b0ba33947b920d0a91bf8391a2b00ec73d60096 | 03:00 |
|---|---|---|
| *** tristan has quit IRC | 03:03 | |
| *** jude has joined #buildstream | 04:18 | |
| *** tristan has joined #buildstream | 04:52 | |
| *** jude has quit IRC | 06:02 | |
| *** tristan has quit IRC | 06:33 | |
| *** jude has joined #buildstream | 07:05 | |
| *** bochecha has joined #buildstream | 07:35 | |
| *** jonathanmaw has joined #buildstream | 08:18 | |
| *** ssam2 has joined #buildstream | 09:00 | |
| *** tlater has joined #buildstream | 09:08 | |
| gitlab-br-bot | buildstream: issue #80 ("Awkward to recover after manually deleting workspace dir") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/80 | 09:51 |
| *** jude has quit IRC | 10:37 | |
| *** jude has joined #buildstream | 10:42 | |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 1 commit (last: _sandboxchroot.py: Fix --rbind issues) https://gitlab.com/BuildStream/buildstream/commit/06472332af11dfc381ef0ef6cd49fb62d0076b42 | 12:24 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: WIP: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 12:24 |
| *** tlater has quit IRC | 12:30 | |
| *** jude has quit IRC | 12:43 | |
| *** jude has joined #buildstream | 12:45 | |
| *** bochecha has quit IRC | 12:59 | |
| *** bochecha has joined #buildstream | 13:00 | |
| *** bochecha has quit IRC | 13:03 | |
| *** bochecha has joined #buildstream | 13:04 | |
| *** bochecha has quit IRC | 13:09 | |
| *** bochecha has joined #buildstream | 13:09 | |
| *** bochecha has quit IRC | 13:25 | |
| *** bochecha has joined #buildstream | 13:25 | |
| *** bochecha has quit IRC | 13:28 | |
| *** bochecha_ has joined #buildstream | 13:28 | |
| *** bochecha_ has joined #buildstream | 13:29 | |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 1 commit (last: _sandboxchroot.py: Fix --rbind issues) https://gitlab.com/BuildStream/buildstream/commit/e42ecad09ba06b0848e81cb5ec7600ec2296ad5b | 14:35 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: WIP: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 14:35 |
| *** jude has quit IRC | 14:35 | |
| *** tlater has joined #buildstream | 14:38 | |
| gitlab-br-bot | buildstream: issue #81 ("Non-empty read-only directories not handled during 'bst build' and others") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/81 | 14:40 |
| *** jude has joined #buildstream | 15:15 | |
| *** tristan has joined #buildstream | 15:21 | |
| tristan | bochecha_, sorry I would have enjoyed to hand off the compose feature to you but I had already been thinking of it and had started when I saw your message | 15:25 |
| tristan | :) | 15:25 |
| tristan | also I had to change public API for it and wasnt sure how that was going to go | 15:26 |
| tristan | regarding directory names in split rules, it's a bit sticky; you can have it one way or the other but not both I think (either you can specify a directory inclusively; or just a directory) | 15:27 |
| tristan | I did think of that before and didnt find something easy; maaybe making some exception about how the dir is specified (like trailing / or not) would do ? | 15:28 |
| tristan | rpm spec files have more complex directives around files, we should also consider whether we're going to need more data around this or not | 15:29 |
| tristan | tlater, ok so lets talk about what to do with cross platform and staging files | 15:30 |
| tlater | Listening :) | 15:31 |
| tristan | I dont like the conclusions we came to on the post-guadec friday; I was frankly exhausted those last days | 15:31 |
| tristan | So to summarize the issue... this has to do with the platform's choice of using hardlinks or forcing direct copies in the absence of fuse | 15:32 |
| tristan | When fuse is not usable... we cant use hardlink staging for anything that will appear in a writable volume in the sandbox | 15:33 |
| tristan | because we cant provide the copy-on-write | 15:33 |
| tristan | Our conclusion was to tie read/write-ness of volumes to the sandbox lifetime | 15:34 |
| tristan | So that the platform can make one choice, and doesnt start staging hardlinks only to find later that it needs copies | 15:35 |
| tristan | The problem with that plan though, is that if we change that then performance suffers on the linux case | 15:35 |
| tristan | because fuse is always running; so even during builds shared libraries and data from the runtime is being loaded and accessed through a single threaded python fuse layer | 15:36 |
| tristan | So to fix that there are two approaches I can see since my head got screwed on this morning... | 15:37 |
| tristan | o Dont remove the existing sandbox for root-read-only-ness, but add something sandbox-lifetime configuration which states that "In this sandbox, root will be read-write at least once" | 15:38 |
| tristan | o Dont change API at all, but deduplicate hardlinks from the artifact cache the first time that the sandbox has been "run" with root read-write | 15:40 |
| tristan | tlater, so far does that make sense ? | 15:40 |
| tlater | Yes, though those still have the issue that not only root can be read-write | 15:40 |
| tristan | Root is only special in that it can be read-write or read-only | 15:41 |
| tristan | tlater, so you haven't dealt with that part yet right ? | 15:41 |
| tlater | Not yet, no | 15:42 |
| tlater | Because /buildstream/{install,build} will always have to be RW | 15:42 |
| tristan | Not true | 15:42 |
| tlater | Unless we don't build anything, I guess? | 15:43 |
| tristan | all the directories which were marked to be read-write via the sandbox need to be read-write, currently | 15:43 |
| tlater | Ah, right, so I could just add a flag to sandbox.mark_directory that says whether or not to mount read write | 15:44 |
| tristan | I.e. Sandbox.mark_directory() | 15:44 |
| tristan | Hmmm | 15:44 |
| tristan | tlater, that sounds about right yes | 15:45 |
| tlater | And then whenever anything is staged check whether it lands in a subdirectory of that or root | 15:45 |
| tristan | I can see there is a tricky bit of finding the relation of paths | 15:45 |
| bochecha_ | tristan: basing the recursivity on whether the trailing / was specified or not seems like it might be a bit too much magic, and could lead to confusion :-/ | 15:45 |
| tristan | tlater, if you need to come up with a function for that, it would be nice to have utils.path_is_ancestor() | 15:47 |
| bochecha_ | tristan: but maybe BuildStream could grow a syntax for specifying options with paths? e.g /path/to/folder (just the folder) vs /path/to/folder:recursive (the folder and its contents) ? | 15:47 |
| bochecha_ | tristan: or dir:/path/to/folder (just the folder) vs /path/to/folder (the folder and its content) ? | 15:47 |
| tristan | tlater, for instance; I guess you will be receiving platform->stage_artifact() with a path, and you will need to lookup that location and see what volume it lands on | 15:47 |
| tristan | tlater, according to the mappings declared by Sandbox.mark_directory() | 15:48 |
| bochecha_ | the second one borrows from rpm's `%dir /path/to/folder`, but the first one has the advantage of being compatible with the existing version | 15:48 |
| tlater | tristan: There's probably a standard algorithm for that, I think I know how to get there now. | 15:48 |
| tlater | I was confused between marked and staged directories, for some reason I thought they were the same thing. | 15:49 |
| tristan | tlater, right okay, and I wanted to ask you; what is the strategy for determining if an element is going to write outside of marked directories ? | 15:50 |
| tristan | tlater, I guess that is the crux of it right; it's not so much that root is read-write, it's more that the plugin needs to write outside of marked dirs | 15:50 |
| tlater | The user needs to know how their element is built - it should not write outside | 15:50 |
| tlater | Or, if it should, the user needs to explicitly state that root needs to be read-write | 15:51 |
| tristan | of course, this is partly a matter of policy yes | 15:51 |
| tristan | Yes | 15:51 |
| tristan | But then there is this icky thing about integration commands | 15:51 |
| tlater | That's why we had no-integrate, right? | 15:52 |
| tristan | So we know that it's going to be more expensive to have read-write root, in which case it is desirable to determine before hand *if* any integration commands should be run | 15:52 |
| tristan | tlater, I was going to get to that | 15:52 |
| tristan | actually I was hoping to ditch it | 15:52 |
| tristan | no-integrate is one of the more undesirable outcomes of that friday meeting, it means users will create projects differently on different platforms in a fundamental way | 15:54 |
| tristan | which kind of kills the opportunity for nice cross platform compatible projects | 15:54 |
| tristan | tlater, we should be able to replace that no-integrate fairly easily I think by reading the element's public data as soon as we're creating a sandbox that builds against it | 15:56 |
| tristan | just check if there are any integration commands basically ? | 15:56 |
| tristan | integration is already like, first class citizen | 15:56 |
| tlater | tristan: I guess if integration commands can be delayed anyway, there's no need to disable them; you just don't define them in the first place? | 15:59 |
| tlater | That would make elements a lot less flexible, though - I think I don't quite follow | 16:00 |
| tristan | tlater, thats basically what I was thinking, I think you would have to use compose elements here and there at checkpoints and integrate once | 16:00 |
| tristan | well | 16:01 |
| tristan | Its required either way that you make these checkpoints | 16:01 |
| tristan | tlater, if you are going to "integrate later", you cant do it at just one place but not store the integration result | 16:01 |
| tristan | things that depend on it further; will also need to integrate | 16:02 |
| tristan | so compose elements could carry the integration data forward | 16:02 |
| tlater | Right, I understand | 16:02 |
| tristan | tlater, so that is *in the case that we dont have /some/ fuse equivalent solution on the platform* | 16:02 |
| tristan | And no weird API is introduced | 16:03 |
| tristan | Which means that if we later find a solution; things will just magically improve for users and no projects will have to change | 16:03 |
| tlater | Yup, I like that solution | 16:03 |
| tristan | In the case that we *do* add API, it looks like we still have the same limitations | 16:03 |
| tlater | In terms of the two approaches to managing read-onlyness, which one do you prefer? | 16:09 |
| tristan | bochecha_, so basically we need syntax for expressing a 'mask of files' such that it can be matched against file lists; in other words we dont ever want the split-rules to gain syntax about what to *do* with these files, I guess this means chances are very low that anything else will be required | 16:09 |
| bochecha_ | tristan: I'm not sure I understand | 16:10 |
| tristan | tlater, the first is cleaner I think; I dont particularly like the idea of hardlinking all files and then another pass of copy/move | 16:11 |
| tristan | bochecha_, well I was considering that a flexible format permits for more metadata in the future, but is bulkier and more verbose | 16:11 |
| tristan | I mean, if it ends up that you need to add more YAML to say this path is recursive, it's not really saving you anything over just having 2 path matches | 16:12 |
| tristan | which is why I thought *maybe* trailing / could indicate that... | 16:13 |
| tristan | not really nice approach either, though | 16:13 |
| tristan | but maybe nicer to the user | 16:13 |
| tristan | ssam2 ran a lot of pipelines today | 16:16 |
| ssam2 | did i ? | 16:16 |
| * tristan hands ssam2 the gitlab pipelines award of the day | 16:16 | |
| bochecha_ | tristan: indeed | 16:17 |
| ssam2 | tristan, you mean in baserock definitions ? i did run many pipelines there recently | 16:17 |
| tristan | ssam2, oops no I had wrong window open, looking at baserock | 16:17 |
| bochecha_ | tristan: in the grand scheme of things, it's not really that bad to have to specify both the folder and the glob for its contents | 16:17 |
| tristan | haha | 16:17 |
| ssam2 | i hope I can keep the award | 16:17 |
| tristan | bochecha_, haha... sticky issue :) | 16:17 |
| bochecha_ | it would be nicer to be able to specify the folder recursively, but maybe it's not that important if it's too complex | 16:17 |
| tristan | complex in terms of format, it wont really hurt the codebase | 16:18 |
| tristan | I think, well... maaaybe years later I'll palmface about adding a trailing / detection there | 16:19 |
| *** waltervargas[m] has quit IRC | 16:25 | |
| *** jjardon[m] has quit IRC | 16:25 | |
| *** mrmcq2u[m] has quit IRC | 16:25 | |
| *** cgmcintyre[m] has quit IRC | 16:25 | |
| *** mattiasb has quit IRC | 16:25 | |
| *** ptomato[m] has quit IRC | 16:25 | |
| *** ptomato[m] has joined #buildstream | 16:26 | |
| bochecha_ | tristan: you did add it? | 16:29 |
| tristan | bochecha_, no no | 16:29 |
| bochecha_ | ok, I thought I had misunderstood this whole conversation :) | 16:30 |
| tristan | was just reflecting | 16:30 |
| tristan | :) | 16:30 |
| bochecha_ | in any case, don't worry too much about it, it's not that big an issue, I'm sure there's much more important to do in BuildStream | 16:30 |
| bochecha_ | overall I'm super happy about it, having to add a few lines in my project.conf won't prevent me from sleeping :) | 16:30 |
| tristan | :) | 16:31 |
| tristan | I'm also thinking of new general format features | 16:31 |
| tristan | wrt dictionary / array composition | 16:32 |
| tristan | currently the engine decides whether arrays are appended or replace source arrays when compositing | 16:32 |
| tristan | and prepend is never an option | 16:32 |
| tristan | But I was thinking maybe a << >> == kinda notation might let the user decide instead | 16:33 |
| tristan | maybe scratch ==, something like foo: would mean "replace this array", and then you might have foo>>: "append to this array" or foo<<: "prepend to this array" | 16:34 |
| tristan | That would let you decide if you wanted to override the split-rules entirely or just augment them | 16:34 |
| tristan | (and would allow stuff like prepending commands to 'build-commands' in build elements) | 16:35 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 7 commits (last: element.py: Changing artifact version API) https://gitlab.com/BuildStream/buildstream/commit/2b0ba33947b920d0a91bf8391a2b00ec73d60096 | 16:39 |
| tristan | Alright the versions have landed ! | 16:39 |
| tristan | Now we have all kinds of versions yay \o/ | 16:39 |
| tlater | tristan: Python has a way to confirm ancestors: | 16:51 |
| tlater | test = pathlib.Path('/home/tlater/Documents') | 16:51 |
| tlater | if pathlib.Path('/home') is in test.parents: | 16:51 |
| tlater | # something | 16:51 |
| tlater | I think that's simple enough that there's no need for a util function | 16:52 |
| tristan | tlater, nice | 17:01 |
| bochecha_ | +1 on using pathlib, it's a pretty great module | 17:09 |
| bochecha_ | be careful about the minimum Python versions, though | 17:10 |
| bochecha_ | it gets new functions every Python release, so you might end up using something that's only in e.g Python 3.6 | 17:10 |
| tristan | we're with 3.4 | 17:11 |
| tristan | yeah :) | 17:11 |
| bochecha_ | like path.write_text() and path.write_bytes() which were added in python 3.5, and are just amazing as they completely remove the need from opening the file (with, etc...) | 17:11 |
| bochecha_ | Path.parents was in 3.4 though, so you're good :) | 17:11 |
| tlater | bochecha_: Thanks for the reminder :) | 17:12 |
| * bochecha_ loves pathlib | 17:12 | |
| * tlater wishes I'd known about it last year | 17:12 | |
| tristan | the old glob implementation was with PurePath.match() | 17:14 |
| *** tlater has quit IRC | 17:14 | |
| tristan | I think it's a bit of a shame that python's glob module is hardwired to the os interface for walking files | 17:15 |
| bochecha_ | tristan: what do you mean? | 17:15 |
| tristan | pathlib was the only thing close | 17:15 |
| tristan | I mean you cant use `import glob` for globbing a list of files | 17:15 |
| tristan | or paths in a tarball | 17:15 |
| bochecha_ | oh | 17:16 |
| tristan | it just calls into os listdir and walk and such | 17:16 |
| bochecha_ | yeah, I see what you mean | 17:16 |
| bochecha_ | you have to use fnmatch on each element of the list -_- | 17:16 |
| bochecha_ | that's pretty terrible, yeah | 17:16 |
| tristan | we do something a bit more exotic but yeah | 17:16 |
| tristan | I think it's from fnmatch we took the code to convert to regex and additionally support globstar (**) | 17:17 |
| tristan | then at least we compile a regex inclusive of all your globs | 17:18 |
| bochecha_ | and all my globs are belong to buildstream | 17:18 |
| tristan | but yeah, I think glob is a bit more optimized, manages to skip directories | 17:18 |
| tristan | haha | 17:18 |
| tristan | yes, regex is eating peoples globs | 17:19 |
| bochecha_ | stop eating my globs! | 17:19 |
| bochecha_ | a long time ago, I had sent a patch to Python upstream about supporting « {foo,bar} » expansion in fnmatch | 17:19 |
| bochecha_ | but then folks started wondering whether that should be in fnmatch or glob or shutil or ... | 17:20 |
| bochecha_ | and eventually nothing was decided, so I just gave up | 17:20 |
| tristan | I wanted that, can we have it ? :) | 17:20 |
| *** jonathanmaw has quit IRC | 17:23 | |
| *** ssam2 has quit IRC | 17:27 | |
| *** jude has quit IRC | 17:28 | |
| bochecha_ | tristan: http://bugs.python.org/issue9584 | 17:28 |
| tristan | bochecha_, ah that one is in glob though | 17:31 |
| bochecha_ | tristan: no, my patch was in fnmatch | 17:31 |
| bochecha_ | tristan: that's the ticket I had opened 7 years ago :) | 17:31 |
| tristan | http://bugs.python.org/file21001/0001-Curly-brace-expansion-in-glob.patch | 17:32 |
| tristan | anyway there is expand_braces() which uses a regex so it's a starting point | 17:33 |
| bochecha_ | tristan: look at my initial comment, the initial patch was for fnmatch | 17:33 |
| bochecha_ | but then I was asked to change it, then it changed again, then... then I lost interest | 17:33 |
| tristan | yeah | 17:34 |
| bochecha_ | do read the whole conversation, because it's very possible the last patch I posted had some issues someone reported, but I never fixed | 17:34 |
| tristan | but the history of patches got lost there ? | 17:34 |
| tristan | I cant find the old obsolete ones | 17:34 |
| bochecha_ | yeah | 17:36 |
| bochecha_ | I think RT loses old patches when they are marked obsolete | 17:36 |
| bochecha_ | or maybe not, but the UI is so confusing that even though they might still be there, they can't be found :D | 17:37 |
| tristan | bochecha_, I think if you created the issue and attached the patch after, we'd have it in the history | 17:38 |
| tristan | the history appears to lose the initial entry | 17:38 |
| tristan | anyway | 17:39 |
| gitlab-br-bot | buildstream: issue #82 ("Store artifact manifests in metadata") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/82 | 17:46 |
| *** cgmcintyre[m] has joined #buildstream | 17:57 | |
| *** mrmcq2u[m] has joined #buildstream | 17:57 | |
| *** mattiasb has joined #buildstream | 17:57 | |
| *** waltervargas[m] has joined #buildstream | 17:57 | |
| *** jjardon[m] has joined #buildstream | 17:57 | |
| *** jude has joined #buildstream | 17:57 | |
| gitlab-br-bot | buildstream: issue #69 ("Core and Plugin format revisioning") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/69 | 18:05 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!