| *** violeta_ has joined #buildstream | 07:22 | |
| *** tiagogomes has joined #buildstream | 07:47 | |
| *** violeta_ has quit IRC | 07:57 | |
| *** violeta_ has joined #buildstream | 08:01 | |
| *** jonathanmaw has joined #buildstream | 08:40 | |
| *** tristan has joined #buildstream | 09:01 | |
| jonathanmaw | morning/evening tristan | 09:02 |
|---|---|---|
| tristan | :) | 09:02 |
| jonathanmaw | I'm having a look at https://gitlab.com/BuildStream/buildstream/issues/16, and I'm not entirely sure what to do with it | 09:02 |
| tristan | I had doctors appointment in the afternoon, arriving a bit late | 09:02 |
| jonathanmaw | AIUI, you feel that the script element is inflexible. In what way? | 09:02 |
| tristan | jonathanmaw, right :) | 09:02 |
| tristan | So, you know how BuildElement works right ? | 09:03 |
| jonathanmaw | tristan: not specifically. that's buildstream/element.py? | 09:03 |
| tristan | I recently documented that pretty well actually: https://buildstream.gitlab.io/buildstream/buildstream.buildelement.html#module-buildstream.buildelement | 09:04 |
| tristan | No, buildstream/element.py is the base class for all elements | 09:04 |
| *** ssam2 has joined #buildstream | 09:04 | |
| tristan | And BuildElement is a derived abstract class which we use to drive all the build related elements, like autotools, cmake, meson etc | 09:04 |
| jonathanmaw | ok | 09:05 |
| tristan | jonathanmaw, So for instance if you look at autotools: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugins/elements/autotools.py | 09:05 |
| tristan | you can see the .py basically does nothing | 09:05 |
| tristan | That is not a hard requirement for what I want from a base script element, but the idea is that you can make many different powerful derivations without having to write much code | 09:06 |
| tristan | A lot can be done just in the yaml | 09:06 |
| tristan | i.e. https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugins/elements/autotools.yaml | 09:06 |
| tristan | autotools plugin is mostly just yaml | 09:06 |
| tristan | jonathanmaw, So... I want something like BuildElement, but something that is configurable for running scripts | 09:07 |
| tristan | where one must stage at least one base dependency set at / | 09:07 |
| tristan | But can stage zero...N other bases in other directories below the /buildstream directory | 09:07 |
| tristan | jonathanmaw, So think of the /buildstream directory as special, it's the only place we ever stage anything other than the base runtime | 09:08 |
| tristan | The reason for this special case... is because we have a Sandbox option which allows staging the / read-only | 09:08 |
| tristan | but keeps /buildstream read-write | 09:08 |
| tristan | jonathanmaw, I recently forced (hacked) the script element to not mount root readonly | 09:09 |
| tristan | This is because I also have a use case for running scripts directly in the root, which modify the root | 09:09 |
| tristan | (i.e. with the GNOME multistrap base system, I need to do a conversion step with a script element that runs dpkg --configure -a) | 09:10 |
| tristan | jonathanmaw, So for instance one option for derived plugins, is to decide whether the root should be readonly or not | 09:10 |
| tristan | jonathanmaw, Also, I think it would be good if subclasses of the base script element can decide on domains of commands to run | 09:11 |
| tristan | I.e., like BuildElement has a set of configure-commands, build-commands, install-commands etc | 09:11 |
| tristan | But this would be more abstract | 09:11 |
| tristan | jonathanmaw, the idea behind that is... to write an image deployment, we probably want to break this down into: Commands to make a boot partition bootable (syslinux and suchlike), Commands to generate an fstab, Commands to create the filesystems, commands to create a disk image and splice the created filesystems into it | 09:13 |
| tristan | like you saw with the crazy script I showed you yesterday | 09:13 |
| jonathanmaw | ok | 09:13 |
| tristan | So we probably want sane defaults for this, and as much configuration in the yaml as possible in the base script element | 09:14 |
| tristan | That means that, projects which have some special need dont have to write a python file to crank out a crazy script if they so desire | 09:15 |
| tristan | But at the same time, we want to create specialized derivatives | 09:15 |
| jonathanmaw | so, we want to have a x86-image element that derives from an enhanced script element and by the magic of an associated yaml file already has the default commands | 09:15 |
| tristan | So that we have an included x86-image plugin in buildstream, which we can use instead of that crazy script in projects | 09:15 |
| tristan | Exactly | 09:16 |
| jonathanmaw | and we want to extend the script element to group commands by domains, so that we can say "only do these groups of commands" | 09:16 |
| tristan | Basically we want to repeat the pattern we did with BuildElement, but with a bit more flexibility, so we can crank out plugins like x86-image | 09:16 |
| tristan | Yeah | 09:16 |
| tristan | Maybe for that, the default is to just have "commands", but if a plugin defines a list of command domains, that overrides the default | 09:17 |
| jonathanmaw | and they can of course override all of the commands in an individual domain, which saves them from having to override every single command and basically rewrite it from scratch. | 09:17 |
| tristan | Well | 09:18 |
| tristan | Derivation in buildstream is not that flexible | 09:18 |
| tristan | plugin-wise | 09:18 |
| tristan | jonathanmaw, the ScriptElement base abstract class will have to live beside BuildElement, in the core and not as a plugin | 09:18 |
| tristan | loaded plugins cannot derive from eachother | 09:19 |
| tristan | because we dont have a plugin system that allows inter plugin dependencies | 09:19 |
| tristan | we probably could, but I dont think it's that important | 09:19 |
| ssam2 | another thing buildstream has in common with gstreamer ;-) | 09:20 |
| tristan | heh | 09:20 |
| tristan | jonathanmaw, so anyway, a plugin will probably define all of its default commands, and the command domains it wants to run | 09:20 |
| tiagogomes | Does buildstream also provide insane amounts of colourised debug information? | 09:20 |
| ssam2 | there are colours! | 09:21 |
| tristan | tiagogomes, yes, but we stick to a format color and content color mostly all around | 09:21 |
| tristan | and dont do too much rainbow I think | 09:21 |
| tiagogomes | :) | 09:22 |
| tristan | jonathanmaw, of course, when someone declares an element of that kind (i.e. writes a file.bst), they can then override commands in their yaml as usual | 09:22 |
| tiagogomes | gstreamer allows plugin writers to define a color for the plugin debug | 09:22 |
| tristan | jonathanmaw, actually for reference, the order of priority for how configurations are applied is documented here: https://buildstream.gitlab.io/buildstream/format.html#element-composition | 09:23 |
| jonathanmaw | tristan: ok, so for example, x86-image would have filesystem-arranging-commands, syslinux-config-commands, and partitioning-commands, and "command-domains: [filesystem-arranging, syslinux-config, partitioning]" | 09:24 |
| tristan | jonathanmaw, right, so that's the basic idea yes, _but_ I dont want command-domains itself to be configurable in the yaml | 09:25 |
| tristan | this is mostly a question about presenting a digestible/understandable API | 09:25 |
| tristan | jonathanmaw, So that would be a one liner that the derived ScriptElement would do in python | 09:25 |
| tristan | And the x86-image plugin would then show only a yaml API that makes sense for itself | 09:26 |
| tristan | jonathanmaw, I know this conflicts a bit with what I said about having mostly things configurable in yaml, but we have to make some judgment calls along the way | 09:27 |
| tristan | jonathanmaw, for instance with this, the default of "command-domains" is [ "commands" ] | 09:27 |
| tristan | that means the raw element could be used | 09:27 |
| tristan | that part itself could actually be class data of the plugin | 09:28 |
| tristan | (as opposed to instance data) | 09:28 |
| tristan | jonathanmaw, I'm not sure I understand why you have to call os.makedirs() in bzr->stage() | 10:03 |
| jonathanmaw | tristan: I was getting file not found errors when trying to stage | 10:04 |
| tristan | jonathanmaw, in normal circumstances, your plugin will stage sources in /buildstream/build/<contents of repo> ... i.e. os.makedirs() is getting called on /buildstream | 10:04 |
| tristan | So then your file not found errors were... entirely from your test case ? | 10:05 |
| jonathanmaw | the full path seemed to be buildstream/build/<tmpdir>/buildstream/build, though I'm not sure why. | 10:05 |
| jonathanmaw | this was not in the test case, iirc | 10:05 |
| tristan | Hmmm, this sounds dangerous, I better try it | 10:06 |
| tristan | Do you have that emacs element lying around anymore ? | 10:06 |
| jonathanmaw | https://pastebin.com/3Qn8kdCa | 10:06 |
| jonathanmaw | tristan: so, apart from taking arbitrary command domans, the enhanced scriptelement must also be able to stage elements to arbitrary places as read-only/read-write? | 10:19 |
| jonathanmaw | to the extent that "/" in the stage can be read-write | 10:19 |
| tristan | jonathanmaw, so the sandbox only has an option to mount / readonly or readwrite, and /buildstream is always read-write | 10:23 |
| jonathanmaw | and we can stage multiple things inside /buildstream | 10:23 |
| tristan | jonathanmaw, so it must stage one element at /, asides from that, it should be allowed to stage an arbitrary number of elements in directories under /buildstream | 10:23 |
| tristan | Exactly | 10:24 |
| jonathanmaw | so "base" will take an element and permissions. "input" becomes multiple element-permission pairs | 10:25 |
| jonathanmaw | would they be installed in a subdir based on the element's name, or allow that to be user-defined as well? | 10:25 |
| tristan | That is one way I suppose | 10:25 |
| tristan | but consider this a complete rewrite of the script element | 10:25 |
| tristan | we have one chance to make the best API | 10:26 |
| tristan | jonathanmaw, I think for instance, if we are not going to specify any additional elements to stage (i.e. we are only staging one element at /), then we should not have to specify which of the dependencies is the base one | 10:26 |
| tristan | however, that may not mean imply that the / is readonly (maybe we stage certain dependencies, to write a script which produces something to be collected in /buildstream/install) | 10:28 |
| tristan | The root should be readonly unless otherwise specified (default readonly root is best I think) | 10:28 |
| tristan | jonathanmaw, also note that... in the case that we are only staging things in / and nothing else, then we should be staging Scope.BUILD of self | 10:29 |
| tristan | Otherwise we stage Scope.RUN of a specified dependency | 10:29 |
| jonathanmaw | tristan: would that mean a "dependencies" list, where the first element defaults to installing to "/" and is read-only, and subsequent elements default to "/buildstream/%{name}" and are read-write? | 10:31 |
| jonathanmaw | and we throw a wobbly if after going through all those elements, none of them install to "/" | 10:31 |
| *** tristan has quit IRC | 10:33 | |
| *** tristan has joined #buildstream | 10:42 | |
| tristan | Here I was whining about how bst track emacs.bst was hanging | 10:43 |
| tristan | turns out I lost my connection :-S | 10:43 |
| jonathanmaw | hrm, I commented out the directory creation and tried again, and the problem seems to have vanished | 10:44 |
| jonathanmaw | so I guess I can take that out | 10:47 |
| tristan | I think so, I'm trying to reproduce here but looks like savannah doesnt wanna give me emacs :-( | 10:47 |
| jonathanmaw | it took me about 30 minutes to clone :/ | 10:47 |
| tristan | maybe they know I was bad mouthing their ancient website https://savannah.gnu.org/ | 10:48 |
| tristan | and they've singled me out ! | 10:48 |
| tristan | jonathanmaw, Ok well, one other thing, functionally I can see that you're two functions for the atomic repodir thing... seems they can be collapsed into the context manager | 10:49 |
| tristan | You only call the context manager from one place (which _ends_ with yield) and in that place, you end up calling that other function at the end | 10:49 |
| *** ChanServ sets mode: +o tristan | 10:51 | |
| jonathanmaw | tristan: hrm, I could make _atomic_repodir call _atomic_replace_mirrordir after the yield | 10:52 |
| tristan | jonathanmaw, that's what I was observing yeah, looks like they belong together | 10:52 |
| jonathanmaw | I think I did it that way because _atomic_repodir could in principle be used for non-mirroring tasks, but it looks like it's too specific here to actually warrant that. | 10:53 |
| jonathanmaw | so I'll move the _atomic_replace_mirrordir | 10:53 |
| tristan | Ok | 10:54 |
| jonathanmaw | I'm not in favour of merging the functions because I'd find it harder to read, what with all the exceptions going on | 10:54 |
| tristan | yeah, sometimes it's better to name a function for readability even if you're not going to reuse it, I agree | 10:54 |
| gitlab-br-bot | buildstream: merge request (jonathan/bzr-source->master: Jonathan/bzr source) #19 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/19 | 11:06 |
| tristan | jonathanmaw, ok I'm just going to wait... have been wasting time looking for a more efficient repository than savannah | 11:14 |
| tristan | Trying to clone emacs and bzr itself (from launchpad) in parallel now | 11:14 |
| tristan | anyway, sorry please dont give too much thought to bzr anymore, I expect it'll work I just wanna see it and feel it myself | 11:14 |
| jonathanmaw | ok | 11:15 |
| tristan | Also, I think there is another bug with bzr but I'll try to figure it out, it's a tough one | 11:15 |
| tristan | Seems to consistently behave badly with cntl-C, I think it's my fault | 11:15 |
| tristan | weird, bzr seems to be doing something in my system /tmp | 11:19 |
| tristan | ls -l /tmp/bzr-index-ClGP4J 3.7M | 11:19 |
| tristan | downloading at 300K, and du -hs the trunk dir from bzr branch still says 20K | 11:20 |
| tristan | Oh, it's going into .bzr *beside* trunk/ | 11:21 |
| tristan | Ok, something is happening | 11:21 |
| jonathanmaw | tristan: do we need to maintain backwards-compatibility for existing script elements? | 11:36 |
| tristan | jonathanmaw, No. | 11:48 |
| tristan | absolutely not, the current is garbage | 11:48 |
| tristan | there are 2 use cases and we'll replace them | 11:49 |
| jonathanmaw | tristan: ok, and if the definitions->bst importer creates an script elements, it's up to them to get it right now. | 11:52 |
| tristan | jonathanmaw, it's up to us to fix that after | 12:05 |
| tristan | there are two, and actually in neither case does the importer generate that | 12:06 |
| tristan | it's static data | 12:06 |
| jonathanmaw | tristan: Do we want script elements to always collect output from the %{install-root} directory, or do we want to be able to override it and specify something else? | 12:25 |
| tristan | jonathanmaw, we definitely want to be allowed to collect from any location | 12:25 |
| tristan | BUT | 12:25 |
| tristan | I have been pondering this a bit | 12:25 |
| tristan | jonathanmaw, and I think that the two are not mutually exclusive | 12:26 |
| tristan | I.e. we can say Yes: Output is always collected from '%{install-root}' | 12:26 |
| tristan | And we can also say Yes: You can always set the value of install-root in your variables section of your element. | 12:26 |
| tristan | The only question here is, is this good API ? | 12:27 |
| jonathanmaw | tristan: the script element currently has config.collect, which defaults to "%{install-root}". I think I'd rather override collect than mangle install-root | 12:30 |
| tristan | Not sure, I think I have scrutiny for using %{install-root} overrides, but not for the same reasons | 12:33 |
| tristan | First, I think it's perfectly valid that an element author should set variables: sections to override default functionality | 12:33 |
| tristan | However, I'm just not convinced that using the word 'install-root' here makes sense; since install-root is mostly used by the build element as a make install location | 12:34 |
| tristan | That said, I think I like the current approach for that in the current script element | 12:35 |
| tristan | To default to %{install-root}, but then; it does seem a bit superficial that we have a parameter for it | 12:35 |
| tristan | jonathanmaw, so there is also another principle here to consider; which is, allowing multiple methods of achieving the same thing | 12:36 |
| tristan | Which is something I tend to dislike as well | 12:36 |
| tristan | jonathanmaw, if we go with the current approach, then it becomes possible to override install-root variable, in order to achieve the same thing as overriding 'collect' in the config: section | 12:37 |
| tristan | which raises the question, why did we add that configuration in the first place ? | 12:37 |
| tristan | I dont know... ssam2 have any thoughts about this ? | 12:37 |
| jonathanmaw | hmm. I was uncertain whether overriding install-root would actually work that way | 12:38 |
| tristan | I'm tending towards actually preferring avoiding adding additional config: parameters when the same can be achieved by setting a variable | 12:38 |
| ssam2 | tristan, I've not thought about this deeply at all. I think 'collect' is a slightly confusing concept though | 12:38 |
| ssam2 | non-intuitive, at least. it's not really confusing once you figure out what it means | 12:39 |
| tristan | variables work all the way down to the project configuration (if declared there) | 12:39 |
| tristan | jonathanmaw, so something with a default in the project, can be overridden by a default on the element type, it can then be overridden again for all "foo" elements of a given project in the project.conf (at the end), and the final word is given to the element declaration (the foo.bst file) itself | 12:40 |
| tristan | actually jonathanmaw... I recently changed some of the build elements with something interesting... | 12:42 |
| tristan | jonathanmaw, see how 'conf-extra' works in https://buildstream.gitlab.io/buildstream/elements/autotools.html#module-elements.autotools for example | 12:42 |
| tristan | This was basically done so that a project can add project wide configure flags, which do not conflict with element specific stuff, but an element can itself also just set conf-extra to add a configure option (but that would override any project wide setting) | 12:43 |
| tristan | jonathanmaw, ok lets go ahead with just stating that output is collected from 'install-root', which a user can set | 12:44 |
| tristan | we have some months ahead of us before hardening, if we find something horrible about it, we'll change that (but hope we dont have to) | 12:45 |
| jonathanmaw | looking at buildelement, they have the command-subdir variable for where to start the command from. I think we would benefit from doing that, too. | 12:46 |
| * tristan notes that after a first stable release, we'll have to define a release cycle and let plugin API additions land early in the cycle to allow API churn in master before releasing stable | 12:46 | |
| tristan | yes, that is a good point too | 12:47 |
| tristan | jonathanmaw, I wonder if we may want that even on a per command-domain basis | 12:47 |
| tristan | having to write multi-line commands starting with 'cd foo' is obnoxious | 12:48 |
| jonathanmaw | ok. with the working directory defaulting to %{build-root} (I have no idea what build-root should sensibly be in a script element though), and definable for every command-domain | 12:49 |
| tristan | maybe per-command-domain CWDs is overkill, not sure :) | 12:49 |
| jonathanmaw | Defining it in yaml gets trickier | 12:49 |
| tristan | indeed | 12:50 |
| jonathanmaw | I think I'll go with working dir being a per-element thing for now | 12:50 |
| jonathanmaw | tristan: what does %{build-root} usually get set to? | 12:50 |
| jonathanmaw | iirc in baserock it was /%{name}.build | 12:51 |
| tristan | Yes, also I sort of feel like %{build-root} is not sensible for script elements | 12:51 |
| tristan | %{build-root} is /buildstream/build | 12:51 |
| tristan | But /buildstream/build exists for building sources | 12:51 |
| tristan | So far, I think we dont allow sources in script elements | 12:51 |
| tristan | And I think that is quite OK | 12:51 |
| tristan | For manual building of sources, we have the 'manual' build element which should provide everything we need | 12:52 |
| tristan | And we can always use a 'compose' element or 'import' element if we want to combine with 'script' | 12:52 |
| jonathanmaw | So for the script element, the default working directory should be "/", and be overridable | 12:53 |
| jonathanmaw | and with any luck, we can ignore the existence of %{build-root} altogether. | 12:53 |
| tristan | Sure, it could be '/' by default, or, it could be: '/' by default if no additional elements are staged (and no /buildstream directory gets created) | 12:53 |
| tristan | Otherwise it could default to /buildstream when other elements get staged "somewhere below that" | 12:54 |
| tristan | But maybe a static default is easier and even more comprehensive | 12:54 |
| tristan | That said, it could still be defined as a variable instead of something collected form the config: section | 12:55 |
| jonathanmaw | tristan: yep. in buildelement, it's the variable command-subdir. | 12:55 |
| tristan | jonathanmaw, right, and iirc it's called 'subdir' because its a subdirectory of %{build-root} (not an absolute path) | 12:57 |
| jonathanmaw | tristan: so, "command-dir" then? | 12:57 |
| tristan | or cwd, even ? | 12:58 |
| tristan | current-working-directory is damn lengthy, but everyone understands cwd (I think) | 12:59 |
| tristan | then again, command-dir is also fine, or work-directory | 12:59 |
| tristan | just throwing around ideas :) | 12:59 |
| jonathanmaw | I think we can get away with "cwd". AIUI, these variables don't get exported as environment variables, so shell commands won't get confused by the environment variable CWD, or the command cwd | 13:01 |
| tristan | No they are not... actually we have environment: section for that :) | 13:03 |
| tristan | The environment dictionary is what ultimately gets set into the environment, and it's values are substituted with the elements variables, too | 13:04 |
| tristan | Ha ! | 13:09 |
| tristan | jonathanmaw, Ok I've been debugging bzr, and I found it ! | 13:09 |
| tristan | mouahahahaha | 13:09 |
| jonathanmaw | erm | 13:10 |
| tristan | Ok I'll fix it myself | 13:10 |
| tristan | you will first facepalm, then you have permission to blame me :) | 13:10 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 6 commits (last: Add bzr source plugin) https://gitlab.com/BuildStream/buildstream/commit/4653feb0c6b63dc97cadaf8c2a593664f990076f | 13:16 |
| gitlab-br-bot | buildstream: merge request (jonathan/bzr-source->master: Jonathan/bzr source) #19 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/19 | 13:16 |
| tristan | jonathanmaw, https://gitlab.com/BuildStream/buildstream/commit/92eab614c940677a64fc4d911b2438a7c2fe5f5a | 13:17 |
| jonathanmaw | wow | 13:17 |
| tristan | yeah | 13:18 |
| tristan | Now I think we need better error tracking in the core for this, those stack traces were hard to follow | 13:18 |
| gitlab-br-bot | buildstream: issue #9 ("Enhancement: bzr source") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/9 | 13:23 |
| jonathanmaw | tristan: are these new dependencies going to have to live in config (and essentially be duplicates of the "dependencies" section earlier on but with more metadata), or can we put it in the root "dependencies" section? | 13:24 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Docs Makefile: Dont generate plugin skeletons for __init__.py) https://gitlab.com/BuildStream/buildstream/commit/c4b5f665bedd1816e2849d899b7d08061f8496cf | 13:25 |
| tristan | jonathanmaw, thats a bit of a constraint for us... we have to have dependencies in the regular place | 13:27 |
| tristan | <tristan> jonathanmaw, I think for instance, if we are not going to specify any additional elements to stage (i.e. we are only staging one element at /), then we should not have to specify which of the dependencies is the base one | 13:28 |
| tristan | I.e. hence that comment from earlier on | 13:28 |
| jonathanmaw | ok. perhaps we should use a different name than "dependencies" for them then. | 13:29 |
| jonathanmaw | "stages"? | 13:29 |
| tristan | jonathanmaw, basically the core / loader will resolve the dependency graph long before the plugin is instantiated and such | 13:29 |
| tristan | Yeah in the config we need a good name | 13:29 |
| tristan | each of them need to specify some things... like A.) What directory under /buildstream to put them at... B.) the element name itself... | 13:30 |
| tristan | jonathanmaw, I guess it's fair to say that, if the directory is not specified, then it is the root dependency ? | 13:31 |
| tristan | And if none at all are specified, then we just stage the Scope.BUILD dependencies of the self element at the root ? | 13:31 |
| tristan | Nice, looks like the CNTL-C (suspend/resume) functionality is fine with bzr, as long as the other bug was fixed | 13:33 |
| jonathanmaw | So, they default to just using the depends in the order they're given (first one is read-only /) | 13:33 |
| jonathanmaw | but can be overridden with greater detail in config | 13:33 |
| tristan | jonathanmaw, I dont believe I follow what you mean | 13:35 |
| tristan | jonathanmaw, I mean that... in the case that you dont specify any elements to stage in subdirs, as a special case, we just stage all the dependencies of self at / | 13:35 |
| tristan | otherwise, you have to specify an element for each thing that you stage, including the thing you stage at / | 13:36 |
| tristan | jonathanmaw, (note that there is no such thing as an order of things specified in depends) | 13:37 |
| tristan | yes it is a list, but that is superficial | 13:37 |
| jonathanmaw | hrm | 13:37 |
| tristan | they are resorted by dependency at load time | 13:37 |
| tristan | so the order has no meaning, except that when iterating over self.dependencies(), you can be sure that the first element does not depend on the second, etc | 13:38 |
| jonathanmaw | ok. So default with multiple elements in depends is to stage them all to /, and / is read-only. | 13:39 |
| tristan | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L631 | 13:39 |
| jonathanmaw | But if we want more, we define it in config | 13:39 |
| tristan | Yes | 13:39 |
| tristan | jonathanmaw, the thing is, there is no way currently to stage more than one element to a given directory | 13:40 |
| tristan | because then the whole dependency resolving thing needs to be rerun, in that case the user need just provide a single element which depends on others; to stage in a subdir | 13:41 |
| jonathanmaw | Do we still want to try and guess at sensible defaults when it's being overridden with config? It's getting complex to explain in the docstring, so I think it's going to be similarly complex to implement | 13:41 |
| tristan | ok hold on... | 13:42 |
| tristan | I think it's not that complex | 13:42 |
| tristan | But looking at this should be more telling: https://gitlab.com/BuildStream/buildstream-tests/blob/gnome-conversion-base/elements/base-configure.bst | 13:43 |
| tristan | jonathanmaw, So basically, that script is pretty straight forward, and I suspect similar things could happen; like just some hack someone wants to run before deploy | 13:44 |
| tristan | jonathanmaw, I feel like specifying base-system.bst twice in that case is redundant | 13:45 |
| tristan | especially since we're not staging anything else | 13:45 |
| jonathanmaw | tristan: ok. For cases where there are more than one element in depends, we can't make the same guess, so those will always require some amount of overriding | 13:48 |
| jonathanmaw | given it inherits element, I should be able to call self.dependencies(scope, recurse=False) | 13:55 |
| tristan | jonathanmaw, I dont think that is the case, conversely, in cases that nothing is specified for staging things | 13:55 |
| jonathanmaw | if I can figure out where scope comes form | 13:56 |
| tristan | so ok, lets have a name for that | 13:56 |
| tristan | I dont like "stages" really | 13:56 |
| tristan | sounds like parts of a process | 13:56 |
| jonathanmaw | depends-layout? | 13:56 |
| tristan | yeah, or just "layout" | 13:57 |
| tristan | So in the case that no "layout" is specified, we stage Scope.BUILD dependencies of self in / | 13:57 |
| tristan | Otherwise we stage Scope.RUN dependencies in every place that is specified in the "layout" | 13:57 |
| tristan | https://buildstream.gitlab.io/buildstream/buildstream.element.html#buildstream.element.Scope | 13:59 |
| tristan | That is scope | 13:59 |
| jonathanmaw | which is defined in the yaml by giving the depends the type "build" or "run"? | 13:59 |
| tristan | Not exactly no, however those are intrinsically linked | 13:59 |
| tristan | jonathanmaw, Scope defines a scope of an element, while the "build" or "run" attributes of a dependency of an element define the nature of that dependency relationship specifically | 14:00 |
| tristan | So, those are dependency types, explained here: https://buildstream.gitlab.io/buildstream/format.html#dependency-types | 14:01 |
| tristan | By default an element is both a runtime and build dependency | 14:01 |
| tristan | Which means: I need this element in order to run, and I need this element in order to build | 14:01 |
| tristan | jonathanmaw, The Scope rather gives us a decision of what dependencies to pull in, depending on their types | 14:02 |
| jonathanmaw | So we'd expect a gnome-system script to have in its depends "deploy-base" with type "build", and "gnome-system" with type "run", and have layout say what to do with gnome-system? | 14:02 |
| tristan | Not really; note that in the current script element I have disallowed at preflight time; any elements which are not explicitly only build type dependencies | 14:03 |
| tristan | I think that is going to continue to make sense | 14:04 |
| tristan | jonathanmaw, can you think of a case; where a script element's output artifact still requires that script element's dependencies afterwards ? | 14:04 |
| tristan | In both known cases, the dependencies of the script element itself must be non transitive | 14:05 |
| jonathanmaw | ok | 14:05 |
| jonathanmaw | I think I've gotten confused somewhere. You think we should stage Scope.BUILD dependencies to / when layout isn't specified. And stage Scope.RUN dependencies according to layout, when it is specified. | 14:08 |
| jonathanmaw | so would that mean that base-configure would have all their depends with type build, and gnome-system-image would have all their depends with type run? | 14:08 |
| persia | Libraries for interpreted languages are a good case where a given element may be both a build-dependency and a runtime-dependency (well, for languages with less specialness. perl, shell, etc. Less so python, because specialness). | 14:14 |
| persia | The moreso for build processes that include `make test` or equivalent. | 14:15 |
| tristan | jonathanmaw, There is a distinction here you missed I think... when we stage Scope.BUILD dependencies to / as a fallback/default... it is the dependencies of _self_ | 14:16 |
| tristan | jonathanmaw, whereas when we stage specific things into a layout, it is Scope.RUN dependencies of _those specified elements_ | 14:17 |
| jonathanmaw | so self is the element, e.g. gnome-system-image.bst, and the Scope.BUILD dependencies of it are deploy-base.bst and gnome-system.bst. | 14:19 |
| jonathanmaw | when we have "layout", each element specified in that layout has its Scope.RUN dependencies staged? | 14:20 |
| tristan | jonathanmaw, ummm, ok so YES to the second thing | 14:21 |
| tristan | jonathanmaw, to the first thing... The Scope.BUILD dependencies of gnome-system-image.bst are what you said, and _additionally_ the transient runtime dependencies of those element | 14:22 |
| tristan | s | 14:22 |
| tristan | jonathanmaw, lemme find a better example... | 14:22 |
| tristan | jonathanmaw, https://gitlab.com/BuildStream/buildstream-tests/blob/build-gnome/elements/gtk3/gtk3.bst | 14:23 |
| tristan | jonathanmaw, ok so gtk3.bst in that tree... has 3 dependencies, which are both runtime/build (default) | 14:24 |
| tristan | jonathanmaw, the Scope.BUILD of gtk3.bst is all the dependencies which _must be staged_ in order to build gtk3 | 14:24 |
| tristan | jonathanmaw, that means, gtk-deps.bst, wayland-generic.bst and x-common.bst... PLUS all of the "runtime" dependencies of those, recursively | 14:25 |
| jonathanmaw | ok, so dependencies of type "build" won't be picked up as part of recursive scoping | 14:27 |
| tristan | jjardon[m], This is the cause of the gitlab build failures btw: https://github.com/ostreedev/ostree/issues/633 I'm pretty sure, now why would hardlinks fail I have no idea, but they seem to also on my aarch64 build machines | 14:28 |
| tristan | jonathanmaw, exactly, they are non transitive | 14:28 |
| tristan | jonathanmaw, well, that is to say, if they are build _only_, think of build & runtime as bits in a bitmask | 14:28 |
| jonathanmaw | so in gnome-system-image, we explicitly build-depend on gnome-system and deploy-base. We also implicitly depend on all of their runtime dependencies. | 14:29 |
| jonathanmaw | which means they'll be present, so we can have "layout" use them | 14:30 |
| tristan | jonathanmaw, Yes, but by explicitly depending on them as "build _only_" it means that anything that depends on gnome-system-image artifact, wont carry in it's dependencies | 14:30 |
| jjardon[m] | thanks for the link tristan | 14:31 |
| * jjardon[m] subscribes | 14:31 | |
| *** tiagogomes has quit IRC | 14:35 | |
| jonathanmaw | tristan: do we allow the Scope.BUILD dependencies of self to be defined in layout as well (in cases like gnome-system-image, which depend on deploy-base and gnome-system)? | 14:35 |
| tristan | jonathanmaw, I think not | 14:35 |
| jonathanmaw | I'm not sure how something like gnome-system-image would be set up, then. A deploy-base that depends on all the things you want to include in layout? | 14:37 |
| tristan | jonathanmaw, in gnome-system-image, we stage deploy-base.bst in / and we stage gnome-system.bst in a subdir | 14:37 |
| tristan | Ok so, there is no wiggling your way out of the fact of life, that an elements dependencies must be resolved waaay before an element plugin comes along and knows what to do with the dependencies | 14:38 |
| tristan | So in the case of an element which says "I want to stage this standard debian base system with some host tools I know will work, which I can use to wrap up a system I build which is staged in /buildstream/input into a booting image" | 14:40 |
| tristan | I.e. remember that these are _two different systems_ | 14:40 |
| tristan | You dont want to overlap them and ever stage the Scope.BUILD dependencies of self, as they will inevitably contradict eachother | 14:40 |
| tristan | However, you still _need_ the "depends:" to be defined at load time, so that when it comes time to assemble the element, you know those deps are built/cached and available to work with | 14:41 |
| jonathanmaw | tristan: yep, so they can't both be staged at /, that'd be completely unhelpful. | 14:42 |
| tristan | jonathanmaw, right, except for the case where you want to script something without staging things in different places | 14:42 |
| tristan | I.e. something that just runs some stuff on / and outputs a modified / | 14:43 |
| jonathanmaw | tristan: that'd be one thing staged as "/", or a bunch of things staged carefully so that they're laid out exactly as you want. | 14:44 |
| jonathanmaw | (for when you're outputting a modified /) | 14:44 |
| tristan | Right, except when you say "one thing" staged as "/", you may be talking about multiple dependencies | 14:44 |
| tristan | That is okay | 14:45 |
| tristan | because the Scope.BUILD dependencies of 'self' are resolve | 14:45 |
| tristan | d | 14:45 |
| tristan | It is always impossible, of course, to ever stage the Scope.RUN dependencies of self in an assemble method of self | 14:45 |
| tristan | because self has not yet produced an artifact | 14:45 |
| jonathanmaw | tristan: by 'one thing staged as "/"' I was referring to the likes of base-configure, which only depends on one element | 14:47 |
| jonathanmaw | a bunch of things staged carefully would be like gnome-system-image | 14:47 |
| tristan | jonathanmaw, yes exactly, I'm just remarking that base-configure only happens to declare one dependency, but it could depend on multiple | 14:49 |
| tristan | it would still be "the Scope.BUILD dependencies of self in /" | 14:49 |
| jonathanmaw | tristan: ok, base-configure has one _declared_ dependency, but also all the implicit dependencies of it as well, which all end up in / | 14:51 |
| tristan | right | 14:51 |
| jonathanmaw | gnome-system-image, however, declares gnome-system and deploy-base, which cannot both be staged directly to / | 14:51 |
| tristan | jonathanmaw, the reason why this distinction I'm making is significant, is because the way that the multiple dependencies declared on self are resolved is significant | 14:52 |
| tristan | I.e. it is impossible to reliably say "stage foo and bar here" | 14:52 |
| tristan | because without having a single element, there is no resolved deterministic staging order | 14:52 |
| tristan | that is done in the loader, before anything else | 14:52 |
| jonathanmaw | right | 14:52 |
| jonathanmaw | so for staging, depends does not guarantee any ordering, so it only gives us useful information when we have only one declared dependency | 14:53 |
| tristan | depends does not decide the order | 14:53 |
| tristan | right | 14:54 |
| tristan | you can reorder your depends as much as you like: same cache key, same staging order | 14:54 |
| jonathanmaw | We seem to be disagreeing on what to do when there's multiple declared dependencies. I think in that case we should stage them according to "layout" | 14:54 |
| tristan | Right, and that is incorrect, the semantic is backwards | 14:55 |
| tristan | If there is a layout defined, then you must decide what to put where, including / | 14:55 |
| tristan | If there is no layout, then you get all the Scope.BUILD dependencies staged in / | 14:56 |
| tristan | jonathanmaw, basically, you are proposing that base-configure can only ever depend on one element | 14:56 |
| tristan | But I dont think that is required to be the case | 14:56 |
| tristan | jonathanmaw, Another thing which may not have been clear | 14:58 |
| tristan | jonathanmaw, you can have only one element listed in "depends:" but you can _still_ have a "layout" | 14:59 |
| tristan | jonathanmaw, for example, you might only need to stage core.bst to have the tools you need in /, in order to do something with a full system in /buildstream/input | 15:00 |
| jonathanmaw | tristan: so, in cases like base-configure, stage everything in depends to /, assuming that they know what they're doing if they have multiple declared depends. In cases like gnome-system-image, there are still multiple declared depends, but how they get staged is defined by layout, right? | 15:00 |
| tristan | Because core.bst can be found somewhere in the dependencies of devel-system.bst (for example), it can still be specified | 15:00 |
| tristan | right, my last comment sort of clarifies this I think | 15:01 |
| tristan | I have to leave this closing coffee shop at present midnight | 15:01 |
| jonathanmaw | tristan: I think I'm getting confused because you said only runtime depends of the declared dependencies can be in layout | 15:02 |
| jonathanmaw | which, aiui, excludes the declared dependencies, themselves | 15:02 |
| tristan | But basically: If the "layout" is there it will define what element (found in self's dependencies, somewhere) to stage as Scope.RUN at which location; _otherwise_ in the absence of a "layout", default is to stage Scope.BUILD of _self_ (in other words, everything) in / | 15:02 |
| tristan | jonathanmaw, I said "staged as Scope.RUN", not runtime-only dependencies :) | 15:03 |
| jonathanmaw | i.e. includes dependencies of type "all", or am I missing something else? | 15:04 |
| tristan | Ok... stop for a moment, I have to go anyway | 15:04 |
| tristan | Without thinking of all of this, please digest this: https://buildstream.gitlab.io/buildstream/format.html#dependency-types | 15:04 |
| tristan | Which is about the types of dependencies an element can have | 15:04 |
| tristan | And then after, digest this: https://buildstream.gitlab.io/buildstream/buildstream.element.html#buildstream.element.Scope | 15:05 |
| tristan | Which is about how we iterate over a set of dependencies | 15:05 |
| tristan | Understand the difference, and by then I will be reconnected from home :) | 15:05 |
| *** tristan has quit IRC | 15:08 | |
| jonathanmaw | aha, an element's run scope includes itself | 15:11 |
| *** tristan has joined #buildstream | 15:41 | |
| tristan | jonathanmaw, "aha, an element's run scope includes itself" yes that is an important detail :) | 15:42 |
| tristan | you need the element you want to run, if you intend to run it | 15:42 |
| tristan | Sorry I stopped off to pick up some bbq chicken | 15:43 |
| tristan | anyway it's pretty late here, I hope things are making a bit more sense, and if there are any suggestions that can help to make dependencies and element scope more clear, I'd be happy to hear them :) | 15:44 |
| tristan | since the clock is quickly ticking on a hardening which there is no turning back from | 15:45 |
| tristan | Note also that many of the frontend commands also have the concept of scope `bst show --deps <scope>`, `bst track --deps <scope>` | 15:46 |
| tristan | for consistency I want to make `bst fetch` also have that, `bst build` is a bit special, though | 15:46 |
| tristan | Then `bst shell` has --scope, but maybe that API should be changed since the semantics of running a shell to build or running a shell to launch some program is a bit different than operating on a pipeline | 15:47 |
| tristan | (for instance the sandbox environment must be much weaker for running gui programs from a bst shell, than when trying to build in a clean environment) | 15:48 |
| jonathanmaw | tristan: yep, I'm getting my head around it now | 15:52 |
| *** jonathanmaw has quit IRC | 17:01 | |
| *** ssam2 has quit IRC | 17:50 | |
| *** tristan has quit IRC | 20:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!