| tristan | juergbi, we exposed it because we needed to allow plugins to manipulate filesystem trees, decide where where to put files, etc | 03:40 |
|---|---|---|
| tristan | i.e. a plugin is allowed to say "take this file from this artifact and put it there in the sandbox" | 03:40 |
| tristan | A plugin was never allowed to modify content or attributes of files | 03:40 |
| tristan | We did have this conversation years ago, jjardon and valentind when they came up with this plugin, it's a very long time ago | 03:41 |
| tristan | I did point out that the plugin should not be modifying this stuff... | 03:42 |
| tristan | And yes, adding the vdir stuff would have been the perfect opportunity to make it impossible to exploit this availability of the directory on disk | 03:42 |
| tristan | Now that we've added these APIs, I definitely think we need to remove them before 2.0, and I'll raise a thread about this | 03:44 |
| tristan | And yes of course the crawling of the hierarchy and invoking non Element/Source APIs is separate, I noticed this bug a long time later, after bst-plugins-experimental was created | 03:45 |
| *** tristan has quit IRC | 04:56 | |
| *** tristan has joined #buildstream | 06:36 | |
| *** ChanServ sets mode: +o tristan | 06:36 | |
| tristan | juergbi, https://mail.gnome.org/archives/buildstream-list/2020-June/msg00017.html, https://lists.apache.org/list.html?dev@buildstream.apache.org | 06:38 |
| tristan | Issue raised on the ML now :-S | 06:38 |
| juergbi | tristan: ta, further 'experimental' plugins are affected, though. bazelize, collect_integration, dpkg_build, dpkg_deploy, flatpak_image and tar_element all also create files in the sandbox directory, in addition to collect_manifest and oci | 06:47 |
| juergbi | i.e., almost every bst-plugins-experimental plugin that isn't just a BuildElement with a yaml file violates this | 06:48 |
| tristan | dpkg_build & dpkg_deploy as I recall they were implemented initially, did not write this stuff directly to the sandbox | 06:52 |
| tristan | Crap maybe dpkg_deploy does ? | 06:52 |
| juergbi | dpkg_deploy creates debian's control file, package-scripts (postinst etc.) and md5sums files | 06:53 |
| juergbi | same for dpkg_build | 06:53 |
| juergbi | ah no, dpkg_build reads them | 06:53 |
| tristan | Yeah | 06:53 |
| tristan | I think we should be able to expose plugins to code which runs in a sandbox and that should fix basically all of these problems | 06:54 |
| tristan | In fact, plugins can already do this by setting up the environment | 06:54 |
| tristan | That is a bit less than ideal though when it comes to lists | 06:54 |
| tristan | It would be better if the core had a way to guarantee that such public data being exposed in some way to a sandbox, is stably sorted and consistent across buildstream versions / execution environments | 06:55 |
| tristan | As it stands, I bet that the outputs of plugins like collect_manifest are already not reproducible due to python's dictionary randomization | 06:56 |
| juergbi | didn't Python make dict ordering stable a while ago? | 06:57 |
| tristan | It did the opposite | 06:57 |
| juergbi | guaranteed to be insertion order, iirc | 06:57 |
| tristan | It used to be stable, and then it was intentionally made random in order to ensure people caught bugs where they relied on ordering | 06:57 |
| tristan | That happened near the beginning of BuildStream | 06:57 |
| tristan | maybe it became random in 3.5, while in 3.4 is was stable | 06:58 |
| juergbi | tristan: " | 06:58 |
| juergbi | Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6." | 06:58 |
| juergbi | since 3.7 it's explicitly guaranteed | 06:59 |
| tristan | Changed back again | 06:59 |
| juergbi | however, effectively it's that way since 3.6 | 06:59 |
| juergbi | which is our minimum version | 06:59 |
| tristan | Well, clearly it's not rational to depend on python to be consistent | 06:59 |
| juergbi | no, unfortunately not | 06:59 |
| tristan | juergbi, We were using OrderedDict for that at some point fwiw | 07:00 |
| tristan | near the close of 3.4 support and adding 3.5 support | 07:01 |
| juergbi | tristan: how do you envision plugins executing code in the sandbox? I don't think we have a good story for this currently | 07:01 |
| juergbi | I definitely like going into that direction | 07:01 |
| tristan | Right now it means running commands in the sandbox | 07:01 |
| juergbi | I mean, code that is essentially part of the plugin | 07:01 |
| tristan | Like autotools BuildElement does, or cmake or whichever | 07:02 |
| juergbi | an element plugin in a bst project-independent repo could place any code in the sandbox | 07:02 |
| tristan | Further than that, there are some proposals on the table already | 07:02 |
| tristan | One of them being default dependencies for element kinds | 07:02 |
| tristan | Another one being a recent idea tied into composite plugins | 07:02 |
| tristan | Which is somewhat related | 07:02 |
| tristan | I.e. it would be nice to consider some plugins to be "Bound to the project they originate from", if you are using cross-junction elements | 07:03 |
| juergbi | yes, something along that line | 07:04 |
| tristan | This was the idea for the BSP composite plugin thread, i.e. "Give me a freedesktop-sdk BSP element to use in my project, which itself internally depends on all of the artifacts it uses for deployment" | 07:04 |
| tristan | And it would be a composite plugin which would do all of the initrd, filesystem root composition, and syslinux bootability stuff with multiple elements internally | 07:04 |
| tristan | Of course that would be an end result of a few different developments | 07:05 |
| tristan | But until that happens, plugins just have a dependency on tools existing in the sandbox, and project authors need to provide those tools | 07:06 |
| tristan | (that's kind of always been the case) | 07:06 |
| tristan | I recall in the early days we discussed preflighting in the sandbox similar to host tooling; i.e. allow build elements to fail with more interesting error messages when a tool they depend on does not exist in the sandbox | 07:07 |
| tristan | (not very preflight but still an interesting assertion to have) | 07:07 |
| tristan | The story of implicit plugin dependencies on elements is a tricky one on it's own to solve - for autotools/cmake etc, we want to run those tools staged into the sysroot of the payload we're building | 07:09 |
| tristan | When considering implicit plugin dependencies for deploying FS images, we don't care at all about that and such bindings could be less error prone | 07:09 |
| tristan | For autotools/cmake etc the idea was always to have the project authors define that once in project.conf (just to save some typing in the project elements) | 07:10 |
| *** tristan has quit IRC | 07:21 | |
| *** tristan has joined #buildstream | 07:26 | |
| *** ChanServ sets mode: +o tristan | 07:26 | |
| *** lantw44 has quit IRC | 16:08 | |
| *** toscalix has joined #buildstream | 19:17 | |
| *** toscalix has quit IRC | 21:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!