| *** juergbi has quit IRC | 06:16 | |
| *** tristan has joined #buildstream | 07:25 | |
| *** ChanServ sets mode: +o tristan | 07:43 | |
| *** juergbi has joined #buildstream | 07:56 | |
| tristan | sigh... so I think this image deployment is going to take more thought | 09:33 |
|---|---|---|
| tristan | I've been brute forcing for a little while and things have went well, but sometimes you need to take a day and let things sink in, before you make some irreversible design disasters | 09:34 |
| tristan | In case some inspiration can come my way, things I need to figure out are, how to deal with elements which dont produce an artifact, and how to communicate parameters to those elements | 09:35 |
| tristan | from outside of regular build configuration (i.e. where to deploy to, or if you want to set a default root password, or include some ssh key in an image deployment) | 09:36 |
| tristan | Also it would be nice to anchor multiple deployments to a target, so that in a single command you might deploy multiple things, but to make that dependency on non-artifact producing elements | 09:37 |
| paulsher1ood | fwiw: deploying multiple things was/is handled in morph/ybd ... they are clusters in definitions | 09:48 |
| paulsher1ood | for secrets/passwords in practice i had to add support for passing them in as environment variables (from, say, ci pipeline control environment) | 09:50 |
| tristan | paulsher1ood, yeah I know this and looking at it, my worry is mostly about API design and convenience | 10:00 |
| tristan | looking at the extensions, they are a bit rigid, for instance they _only_ do btrfs | 10:00 |
| tristan | and how to do this in a pipeline scenario, I think an "anchor" element might be interesting; i.e. just a sort of noop which anchors your pipeline (I'm thinking, how to do a flatpak deployment, which is more like 3 or more separate deployments) | 10:01 |
| tristan | the secrets and passwords, and any configuration which is not a constant but an argument for a deployment, is a trick about providing the most convenient interface for users | 10:02 |
| tristan | i.e. can we avoid having the user create local configuration files which live outside of the bst project ? | 10:03 |
| tristan | Can we avoid environment variables ? | 10:03 |
| *** ssam2 has joined #buildstream | 10:04 | |
| tristan | so I can just do: `bst build gnome-deploy.bst --root-password pony` ? | 10:04 |
| tristan | or whatnot | 10:04 |
| tristan | of course its a bit mind numbing, but the result needs thought if we want something convenient and great for the long term | 10:05 |
| tristan | anyway for the actual juice of what a deployment *does*, this can be quite simple; I'm thinking first to start with a general purpose scripting element which A.) Lets you stage tools in '/' B.) Lets you stage what you want to manipulate in /buildstream/install C.) Lets you run scripts | 10:07 |
| tristan | to modify the /buildstream/install stuff and mutate the output | 10:07 |
| tristan | with that only, its very easy to test creating bootable images using scripts, and from that point, you can create a nice parameterized deployment element | 10:08 |
| tristan | which asks, what size of image you want to deploy, what rootfs filesystem type to use, what kind of bootloader to user, etc | 10:08 |
| tristan | and partitioning scheme configuration | 10:08 |
| tristan | as a side thought, it could be interesting to create a graphical tool to design pipelines at some point :) | 10:11 |
| * tristan ponders one possibility... instead of having elements which do not get cached; produce cache keys for deployments and cache them, but have them "tainted" so that they are never shared with remote artifact caches | 10:16 | |
| * tristan already has some other uses for "tainted" artifacts | 10:17 | |
| ssam2 | mmm, that would be useful for dynamically linked bootstrap artifacts too | 10:17 |
| tristan | this way, you never produce the same deployment twice (i.e. when deploying with same ssh key or such), but they are safely local and not shared | 10:18 |
| tristan | ssam2, what are those ? | 10:18 |
| ssam2 | things like stage1-gcc from the current Baserock bootstrap | 10:18 |
| ssam2 | I guess in Buildstream you don't bootstrap, you use a prebuilt binary sysroot insetad | 10:19 |
| tristan | you mean mark those for not being shared ? | 10:19 |
| tristan | ssam2, we certainly bootstrap | 10:19 |
| ssam2 | yeah, cus if they're linked against stuff on the host they definitely shouldn't be | 10:19 |
| tristan | We just never use host tools to do it | 10:19 |
| ssam2 | yeah, I guess just a different kind of bootstrap | 10:19 |
| tristan | ssam2, the whole bootstrapping procedure to seal off from host tools is just as relevant with buildstream | 10:19 |
| tristan | it's only that you always use a deterministic "host tool set" to start with | 10:20 |
| ssam2 | there are still cases where bootstrapping from the host is needed, like proprietary OSes or brand new architectures, so might come in handy | 10:20 |
| tristan | ssam2, I dont think so | 10:20 |
| tristan | ssam2, if you can bootstrap whatever it is you have in mind from your host, then you can do it from a deterministic set of tools instead | 10:21 |
| ssam2 | i have a use case right now of building self-contained compiler packages on some proprietary OSes | 10:22 |
| tristan | the uses I have for 'tainted' artifacts are different, one of them is when we have 'workspaces' (i.e. https://wiki.gnome.org/Projects/BuildStream/Roadmap/Workspaces) | 10:22 |
| ssam2 | right now using Omnibus, which means I keep having to track down bugs in Ruby | 10:22 |
| ssam2 | would be nice to use BuildStream instead, but I don't see how I can provide an OSTree repo containing a deterministic set of tools from a proprietary OS | 10:22 |
| ssam2 | other than by creating it on the host using a script.. but then is that any better than just using the host tools? | 10:22 |
| tristan | ssam2, yes it is actually, because your proprietary OS will inevitably receive updates | 10:23 |
| ssam2 | ha, well in theory at least ;-) | 10:23 |
| ssam2 | fair enough | 10:23 |
| tristan | ssam2, so it's questionable whether the bootstrap you created today, will still be doable in 10 years | 10:23 |
| tristan | in any case for the near future BuildStream is linux only, due to some things... hardlinks, ostree, bubblewrap | 10:24 |
| ssam2 | ah well | 10:24 |
| tristan | that hopefully can change, and for the crazier cases we can use VM sandboxes | 10:24 |
| ssam2 | better to be focused | 10:24 |
| ssam2 | didn't realise OSTree was Linux only... i might see how it runs on AIX just for the craic ;-) | 10:25 |
| tristan | We can have a dual artifact cache implementation for non-linux (ostree is actually great, after building a few times, my artifact cache does not grow significantly) | 10:25 |
| ssam2 | I do remember seeing linux syscalls in the code I suppose | 10:25 |
| tristan | in any case the internal API we use for the artifacts is well sealed so it _can_ be dual implemented | 10:26 |
| ssam2 | cool | 10:26 |
| tristan | but I'd rather not think of these things until completing our milestones :) | 10:26 |
| tristan | maybe it would work on AIX yes, certainly not bubblewrap, but sandboxes are intended to have multiple implementations too | 10:27 |
| ssam2 | great | 10:27 |
| ssam2 | I mean, Omnibus has no sandboxing and its artifact cache is a Git repo | 10:27 |
| tristan | another use for tainted artifacts is... hand patched build failures if we support that | 10:28 |
| tristan | i.e. "Your build failed, would you like to drop into a shell and fix it ?" | 10:29 |
| ssam2 | all useful debug tools | 10:29 |
| ssam2 | so would be useful in some cases, the hard part is discouraging people from hacking it into working, then pretending it's finished :-) | 10:29 |
| tristan | So then you fix it and make install into /buildstream/install, exit shell "Would you like to cache this build result and continue with the build ? All elements depending on this element will be tainted" | 10:29 |
| ssam2 | Perhaps recording the reason *why* it's tainted would be useful | 10:30 |
| tristan | I would like to be able to spawn an editor in that case and let the user edit the 'element.bst' in question, but that would mean recomputing the entire pipeline in mid-build | 10:30 |
| ssam2 | then after you've spent 5 days hacking something into working, your final build says "A is tainted because of manual build fixing", "B is tainted because it was built from a local dir", etc. | 10:30 |
| tristan | which... would be freaking awesome, but it's a bit beyond what I can do right in the immediate future :) | 10:30 |
| tristan | ssam2, yeah, well we need it for workspaces, and this is just what I was thinking the other day when I was talking about added metadata in the artifacts | 10:31 |
| tristan | so I want the artifacts to contain some metadata about the artifact, including taintedness and such, also include a build log for provenance, and the content separately | 10:32 |
| *** tristan has quit IRC | 10:35 | |
| *** tristan has joined #buildstream | 10:43 | |
| *** ChanServ sets mode: +o tristan | 10:43 | |
| jjardon[m] | tristan: reading about https://wiki.gnome.org/Projects/BuildStream/Roadmap/GnomeContinuous. In case if of any interest, the machine that is running continuous is a 48-core xeon server atm | 12:25 |
| jjardon[m] | tristan: Also, I'd recommend you to join #testable to get a taste of the pace of changes continuous handle atm | 12:26 |
| tristan | jjardon[m], thanks ! I will join that channel didnt know about it | 14:45 |
| *** ssam2 has quit IRC | 14:51 | |
| *** ssam2 has joined #buildstream | 15:06 | |
| *** ssam2 has quit IRC | 18:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!