| *** tristan has joined #buildstream | 04:04 | |
| *** ChanServ sets mode: +o tristan | 04:13 | |
| *** tristan has quit IRC | 05:34 | |
| *** tristan has joined #buildstream | 05:39 | |
| *** ChanServ sets mode: +o tristan | 05:39 | |
| tristan | juergbi, just thought this up btw: https://gitlab.com/tristanvb/buildstream/issues/6 | 05:49 |
|---|---|---|
| tristan | hergertme, welcome to the party :) | 05:51 |
| tristan | hergertme, our timezones are probably not matching up very well... | 05:51 |
| tristan | but I'm sorry if I was in any way less than forthcoming to the community, I am racing against time to deliver something to propose more formally, as I'm sure you understand what that's like :) | 05:52 |
| hergertme | tristan: no worries | 06:23 |
| hergertme | tristan: i only have a partial view of what's going on, but what i see is that it will make a lot of sense for us to spin up images for things like mutter, shell, gtk, etc etc (which are certainly needed, but not our primary users we cater to in Builder) | 06:24 |
| hergertme | but it would be cool if we can have a simulator or equivalent for that type of thing | 06:25 |
| hergertme | but builder already has about as much code for flatpak-builder as flatpak-builder itself, so its a non-trivial amount of work for us to go change that plumbing. anything that would replace it (for applications at least), needs to allow us to keep the workflow we just finally got to of: clone/build/run. getting to 1 step to testing software has been a ton of work over the last year and a half, and i'd hate for it to be for not. | 06:26 |
| tristan | hergertme, understood yes | 06:31 |
| tristan | I suppose the bulk of the work is generating json and running flatpak-builder ? | 06:31 |
| hergertme | those are both relatively easy give the stuff we pushed up into flatpak-builder | 06:32 |
| tristan | I see, what kind of features are we talking about ? I was quite fluent with the flatpak (and builder) code base many months ago but not up to speed on the more recent things | 06:33 |
| tristan | A large part of what the builder does is encode metadata that flatpak itself will use (asides from building) | 06:34 |
| tristan | those things should be very trivial | 06:34 |
| hergertme | incremental builds for the top-most project, running/testing inside of runtimes before having to go through ostree commit stages, non-destructive editing of json files, integration with ptrace/perf, cloning from flatpak manifests (so you can edit a running program), runtime management, ... | 06:34 |
| hergertme | some amount will be re-usable of course | 06:34 |
| hergertme | being able to work with the partial built pipelines is pretty important for us, ostree commit stages are too slow for us to use "flatpak run foo" every time | 06:35 |
| tristan | for incremental builds for the topmost project (or any of them actually), that will be a hard requirement for BuildStream | 06:35 |
| tristan | and of course shelling in and testing/debugging etc | 06:36 |
| hergertme | its absolutely critical for us | 06:36 |
| hergertme | because otherwise every time you click build you'd have to rebuild your whole project | 06:36 |
| hergertme | so we use flatpak-builder for 0..N-1 | 06:36 |
| hergertme | and then we manage the build of your project using the internal build operations in builder | 06:36 |
| tristan | we dont use the same serial build model, so we should be able to have say, the whole stack which composes a desktop and a checkout of GTK+, and "just build GTK+ and launch the VM" kind of thing | 06:37 |
| hergertme | that will be cool for platform libraries and such | 06:37 |
| hergertme | i still very much want to, someday, have a simulator/emulator for Builder with "GNOME OS" (for whatever that means) | 06:37 |
| tristan | with flatpaks, it will be helpful if you want to say, include something that your embedded webkit depends on, and only rebuild that and test in a tight developer loop | 06:38 |
| tristan | So regarding the GNOME OS simulator big picture, we want to have continuous contributing to that | 06:38 |
| tristan | Currently the base runtime of both flatpak and continuous is a fork from the same original yocto compile | 06:38 |
| hergertme | at some point, we need to get to where we can allow you to work on multiple projects in the same build pipeline for Builder (you can open multiple projects, but not in the same workbench). that way people can hack on GStreamer and their app at the same time | 06:39 |
| tristan | What we want to do for that is just pull in the necessary things continuous does, to the same yocto build which flatpak uses (so you can build the "only runtime" or the "runtime + some services and kernel") | 06:39 |
| hergertme | excellent. if we had access to the continuous OSTree, we can actually have updated git-master to people in a matter of hours | 06:39 |
| tristan | so that will be unified, and GNOME Continuous builds should be contributing to a shared artifact cache | 06:39 |
| tristan | So developers dont have to build everything (if they wire to the continuous artifact cache, they download prebuilds of anything available) | 06:40 |
| tristan | So migrating continuous is on the todo list, as we will want it to also use the same build metadata as developers, and flatpak sdk/runtime builds etc | 06:41 |
| hergertme | well, the serial nature of flatpak-builder had a very specific purpose, to get things closer towards reproducible i thought? | 06:41 |
| tristan | We are actually all about reproducible hehe | 06:41 |
| tristan | Except ! | 06:41 |
| tristan | For GNOME developer purposes we need an exception to that | 06:41 |
| tristan | because reprodicible builds are not convenient for developers | 06:42 |
| tristan | For that we need a dual cache key calculation mode | 06:42 |
| tristan | hergertme, so basically dependencies are built and staged in the same deterministic order, but many things can be built in parallel when their dependencies are orthogonal | 06:42 |
| hergertme | well i do definitely want to see the runtimes being reproducible, even if testing systems aren't for speed issues | 06:43 |
| hergertme | that seems reasonable | 06:43 |
| tristan | A cache key for a given element is A.) It's sources (commit sha usually) B.) it's build instructions and C.) The sum of it's direct dependency cache keys | 06:43 |
| tristan | So yeah, we need to think about the solid release case where everything is deterministic (thats the mentality we started out with) | 06:44 |
| tristan | But we also have to consider that a developer doesnt want to rebuild webkit on their machine to test a code change in glib | 06:44 |
| hergertme | i also really really really want tinderbox for gnome | 06:46 |
| tristan | One thing which was inspired by the serial nature of flatpak-builder, one of the advantages there; is that you get to run cache updates (like fc-cache, ldconfig, gsettings schema compiles etc) directly on something you commit | 06:46 |
| hergertme | so i can just take my working tree + changes, and submit a build | 06:46 |
| tristan | Ah, ok like mozilla (took me a moment sorry :)) | 06:49 |
| tristan | hergertme, Interestingly... that is something we have previously experimented with using gitlab | 06:49 |
| tristan | except you need to submit a branch to trigger a build | 06:50 |
| tristan | which is a bit different | 06:50 |
| hergertme | at least that i can automate in Builder | 06:50 |
| tristan | nod | 06:50 |
| tristan | I dont want to step on colin's toes too much but when looking into the continuous things, it's possible that "run a gitlab instance" might turn out to be an awesome alternative | 06:52 |
| tristan | but that needs more investigation anyway | 06:52 |
| hergertme | agreed | 06:53 |
| *** tristan has quit IRC | 08:15 | |
| *** tristan has joined #buildstream | 09:05 | |
| *** ChanServ sets mode: +o tristan | 09:07 | |
| *** csoriano has joined #buildstream | 10:10 | |
| *** ssam2 has joined #buildstream | 10:18 | |
| tristan | ssam2, checkout my latest crazy idea: https://gitlab.com/tristanvb/buildstream/issues/6 :) | 10:25 |
| ssam2 | yeah that would be nice | 10:26 |
| tristan | ironfoot, if you have some time to think about conversions... I have a small challenge for you :) | 11:10 |
| ironfoot | defs2bst? | 11:10 |
| tristan | not exactly, I'm going to pick that up again very soon... but related yes | 11:10 |
| ironfoot | yeah, I have some time to think, maybe not to implement things | 11:11 |
| tristan | ironfoot, its an implement thing, but I think it's not very involving and will be attractive to baserock | 11:11 |
| tristan | so basically | 11:11 |
| tristan | Right now we run the conversion on top of the manually converted base https://gitlab.com/tristanvb/buildstream-tests/tree/build-essential | 11:12 |
| tristan | ironfoot, What I would like to do, is A.) run that build... B.) Stage the stack (could be using bst shell) ... C.) Commit that sysroot to a git repository... D.) Replace the gnome platform/sdk in build-essential with *that git* | 11:13 |
| tristan | ironfoot, in this way baserock continues to be circular and self reliant | 11:14 |
| ironfoot | So we would be producing the "host tools" needed | 11:15 |
| tristan | exact | 11:15 |
| tristan | ironfoot, it may be tricky for other arches than x86_32 and x86_64, but that could be postponed I guess | 11:15 |
| ironfoot | hm.. it will be a bit big (hundreds of MB) | 11:16 |
| tristan | there are various ways to get a runtime for other arches, for armv7a and aarch64 there are freedesktop runtimes | 11:16 |
| tristan | yeah it *should* be ostree really | 11:16 |
| tristan | but if baserock wants to continue with the "everything repeatable in only-git trove"... | 11:16 |
| ssam2 | I don't know that it does | 11:16 |
| ssam2 | there was originally a "you need to download a Baserock image from download.baserock.org" step to using it | 11:17 |
| ssam2 | this is basically going back to those days, but automating the initial setup instead of requiring the user to faff around setting up a VM for 3 hours | 11:17 |
| tristan | Indeed, but I think the trove idea is to "have always everything you need in one place, so you can reproduce exactly this 10 years later" | 11:17 |
| ssam2 | sure, but Trove already has a HTTP server, so could easily host OStree repos | 11:17 |
| tristan | sure, that would also be fine | 11:18 |
| tristan | I'm just thinking people will like it more if it's circular | 11:18 |
| tristan | and not rooted in something else | 11:18 |
| ssam2 | I think I care more about the base being small | 11:19 |
| ssam2 | and easy to maintain | 11:19 |
| ssam2 | which doesn't exclude Baserock, but doesn't require it either | 11:19 |
| tristan | I personally dont really care heh | 11:19 |
| ironfoot | at the moment is not small with that sdk ostree repo | 11:20 |
| tristan | yes that ostree can probably be replaced with the base runtime directly | 11:20 |
| tristan | org.freedesktop.BaseSdk and org.freedesktop.BaseRuntime | 11:20 |
| ssam2 | the Baserock definitions *were* organised in such a way that you could bootstrap with nothing but a GCC toolchain, Busybox and M4 | 11:20 |
| tristan | (from the same ostree host) | 11:20 |
| tristan | ssam2, and that's what the BaseSdk will basically give you | 11:21 |
| ssam2 | of course, that made it hard to integrate things like Rust which bootstrap themselves from their older binary images | 11:21 |
| ssam2 | ah, cool | 11:21 |
| ssam2 | ah, so you're basically talking about committing the stage2 binaries to this repo? in that case, I like it :-) | 11:21 |
| tristan | it sits below org.freedesktop.Sdk and org.freedesktop.Runtime | 11:21 |
| ssam2 | and you could then inject extra stuff (e.g. rust bootstrap binaries) if you wanted Rust | 11:21 |
| tristan | ssam2, basically yeah, well I would have went with a full build-essential commit | 11:21 |
| ssam2 | of course you lose portability if you're depending on Rust, but it's optional | 11:22 |
| ssam2 | yeah I guess build-essential stage 3 makes more sense | 11:22 |
| ssam2 | well, there's a tradeoff | 11:22 |
| ssam2 | if you commit stage 2, the user then gets to build their own GCC, which means stage2 can be a much older GCC | 11:22 |
| ssam2 | and we only have to update the ref for stage3 gcc, not the bootstrap binaries | 11:22 |
| tristan | ssam2, the thing about stage2, isnt it still this weird /tools directory thing ? | 11:23 |
| ssam2 | ah, true | 11:23 |
| tristan | ssam2, I think that's a bad base if so | 11:23 |
| tristan | right | 11:23 |
| tristan | better have build instructions from scratch | 11:23 |
| ssam2 | yeah I guess we can just leave the stage3 GCC at some stable release, and say "build your own GCC from Git and install it over the top of ours if you want" | 11:24 |
| ironfoot | yep, and baserock could continue building the 3 stages on top of that | 11:25 |
| tristan | ssam2, not sure its clear but I want to keep build instructions for the whole bootstrap in baserock bst project | 11:25 |
| tristan | I think it's clear | 11:25 |
| ssam2 | there's not much point though, the idea of the 3 stage bootstrap is (a) so you can cross-bootstrap and (b) so your compiler is isolated from the host OS | 11:25 |
| tristan | ssam2, when moving the OS forward, you'll sometimes need to recommit build-essential into your base | 11:26 |
| tristan | when you may reach a point where your toolchain is too old | 11:26 |
| tristan | to build something very new | 11:26 |
| ssam2 | yeah | 11:26 |
| tristan | I dont want to lose bootstrapping really, sure it's complicated but we should have a way to go from nothing to everything and back around again | 11:27 |
| ssam2 | no, it's one of the really interesting things about Baserock | 11:27 |
| * paulsher1ood *really* doesn't want to lose bootstrapping | 11:28 | |
| ironfoot | right, so are we calling this bs-sdk? | 11:28 |
| tristan | yes, no loss of bootstrapping please | 11:28 |
| paulsher1ood | or multip-arch | 11:28 |
| tristan | ironfoot, I want to call it something else | 11:28 |
| paulsher1ood | bulls***-sdk? :-) | 11:28 |
| ssam2 | Aboriginal 2 :-) | 11:28 |
| tristan | ironfoot, I want to call it gnu, or gnu-runtime | 11:28 |
| tristan | or gnu-sdk | 11:28 |
| ssam2 | gnu-toolchain ? | 11:29 |
| tristan | And then when we do a toybox / musl version, it will be a different base | 11:29 |
| tristan | gnu-toolchain sounds good | 11:29 |
| paulsher1ood | where does this sit relative to rob landley's new thing? mkroot, irrc | 11:29 |
| ironfoot | it will live in an ostree repo? | 11:29 |
| ssam2 | I guess it's kinda the same, but presumably Rob still wants to write everything in Bash | 11:29 |
| tristan | landleys thing, well, is mostly about cross bootstrapping | 11:29 |
| tristan | and his dropping of the aboriginal way (which was a bit more convenient, less moving parts) was mostly his holy war | 11:30 |
| paulsher1ood | https://github.com/landley/mkroot | 11:30 |
| paulsher1ood | got to give him credit... he has it down to <500 lines of shell | 11:31 |
| ssam2 | true | 11:31 |
| tristan | paulsher1ood, So, what I want, is to first reach our targets; and then after that we turn the world upside down by A.) Introducing a base which we build in an aboriginal like way, and then B.) Make a sandbox implementation which depends on a cross built root | 11:31 |
| paulsher1ood | ack | 11:31 |
| tristan | And that sandbox implementation cross compiles | 11:31 |
| paulsher1ood | may the force be with you, young jedi | 11:32 |
| tristan | :) | 11:32 |
| tristan | everything along the way is already proven, but needs dusting up | 11:32 |
| ironfoot | so, ostree repo, with different branches for different versions/arches? | 11:36 |
| tristan | ironfoot, for different arches at least yeah | 11:37 |
| tristan | as you can see there is already an arch conditional in gnome-platform.bst and gnome-sdk.bst to checkout the right base for building on | 11:38 |
| tristan | it's missing the arm and aarch64 variants, but those are available also at sdk.gnome.org | 11:38 |
| tristan | actually if we want to go with baserock terminology, we should use x86_32 instead of i386 in that conditional | 11:39 |
| tristan | changes of the day include shorter log lines, abbreviated cache keys, rework of how logging works so I can improve it more, but not quite what I want it to be yet | 14:56 |
| tristan | closer | 14:56 |
| * tristan signing out... | 14:56 | |
| ironfoot | o/ | 14:57 |
| *** tristan has quit IRC | 14:59 | |
| *** tristan has joined #buildstream | 15:25 | |
| *** ChanServ sets mode: +o tristan | 15:25 | |
| *** csoriano has quit IRC | 15:41 | |
| *** csoriano has joined #buildstream | 15:59 | |
| *** ghishadow_ has joined #buildstream | 16:36 | |
| *** csoriano has quit IRC | 16:43 | |
| *** csoriano has joined #buildstream | 16:57 | |
| *** csoriano has quit IRC | 17:59 | |
| *** csoriano has joined #buildstream | 18:04 | |
| *** ssam2 has quit IRC | 18:09 | |
| *** csoriano has quit IRC | 19:14 | |
| *** tristan has quit IRC | 21:20 | |
| *** jjardon[m] has joined #buildstream | 22:57 | |
| *** jjardon has left #buildstream | 22:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!