| *** tristan has joined #buildstream | 03:08 | |
| *** ChanServ sets mode: +o tristan | 03:09 | |
| *** tristan has quit IRC | 10:13 | |
| *** tiagogomes has joined #buildstream | 10:16 | |
| *** tiagogomes has quit IRC | 10:18 | |
| *** ssam2 has joined #buildstream | 10:24 | |
| *** tiagogomes has joined #buildstream | 10:24 | |
| *** tristan has joined #buildstream | 10:48 | |
| *** ChanServ sets mode: +o tristan | 10:49 | |
| tristan | Mkay... so juergbi paulsher1ood ironfoot ... ssam2, anyone who is interested... would be great if you subscribed to our mailing list: https://mail.gnome.org/mailman/listinfo/buildstream-list | 11:49 |
|---|---|---|
| tristan | irc not great for all conversation formats... | 11:49 |
| tristan | For today, I have added something called 'split-rules', which appears as additional public data in the 'bst' domain (i.e. reachable by Element.get_public_data('bst')) | 11:50 |
| tristan | And I'm considering, searching for a name, for an element I want to add as a pre-deployment element which makes use of this | 11:51 |
| tristan | For a peek at what this looks like, see the defaults for project.conf files (can be overridden for any project, and for any element type, or element, like most everything) | 11:52 |
| tristan | https://buildstream.gitlab.io/buildstream/buildstream.project.html#module-buildstream.project | 11:52 |
| tristan | It gets substituted with variables in context for any given element that split-rules are resolved for | 11:53 |
| tristan | So changing a variable like prefix, or libdir, has the desired consequences | 11:53 |
| ssam2 | yay mailing list | 11:54 |
| tristan | Now the element I'm thinking, maybe 'composer' ? will A.) Have a boolean option for whether integration commands should be run... B.) May optionally list which 'domains' are to be included in the output | 11:54 |
| tristan | ssam2, \o/ :) | 11:55 |
| ssam2 | seems a sane design | 11:55 |
| tristan | So it's not the same exactly as baserock, but inspired from there, also with consideration of how domains need to be split for deploying different flatpakish thingies | 11:55 |
| tristan | So .Debug and .Locale can be generated from that | 11:56 |
| tristan | For baserock, it can be used to just generate a slimmed down, or devel bootable atomically upgradable system | 11:56 |
| tristan | which seems to be the primary use case | 11:56 |
| ssam2 | yeah, well actually for Baserock i'd like a simialr thing | 11:56 |
| ssam2 | optional debug info | 11:56 |
| tristan | Also, for other things, like say generating an rpm for a given build output, it makes sense, and one would have to add rpm-specific metadata for the scriptlets which go into a package based system (and opt to not run the integration commands) | 11:57 |
| ssam2 | right, so this element is basically the "output to any format" device ? | 11:58 |
| tristan | ssam2, yeah I think the debugging info in baserock gets dumped into misc, as it's not picked up by the -bins or -libs | 11:58 |
| ssam2 | or will you have another element after it that generates the actual package ? | 11:58 |
| tristan | ssam2, it's a permutation to run before a deployment, at least thats my use case for it now | 11:58 |
| tristan | exact | 11:58 |
| tristan | a mutation rather, a composition | 11:59 |
| tristan | permutation not the best word :) | 11:59 |
| tristan | I guess composer is decent word for that | 11:59 |
| tristan | ssam2, right so with flatpaks and its runtimes, I would think build elements -> composer -> flatpak -> ostree | 12:00 |
| tristan | where flatpak is a thing that basically adds metadata to it | 12:00 |
| tristan | for baserock probably build elements -> composer -> image deployment | 12:01 |
| ssam2 | ok, makes sense | 12:01 |
| ssam2 | so the composer can output multiple artifacts I guess ? | 12:01 |
| tristan | nope | 12:01 |
| tristan | But it can be used many times | 12:01 |
| ssam2 | ok | 12:02 |
| tristan | I also suppose it may be interesting to have some control on the depth of dependencies it composes | 12:02 |
| tristan | So the default might be: Take the dependencies of this element, in the Scope.RUN scope (everything required to run this element) | 12:02 |
| tristan | But one might want to just have single depth, or stack depth | 12:03 |
| ssam2 | I'm completely lost | 12:03 |
| ssam2 | oh, I think I see | 12:03 |
| ssam2 | so I could build an RPM that only contains the actual package, not the stuff required to build that package | 12:03 |
| tristan | Exact | 12:04 |
| ssam2 | which is usual for an RPM (although only makes sense if you can also package up all the dependencies into their own packages, or you have an existing package repo containing them) | 12:04 |
| tristan | Right, for rpm it may make more sense to generate all at once, in any case the idea would be a potential rpm plugin would demand that users sprinkle additional metadata on their elements in the rpm domain | 12:05 |
| ssam2 | yes, that makes sense | 12:05 |
| tristan | so one could iterate over all the dependencies and pull out metadata | 12:05 |
| tristan | Or other approaches could be used, one use case for "stack depth" (which would mean... I list stacks as dependencies, and only want to stage direct dependencies of those) | 12:06 |
| tristan | Would be a more exact replica of what baserock calls a "system" | 12:06 |
| tristan | But, I dont really wanna do that | 12:06 |
| tristan | Dont wanna encourage the "lets list every stack by hand" culture | 12:07 |
| tristan | I guess I can leave depth out of the picture for now until some need arises for it | 12:07 |
| ssam2 | I think one of the main drivers for the way systems currently work in Baserock is simplicity in the build tool | 12:11 |
| ssam2 | it's pretty bad having to list every stratum manually because when the run-depends of a shared stratum change you have to update literally every system that uses it | 12:12 |
| tristan | exactly, I want that gone | 12:12 |
| tristan | since we offer the build/run distinction, the technical need for that approach disappears I think | 12:12 |
| ssam2 | it means downstream forks get broken systems every time they pull updates, and they have to manually comb the git logs to figure out what new run-depends the strata have | 12:12 |
| ssam2 | so I think few people will argue in favour of preserving that 'feature' :-) | 12:13 |
| tristan | heh | 12:13 |
| ironfoot | otoh, easier to track when things were added | 12:13 |
| tristan | ironfoot, for that we could add sugar to help with provenance and release notes and the like | 12:14 |
| ironfoot | but, I won't be one of those arguing to preserve that feature | 12:14 |
| ssam2 | I think there are better ways to communicate "we added dependencies" than breaking everyone's builds :-) | 12:14 |
| ironfoot | sure :) | 12:14 |
| tristan | I'm hoping that `bst show`, which currently is not perfect, will evolve into something that is cleanly scriptable too | 12:15 |
| tristan | so if one wanted, they could easily wrap up a build and deploy and release "job" which finishes with something that uses `bst show` to publish something about a release | 12:16 |
| tristan | these are the cache keys, this exactly what went into it, etc etc | 12:16 |
| ironfoot | something like: https://download.baserock.org/baserock/baserock-current-genivi-baseline-system-armv7lhf-jetson-manifest.txt | 12:17 |
| tristan | ironfoot, indeed | 12:20 |
| tristan | So besides that... any thoughts on the name 'composer' for the functionality it does ? | 12:21 |
| tristan | or 'compose' ? ... I was thinking 'filter' at one point but doesnt truly satisfy it... 'flattener' was another thought hehe | 12:22 |
| tristan | squash | 12:22 |
| tristan | assembler | 12:22 |
| tristan | squash is kind of like, squashing commits... squashing elements into a single one | 12:22 |
| * tristan thinks so far composer is the sanest idea | 12:23 | |
| tristan | and perhaps 'compose' is better | 12:23 |
| tristan | use the simple verb form for everything, might set a better precedent | 12:24 |
| tristan | yes, we have 'import', not 'importer' | 12:24 |
| tristan | ironfoot, fyi the conversion script received a lot of love | 12:26 |
| tristan | notably, that uglification/escaping of strings is gone | 12:26 |
| tristan | and a bit of an overhaul on the whole script | 12:26 |
| ironfoot | yay :) | 12:27 |
| ironfoot | i'd like to find some time for this soon | 12:28 |
| tristan | ironfoot, you might be late to the party as far as conversions go, but I have plenty to offer haha :) | 12:28 |
| ironfoot | i saw the roadmap, yeah :P | 12:29 |
| tristan | next on the list is after this composer, deployments, and handle conversions of the "system" morphs | 12:29 |
| tristan | I want to at least automatically handle the 'install-files' extensions | 12:29 |
| tristan | for clusters, I dont know if it's even worth the time | 12:30 |
| tristan | (just having a deployment seems to me enough) | 12:30 |
| ironfoot | btw, given a rootfs, you would be able to run definitions deployment extensions to test a deployment of something built with buildstream | 12:30 |
| tristan | not sure you would, you would at least lack the static dev nodes | 12:31 |
| tristan | I guess you could try running an extension in a bst shell, but then you dont have host tools that those extensions also enjoy | 12:31 |
| tristan | or you could just root yourself, and do the mknods | 12:32 |
| tristan | meh, its damn close :) | 12:32 |
| tristan | this week we try to boot images | 12:32 |
| tristan | ironfoot, on the other hand I just thought, it could be very cool if you looked into post-conversion stuff in baserock | 12:39 |
| tristan | ironfoot, like, things which will really improve the experience would be... | 12:40 |
| tristan | o Use arch conditionals to remove all those different arch specific systems it creates | 12:40 |
| tristan | o Use build type dependencies to ensure that busybox is never staged after foundation is built | 12:40 |
| ironfoot | ?o Remove old strata that only contains one chunk | 12:42 |
| tristan | Well, also a lot of dependency untangling | 12:43 |
| tristan | ironfoot, i.e. baserock strata 'contain' chunks, buildstream stacks only depend on elements | 12:44 |
| tristan | this containing-ism is lost knowledge base of what things really depend on, and consequently slow down builds which could otherwise be parallelized | 12:45 |
| ironfoot | I wonder if we would go to a finer dependency grain now that we can | 12:45 |
| tristan | In the abstract, I think: | 12:47 |
| tristan | o Stacks should not disappear, they are convenient | 12:47 |
| tristan | o The elements a stack depends on should not depend on other stacks... | 12:47 |
| tristan | o Unless, those other stacks are from a pipeline element (other project) | 12:47 |
| tristan | So, you keep the convenience of offering a stack which does something, you get finer grained dependencies mostly at build time, but you lose the fine grain when we introduce pipeline elements (but in larger groups than we did with strata) | 12:48 |
| tristan | So for instance the ideal I think for 'gnu-toolchain', is that it offers 2 stacks for consumers; A.) The runtime ... and B.) busybox and friends | 12:49 |
| tristan | Then, if it's a separate project... 'core' or 'foundation' which sits on top of it, depends on the 'runtime' stack (both build & run), but only 'build' depends on the tooling (so it can be discarded after they are replaced with better tools) | 12:50 |
| tristan | oh /me didnt notice he had built a webkit in the background... second webkit brewing hehe | 13:12 |
| *** ssam2 has quit IRC | 13:21 | |
| *** ssam2 has joined #buildstream | 13:38 | |
| tristan | interesting technical detail of compose element | 14:18 |
| tristan | If integrate is True, then everything must be staged first, and the domains which are not listed are to be removed/filtered out | 14:19 |
| tristan | If integrate is False, then including specific domains only is possible | 14:19 |
| tristan | However, the output is different | 14:19 |
| tristan | Because one can never guarantee that split rules will be able to cover all files | 14:20 |
| tristan | I suppose there are two approaches, either make this behavior explicit in the configuration, or, have a semantic for describing "any files not covered by defined split rules" | 14:21 |
| tristan | perhaps the latter is more desirable | 14:21 |
| ssam2 | would it make sense to have two different elements rather than an 'integrate' boolean? | 14:36 |
| ssam2 | it seems like the use cases are different | 14:36 |
| ssam2 | one would be useful for making finished systems, the other useful for making packages | 14:37 |
| tristan | ssam2, well, integration happens at every step fwiw | 14:41 |
| tristan | ssam2, stage -> integrate -> build, every time | 14:41 |
| tristan | it *might* make sense to split this into separate things, but so far I havent found real cause to | 14:41 |
| ssam2 | ah, ok | 14:41 |
| tristan | also, it would potentially cost a lot more disk space | 14:42 |
| ssam2 | my only worry was 'compose' becoming too complex from having many options | 14:42 |
| tristan | yeah I understand | 14:42 |
| ssam2 | but if 'integrate' is standard, it doesn't make sense to split based on that | 14:42 |
| tristan | it's a tricky problem, though | 14:43 |
| tristan | the chaos of trying to manage split in one step and integrate separately, is something that has left baserock with an unsatisfactory result, regardless of which order it's done in | 14:43 |
| tristan | for instance (warts ahead: ...) currently it's been reversed so that integration commands in baserock run first directly in the sandbox | 14:44 |
| ssam2 | right.. splitting should really be last I guess | 14:44 |
| tristan | and then, instead of removing things that are not in the splits, the split rules are used to copy files over from / -> /<system>.inst/ | 14:44 |
| tristan | So, everything that was done in the integration phase, is actually just lost | 14:45 |
| tristan | in the final system :) | 14:45 |
| ssam2 | ha | 14:45 |
| * ssam2 never dared look at how splitting worked in YBD | 14:45 | |
| tristan | well, I wrote an email to the list months ago, and I think in the end nobody wanted to revisit the thing | 14:46 |
| tristan | I'm not convinced (or nobody has taken the time to convince me) that the problem was ever actually solved in morph, either | 14:46 |
| *** tristan has quit IRC | 14:57 | |
| * paulsher1ood is convinced it wasn't | 15:13 | |
| *** tristan has joined #buildstream | 15:13 | |
| *** ChanServ sets mode: +o tristan | 15:14 | |
| *** leeming has quit IRC | 15:47 | |
| *** tiagogomes has quit IRC | 15:47 | |
| *** ssam2 has quit IRC | 15:48 | |
| *** ssam2 has joined #buildstream | 16:01 | |
| *** tiagogomes has joined #buildstream | 16:02 | |
| *** leeming has joined #buildstream | 16:02 | |
| *** leeming has quit IRC | 16:25 | |
| *** leeming has joined #buildstream | 16:41 | |
| *** ssam2 has quit IRC | 18:57 | |
| *** tristan has quit IRC | 21:18 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!