*** gtristan has joined #baserock | 04:36 | |
gtristan | Can I use the 'verbatim' build system and extend it ? | 05:09 |
---|---|---|
gtristan | I cant seem to find any chunks which have build-system: verbatim and also a custom morph that does some extra post frobnication | 05:10 |
gtristan | ok nevermind I got it | 05:22 |
gtristan | it's defined in the DEFAULTS | 05:22 |
paulsherwood | gtristan: afaik verbatim hasn't been upstreamed to definitions | 06:17 |
*** fay_ has joined #baserock | 06:48 | |
gtristan | paulsherwood, ah, anyway I got it, just doing stuff in post-install-commands there | 06:50 |
*** paulwaters_ has joined #baserock | 07:03 | |
*** paulwaters_ has quit IRC | 07:03 | |
jjardon | Hi, what is the var to refer to the src folder in a .morph file? Does something like $SRCDIR exits? | 07:12 |
*** toscalix_ has joined #baserock | 07:31 | |
pedroalvarez | jjardon: every command is executed from the src folder | 07:48 |
pedroalvarez | there should be a var for that, though | 07:48 |
gtristan | I agree, I've wanted one before more than once | 07:48 |
gtristan | and have resorted to saving `pwd` | 07:48 |
gtristan | but I just checked and can confirm at least in this build I'm running, there isnt one | 07:49 |
jjardon | Is this a bug in definitions or in ybd/morph? | 07:52 |
*** locallycompact has joined #baserock | 07:55 | |
*** locallycompact has quit IRC | 07:56 | |
*** locallycompact has joined #baserock | 07:56 | |
paulsherwood | all of the above :) | 07:56 |
*** locallycompact has quit IRC | 08:04 | |
*** locallycompact has joined #baserock | 08:48 | |
*** tiagogomes has joined #baserock | 09:02 | |
*** ctbruce has joined #baserock | 09:15 | |
richard_maw | today's lwn has an article about a google thing called minjail, pulls together all the various sandboxing tech in an attempt to do as much containment as possible | 10:12 |
edcragg | nice | 10:12 |
richard_maw | https://lwn.net/Articles/700557/ is the article. Couldn't find any links to the code for it though. Google just gives depressing results. | 10:13 |
locallycompact | only avaliable to subscribers :( | 10:13 |
richard_maw | locallycompact: are you not on Codethink's corporate subscription then? | 10:15 |
*** toscalix_ has quit IRC | 10:16 | |
leeming | ah, sounds like sandboxlib | 10:16 |
* leeming also hit the paywall | 10:16 | |
paulsherwood | https://lwn.net/SubscriberLink/700557/091f804480fab479/ | 10:17 |
* leeming wonders if this is it https://github.com/omegaup/minijail | 10:18 | |
*** toscalix_ has joined #baserock | 10:18 | |
leeming | actually.. google : https://chromium.googlesource.com/chromiumos/platform/minijail/ | 10:18 |
*** toscalix has joined #baserock | 10:24 | |
*** toscalix_ has quit IRC | 10:26 | |
leeming | good catch richard_maw, this is an interesting read | 10:31 |
pedroalvarez | maybe this is the most recent one https://chromium.googlesource.com/aosp/platform/external/minijail/ | 10:33 |
leeming | ah yes, good catch | 10:39 |
leeming | is brillo/weave of interest to anyone here? https://developers.google.com/brillo/ | 10:45 |
*** toscalix has quit IRC | 10:48 | |
*** toscalix has joined #baserock | 10:59 | |
jjardon | this is a good article as well: http://lwn.net/Articles/700530/ "Greg Kroah-Hartman, the maintainer of the long-term support kernels, agreed with this assessment of the situation, noting that even serious security fixes don't find their way through to the kernels shipped by vendors despite being promptly included in the LTS kernels. So he is mulling the idea | 11:40 |
jjardon | of stopping the the maintenance of the LTS kernels entirely: " | 11:40 |
jjardon | mainline FTW | 11:40 |
jmacs | non-subscriber link? | 11:41 |
jjardon | https://lwn.net/SubscriberLink/700530/7f1c21b0fd40489e/ | 11:42 |
*** ctbruce has quit IRC | 11:44 | |
*** gtristan has quit IRC | 11:57 | |
*** gtristan has joined #baserock | 12:16 | |
locallycompact | Here's a first pass at the 009 migration, I would have loved to one million things at once but this change is big enough as it is. https://gitlab.com/baserock/definitions/commit/4b3be68d4fdf5357c9a765cc7c7194c369feaa00 | 12:18 |
locallycompact | I also guess that people prefer non-implicit transitivity in b-ds at the system level which this isn't | 12:19 |
jjardon | locallycompact: what v009 tries to achieve? | 12:21 |
locallycompact | Move build-dependencies from the stratum level to the system level | 12:21 |
locallycompact | So if you want to change to a new root it's now a one line change | 12:22 |
locallycompact | just change which stratum the 'sysroot' entry points to, for example | 12:22 |
locallycompact | or add a few and set the b-ds up in the system | 12:22 |
locallycompact | But it also means there must be inclusion semantics to decide which strata actually end up in the final system | 12:22 |
paulsherwood | what's 'include-mode' ? | 12:24 |
paulsherwood | this is the first thing that ybd complains about | 12:24 |
locallycompact | so previously you are relying on the fact that the stratum is in the list at all to govern whether or not it ends up in the final system | 12:25 |
locallycompact | you can't use that anymore | 12:25 |
locallycompact | so I split it into 'relevant', which is the default, and 'affine' which gets discarded implicitly by the system | 12:25 |
paulsherwood | -1 | 12:25 |
locallycompact | + a third which doesn't exist yet | 12:26 |
locallycompact | paulsherwood, you have a simpler solution? | 12:26 |
paulsherwood | i don't know what affine means and am not about to to google it | 12:27 |
locallycompact | here it means doesn't go in the final system | 12:27 |
paulsherwood | please explain in terms a normal human can understand :) | 12:27 |
richard_maw | locallycompact: they are build-time-only dependencies, right? | 12:28 |
locallycompact | yes | 12:28 |
locallycompact | I wanted this also for chunks because for example we have a chunk which is a rootfs tarball and just runs cp -R * $DESTDIR, but we have to do this in two stages, one in bootstrap mode and once in regular mode, and it takes 20 minutes each to split | 12:28 |
locallycompact | but bootstrap mode implicitly doesn't get included | 12:29 |
paulsherwood | so build-time, run-time? where run-time implies also build-time? | 12:29 |
locallycompact | so we are doing a whole tarball just for the sake of "cp" | 12:29 |
locallycompact | why not make the initial rootfs tarball relevant | 12:29 |
locallycompact | and have only the one | 12:29 |
locallycompact | include-mode: "build-time / run-time" | 12:30 |
richard_maw | gotcha, my previous thoughts on this was moving more stuff out of system definitions and making chunks/strata less composable, but I see you're goign in a different direction | 12:30 |
locallycompact | it puts one slight dampener in a plan I have for more composable strata assemblages | 12:31 |
locallycompact | for example, what happens whan you have a build-time dependency which has a run-time component in order to work | 12:31 |
paulsherwood | sorry, what does 'composable' mean? | 12:31 |
locallycompact | so for example you might want a file containing some strata which doesn't in itself constitute a full system | 12:32 |
locallycompact | like all the "base" components" | 12:32 |
locallycompact | whjich are unlikely to change across systems | 12:32 |
locallycompact | this isn't part of 009 but it is in my mind | 12:32 |
locallycompact | but in so far as say "graphics stuff" assemblage contains some things that are only build-time components, are they build-time componetns for strat that depend on graphics stuff or for strata *within* graphics stuff | 12:33 |
locallycompact | x-org whatever they are | 12:33 |
* paulsherwood had hoped for generic nestable groups/sets/collections, some of which are systems | 12:33 | |
locallycompact | right | 12:33 |
locallycompact | essentially this | 12:33 |
locallycompact | but the precendence of include-mode is important here | 12:33 |
locallycompact | what does "foo" end up in the staging area for | 12:34 |
locallycompact | does it just provide locally | 12:34 |
locallycompact | until the local collection is build | 12:34 |
locallycompact | then gets discarded in the staging for higher collections | 12:34 |
locallycompact | or stay in the staging area for everything and also go in the system | 12:35 |
locallycompact | or somewhere between | 12:35 |
locallycompact | two modes gets you some of the way I'm not sure all of the way | 12:35 |
paulsherwood | that requires that anything higher re-specifies foo if it needs it, which for some reason was not the way we originally approached this... | 12:35 |
* paulsherwood was surprised to discover that our dependencies where 'transitive' but lots of folks seemed to think this was obviously correct | 12:36 | |
persia | locallycompact: My concern with moving build-depends to the system level is how the artifact is defined: is this sufficiently abstracted that if I build the same chunk for two systems with different expressed build dependencies, I get separate artifacts? | 12:41 |
locallycompact | that's on the cache key generator to figure that out | 12:41 |
locallycompact | and it should take into account everything that could potentially make a difference | 12:41 |
persia | It's the "should" that worries me :) | 12:42 |
locallycompact | one can't cheat the doing things right of the build tool | 12:42 |
persia | I suppose it's an even tradeoff between the need to duplicate lots of entries to switch between e.g. glibc/gcc and musl/clang and the need to maintain duplicate chunk entries for the same. | 12:43 |
persia | My thought was more about the complexity of thought required by the build tool developers: abstracting things to higher levels means that one needs to manage the cache key entirely symbolically, rather than being able to assert that the contents of files are meaningful input. I suppose that as long as the build tool developers are willing to ignore a "file hash" as an important expression this doesn't matter. | 12:44 |
persia | Thinking about it more, I think this also complicates the creation of systems. | 12:48 |
persia | One of the original ideas was that a system definition should be fairly short and easy to write, although the tooling never actually achieved the goal of making that entirely true (for various reasons). | 12:48 |
locallycompact | avoiding having to duplicate an entire folder of 40 strata in order to replace one lower stratum was the goal here | 12:49 |
persia | And we never established a good model for definitions that would let folk maintain lots of systems effectively. | 12:49 |
persia | So maybe that goal can be set aside. | 12:49 |
persia | locallycompact: Indeed: from a maintenance point of view, that makes life easier. | 12:49 |
paulsherwood | that's a bit harsh.. i think definitions is 'ok', just not 'great' | 12:49 |
persia | paulsherwood: Which is harsh? | 12:49 |
paulsherwood | 13:49 < persia> And we never established a good model for definitions that would let folk maintain lots of systems effectively. | 12:50 |
persia | paulsherwood: Yes. I'm specifically speaking about the issue of how to maintain downstream systems. I remember folk working on this for a while, but the consensus seemed to be that it was just easier to do work upstream, or fork and resync later. | 12:51 |
persia | I don't consider either of these as easy to manage as other tooling for package-based systems to maintain custom configurations downstream. | 12:51 |
persia | I very much do not mean to criticise other aspects of definitions by that statement. | 12:52 |
locallycompact | it's true about the downstream thing | 12:52 |
persia | I think this change makes downstream management harder. I'm not sure that we care, really. | 12:52 |
locallycompact | no | 12:52 |
persia | Because we never really solved the downstream issue, so this doesn't make it worse. | 12:52 |
locallycompact | what happens previously is upstream updates al lthe refs for a stratum and we can't absorbd it because the build-dependencies conflict | 12:53 |
locallycompact | now we can use identical stratum files because there's no downstream particular information in them | 12:53 |
locallycompact | we might change a sha for a component, but we also have the capability to treat that as a different stratum entirely | 12:54 |
persia | Yes, but they no longer mean anything, so it isn't importnat. | 12:54 |
locallycompact | They represent what somebody expects to work in some capacity | 12:55 |
locallycompact | You can slot that into your system and yay or nay | 12:55 |
persia | Plus, although the declaration of build-dependencies has been moved to the system level, there are still many cases where the build-dependencies are required at the chunk level (because of the build commands), so things don't reliably work when the underlying implementation changes. | 12:55 |
locallycompact | if nay, maintain another stratum, just not forty | 12:55 |
persia | I agree it solves the forty-stratum problem. | 12:55 |
persia | I just believe that this change means that it is no longer interesting to look at the downstream issue, because it cannot be safely solved: folk needing that sort of thing would be better served with packages. | 12:56 |
persia | (ideally packages that don't have runtime installation scripts, and in combination with pregenerating system artifacts for validation before releasing, rather than installing packages at system instantiation time, but there exist systems that work this way) | 12:57 |
locallycompact | I would have thought the underlying implementation thing is solved by strata being architecture specific and pointing their chunks at the relevant build instructions | 12:57 |
persia | The build instructions depend on the set of build dependendencies. | 12:58 |
locallycompact | i.e build-essential-armv7, build-essential-x86, but the higher strata need not change necessarily | 12:58 |
locallycompact | That's not always the case | 12:58 |
persia | No. | 12:58 |
persia | Although I believe it also a rare case that one needs different build-essentials on a per-ISA basis. | 12:58 |
locallycompact | ISA? | 12:58 |
* jjardon thinks this will solve the problem on upstreaming the work Ive been doing lately: replace busybox stage3 for GNU utils | 12:59 | |
persia | In most cases, upstream tooling is designed to generate completely different results based on the passed ISA, rather than needing different bits installed. | 12:59 |
persia | Instruction Set Architecture. | 12:59 |
persia | jjardon: It will help massively with that effort, indeed. | 13:00 |
persia | jjardon: Further, it will make it easy for people to choose busybox or GNU at the system level, rather than needing to duplicate the hierarchy, so it makes it easier to run them in parallel. | 13:01 |
*** brlogger has joined #baserock | 13:02 | |
*** brlogger has joined #baserock | 13:02 | |
jjardon | persia: patches are almost ready, but I have that problem: or we always use busybox, or we always use GNU utils. You could build always with GNU utils and then build busybox but that defeats the pourpose of busybox I think | 13:03 |
locallycompact | It makes system definitions a lot bigger I'll admit, but I think working towards nested composableness will reduce it to I want some compilery things and some graphicsy things and maybe I'll decompose it to exchange one layer out when I need to | 13:03 |
persia | locallycompact: So, igoring the x86 vs. armv8 issue (because it's safely separable), contrast the new system definition with some of the other tooling: here's the germinate system definition: http://people.canonical.com/~ubuntu-archive/seeds/lubuntu.yakkety/desktop-gtk | 13:03 |
locallycompact | good lord what is this | 13:04 |
persia | A "seed file" for germinate. It's the tool used for defining the systems that LP generates. | 13:04 |
locallycompact | LP is? | 13:04 |
persia | I think germinate is also in Debian, so you don't need to use Ubuntu if you want to use this sort of format. | 13:04 |
persia | launchpad.net | 13:04 |
persia | Germinate is a tool that consumes "seed" files (like the one linked), and then creates package lists, to define systems (where the packages can be installed to create a deployable system or they can be copied somewhere to create installation media, or whatever) | 13:06 |
locallycompact | That's interesting | 13:07 |
persia | https://wiki.ubuntu.com/Germinate and https://wiki.ubuntu.com/SeedManagement are the relevant docs, although I don't know if they are up-to-date | 13:08 |
* persia hasn't used germinate to build systems in over 5 years | 13:08 | |
persia | Note that the specific structure used in Ubuntu also tries to enforce the "ogre model", wherein subsets of the available sources are required to be self-contained sets in terms of build- and run-time dependencies. | 13:09 |
persia | Some subsets may depend on others (so it makes layers), but by separating them, this allows organisations to be able to select set-theoretic collections of packages for support with confidence that they would also support the necessary dependencies (both types). | 13:10 |
*** jonathanmaw has joined #baserock | 13:10 | |
persia | Mind you, since I was involved, lots of things have happened, including a reduction in the number of organisations providing support, so I encourage you to take ideas from it, rather than digging too deeply into the current implementation and practice. | 13:11 |
persia | And, of course, as with any debian-format package based approach, the resultant systems are subject to the failure rate of apt (about 1/50th of 1%) and the uncertainties of maintainer scripts, so there's plenty to criticise when compared to precomposed system generators (like morph or ybd). | 13:12 |
locallycompact | mmm | 13:14 |
persia | I've not been following recently, but I believe Debian still uses tasksel to define the tasks that end up on live images, install images, etc. | 13:17 |
persia | Here's a link to the relevant README in the VCS: https://sources.debian.net/src/tasksel/3.35/README/ | 13:17 |
persia | Anyway, the point of this is that past system definitions were more closely aligned with the simpliciy of these sorts of formats, and the expressed goal was to make them even easier to write. | 13:18 |
persia | I think this change moves in the other direction, making them potentially more complex than other tooling. | 13:19 |
persia | But, as I said, given the issues with downstream definitions anyway, I'm not sure this is entirely a bad thing: the ways in which this definitions format change makes it easier to work downstream may well offset the ways in which it makes it harder. | 13:20 |
rjek | Who on earth thinks Markdown or similar is a sensible data format | 13:22 |
persia | I don't know how the Fedora infrastructure creates livecds (and searching for this keeps sending me to instructions to modify livecds, without pointers to the source), but I believe it is custom kickstart files that are used to define the contents of the liveCD or USB image (or cloud image). | 13:23 |
jonathanmaw | Is /usr/include/linux/a.out.h a sensibly-named file, or has something gone wrong with my gcc chunk? | 13:23 |
locallycompact | rjek, people who haven't yet discovered rst :) | 13:23 |
persia | rjek: It's moinmoin, actually. | 13:23 |
richard_maw | jonathanmaw: I've got one. | 13:23 |
jonathanmaw | ok, just need to figure out why this version of systemd isn't able to find something that exists in a.out.h | 13:24 |
jjardon | locallycompact: in your new model, is it easier/possible to express: "I want this stratum depending in this other for the arch, but not for other arch" (ie, build llvm as a dependency of mesa only in x86) | 13:35 |
locallycompact | yes | 13:36 |
persia | Err, not really. | 13:36 |
locallycompact | you need two system definitions of course | 13:36 |
persia | One can have a system that is x86 where mesa depends on llvm, and another that is armv7 where it doesn't. | 13:36 |
locallycompact | yes that | 13:36 |
persia | One cannot express architecture dependence for build-dependencies directly. | 13:36 |
locallycompact | right | 13:36 |
jjardon | ok, cool; we currently always build llvm, even if its not needed | 13:37 |
persia | Right. THe new system is more elegant, and gives system definers greater control over the result. | 13:37 |
persia | It is just also more complex, requiring a greater understanding by system definers. | 13:37 |
persia | (hence my reservations) | 13:37 |
locallycompact | There's no additional information that wasn't in there before | 13:38 |
persia | locallycompact: In definitions, true. In system definitions, yes there is. | 13:38 |
persia | So for folk that wanted to ignore strata, and just define systems from strata, there is a higher level of understanding required. | 13:38 |
persia | One could (safely) argue that they shouldn't be defining systems without that understanding, of course :) | 13:38 |
locallycompact | They can also start from an example | 13:39 |
persia | cargo culting is less reliable with the new model, because inserting things requires understanding the concepts of build-time and run-time dependencies. | 13:39 |
persia | Under the old model, one could just duplicate a line and change the name and have a reasonable expectation of things working. | 13:40 |
jjardon | locallycompact: another: is it possible to have same/very similar definitions but change the configuration options of some chunks depending on what you want to build? for example: "only support wayland, only support xorg, support both" without the need of duplicating all the strata? | 13:40 |
* persia hopes not | 13:40 | |
locallycompact | jjardon, you must duplicate the strata containing chunks that have different build instructions | 13:41 |
persia | If you really want to optimise in that direction, it would be possible to create a chunk that is a build-dependency of those, which generates a file based on it's own build-dependencies, which file is sourced in the build-instructions for the chunks that should vary. | 13:42 |
persia | Please don't do that :) | 13:42 |
jjardon | well, its not exactly true at the moment: we have the same mesa chunk, but it build different drivers depending if Im building for ARM or x86 | 13:42 |
locallycompact | personally I'm fine with parameterising build-instructions at the stratum level, but it got bombed quite recently | 13:42 |
* locallycompact checks mesa | 13:42 | |
persia | locallycompact: Did you ever solve the issue of being able to assert the correct cache key in that case? | 13:42 |
* persia was involved in the bombing, but is more concerned about cryptographic reliability than the file formats | 13:43 | |
locallycompact | One possibility is only allow parameterisation via environnment variables and hash those | 13:43 |
locallycompact | so in the chunk entry, envs: [dict] | 13:43 |
persia | Doesn't the sandbox clear the environment though? | 13:44 |
locallycompact | and the build-instructions are explicitly coeffectful | 13:44 |
locallycompact | thing propagate through, like DESTDIR | 13:44 |
richard_maw | locallycompact: parameterising at build command environment variable level doesn't solve the "no llvm on arm" problem | 13:44 |
locallycompact | you have to set that from the juresdiction of the build tool | 13:44 |
persia | That's not propagation: that's defined by the build tool. | 13:44 |
locallycompact | richard_maw, no it was for the mesa thing | 13:44 |
persia | richard_maw: What is the "no llvm on ARM problem"? | 13:45 |
locallycompact | no llvm on arm is solved by having a system that doens't have llvm on arm | 13:45 |
* richard_maw meant the mesa having different dependencies on different architectures problem | 13:45 | |
locallycompact | system's problem now | 13:45 |
jjardon | persia: 14:35 <jjardon> locallycompact: in your new model, is it easier/possible to express: "I want this stratum depending in this other for the arch, but not for other arch" (ie, build llvm as a dependency of mesa only in x86) | 13:45 |
persia | I beleive that to be a foolsih optimisation, but yes, it's a system issue. | 13:45 |
persia | For instance, I probably want LLVM on ARM if I'm using AMD or NVIDIA graphics cards. | 13:46 |
persia | locallycompact: I usually like your words, but "coeffectful" bothers me. Do you maybe mean "coeffectual"? | 13:47 |
locallycompact | I used coeffectful in this sense http://tomasp.net/coeffects/ | 13:48 |
locallycompact | The build instructions expect a coeffect from the environment and fail if they don't recieve it | 13:48 |
persia | locallycompact: Ah, yes, thanks for the reference. I suppose it is Tomas' use of English with which I have argument (although that "effect" has an irregular adjectival form isn't necessarily widely taught). | 13:50 |
locallycompact | he will have gotten it from purescript developers | 13:53 |
locallycompact | http://www.purescript.org/learn/eff | 13:53 |
locallycompact | I don't know what else has support for typed effects, I think maybe Idris | 13:54 |
persia | No worries. There's too much history to fix this now. | 13:54 |
locallycompact | yeah good old idris http://docs.idris-lang.org/en/latest/effects/index.html | 13:54 |
persia | English appears to have gained a new word, which just happens to be one that I was told was incorrect when I was in school. | 13:55 |
persia | More synonyms are fun ("effective" and "effectual" were disambiguated in the 14th century. "efficacious" was considered flowery when invented, and can mean either of those. "effectful" is new, and seems to mean the same thing as "effectual", at least for now, except restricted to computer science as technical jargon) | 13:57 |
locallycompact | really could use a dedicated baserock gitlab instead of this public hooha | 14:56 |
persia | You'd want to have a discussion with other folk: lots of folk are using the gerrit managed by the baserock infra team :) | 14:58 |
locallycompact | gerrout | 14:58 |
*** ctbruce has joined #baserock | 15:08 | |
*** jonathanmaw has quit IRC | 15:12 | |
*** jjardon_matrix has quit IRC | 15:21 | |
*** gtristan has quit IRC | 15:32 | |
*** locallycompact has quit IRC | 15:56 | |
*** rdale has quit IRC | 16:09 | |
*** rdale has joined #baserock | 16:10 | |
leeming | question for anyone that has actually used linux-user-chroot or sandboxlib... | 16:19 |
leeming | the run_sandbox method seems to return back exit, out, err... but, im just wondering how it returns anyway, as you usually need to ctrl-D or exit from sandboxes? | 16:20 |
leeming | not like it is synchronous call | 16:20 |
richard_maw | it terminates after it finishes running the provided command | 16:21 |
leeming | just trying to understand that, as I am getting failures when testing bubblewrap integraton and do no know if i am doing it right | 16:21 |
leeming | oh | 16:21 |
leeming | so it doesn't keep the sandbox/session open? | 16:21 |
richard_maw | nopety nope | 16:21 |
* leeming consults the bubblewrap doc | 16:22 | |
*** jjardon_matrix has joined #baserock | 16:44 | |
*** jjardon_matrix has quit IRC | 16:50 | |
*** gtristan has joined #baserock | 16:52 | |
*** toscalix has quit IRC | 17:20 | |
*** jjardon_matrix has joined #baserock | 17:44 | |
*** rdale has quit IRC | 17:51 | |
*** ctbruce has quit IRC | 18:01 | |
*** jjardon has quit IRC | 21:46 | |
*** jjardon has joined #baserock | 21:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!