IRC logs for #baserock for Thursday, 2016-09-15

*** gtristan has joined #baserock04:36
gtristanCan I use the 'verbatim' build system and extend it ?05:09
gtristanI cant seem to find any chunks which have build-system: verbatim and also a custom morph that does some extra post frobnication05:10
gtristanok nevermind I got it05:22
gtristanit's defined in the DEFAULTS05:22
paulsherwoodgtristan: afaik verbatim hasn't been upstreamed to definitions06:17
*** fay_ has joined #baserock06:48
gtristanpaulsherwood, ah, anyway I got it, just doing stuff in post-install-commands there06:50
*** paulwaters_ has joined #baserock07:03
*** paulwaters_ has quit IRC07:03
jjardonHi, what is the var to refer to the src folder in a .morph file? Does something like $SRCDIR exits?07:12
*** toscalix_ has joined #baserock07:31
pedroalvarezjjardon: every command is executed from the src folder07:48
pedroalvarezthere should be a var for that, though07:48
gtristanI agree, I've wanted one before more than once07:48
gtristanand have resorted to saving `pwd`07:48
gtristanbut I just checked and can confirm at least in this build I'm running, there isnt one07:49
jjardonIs this a bug in definitions or in ybd/morph?07:52
*** locallycompact has joined #baserock07:55
*** locallycompact has quit IRC07:56
*** locallycompact has joined #baserock07:56
paulsherwoodall of the above :)07:56
*** locallycompact has quit IRC08:04
*** locallycompact has joined #baserock08:48
*** tiagogomes has joined #baserock09:02
*** ctbruce has joined #baserock09:15
richard_mawtoday's lwn has an article about a google thing called minjail, pulls together all the various sandboxing tech in an attempt to do as much containment as possible10:12
edcraggnice10:12
richard_mawhttps://lwn.net/Articles/700557/ is the article. Couldn't find any links to the code for it though. Google just gives depressing results.10:13
locallycompactonly avaliable to subscribers :(10:13
richard_mawlocallycompact: are you not on Codethink's corporate subscription then?10:15
*** toscalix_ has quit IRC10:16
leemingah, sounds like sandboxlib10:16
* leeming also hit the paywall10:16
paulsherwoodhttps://lwn.net/SubscriberLink/700557/091f804480fab479/10:17
* leeming wonders if this is it https://github.com/omegaup/minijail10:18
*** toscalix_ has joined #baserock10:18
leemingactually.. google : https://chromium.googlesource.com/chromiumos/platform/minijail/10:18
*** toscalix has joined #baserock10:24
*** toscalix_ has quit IRC10:26
leeminggood catch richard_maw, this is an interesting read10:31
pedroalvarezmaybe this is the most recent one https://chromium.googlesource.com/aosp/platform/external/minijail/10:33
leemingah yes, good catch10:39
leemingis brillo/weave of interest to anyone here? https://developers.google.com/brillo/10:45
*** toscalix has quit IRC10:48
*** toscalix has joined #baserock10:59
jjardonthis is a good article as well: http://lwn.net/Articles/700530/ "Greg Kroah-Hartman, the maintainer of the long-term support kernels, agreed with this assessment of the situation, noting that even serious security fixes don't find their way through to the kernels shipped by vendors despite being promptly included in the LTS kernels. So he is mulling the idea11:40
jjardonof stopping the the maintenance of the LTS kernels entirely: "11:40
jjardonmainline FTW11:40
jmacsnon-subscriber link?11:41
jjardonhttps://lwn.net/SubscriberLink/700530/7f1c21b0fd40489e/11:42
*** ctbruce has quit IRC11:44
*** gtristan has quit IRC11:57
*** gtristan has joined #baserock12:16
locallycompactHere's a first pass at the 009 migration, I would have loved to one million things at once but this change is big enough as it is. https://gitlab.com/baserock/definitions/commit/4b3be68d4fdf5357c9a765cc7c7194c369feaa0012:18
locallycompactI also guess that people prefer non-implicit transitivity in b-ds at the system level which this isn't12:19
jjardonlocallycompact: what v009 tries to achieve?12:21
locallycompactMove build-dependencies from the stratum level to the system level12:21
locallycompactSo if you want to change to a new root it's now a one line change12:22
locallycompactjust change which stratum the 'sysroot' entry points to, for example12:22
locallycompactor add a few and set the b-ds up in the system12:22
locallycompactBut it also means there must be inclusion semantics to decide which strata actually end up in the final system12:22
paulsherwoodwhat's 'include-mode' ?12:24
paulsherwoodthis is the first thing that ybd complains about12:24
locallycompactso previously you are relying on the fact that the stratum is in the list at all to govern whether or not it ends up in the final system12:25
locallycompactyou can't use that anymore12:25
locallycompactso I split it into 'relevant', which is the default, and 'affine' which gets discarded implicitly by the system12:25
paulsherwood-112:25
locallycompact+ a third which doesn't exist yet12:26
locallycompactpaulsherwood, you have a simpler solution?12:26
paulsherwoodi don't know what affine means and am not about to to google it12:27
locallycompacthere it means doesn't go in the final system12:27
paulsherwoodplease explain in terms a normal human can understand :)12:27
richard_mawlocallycompact: they are build-time-only dependencies, right?12:28
locallycompactyes12:28
locallycompactI wanted this also for chunks because for example we have a chunk which is a rootfs tarball and just runs cp -R * $DESTDIR, but we have to do this in two stages, one in bootstrap mode and once in regular mode, and it takes 20 minutes each to split12:28
locallycompactbut bootstrap mode implicitly doesn't get included12:29
paulsherwoodso build-time, run-time? where run-time implies also build-time?12:29
locallycompactso we are doing a whole tarball just for the sake of "cp"12:29
locallycompactwhy not make the initial rootfs tarball relevant12:29
locallycompactand have only the one12:29
locallycompactinclude-mode: "build-time / run-time"12:30
richard_mawgotcha, my previous thoughts on this was moving more stuff out of system definitions and making chunks/strata less composable, but I see you're goign in a different direction12:30
locallycompactit puts one slight dampener in a plan I have for more composable strata assemblages12:31
locallycompactfor example, what happens whan you have a build-time dependency which has a run-time component in order to work12:31
paulsherwoodsorry, what does 'composable' mean?12:31
locallycompactso for example you might want a file containing some strata which doesn't in itself constitute a full system12:32
locallycompactlike all the "base" components"12:32
locallycompactwhjich are unlikely to change across systems12:32
locallycompactthis isn't part of 009 but it is in my mind12:32
locallycompactbut in so far as say "graphics stuff" assemblage contains some things that are only build-time components, are they build-time componetns for strat that depend on graphics stuff or for strata *within* graphics stuff12:33
locallycompactx-org whatever they are12:33
* paulsherwood had hoped for generic nestable groups/sets/collections, some of which are systems12:33
locallycompactright12:33
locallycompactessentially this12:33
locallycompactbut the precendence of include-mode is important here12:33
locallycompactwhat does "foo" end up in the staging area for12:34
locallycompactdoes it just provide locally12:34
locallycompactuntil the local collection is build12:34
locallycompactthen gets discarded in the staging for higher collections12:34
locallycompactor stay in the staging area for everything and also go in the system12:35
locallycompactor somewhere between12:35
locallycompacttwo modes gets you some of the way I'm not sure all of the way12:35
paulsherwoodthat requires that anything higher re-specifies foo if it needs it, which for some reason was not the way we originally approached this...12:35
* paulsherwood was surprised to discover that our dependencies where 'transitive' but lots of folks seemed to think this was obviously correct12:36
persialocallycompact: My concern with moving build-depends to the system level is how the artifact is defined: is this sufficiently abstracted that if I build the same chunk for two systems with different expressed build dependencies, I get separate artifacts?12:41
locallycompactthat's on the cache key generator to figure that out12:41
locallycompactand it should take into account everything that could potentially make a difference12:41
persiaIt's the "should" that worries me :)12:42
locallycompactone can't cheat the doing things right of the build tool12:42
persiaI suppose it's an even tradeoff between the need to duplicate lots of entries to switch between e.g. glibc/gcc and musl/clang and the need to maintain duplicate chunk entries for the same.12:43
persiaMy thought was more about the complexity of thought required by the build tool developers: abstracting things to higher levels means that one needs to manage the cache key entirely symbolically, rather than being able to assert that the contents of files are meaningful input.  I suppose that as long as the build tool developers are willing to ignore a "file hash" as an important expression this doesn't matter.12:44
persiaThinking about it more, I think this also complicates the creation of systems.12:48
persiaOne of the original ideas was that a system definition should be fairly short and easy to write, although the tooling never actually achieved the goal of making that entirely true (for various reasons).12:48
locallycompactavoiding having to duplicate an entire folder of 40 strata in order to replace one lower stratum was the goal here12:49
persiaAnd we never established a good model for definitions that would let folk maintain lots of systems effectively.12:49
persiaSo maybe that goal can be set aside.12:49
persialocallycompact: Indeed: from a maintenance point of view, that makes life easier.12:49
paulsherwoodthat's a bit harsh.. i think definitions is 'ok', just not 'great'12:49
persiapaulsherwood: Which is harsh?12:49
paulsherwood13:49 < persia> And we never established a good model for definitions that would let folk maintain lots of systems effectively.12:50
persiapaulsherwood: Yes.  I'm specifically speaking about the issue of how to maintain downstream systems.  I remember folk working on this for a while, but the consensus seemed to be that it was just easier to do work upstream, or fork and resync later.12:51
persiaI don't consider either of these as easy to manage as other tooling for package-based systems to maintain custom configurations downstream.12:51
persiaI very much do not mean to criticise other aspects of definitions by that statement.12:52
locallycompactit's true about the downstream thing12:52
persiaI think this change makes downstream management harder.  I'm not sure that we care, really.12:52
locallycompactno12:52
persiaBecause we never really solved the downstream issue, so this doesn't make it worse.12:52
locallycompactwhat happens previously is upstream updates al lthe refs for a stratum and we can't absorbd it because the build-dependencies conflict12:53
locallycompactnow we can use identical stratum files because there's no downstream particular information in them12:53
locallycompactwe might change a sha for a component, but we also have the capability to  treat that as a different stratum entirely12:54
persiaYes, but they no longer mean anything, so it isn't importnat.12:54
locallycompactThey represent what somebody expects to work in some capacity12:55
locallycompactYou can slot that into your system and yay or nay12:55
persiaPlus, although the declaration of build-dependencies has been moved to the system level, there are still many cases where the build-dependencies are required at the chunk level (because of the build commands), so things don't reliably work when the underlying implementation changes.12:55
locallycompactif nay, maintain another stratum, just not forty12:55
persiaI agree it solves the forty-stratum problem.12:55
persiaI just believe that this change means that it is no longer interesting to look at the downstream issue, because it cannot be safely solved: folk needing that sort of thing would be better served with packages.12:56
persia(ideally packages that don't have runtime installation scripts, and in combination with pregenerating system artifacts for validation before releasing, rather than installing packages at system instantiation time, but there exist systems that work this way)12:57
locallycompactI would have thought the underlying implementation thing is solved by strata being architecture specific and pointing their chunks at the relevant build instructions12:57
persiaThe build instructions depend on the set of build dependendencies.12:58
locallycompacti.e build-essential-armv7, build-essential-x86, but the higher strata need not change necessarily12:58
locallycompactThat's not always the case12:58
persiaNo.12:58
persiaAlthough I believe it also a rare case that one needs different build-essentials on a per-ISA basis.12:58
locallycompactISA?12:58
* jjardon thinks this will solve the problem on upstreaming the work Ive been doing lately: replace busybox stage3 for GNU utils12:59
persiaIn most cases, upstream tooling is designed to generate completely different results based on the passed ISA, rather than needing different bits installed.12:59
persiaInstruction Set Architecture.12:59
persiajjardon: It will help massively with that effort, indeed.13:00
persiajjardon: Further, it will make it easy for people to choose busybox or GNU at the system level, rather than needing to duplicate the hierarchy, so it makes it easier to run them in parallel.13:01
*** brlogger has joined #baserock13:02
*** brlogger has joined #baserock13:02
jjardonpersia: patches are almost ready, but I have that problem: or we always use busybox, or we always use GNU utils. You could build always with GNU utils and then build busybox but that defeats the pourpose of busybox I think13:03
locallycompactIt makes system definitions a lot bigger I'll admit, but I think working towards nested composableness will reduce it to I want some compilery things and some graphicsy things and maybe I'll decompose it to exchange one layer out when I need to13:03
persialocallycompact: So, igoring the x86 vs. armv8 issue (because it's safely separable), contrast the new system definition with some of the other tooling: here's the germinate system definition: http://people.canonical.com/~ubuntu-archive/seeds/lubuntu.yakkety/desktop-gtk13:03
locallycompactgood lord what is this13:04
persiaA "seed file" for germinate.  It's the tool used for defining the systems that LP generates.13:04
locallycompactLP is?13:04
persiaI think germinate is also in Debian, so you don't need to use Ubuntu if you want to use this sort of format.13:04
persialaunchpad.net13:04
persiaGerminate is a tool that consumes "seed" files (like the one linked), and then creates package lists, to define systems (where the packages can be installed to create a deployable system or they can be copied somewhere to create installation media, or whatever)13:06
locallycompactThat's interesting13:07
persiahttps://wiki.ubuntu.com/Germinate  and https://wiki.ubuntu.com/SeedManagement are the relevant docs, although I don't know if they are up-to-date13:08
* persia hasn't used germinate to build systems in over 5 years13:08
persiaNote that the specific structure used in Ubuntu also tries to enforce the "ogre model", wherein subsets of the available sources are required to be self-contained sets in terms of build- and run-time dependencies.13:09
persiaSome subsets may depend on others (so it makes layers), but by separating them, this allows organisations to be able to select set-theoretic collections of packages for support with confidence that they would also support the necessary dependencies (both types).13:10
*** jonathanmaw has joined #baserock13:10
persiaMind you, since I was involved, lots of things have happened, including a reduction in the number of organisations providing support, so I encourage you to take ideas from it, rather than digging too deeply into the current implementation and practice.13:11
persiaAnd, of course, as with any debian-format package based approach, the resultant systems are subject to the failure rate of apt (about 1/50th of 1%) and the uncertainties of maintainer scripts, so there's plenty to criticise when compared to precomposed system generators (like morph or ybd).13:12
locallycompactmmm13:14
persiaI've not been following recently, but I believe Debian still uses tasksel to define the tasks that end up on live images, install images, etc.13:17
persiaHere's a link to the relevant README in the VCS: https://sources.debian.net/src/tasksel/3.35/README/13:17
persiaAnyway, the point of this is that past system definitions were more closely aligned with the simpliciy of these sorts of formats, and the expressed goal was to make them even easier to write.13:18
persiaI think this change moves in the other direction, making them potentially more complex than other tooling.13:19
persiaBut, as I said, given the issues with downstream definitions anyway, I'm not sure this is entirely a bad thing: the ways in which this definitions format change makes it easier to work downstream may well offset the ways in which it makes it harder.13:20
rjekWho on earth thinks Markdown or similar is a sensible data format13:22
persiaI don't know how the Fedora infrastructure creates livecds (and searching for this keeps sending me to instructions to modify livecds, without pointers to the source), but I believe it is custom kickstart files that are used to define the contents of the liveCD or USB image (or cloud image).13:23
jonathanmawIs /usr/include/linux/a.out.h a sensibly-named file, or has something gone wrong with my gcc chunk?13:23
locallycompactrjek, people who haven't yet discovered rst :)13:23
persiarjek: It's moinmoin, actually.13:23
richard_mawjonathanmaw: I've got one.13:23
jonathanmawok, just need to figure out why this version of systemd isn't able to find something that exists in a.out.h13:24
jjardonlocallycompact: in your new model, is it easier/possible to express: "I want this stratum depending in this other for the arch, but not for other arch" (ie, build llvm as a dependency of mesa only in x86)13:35
locallycompactyes13:36
persiaErr, not really.13:36
locallycompactyou need two system definitions of course13:36
persiaOne can have a system that is x86 where mesa depends on llvm, and another that is armv7 where it doesn't.13:36
locallycompactyes that13:36
persiaOne cannot express architecture dependence for build-dependencies directly.13:36
locallycompactright13:36
jjardonok, cool; we currently always build llvm, even if its not needed13:37
persiaRight.  THe new system is more elegant, and gives system definers greater control over the result.13:37
persiaIt is just also more complex, requiring a greater understanding by system definers.13:37
persia(hence my reservations)13:37
locallycompactThere's no additional information that wasn't in there before13:38
persialocallycompact: In definitions, true.  In system definitions, yes there is.13:38
persiaSo for folk that wanted to ignore strata, and just define systems from strata, there is a higher level of understanding required.13:38
persiaOne could (safely) argue that they shouldn't be defining systems without that understanding, of course :)13:38
locallycompactThey can also start from an example13:39
persiacargo culting is less reliable with the new model, because inserting things requires understanding the concepts of build-time and run-time dependencies.13:39
persiaUnder the old model, one could just duplicate a line and change the name and have a reasonable expectation of things working.13:40
jjardonlocallycompact: another: is it possible to have same/very similar definitions but change the configuration options of some chunks depending on what you want to build? for example: "only support wayland, only support xorg, support both" without the need of duplicating all the strata?13:40
* persia hopes not13:40
locallycompactjjardon, you must duplicate the strata containing chunks that have different build instructions13:41
persiaIf you really want to optimise in that direction, it would be possible to create a chunk that is a build-dependency of those, which generates a file based on it's own build-dependencies, which file is sourced in the build-instructions for the chunks that should vary.13:42
persiaPlease don't do that :)13:42
jjardonwell, its not exactly true at the moment: we have the same mesa chunk, but it build different drivers depending if Im building for ARM or x8613:42
locallycompactpersonally I'm fine with parameterising build-instructions at the stratum level, but it got bombed quite recently13:42
* locallycompact checks mesa13:42
persialocallycompact: Did you ever solve the issue of being able to assert the correct cache key in that case?13:42
* persia was involved in the bombing, but is more concerned about cryptographic reliability than the file formats13:43
locallycompactOne possibility is only allow parameterisation via environnment variables and hash those13:43
locallycompactso in the chunk entry, envs: [dict]13:43
persiaDoesn't the sandbox clear the environment though?13:44
locallycompactand the build-instructions are explicitly coeffectful13:44
locallycompactthing propagate through, like DESTDIR13:44
richard_mawlocallycompact: parameterising at build command environment variable level doesn't solve the "no llvm on arm" problem13:44
locallycompactyou have to set that from the juresdiction of the build tool13:44
persiaThat's not propagation: that's defined by the build tool.13:44
locallycompactrichard_maw, no it was for the mesa thing13:44
persiarichard_maw: What is the "no llvm on ARM problem"?13:45
locallycompactno llvm on arm is solved by having a system that doens't have llvm on arm13:45
* richard_maw meant the mesa having different dependencies on different architectures problem13:45
locallycompactsystem's problem now13:45
jjardonpersia: 14:35 <jjardon> locallycompact: in your new model, is it easier/possible to express: "I want this stratum depending in this other for the arch, but not for other arch" (ie, build llvm as a dependency of mesa only in x86)13:45
persiaI beleive that to be a foolsih optimisation, but yes, it's a system issue.13:45
persiaFor instance, I probably want LLVM on ARM if I'm using AMD or NVIDIA graphics cards.13:46
persialocallycompact: I usually like your words, but "coeffectful" bothers me.  Do you maybe mean "coeffectual"?13:47
locallycompactI used coeffectful in this sense http://tomasp.net/coeffects/13:48
locallycompactThe build instructions expect a coeffect from the environment and fail if they don't recieve it13:48
persialocallycompact: Ah, yes, thanks for the reference.  I suppose it is Tomas' use of English with which I have argument (although that "effect" has an irregular adjectival form isn't necessarily widely taught).13:50
locallycompacthe will have gotten it from purescript developers13:53
locallycompacthttp://www.purescript.org/learn/eff13:53
locallycompactI don't know what else has support for typed effects, I think maybe Idris13:54
persiaNo worries.  There's too much history to fix this now.13:54
locallycompactyeah good old idris http://docs.idris-lang.org/en/latest/effects/index.html13:54
persiaEnglish appears to have gained a new word, which just happens to be one that I was told was incorrect when I was in school.13:55
persiaMore synonyms are fun ("effective" and "effectual" were disambiguated in the 14th century.  "efficacious" was considered flowery when invented, and can mean either of those.  "effectful" is new, and seems to mean the same thing as "effectual", at least for now, except restricted to computer science as technical jargon)13:57
locallycompactreally could use a dedicated baserock gitlab instead of this public hooha14:56
persiaYou'd want to have a discussion with other folk: lots of folk are using the gerrit managed by the baserock infra team :)14:58
locallycompactgerrout14:58
*** ctbruce has joined #baserock15:08
*** jonathanmaw has quit IRC15:12
*** jjardon_matrix has quit IRC15:21
*** gtristan has quit IRC15:32
*** locallycompact has quit IRC15:56
*** rdale has quit IRC16:09
*** rdale has joined #baserock16:10
leemingquestion for anyone that has actually used linux-user-chroot or sandboxlib...16:19
leemingthe run_sandbox method seems to return back exit, out, err... but, im just wondering how it returns anyway, as you usually need to ctrl-D or exit from sandboxes?16:20
leemingnot like it is synchronous call16:20
richard_mawit terminates after it finishes running the provided command16:21
leemingjust trying to understand that, as I am getting failures when testing bubblewrap integraton and do no know if i am doing it right16:21
leemingoh16:21
leemingso it doesn't keep the sandbox/session open?16:21
richard_mawnopety nope16:21
* leeming consults the bubblewrap doc16:22
*** jjardon_matrix has joined #baserock16:44
*** jjardon_matrix has quit IRC16:50
*** gtristan has joined #baserock16:52
*** toscalix has quit IRC17:20
*** jjardon_matrix has joined #baserock17:44
*** rdale has quit IRC17:51
*** ctbruce has quit IRC18:01
*** jjardon has quit IRC21:46
*** jjardon has joined #baserock21:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!