IRC logs for #baserock for Friday, 2016-10-21

*** gtristan has quit IRC04:23
*** gtristan has joined #baserock04:37
*** gtristan has quit IRC05:09
*** gtristan has joined #baserock05:26
*** ctbruce has joined #baserock06:50
*** rdale has quit IRC07:23
*** juergbi has quit IRC07:34
*** juergbi has joined #baserock07:40
*** locallycompact has joined #baserock07:59
*** rdale has joined #baserock08:38
*** ctbruce has quit IRC08:46
*** ctbruce has joined #baserock08:46
*** gtristan has quit IRC09:02
tiagogomespaulsherwood do you remember which system fails if the integration commands are run after the splitting?09:12
locallycompactwhat is subsystems?09:14
pedroalvarezlocallycompact: I think this explains some things about subsystems: https://docs.baserock.org/deployment/09:15
pedroalvarezalso mentioned here: https://docs.baserock.org/spec/#deployment-definitions-clusters09:16
locallycompactmmm09:17
tiagogomesI didn't know about docs.baserock.org. Looks nice09:17
locallycompactI might submit this with clusters broken as a first pass09:18
*** tiagogomes has quit IRC09:30
*** tiagogomes has joined #baserock09:32
* paulsherwood notices tiagogomes has asked, and then disappeared09:32
tiagogomespaulsherwood I did, but not because I was afraid of the answer :)09:34
paulsherwoodthe idea was to be able to apply default splits to any system. so we should be able to create devel-runtime devel-minimal09:34
paulsherwoodi can't remember, but i'm pretty sure those systems don't have everything required for si-commands09:35
paulsherwoods/remember/remember the dtails/09:35
persiaThen they should have different si-commands.09:36
paulsherwoodright. but i've been aiming at tooling to work with definitions as they are09:37
persiaThe problem is that if the system-integration commands run before split, there's no way for the commands themselves to know what will be in the final system, meaning it is unsafe to populate caches, set configuration to use preferred implementations for pluggable ABIs, etc.  If these cannot be done at si-time, when are they to be done?09:37
*** gtristan has joined #baserock09:37
paulsherwoodpersia: caches only contain chunk artifacts09:38
persiaI agree that running system-integration after splitting is a handy shortcut to avoid adjusting definitions, it just happens to create wrong results.09:38
persiapaulsherwood: In-system caches, most importantly the ld.so cache.09:38
persia(but also things like the ca-certificates cache, correctly composing the initrd contents, etc.)09:42
tiagogomespaulsherwood I am not sure if I follow you. Are you referring to creating two system artifacts (devel-runtime and devel-minimal) every time a system is built?09:51
paulsherwoodtiagogomes: no. i'm suggesting that if someone specifies default-splits: minimal, they'd get minimal systems10:18
* SotK wonders where that gets specified10:21
tiagogomesI guess is some ybd flag10:21
paulsherwoodhttps://gitlab.com/baserock/ybd/blob/master/ybd/config/ybd.conf#L8410:22
* paulsherwood believes this works better than what was there before, but it still needs improvements10:23
SotKI feel like it should at least be set in definitions rather than ybd, else I have to tell someone which default-splits value to use when they try to reproduce a thing, rather than everything they need to know being defined in the system definition10:23
* paulsherwood knows that radiofree concluded that our splits are not splitting enough10:23
tiagogomesI am not convinced by that flag . What dictates what goes into a system artifact should be mandated by definitions and definitions only no?10:24
SotKtiagogomes: +110:24
paulsherwoodthe splits themselves are in the defaults in definitions10:27
paulsherwoods/splits/splitting rules/10:27
paulsherwoodthe cache-key notes which actual splits are applied10:28
gtristanthe system integration commands is an interesting problem10:28
* gtristan missed part of the conversation yesterday, but admits there are things to work out there10:28
paulsherwoodi'm happy for folks to improve the approach of course10:28
gtristanfor instance; if one wanted to deploy to rpm; one would probably want the rpms to pickup the artifacts before system integration and implement the si commands as %post scripts10:29
tiagogomesEven in that case, the system integration commands would run after the splitting10:32
gtristantiagogomes, right, they would10:35
persiagtristan: Precisely.  The s-i commands are intended to represent the results of maintainer scripts (both pre- and post-) to make system-specific adjustments to the content.  We do them at build time in an attempt to ensure we can repeat systems, rather than relying on the right balance of cosmic rays in the target system at package installation time.10:37
gtristanthe problem we had with s-i commands was worse than just adjusting definitions (I think, unless the systems we were defining actually did not make sense), we ended up with building a minimal system and then running s-i commands in a sandbox with no shell10:38
gtristanI believe that's when it changed10:38
persiaThe minimal system had no shell?10:39
persiaIf so, the minimal system integration commands should not require a shell :)10:40
gtristanwhich is quite impossible heh10:40
persia(but I'd assert that the minimal system should have a shell)10:40
persiagtristan: Not really: just have them written in a compiled language, and fork into them.10:40
gtristanOk, well then we need our sandboxing code to improve to handle that, I think s-i commands, and all others, are given to the shell10:41
pedroalvarezi was going to say that10:41
pedroalvarezI believe they do need shell, allways10:41
persiaHistorically, Debian's debian/rules file is defined as causing certain behaviours in response to certain arguments.  While >99.99% of them are makefiles, the definition was chosen to allow one to use other sorts of files in the case where one needed to build something without having `make` available.10:42
gtristanYes, and arguably they should continue to do so, because we would want to use shell variable expansion more, not less10:42
tiagogomesWhat's a minimal system here. For me a minimal system what e.g. systems/minimal-system-x86_64-generic.morph is about. Is a minimal system, in ybd context, a system that only includes the $strata-minimal artifacts?10:42
gtristanHowever, that does not rule out asserting that the sandbox is responsible for having an executable shell, regardless of what content is actually staged10:42
gtristantiagogomes, we're talking about using the split-artifact rules indeed10:43
tiagogomesIn that case, if ybd default-splits went away, probably there would be no problem of running the system integration commands after with current definitions.10:45
paulsherwoodnot just that. it should be possible to build *-system as a runtime or minimal version10:45
paulsherwoodif we drop the default-splits idea, we're requiring that users explicitly define their splits for all systems10:46
* paulsherwood would appreciate radiofree comments on this, but believes he's travelling10:46
gtristanI think this needs some semantics, and running system integration commands is an optional transformation one might perform on a system artifact; turning it into something else. On the one hand, one wants to be able to split the system artifact and encode system-integration commands into packages, if that is what their deployment requires; on the other hand; one wants to be able to run the system integration commands on a system before pruning out10:49
gtristanunneeded cruft, like when building a runtime10:49
gtristanSay where the runtime is defined as something like a flatpak runtime (a chrootable or such area where one can run a program with a predetermined set of libraries present, but no tools/shells/dev files or such)10:50
persiaMy concern with asserting that the sandbox has a shell is that the results of system integration may create artifacts that would expect a shell on the target system.10:50
gtristanpersia, whether it is a shell, or a shared library (say pango-querymodules or fontconfig programs), there is a dependency there10:52
*** anahuelamo has quit IRC10:52
gtristanso either there is syntax to account for that dependency, or the burden is on the definitions author to know that ?10:53
gtristanof course, if the sandbox makes a guarantee of the shell being present; s-i commands would only work within the sandbox on on targets that have a shell (so of course your point still stands)10:56
gtristans/on on/or on10:56
*** vgrade12 has quit IRC10:56
gtristanpersia, I'm curious what project atomic is doing with such commands10:59
gtristanprobably similar, installing everything with %post scripts run from package builds before making an ostree commit11:00
gtristanbut really not sure11:00
rdaleit just seems to be like a build dependency to me - you may need bash to run the system integration commands on a minimal system, and not need bash in the final system. so you should just create a 'system integration artifact' against the system made of split artifacts, and just add the system integration one when built11:00
*** vgrade12 has joined #baserock11:01
persiagtristan: I believe they are running %post before commit, yes, but I don't know for sure.11:08
persiardale: There may be a need for code run before splitting, but it's not the right time to run things that are usually in the post-install scripts.11:08
gtristanlooks like it happens in this monstrosity: https://github.com/projectatomic/rpm-ostree/blob/master/src/libpriv/rpmostree-core.c#L184711:11
* gtristan squirms looking at rpmTs again11:11
gtristanrdale, indeed with rpm we even have explicit Requires(post)/Requires(postun) in order to specify what is required to run a given script11:14
gtristanrdale, but I think that comes down to what you're going to do with the output - anyway if you assume the output is always going to be a system image, then a single strategy will do11:15
jjardonHi! Any idea why ybd is failing in this runner? Its using the same docker image as the public runners... https://gitlab.com/baserock/definitions/builds/538423111:16
gtristanrdale, which brings me back to the above, I have a feeling ultimately the definitions user may or may not want to perform a system integration transformation before doing other things, like splitting up and deploying chunks in whatever fashion11:16
jjardonlocallycompact: be aware of that ^11:17
locallycompactthe public runners use ruby2:1 no?11:19
locallycompactruby:2.111:19
leemingstill pending https://github.com/CodethinkLabs/sandboxlib/pull/25 any mayor concerns?11:20
jjardonlocallycompact: you can change that in your .gitlab-ci,yml11:20
locallycompactYou said11:21
locallycompact Its using the same docker image as the public runners..11:21
jjardonlocallycompact: when I use the public runners (with the python:2.7 image), It's not a problem11:23
locallycompactoh right11:23
jjardonlocallycompact: we have our own elastic runners now (still experimenting with it)11:24
locallycompactbuilding some assemblargs https://gitlab.com/baserock/ybd/commits/staging/01011:31
jjardonlocallycompact: I will come back to public runners until we know why this is happening11:32
jjardonbtw, didnt test it yet but I think it would be possible to replace KBAS with the distributed cache gitlab provides, I will get some numbers at some point to see what is faster11:34
paulsherwoodtiagogomes: are you ok with leeming's sandboxlib changes now?11:37
locallycompactjjardon, nice11:38
tiagogomespaulsherwood, I'll have a look. leeming have you successfully built a system by with brwap by now?11:39
leemingit built before with root yes, not without root, but that is a ybd issue not sandboxlib11:40
tiagogomesIf you comment out the code on ybd to create the device nodes, wouldn't it build as non-root?11:41
locallycompactwhat do we want to do about moving splitting11:41
paulsherwoodjjardon: not running as root?11:45
locallycompactybd not lets you11:46
paulsherwood?11:46
paulsherwoodjjardon: `$ ybd/ybd.py systems/build-system-x86_64-chroot.morph x86_6411:47
paulsherwoodsuggests it's not running as root11:47
locallycompactybd stops before anything if you're not root11:47
leemingtiagogomes, no, but gtristan started to look into it. but that is a ybd issue11:47
locallycompact16-10-21 00:00:00 [SETUP] ERROR: ../ybd/ybd.py needs root permissions11:48
leemingyes, standard ybd catches that11:48
locallycompactthose python docker images are only root11:48
locallycompactthey don't have sudo11:48
gtristanleeming, tiagogomes, besides device nodes, there are other reasons ybd needs to be root, some chunks for instance need to install things as specific users; possibly things need to be installed as setuid root binaries as well11:49
tiagogomespaulsherwood, What's the user case for -mininal, -runtime and -devel artifacts of the same system? Let's consider embedded. You want to have a system definition for the production system with the target bsp, the vendor software, no debug symbols and no headers. In addition, you want a system definition for debugging the system, with the vendor software built without optimizations, with debug symbols, headers and a couple of de11:49
tiagogomesbbuging tools (e.g gdb-server, strace…). That looks to me two different system morphologies.11:49
gtristanthe bubblewrap approach solves regular file ownership but not device nodes11:49
paulsherwoodthe actual error in jjardon's run is RuntimeError: ['mount', '-t', 'tmpfs', '-o', '', 'tmpfs', '/root/ybd/tmp/tmptDaWrZ/dev/shm'] failed: mount: permission denied11:50
gtristanyocto does not do device nodes afaik (rburton says that those are entirely left to udev / systemd or such at boot time)11:50
* leeming just wants the PR to be reviewed so he can move on :)11:50
* paulsherwood is tempted to merge it11:50
locallycompactcertainly looks like it's root then11:50
locallycompactif it was able to mkdir /root/ybd/tmp11:50
leemingI am writing a bit of a writeup atm, so remaining features can be patched11:51
paulsherwoodtiagogomes: having two different systems means keeping them in sync. the use-case is developer wants all the debug etc, but also wants to deploy a minimal/production version for testing11:53
paulsherwoodbut maybe you're right11:59
gtristanI dont think so tbh, the problem of debugging info is typically solved nowadays by stripping debugging info into separate files (and we are doing this already)12:03
gtristanAnd the argument for compiling with/without optimization shouldnt really necessitate a whole different set of definitions (or "morphology"), rather, would should be able to adjust the "optflags" or such variable in DEFAULTS (or override a variable in DEFAULTS from the ybd command line)12:04
tiagogomesTrue about debug symbols, but besides them, you would want some debug tools installed, including possibly vendor-specific debugging tools.12:05
SotKwhat does keeping two systems in sync entail beyond adding strata to both instead of just one of them, which doesn't seem hugely onerous to me?12:05
locallycompactThis behaviour is what coeffects are for12:06
SotKit'd be even easier with v10, as you could make the larger system contain the smaller system and eliminate that issue12:06
locallycompactwe could be able to set optflags arbitrarily if build instructions expect it and they get it delivered by runtime or minimal12:06
gtristantiagogomes, yes indeed; still, you cannot begin to choose what vector of split-artifacts (including debugging symbols and debuggers as a complete set), had you not split the artifacts in the first place12:07
gtristantiagogomes, however I may have misread your question: argument of -minimal -devel etc of a given *system* is questionable12:08
gtristantiagogomes, i.e. there are various ways that deployment of built software is usable; those useful outputs are not always bootable things, and probably dont fall directly into 3 simple categories "minimal/runtime/devel"12:09
tiagogomesleeming, the sandboxlib tests are failing http://paste.baserock.org/nasoyatalo12:15
*** anahuelamo has joined #baserock12:15
*** anahuelamo_ has joined #baserock12:17
*** anahuelamo has quit IRC12:17
leemingarg thought you raised this the other day and i fixed it :(12:17
locallycompactwhat is this aboiut s-i commands  # 3. asciibetically sort them12:30
locallycompactseems mighty precarious?12:30
*** gtristan has quit IRC12:31
*** paulwaters_ has joined #baserock12:51
*** inara has quit IRC13:00
persialocallycompact: Ordering of s-i commands needs to be deterministic.  How this is determined is an arbitrary choice.  asciibetically is as good as anything else.13:14
persiaBut it is not safe to have the ordering of s-i commands differ depending on system construction, if one is intending to expect that a given chunk integration is going to be similar between systems.13:14
persia(for example, to have s-i commands ordered based on the ordering of artifact application used to compose the system)13:15
locallycompactI don't like it but too much to solve now13:19
*** inara has joined #baserock13:27
*** paulwaters_ has quit IRC13:33
* tiagogomes is sad that paulsherwood merged a pull request that doesn't even pass the tests.14:03
pedroalvareztiagogomes: which one?14:05
SotKdoes gitlab not do gating of merges?14:07
tiagogomespedroalvarez, https://github.com/CodethinkLabs/sandboxlib/pull/2514:07
SotKoh, its still github14:08
tiagogomesThat PR was not even properly reviewed. I didn't review it, I doubt anyone else did it.14:08
pedroalvarez¯\_(ツ)_/¯14:09
leemingtiagogomes, I was just investigating your latest issue... oyu sure you pulled the latest version btw? I tried it on a remote machine14:09
leemingactually scrap that14:13
*** CTtpollard has joined #baserock14:19
leemingtiagogomes, I've patched that now....14:21
leemingnew PR or can reopen?14:21
* tiagogomes doesn't care anymore14:22
leeminghmm well this is the patched PR https://github.com/CodethinkLabs/sandboxlib/pull/2714:30
leemingI included a recorded reminder of the bubblewrap bug i mentioned the other day here14:31
*** CTtpollard has quit IRC14:35
paulsherwoodtiagogomes: i was under the impression that leeming had addressed your comments14:51
leemingi was in the middle of addressing one of them (the patch PR)14:51
paulsherwoodi'm happy to revert it if that's the best thing14:54
paulsherwoodtiagogomes: your call14:55
* paulsherwood guesses that tiagogomes must have meant it when he said he didn't care any more15:08
paulsherwoodleeming: you got me into this mess.. can you help me get out of it, please? :)15:08
paulsherwooddoes your latest pr fix the tests, and leave users in a working state? or do we revert?15:09
leemingit does fix the 'mess' yes15:10
tiagogomespaulsherwood well it is not breaking anything I suppose... unless someone does a new release of sandboxlib to pip15:10
leeming^ i was going to15:10
leemingnot with broken version btw15:10
paulsherwoodtiagogomes: afaik the only user of sandboxlib is ybd, and i was under the impression that leeming's stuff would only have any impact if bubblewrap is present15:11
paulsherwoodthese ar just excuses, though :)15:11
leemingdo we know that 'we' are the only users?15:11
paulsherwoodas i said above, ybd uses it. no other projects afaik15:12
persiaLots of folk don't disclose when they use things, although a lack of bug reports from non-ybd uses is a very strong indicator15:13
leeminganyway, #27 fixes the issue that was raised. The only thing to note is a (non blocking) bug with bubblewrap leaking PWD into the sandbox/reporting back afterwards15:14
leemingcurrently, I think it would be a blocker/ refuse to work, due to a missing config file15:15
leemingwhich I mistakenly missed out15:15
pedroalvareztests to travis!15:15
paulsherwood+115:15
leemingyes, I should have pushed to gitlab's ci.. but i wasn;t sure what we were doing with the github vs gitlab thing. I had asked a few times15:15
leemingif there was an 'official' word on it15:15
paulsherwoodthere's multiple steps in that. let's do one at a time15:16
paulsherwoodso either establish ci at travis (which is for github), or in the fork at gitlab.15:17
pedroalvarezif pull requests are happening in github, please use travis15:18
paulsherwoodack. i expect after this there won't be any pullrequests until the ci is in place :)15:19
pedroalvarezI would expect a PR to configure the CI :)15:19
* leeming knows nothing of travis15:20
pedroalvarezbut yes, that PR will be already tested15:20
paulsherwoodgah ;)15:20
paulsherwoodleeming: it's trivial, similar to gitlab ci15:20
leemingyeah, i woulda thought so too.. was just thinknig of work commitments *looks at remaining time for today* :)15:21
*** anahuelamo_ has quit IRC15:24
*** anahuelamo has joined #baserock15:25
paulsherwoodleeming: is the process for releasing to pypi properly documented?15:28
tiagogomesThere are a few issues in the bubblewrap in sandboxlib. e.g the filesystem is not read only15:29
leemingthere is a "hacking" file that explains it15:29
leemingit should be RO15:30
leemingbut i was told originally it didn't matter anyway, but i did it cos that is the way it should be :P15:30
leeminghttps://github.com/CodethinkLabs/sandboxlib/blob/master/sandboxlib/bubblewrap.py#L27815:31
tiagogomesthat is not working15:32
leeminghow so?15:33
leemingpaulsherwood, I know a few people know have done it too, but i was waiting for a thumbs up before going ahead15:33
*** edcragg has quit IRC16:38
*** anahuelamo_ has joined #baserock16:57
*** anahuelamo has quit IRC16:57
*** edcragg has joined #baserock17:04
*** franred has quit IRC17:10
*** tiagogomes has quit IRC17:11
*** locallycompact has quit IRC18:06
*** rdale has quit IRC18:22
*** ctbruce has quit IRC19:43
*** lachlanmackenzie has joined #baserock20:00
*** lachlanmackenzie is now known as lchlan20:11
*** lchlan has quit IRC22:23
*** vgrade12 has quit IRC23:01
*** ChanServ has quit IRC23:01
*** edcragg has quit IRC23:01
*** mwilliams_ct has quit IRC23:01
*** ChrisPolin has quit IRC23:01
*** jmacs has quit IRC23:01
*** cosm has quit IRC23:01
*** jjardon_matrix has quit IRC23:01
*** tardyp has quit IRC23:01
*** gary_perkins has quit IRC23:02
*** bjdooks has quit IRC23:02
*** persia_ has quit IRC23:02
*** fay_ has quit IRC23:02
*** inara has quit IRC23:02
*** juergbi has quit IRC23:02
*** persia has quit IRC23:02
*** jude_ has quit IRC23:02
*** leeming has quit IRC23:02
*** dabukalam has quit IRC23:02
*** benbrown_ has quit IRC23:02
*** vgrade has quit IRC23:02
*** malinus has quit IRC23:02
*** perryl has quit IRC23:02
*** bfletcher has quit IRC23:02
*** anahuelamo_ has quit IRC23:02
*** cyndis has quit IRC23:03
*** _longines has quit IRC23:03
*** JPohlman1 has quit IRC23:03
*** inara has joined #baserock23:08
*** jude_ has joined #baserock23:08
*** persia has joined #baserock23:08
*** juergbi has joined #baserock23:08
*** mwilliams_ct has joined #baserock23:08
*** edcragg has joined #baserock23:08
*** jmacs has joined #baserock23:08
*** ChrisPolin has joined #baserock23:08
*** vgrade12 has joined #baserock23:08
*** fay_ has joined #baserock23:08
*** cyndis has joined #baserock23:08
*** _longines has joined #baserock23:08
*** JPohlman1 has joined #baserock23:08
*** perryl has joined #baserock23:08
*** bfletcher has joined #baserock23:08
*** dabukalam has joined #baserock23:08
*** benbrown_ has joined #baserock23:08
*** vgrade has joined #baserock23:08
*** malinus has joined #baserock23:08
*** leeming has joined #baserock23:08
*** anahuelamo_ has joined #baserock23:08
*** cosm has joined #baserock23:08
*** jjardon_matrix has joined #baserock23:08
*** tardyp has joined #baserock23:08
*** gary_perkins has joined #baserock23:08
*** bjdooks has joined #baserock23:08
*** persia_ has joined #baserock23:08
*** ChanServ has joined #baserock23:11
*** niven.freenode.net sets mode: +o ChanServ23:11

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!