IRC logs for #baserock for Thursday, 2015-07-23

*** zoli__ has quit IRC00:57
*** zoli__ has joined #baserock05:50
*** paulw has joined #baserock06:12
*** mariaderidder has joined #baserock08:02
*** bashrc_ has joined #baserock08:08
*** mdunford has joined #baserock08:12
*** jonathanmaw has joined #baserock08:18
*** ssam2 has joined #baserock08:32
*** ChanServ sets mode: +v ssam208:32
*** edcragg has joined #baserock08:51
*** gary_perkins has joined #baserock08:56
paulsherwoodssam2: hi there.... how was the conference?08:57
ssam2paulsherwood: hello09:02
ssam2it was interesting09:02
ssam2got various interesting questions and pointers after my Baserock talk09:02
ssam2pointers included: NixOS (of course), but also something OpenStack develop called 'diskimagebuilder'09:03
ssam2someone also asked about integration with Packer, which would be cool to have09:04
ssam2I didn't get my head around diskimagebuilder yet, but it seems to have some useful overlap with Baserock tools:
ssam2seems to be used by OpenStack test infrastructure to build OS images that are then used to test OpenStack (perhaps you could call it 'inception testing')09:05
paulsherwood'inception testing'... wow! :)09:09
paulsherwoodi like it09:10
paulsherwoodi should google packer properly too :)09:10
paulsherwoodwas your talk well received, do you think?09:10
ssam2I think it was, there wasn't a huge crowd, but lots of well-informed questions after09:11
ssam2so a lot of people are interested in the same area09:11
ssam2it was videod, hopefully the video will be out before too long. although with community-led conferences there can often be a long gap before the organisers have any energy to edit videos :)09:12
paulsherwoodediting videos is surprisingly timeconsuming and dull :)09:13
*** lachlanmackenzie has joined #baserock09:14
tiagogomes_I have used diskimage-builder to build images for Ironic09:19
* edcragg too, never thought it would be comparable to baserock09:20
ssam2it's not really comparable to "Baserock", as "Baserock" refers to a project rather than a specific piece of software09:21
edcraggok, morph/ybd09:21
ssam2how does it compare with the disk image creation functionality we have?09:21
ssam2it's cool that you already know it, anyway09:22
edcraggbaserock assembles the os from scratch, where diskimagebuilder, as far as i know, makes ironic disk images, from distribution releases09:22
ssam2right, but how does it compare with `morph deploy`?09:23
ssam2I know it doesn't overlap at all with `morph build`09:23
edcraggi guess it's equivalent to rawdisk.write09:23
edcraggironic itself is a deployment tool09:24
ssam2seems to have a lot more code than rawdisk.write! and a lot of it is shell.. :(09:24
paulsherwoodless code  for the win :)09:25
ssam2one of the main developers of diskimage-builder was in the talk and was saying it might be nice if Baserock could be integrated with it in the same way distros like Ubuntu are. Not sure how easy that would be09:25
ssam2he said it could allow us to reuse some of OpenStack's infrastructure more easily09:25
paulsherwoodwe could look at that. aren't we already able to spin up our own OpenStack infrastructure trivially though?09:26
edcraggthat sounds feasible, it seems to use plugins/modules for different OSs09:26
edcraggdo we have baserock ironic images already?09:28
ssam2paulsherwood: we can deploy an OpenStack cloud easily. But they have lots of components doing automated integration and testing, which I don't know much about at present09:29
paulsherwoodtempest etc...  do our default definitions include any of that?09:29
paulsherwoodpedroalvarez: ^^09:29
ssam2I think we have tempest already]09:29
* rjek assumes this is not related to reading the contents of a CRT from a distance09:30
paulsherwoodi think we need to understand this inception testing concept better maybe.09:30
ssam2seems to be called TripleO:
paulsherwoodit's like turtles all the way down... what tests the tests... what's the infrastructure for the next infrastructure?09:31
ssam2maybe we could run it all on CloudFoundry09:32
paulsherwoodand run that on Azure?09:34
ssam2we have a lot to learn from the financial industry. We could sell this as "cloud derivatives" and make a fortune09:35
dabukalamMy understanding is that tripleO is no longer an ongoing project09:36
ssam2hmm, OK09:36
paulsherwoodat a non-openstack conference i saw a talk which included claims that OpenStack didn't have a solid way to deal with updates to itself yet... do we know if that's true?09:40
nowsterssam2: The reason for libogg-git.lorry was because we already had a libogg.lorry (fixed to tarball). I hadn't realised we now had an inconsistently named ogg.lorry too.09:45
ssam2nowster: ah, I see09:46
pedroalvarezI still type "open" and press tab when I don't remember where I wanted to go in a definitions.git checkout10:48
pedroalvarezI've been working on openstack & baserock for too long10:49
*** bjdooks_ is now known as bjdooks11:20
*** lachlanmackenzie has quit IRC12:05
*** bwh_ is now known as bwh12:15
*** mdunford has quit IRC12:35
*** lachlanmackenzie has joined #baserock13:00
*** jonathanmaw has quit IRC13:32
*** jonathanmaw has joined #baserock13:35
*** jonathanmaw has quit IRC13:37
*** jonathanmaw has joined #baserock13:38
rjekSo.  linux-user-chroot apparently drops the ability to use chown, for good reason (it's designed to let untrusted users have chroots without letting them easily escalate their privilege using the S bit)13:39
*** jonathanmaw has quit IRC13:40
rjekHowever, this makes it impossible to build and "install" stuff that needs to install files with different ownerships.13:40
rjekThis is very sadmaking.13:40
rjekIs there a work-around?13:40
rjekOther than the hideousness that would be "do it on first boot"13:41
rjek(doing it on first boot might be a reproducability problem, and it requires a mutable root file system)13:42
ssam2rjek: not got a clear answer, but in general I think setting permissions is an 'integration' or 'deploy' thing rather than a 'build' thing13:47
ssam2you could do it in a .configure extension13:47
ssam2and I think in theory you should be able to do it directly in the chunk's .morph file with system-integration-commands13:47
rjekssam2: So one must reverse-engineer a piece of software's build/install targets and reproduce them somewhere else for deploy-time?13:47
ssam2but I have a feeling that won't work13:47
pedroalvarezssam2: nope, system integration commands also run in l-u-c13:48
ssam2rjek: if you're depending on it settings user permissions in 'make install', yes. what software is doing that, out of interest?13:48
ssam2sounds a bit broken, since you'd need to be root to 'make install'13:48
*** jonathanmaw has joined #baserock13:48
rjekLots.  :)13:48
rjekThat's what the "install" command in coreutils is for, after all :)13:49
ssam2that's not really an accurate statement13:49
ssam2but it does allow you to change ownership, if you want13:49
rjekBut, fundamentally, one must reimplement a product's "make install" target such that it occurs at deploy time?13:49
ssam2are you asking as a general question, or in the context of a specific piece of software?13:50
ssam2i've packaged lots of stuff in Baserock, and have never had to do that13:50
rjekBoth, nagios and exim in this situation13:50
ssam2from what i've heard, exim will be a nightmare no matter what the tooling does :)13:50
rjekLots of software is a nightmare.13:51
ssam2indeed. Well, to answer your question, yes, you have to set ownership at deploy time instead13:51
rjekOK, thanks.  Is there a plan to make this easier?13:51
ssam2it's tricky13:52
pedroalvarezI'd like to have one13:52
pedroalvarezbut indeed is tricky13:52
ssam2some people want the build tool to be able to run as non-root13:52
rjekThat's how Debian solves this13:52
ssam2which basically makes this impossible without fakeroot, which is a fragile, complex hack13:52
ssam2it kind of goes against the "work with upstreams, not against them" idea13:52
rjekOr perhaps a vm13:52
rjekUpstreams are humans, thus awful.13:53
ssam2software is made by humans, so that is awful too13:53
ssam2I'd prefer to write patches for projects which expect that `make install` to always have root permissions, so that they don't assume that13:53
rjekSadly for many pieces of software root is non-negotiable as it's a fundamental assumption on what they are doing13:55
ssam2it seems quite wrong to me that system-integration commands can't set user permissions. If they could, it'd mean that building a system image would require root13:55
rjekie, installing files with differing ownerships13:55
ssam2i'm interested in actual data on that. like I said, we've got a long way without support for this13:55
ssam2i'm not disagreeing, I just want an idea of the scale of things13:56
rjekexim, for example, need a dedicated user and group, needs to create directories for mail spool that are owned by that user and have the sticky bit set, and sometimes needs to install a setuid root executable.13:56
rjekAnd it needs all that to function at all13:56
ssam2systemd has some nice tricks for solving all but the last part13:57
ssam2well, tools rather than tricks13:57
rjekI'm sure systemd has all sorts of fragile and ill-documented tricks :)13:57
ssam2systemd-tmpfiles exists to create directories that are empty and owned by a specific user13:57
ssam2and systemd-sysusers will create users and groups on first boot13:58
rjekYeah, first boot root file system modifications.13:58
ssam2I don't think modifying the root file system on first boot is a bad thing13:59
ssam2modifying /usr on first boot would be bad13:59
rjekWe'll have to disagree there then :)13:59
ssam2what am I missing?13:59
rjekYou're assuming the root file system is mutable13:59
rjekOr that you want it to be mutable13:59
ssam2i'm assuming /var and /etc are mutable14:00
ssam2why wouldn't they be?14:00
rjekwhy should /etc be mutable?14:00
ssam2because it's for runtime configuration14:00
rjekAnyway.  Thanks.14:00
* radiofree is currently working on a system where /etc isn't mutable14:09
rjekAt least half the embedded systems I've worked on don't have mutable /etc14:12
ssam2I stand corrected14:12
rjekAnd some of them didn't even have mutable /var14:12
ssam2did they have files owned by multiple users? :)14:13
rjekBecause running everything as root is loopy14:13
paulsherwoodseems i find myself agreeing with rjek again in spite of myself :)14:15
rjekAgreeing with me is perfectly natural, paulsherwood.14:15
paulsherwoodthis does seem like a hole we need to fill14:15
ssam2I tried to write it up here:!/story/5414:25
ssam2contributions welcome to turn it from confused nonsense into an actual useful story14:25
pedroalvarezOk, I understand why cloudfoundry, when generating the disk image for the stemcell, they set a partition offset for installing grub later.14:55
pedroalvarezwhy this is not needed with extlinux in baserock?14:55
ssam2i don't really know. maybe extlinux is small enough to fit in the boot sector?14:59
pedroalvarezthat makes sense15:00
pdarheya, how long should I expect it to take for ybd to build a build sytem?15:03
tiagogomes_is not because the OS files are in btrfs subvolumes, and the booloader is being installed outside the subvolumes?15:03
tiagogomes_So you don't need an offset for your data15:04
tiagogomes_As btrfs takes care that the subvolumes files are not overwritten when writing to the top level15:05
ssam2pdar: not sure, but if you're going from scratch, hours15:10
pedroalvarezthat, and (and maybe more things), is why I still prefer morph :)15:12
pdaralas, I started at 10am ish, I'm on step 5/232/241.15:13
pdarI am too used to the lightning speeds of cache.baserock.org15:14
tiagogomes_pedroalvarez, if Paul reads that...15:14
tiagogomes_step 5/232/241 ?15:15
pedroalvarezindeed, wha is the 5?15:15
tiagogomes_-> 5 <- /232/24115:15
pedroalvarezpdar: we don't understand the meaning of that15:16
pedroalvarezi would understand 5/23215:16
pedroalvarezor 232/24115:16
pdarI assume where I am in the number of the things to build?15:16
pdarIts my first try of ybd15:16
pdar15-07-23 14:59:33 [5/232/241] [linux-api-headers] Starting build15:16
tiagogomes_you started a build at 10am and it is still on build-essential !!?15:17
pdarI had to stop and start it around 2pm15:18
pdarSo, I was looking at using ybd as I wanted to build something in a docker container, I didnt see that we already had instructions for making a baserock docker thing, I guess I can do that and just use morph in a baserock docker container instead15:23
*** paulw has quit IRC15:33
pedroalvarezso, concourse uses docker to set up the environment for testing?15:34
pdarFrom what I've looked at you seem to run all your bits in in docker-containers15:37
*** zoli__ has quit IRC15:37
*** zoli__ has joined #baserock15:38
pedroalvarezif this is going to be to build stemcells, it might be better to assume Ubuntu15:41
pdarI think I should be able to use a different docker container for the construction of the stemcell itself. And use a baserock container to construct the os-image that'll be used to make said stemcell.15:45
ssam2pdar: it might have taken a long time because of having to clone all the Git repos before it did anything else15:48
ssam2pdar: if so, you could gain a bit of speed by proving that cache yourself15:48
tiagogomes_this may be of interest here:
ssam2i've always wanted components to come with 'run this component in a testing environment' scripts15:49
ssam2since each component seems to need isolating from the host in different ways15:50
ssam2like this one:
ssam2(although that's quite a complex example, I imagined something simpler)15:50
*** fay_ has quit IRC15:54
tiagogomes_but wouldn't be enough when you need to test changes in more than one component15:54
pdarssam2: Ooh, how could I populate the cache myself?15:57
ssam2pdar: it's just a bunch of files under /src/cache/gits15:57
ssam2a bunch of Git repos in a specific format, to be more precise15:57
ssam2the format is the same one Morph uses15:57
ssam2so if you've used YBD anywhere else, or used Morph anywhere else, just copy all the git__git_baserock_org_xxx directories you need into the system where you're running the build15:58
ssam2or make them available in some other way15:58
pedroalvarezpdar: also, for speed, you can try to use another trove, if you have one lorrying from g.b.o15:58
*** mariaderidder has quit IRC16:10
*** jonathanmaw has quit IRC16:27
ssam2the dream has finally come true!16:45
ssam2next stop: 1000016:45
*** bashrc_ has quit IRC16:54
pedroalvarezthat means gerrit has been accepted by the baserock community :)17:14
*** edcragg has quit IRC17:42
*** lachlanmackenzie has quit IRC18:43
*** ssam2 has quit IRC19:09
*** zoli__ has quit IRC19:16
*** zoli__ has joined #baserock19:16
*** zoli__ has quit IRC19:54
*** edcragg has joined #baserock21:44
*** zoli__ has joined #baserock22:05
*** zoli__ has quit IRC22:16
*** edcragg has quit IRC23:05
*** sebh has quit IRC23:32
*** Zara_ has quit IRC23:32
*** kejiahu has quit IRC23:32
*** sebh has joined #baserock23:32
*** kejiahu has joined #baserock23:33
*** petefotheringham has quit IRC23:33
*** Zara has joined #baserock23:33
*** jmacs has quit IRC23:33
*** petefotheringham has joined #baserock23:33
*** jmacs has joined #baserock23:33

Generated by 2.15.3 by Marius Gedminas - find it at!