IRC logs for #baserock for Monday, 2015-06-01

*** edcragg has quit IRC00:12
*** seb__ has joined #baserock01:13
*** seb__ has quit IRC02:09
*** seb__ has joined #baserock03:27
*** zoli__ has quit IRC03:27
*** zoli__ has joined #baserock03:28
*** seb__ has quit IRC03:36
*** seb__ has joined #baserock05:03
*** zoli__ has quit IRC05:47
*** zoli__ has joined #baserock05:48
*** zoli__ has quit IRC05:54
*** seb__ has quit IRC06:45
*** seb__ has joined #baserock06:47
*** a1exhughe5 has joined #baserock06:48
*** seb__ has quit IRC06:52
*** mike has joined #baserock06:57
*** mike is now known as Guest1573106:57
*** paulw has joined #baserock07:01
*** rdale has joined #baserock07:21
*** zoli__ has joined #baserock07:25
*** seb__ has joined #baserock07:31
*** zoli__ has quit IRC07:45
*** mariaderidder has joined #baserock07:46
*** a1exhughe5 has quit IRC07:50
*** seb__ has quit IRC07:58
*** gary_perkins has joined #baserock07:59
straycat is compelling, i'd not really considered using bitbucket and the like as harmful, but that argument at least persuaded me that the savings i make by hosting on bitbucket aren't worth the potential costs to our freedom08:02
* paulsher1ood agrees08:04
paulsher1oodbut it'd be hard to give up googledocs, github etc...08:04
rjek<insert normal rjek cloud skepticism here>08:05
*** zoli__ has joined #baserock08:05
paulsher1oodi've got that set as my global default now, rjek - no need to explicitly enable it here :)08:06
*** a1exhughe5 has joined #baserock08:08
*** bashrc_ has joined #baserock08:09
paulsher1oodany thoughts on 15:24 < paulsher1ood> 'mount: only root can use "--types" option'... but i'm root!?08:09
rjekpaulsher1ood: How did you become root?08:10
rjekIf with login/su/sudo, then boggle.  If via a chroot/container/fakeroot, then you may not have had the capability transfered.08:11
* rjek would strace mount and see which system call is failing and with what error08:11
paulsher1oodthat's beyond my skills08:12
rjekstrace mount -t ...08:12
rjekThen you can paste the last bit of output and we can have a look08:12
paulsher1oodthe problem is occuring during execution of one of the morph/ybd 'containerised' commands08:14
paulsher1oods/one of/all of/08:14
rjekAh, if it's containerised you may not /actually/ be root08:14
rjekcontainers have much finer-grained permissions08:15
paulsher1oodno, i'm sudo'd first. then running ybd08:15
paulsher1oodwhoami tells me i'm root08:15
rjekAnd ybd then creates a container to run the build in, yeah?08:15
paulsher1oods/container/magic long command line of doom/08:15
rjekIf so, the container may not have all of root's privileges and capabilities, but the "whoami" command is ignorant of fine-grained capabilities.08:15
*** jonathanmaw has joined #baserock08:16
paulsher1oodpoint is, this works in a baserock vm, not on ec208:16
rjekThat seems surprising.08:17
rjeklinux-user-chroot should be doing the same thing in both a Baserock VM and EC2.08:17
rjekBecause EC2 is just a Baserock VM :)08:17
rjekAnyway, my understanding is that linux-user-chroot throws away almost all capiabilities and things like mount probably won't work inside one.08:18
rjek(For good reason; you could mount something that lets you escape the container.)08:18
jjardonhopefully this will be a good alternative to Google docs: (owncloud has a app to edit docs but it doesnt work very well, a least the version I tested)08:18
bashrc_what about etherpad?08:19
rjekIt's fine if you want a text editor.  Less good if you want a spreadsheet.08:20
paulsher1oodrjek: no, this EC2 is not baserock, it's whatever AWS does as default08:21
rjekpaulsher1ood: I have no idea what AWS does by default, sorry.08:21
rjekBut it would not surprise me if the defaults for linux-user-chroot differed wildly between different Linux distributions.08:21
paulsher1oodLinux ip-172-31-27-107 3.14.35-28.38.amzn1.x86_64 #1 SMP Wed Mar 11 22:50:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux08:22
rjekBut mounting things during builds should be really avoided.08:22
rjekThere's a reproducability problem there too08:22
rjek(Block allocation in modern file systems is not deterministic.)08:22
jjardonbashrc_: nice, I didnt know that one08:23
rjek(as they delay as long as they can before writing in order to minimise fragmentation and maximise write throughput, so what blocks get allocated where and when is hightly dependant an amount of RAM and other CPU load.)08:23
paulsher1oodrjek: why would that affect reproducibility?08:25
rjekpaulsher1ood: One assumes the file system you're mounting has some input to your build, otherwise why would you be mounting it?08:26
* paulsher1ood is not mounting it. just re-implementing what morph does08:26
rjekRegardless, the reason you can't mount is because linux-user-chroot has forbidden it.08:26
*** ssam2 has joined #baserock08:30
*** ChanServ sets mode: +v ssam208:30
jjardon209466s in the last build in mason, I think that's a new record08:47
pedroalvarezjjardon: 2 days and a half08:49
pedroalvarezIs a bit worrying, isn't it?08:49
tlsait doesn't have a distbuild network, does it?08:50
*** CTtpollard has joined #baserock08:50
rjekAnd is it CPU, I/O, or task-bound?08:50
ssam2and, no distbuild network08:52
mwilliams_ctstraycat: just to check, is there any more work you think I need to do on 729?09:05
radiofreei'm pretty sure it didn't take me 2 1/2 days to build a weston system on a jetson09:06
radiofreei know it's building devel + openstack as well, but something is happening there09:06
rjekradiofree: Your Jetson probably has better I/O than the mason systems09:06
rjekLast time I measured it, it was 10MB/sec with ~60ms latency09:07
rjek(The VMs have no local storage.)09:08
*** Krin has joined #baserock09:17
straycatmwilliams_ct, biff09:18
mwilliams_ctstraycat: you're a star, thanks :)09:19
pedroalvarezjust realised that mason built twice some components09:23
pedroalvareze.g. glibc09:23
*** lachlanmackenzie has joined #baserock09:23
*** seb__ has joined #baserock09:23
* richard_maw had heard of some race conditions in some versions of distbuild that would allow that to happen09:26
*** mariaderidder has quit IRC09:32
*** seb__ has quit IRC09:34
*** edcragg has joined #baserock09:36
straycatrichard_maw, are you by any chance planning to review stuff tomorrow?09:41
*** mariaderidder has joined #baserock09:47
*** zoli__ has quit IRC10:04
richard_mawstraycat: I usually do my reviews on Wednesdays, not tuesdays, though there's a couple of things that may require I defer it further.10:25
straycati'll try and help clear the backlog tomorrow then10:25
richard_mawthanks straycat10:25
straycatnp :)10:26
tlsapedroalvarez: "Started building glibc on mason-x86-64:3434" occurs 4 times in
tlsaeach time it takes over 10 mins and then transfers glibc-doc to shared artifact cache10:40
pedroalvarez4 times10:40
tlsa4 times for zlib too10:41
*** kejiahu_ has joined #baserock10:41
pedroalvarezalso gcc, so I guess that everything else is also quadruplicated10:41
*** ssam2_ has joined #baserock10:41
*** tpollard_ has joined #baserock10:41
tlsaand GCC takes half an hour each time10:42
*** Zara_ has joined #baserock10:43
*** kejiahu has quit IRC10:43
*** CTtpollard has quit IRC10:43
*** ssam2 has quit IRC10:43
*** Zara has quit IRC10:43
*** zoli__ has joined #baserock10:54
*** pacon has joined #baserock11:05
ssam2_richard_maw: seems that you were right about linux-user-chroot, it does actually create a new mount namespace unconditionally11:17
ssam2_so the call to `unshare --mounts` in unshared_commandline is unneeded when linux-user-chroot is used11:18
ssam2_perhaps the reason it's there is for deployments, because we can't run all deployments inside linux-user-chroot, but still want to use a separate mount namespace...11:18
ssam2_and also because we can't tell linux-user-chroot to mount a devtmpfs or a tmpfs right now inside that namespace, I guess11:21
paulsher1ooddoes morph even use this stuff for deployment?11:21
paulsher1oodssam2_: ^^11:22
ssam2_paulsher1ood: it uses the 'unshare' machinery (to create a new mount namespace) but not linux-user-chroot11:26
ssam2_it's nice to have a new mount namespace for deployments because then stale mounts get cleaned up no matter what11:27
ssam2_but deployment extensions can still delete the entire host system or whatever else, so we can't really say that we sandbox them at all11:27
ssam2_and it's hard to sandbox them because they each need different capabilities (e.g. to deploy to openstack you need to access the network; to deploy to a USB stick you need to access a block device as root, ...)11:28
*** pacon has quit IRC11:38
*** pacon has joined #baserock11:40
*** Krin has quit IRC11:40
*** pacon has quit IRC11:43
*** pacon has joined #baserock11:48
*** Krin has joined #baserock12:00
richard_mawssam2_: the reason for the unshare was so that the mounts that weren't possible with linux-user-chroot could be done without affecting the mounts of the parent namespace12:00
ssam2_so adding support for tmpfs mounts to linux-user-chroot should be enough to remove the need for it?12:01
*** gary_perkins has quit IRC12:01
*** Krin has quit IRC12:01
*** franred has quit IRC12:01
*** ssam2_ has quit IRC12:01
*** bashrc_ has quit IRC12:01
*** tpollard_ has quit IRC12:01
*** paulw has quit IRC12:01
*** flatmush has quit IRC12:01
*** edcragg_ has joined #baserock12:01
*** mariaderidder_ has joined #baserock12:01
*** paulw has joined #baserock12:01
*** sherm_ has joined #baserock12:01
*** Krin has joined #baserock12:01
*** nowster_ has joined #baserock12:01
*** franred has joined #baserock12:02
*** ssam2_ has joined #baserock12:02
*** tpollard_ has joined #baserock12:02
*** jonathanmaw_ has joined #baserock12:02
*** edcragg has quit IRC12:02
*** jonathanmaw has quit IRC12:02
*** Guest15731 has quit IRC12:02
*** a1exhughe5 has quit IRC12:02
*** bashrc_ has joined #baserock12:02
*** mariaderidder has quit IRC12:02
*** nowster has quit IRC12:02
*** lachlanmackenzie has quit IRC12:02
ssam2_assuming we don't start to do anything with devtmpfs... maybe we'd need to add support for devtmpfs mounts to so we can avoid having to pre-create device nodes in the fhs-dirs chunk12:02
richard_mawpossibly, I don't recall every mount that is attempted, but the overlayfs work also added a new mount that should be isolated12:02
*** lachlanmackenzie has joined #baserock12:02
ssam2_hmm, that's true12:02
*** gary_perkins has joined #baserock12:02
*** a1exhughe5 has joined #baserock12:03
ssam2_i guess there's not much we can do except try to add those mount types to linux-user-chroot as we need to12:03
*** flatmush has joined #baserock12:03
ssam2_anything else will continue to require the user to be 'root', which my goal is to avoid12:03
richard_mawyeah, since enabling unprivileged user namespaces in kernels is not a viable option, since we need to support chroots and distros don't enable unprivileged user namespaces because it's not considered to be secure yet12:04
richard_mawwhich means we also shouldn't use it in our kernels12:04
SotKAnyone know what the image-package-example directory and its contents are for in definitions?12:19
richard_mawSotK: I used to, let me refresh my memory.12:21
*** zoli___ has joined #baserock12:21
richard_mawSotK: works with image-package.write12:21
richard_mawSotK: See clusters/image-package-example.morph12:22
SotKrichard_maw: thanks12:22
*** zoli__ has quit IRC12:25
ssam2_it's just an example, I'm not sure anyone uses it direclty12:35
*** pacon has quit IRC12:49
* SotK notes that the description field in "clusters/installer-build-system-x86_64.morph" says that it uses install-files when actually it doesn't12:50
pedroalvarezI moved it to its own chunk :/12:53
ssam2_seems my proposal to talk about Baserock at EuroPython got accepted13:19
ssam2_anyone up for a trip to Bilbao ?13:19
mwilliams_ctssam2_: congrats!13:21
pedroalvarezoh that's great13:23
franredssam2_, congrats!! :)13:23
Zara_yay! :D13:25
straycatssam2_, awesome :)13:29
perryli'm currently looking at bit-for-bit build reproducibility with regards to morph/ybd, and using the tor project's approach as a kind of guideline of steps i may need to take; a couple of steps involved setting localisation and timezone as LC_ALL=C and TZ=UTC, would these be valid for morph/ybd and if so, where should they be set? i.e. in configuration, for each chunk/strata/system or elsewhere13:36
ssam2_they need to be set when you run all the configure/build/install commands13:38
ssam2_in ybd I think that's done in the clean_env() function13:38
ssam2_in Morph I think it's done in the buildenvironment module13:38
perrylssam2_: thanks, i'll take a look at those now13:39
pedroalvarezI need a couple of lorries to fix pyeclib build
pedroalvarezpyeclib builds a copy of them that embedded on its repo if their are not present at build time13:43
pedroalvarezand pyeclib does it really wrong13:43
pedroalvarezinstalling things in /pyeclib.inst/usr/local/lib13:43
pedroalvarezAlso I found that wpa_supplicant is being installed in /usr/local/sbin :) I might fix that later13:46
pedroalvarezthanks franred ssam2_!13:47
rdaleoh, by the time i had typed in my comment and given you a +1, it had already been merged13:47
pedroalvarezrdale: thank you anyway :)13:50
Zara_does the web editor on the baserock wiki work for anyone else? I think it hasn't been working for me for a while.13:59
*** Zara_ is now known as Zara14:00
jmacsIt did last time I tried...14:00
ssam2_zara: just worked for me now14:01
jmacsYes, works for me14:01
Zaraah, the buttons had just moved to the bottom of my screen; I found them when I scrolled down.14:03
SotKthis is not vim14:12
*** paulw has quit IRC14:14
SotKDoes anyone know why morph copies the contents of deployment extensions that are in definitions into a temporary file and runs that temporary file, rather than just running the script?14:16
*** paulw has joined #baserock14:17
Kinnisoni think it's to do with consistency14:17
Kinnisonand ensuring that files have the right permissions14:17
* Kinnison hmms14:18
KinnisonSince last week was a short week, I shall not do my weekly trawl-through-old-patches this week14:18
KinnisonEnjoy the respite from harranguing14:18
richard_mawSotK: also because they were (possibly still are) loaded out of a repository without creating a checkout14:18
SotKthey are loaded from the definitions checkout in the system branch now I think14:20
SotKyes, they are14:21
SotKthanks Kinnison, richard_maw14:22
pedroalvarezwhat people would appreciate to have tested before moving to new systemd 220?14:34
pedroalvarezMy list would be:14:34
pedroalvarez- trove14:34
pedroalvarez- openstack14:34
pedroalvarez- jetson14:34
Kinnisondevel system14:36
Kinnisonupgrading and downgrading14:36
Kinnisonverification that distbuild clusters still come up properly14:36
Kinnisonand mason14:36
pedroalvarezgood list, thanks Kinnison14:38
SotKhmm, what is version 4 of definitions?14:53
SotKI ask because morph claims to support it, but VERSION and the wiki say version 3 is the most recent14:55
SotKs/most recent/current/14:55
Kinnisonyay for git diff14:55
Kinnisonand git log14:55
Kinnisonand git annotate14:55
SotKoh, its for a change in the install-files configure extension14:58
Kinnisondo the extensions get told what definitions version they get?14:58
straycatiirc, it's not really accessible outside of the source resolver14:59
*** jonathanmaw_ is now known as jonathanmaw14:59
SotKI guess if I want to increment the version because of a change I'm making (moving deployment extensions into a subdirectory - which won't work without an updated morph) then I bump it straight from 3 to 514:59
pedroalvarezSotK: you can't bump definitions version, until there is a release that supports that version of definitions15:00
pedroalvarezthat's why definitions versions hasn't changed yert15:01
pedroalvarezbut go ahead, and make sure that we don't merge the definitions change :)15:01
pedroalvarezSotK: also, I believe I moved configuration extensions to a subfolder in the past, and they worked15:02
SotKpedroalvarez: I see15:03
SotKhmm, I was under the impression that they wouldn't, I'll investigate :)15:03
pedroalvarezwrite exensions may fail, yes15:03
SotKafaict the code to locate and run them is the same for both kinds?15:04
straycatssam2_, speaking of unshare the import tool seems to require root because it uses morphlib to run the extensions, do we really want this for the import tool?15:05
ssam2_it's not ideal15:06
ssam2_probably it's ok to not sandbox the import tool extensions for now15:06
ssam2_SotK: maybe moving .configure and .write extensions into definitions doesn't need to require an incompatible change to definitions15:07
ssam2_I guess the incompatible thing is moving them into a subdirectory, right?15:07
KinnisonSotK: OOI are you making it so that the deployment mechanisms take paths?15:08
ssam2_so maybe separate that task out. a patch to move all the .configure and .write extensions into the /root/ of definitions.git wouldn't be incompatible (although it would be ugly)15:08
KinnisonSotK: or automagically finding the extensions wherever they are?15:08
SotKKinnison: at the moment automatically finding them wherever they are in definitions15:09
straycatssam2_, *nod* i think so15:09
SotKand planning that they all live in definitions in a subdirectory15:09
SotKssam2_: that will be doable, but it'll make definitions really messy :(15:10
ssam2_yes. I guess we'll have to decide whether we prefer to make a release, have messy definitions, or miss out on this feature15:14
*** paulw has quit IRC15:14
ssam2_or give up on backwards compatibility again, but I think that would be dumb15:15
KinnisonAre releases hard?15:15
ssam2_they're not simple15:16
*** paulw has joined #baserock15:16
jjardonHi, anyone more have issues trying to log in the wiki? I'm getting this error: "Error: OpenID failure: naive_verify_failed_return: Direct contact invalidated ID provider response."15:21
ssam2_what openID URL and what wiki URL?15:22
ssam2_worked for me with 2 different openIDs15:23
pedroalvarezjjardon: nice hair15:29
rjekwow much hair15:29
ssam2_jjardon: has it worked in the past?15:30
pedroalvarezlaunchpad openID just worked for me15:31
*** seb__ has joined #baserock15:31
jjardonssam2_: I think so, not completely sure though.15:32
jjardonpedroalvarez: thanks :)15:32
* SotK confirms that moving even configure extensions is incompatible with old morph :(15:39
pedroalvarezSotK: I promise I'm doing that in infrastrucutre.git15:40
pedroalvarezbeing this the system:
pedroalvarezand having the configuration exts in "baserock_irclogs/irclogs.configure"15:42
SotKaha, by giving the relative path as the name15:42
SotKto move all the existing ones into a subdirectory I'd need to do that in every system for the extensions which are in definitions then15:44
pedroalvareznot sure if this is the right approach, although it doesn't look bad to me15:45
SotKI think I'd prefer to patch morph, but I wonder if other people think that the above method would be better?15:47
ssam2_the above method looks OK if it works15:48
ssam2_would be nasty for .write extensions though15:48
richard_mawI'd be perfectly happy with patching all our definitions to set subdirectory paths to the configuration extensions in the system definitions and subdirectory paths in clusters for the write extensions15:50
straycatseems reasonable to me too15:51
SotKI shall do that then15:51
*** a1exhughe5 has quit IRC15:53
jmacsJDK6 requires JDK6 to bootstrap it, apparently. Neither jikes or GCJ come with JDKs later than 1.5. I think we'll have to live with the binary package.15:53
*** gary_perkins has quit IRC15:59
perryli'm trying to build python-core with ybd and getting the following error, can anyone assist?
paulsher1ooddoes your machine have network?16:10
paulsher1oodperryl: ^^16:10
* paulsher1ood notices that the error is very unhelpful16:10
perrylpaulsher1ood: as far as i know, yes16:11
straycatperryl, if you can find the gitdir in the ybd cachedir, check whether the contents look sane16:11
paulsher1oodcan you ping from it?16:11
straycat(the gitdir for stage2-linux-api-headers)16:11
ssam2_perryl: did you ctrl+c a build while it was downloading a git repo? that can break ybd16:11
perrylpaulsher1ood: yes, no packet loss16:12
ssam2_have a look in /src/cache/gits/git___git_baserock_org_delta_linux and see if it's a git repo, or some corrupt imitation of a git repo16:12
straycatindeed, it will only check for the existence of the directory16:12
ssam2_it might be in ~/.ybd/gits  either16:12
perryljust checked in git__git_baserock_org_delta_linux and it's a tar file rather than a git directory, i assume that is the error16:13
paulsher1oodperryl: pls remove the directory and try again. in the meantime i've raised an issue, will fix it soonish unless someone else beats me to it16:14
perrylpaulsher1ood: done, ybd seems to be working fine now, thanks all!16:16
*** jonathanmaw has quit IRC16:16
ssam2_i've successfully managed to build past stage2-glibc as non-root using YBD16:18
ssam2_I had to add 'fakeroot' into the mix though16:18
paulsher1oodssam2_: w00t! :)16:19
ssam2_i imagine it'll fail at stage2-fhs-dirs, or just past it, because it won't be able to create proper device nodes16:19
paulsher1oodssam2_: i live in hope that you'll be wrong :)16:20
ssam2_if it doesn't, it'll fail at the first non-bootstrap chunk because it won't have 'fakeroot' in the staging area16:22
ssam2_this approach would require us to add 'fakeroot' into build-essential16:22
ssam2_but that probably isn't too hard16:22
richard_mawssam2_: fakeroot also fakes up the device nodes AIUI16:23
ssam2_so it'll be fine up until something tries to actually use one16:24
KinnisonHow are you using fakeroot?16:24
Kinnisoni.e. where in the command chains16:24
ssam2_it might be possible to use it outside linux-user-chroot instead, actually16:25
ssam2_it didn't work when I tried it before, but that may have been because I still had '--unshare-net' (which breaks fakeroot either way)16:26
Kinnisonssam2_: and how are you managing the faked?16:26
ssam2_ignoring it completely16:27
ssam2_seems to be started by 'fakeroot'16:27
Kinnisonand then shut down again after16:27
Kinnisonso you'll lose session info16:27
Kinnisonbetween the {pre-,,post-}install-commands16:27
ssam2_right, that's true16:27
ssam2_creating the tarball happens outside the sandbox, so I guess this won't work at all16:28
ssam2_thanks for pointing that out!16:28
KinnisonDebian's approach is to fakeroot around the entire build process16:28
straycatspeaking of running things as non-root, will allow us to run the import tool as non-root16:29
*** mariaderidder_ has quit IRC16:32
*** sebh- has joined #baserock16:38
*** sebh has quit IRC16:38
*** sebh- is now known as sebh16:38
*** Krin has quit IRC16:43
paulsher1oodKinnison: can fakeroot be used instead of linux-user-chroot, so drop linux-user-chroot entirely?16:43
rjekpaulsher1ood: fakeroot won't solve your mount problem either; you're not root at all under fakeroot.16:47
ssam2_paulsher1ood: there is a 'fakechroot' program, which could be used instead of linux-user-chroot to provide a subset of the sandboxing linux-user-chroot does16:48
ssam2_it relies on LD_PRELOAD instead of linux namespacing. I think Linux namespaces are less fragile.16:49
ssam2_and fakechroot doesn't isolate the sandbox from the hosts network, PIDs, mounts, etc. so it'd be a step backwards16:50
*** bashrc_ has quit IRC17:04
*** ssam2_ has quit IRC17:15
*** franred has quit IRC17:21
*** sherm_ has quit IRC17:53
*** edcragg_ has quit IRC18:03
*** lachlanmackenzie has quit IRC18:20
*** zoli__ has joined #baserock18:20
*** zoli___ has quit IRC18:22
*** zoli__ has quit IRC21:26
*** zoli__ has joined #baserock21:46
*** seb__ has quit IRC22:22
*** brlogger has joined #baserock22:23
*** zoli__ has quit IRC22:27
*** seb__ has joined #baserock22:35
*** seb__ has quit IRC22:48

Generated by 2.15.3 by Marius Gedminas - find it at!