IRC logs for #baserock for Tuesday, 2015-02-24

paulsherwoodssam2: ///win 2200:02
paulsherwoodbah! :)00:02
*** zoli__ has joined #baserock00:02
*** zoli__ has quit IRC00:18
*** zoli__ has joined #baserock00:25
*** zoli__ has quit IRC02:21
*** zoli__ has joined #baserock02:30
*** zoli__ has quit IRC02:32
*** zoli__ has joined #baserock03:45
*** zoli__ has quit IRC04:02
*** petefoth has quit IRC04:02
*** CTtpollard has quit IRC04:02
*** fay_ has quit IRC04:02
*** mauricemoss_ has quit IRC04:02
*** sambishop has quit IRC04:02
*** nowster has quit IRC04:02
*** paulw has quit IRC04:02
*** flatmush has quit IRC04:02
*** paulw has joined #baserock04:02
*** CTtpollard has joined #baserock04:02
*** sambishop has joined #baserock04:02
*** fay_ has joined #baserock04:03
*** mauricemoss_ has joined #baserock04:03
*** nowster has joined #baserock04:03
*** petefoth has joined #baserock04:03
*** flatmush has joined #baserock04:03
*** juergbi has quit IRC05:27
*** juergbi has joined #baserock05:36
*** mike has joined #baserock07:18
*** mike is now known as Guest338007:19
*** franred has joined #baserock07:32
*** a1exhughe5 has joined #baserock07:49
*** gfinney__ has joined #baserock08:16
*** gfinney__ has quit IRC08:19
*** gfinney__ has joined #baserock08:19
*** gfinney__ has quit IRC08:20
*** gfinney has joined #baserock08:20
*** gfinney_ has joined #baserock08:20
gfinney_Good morning folks08:20
*** gfinney has quit IRC08:23
*** gfinney_ has quit IRC08:23
*** gfinney has joined #baserock08:23
*** gfinney_ has joined #baserock08:23
*** mdizzle has joined #baserock08:35
*** bashrc has joined #baserock08:58
*** gfinney_ has quit IRC09:03
*** rdale has joined #baserock09:06
*** jonathanmaw has joined #baserock09:12
*** zoli__ has joined #baserock09:27
*** tiagogomes_ has joined #baserock09:29
*** gary_perkins has joined #baserock09:34
*** ssam2 has joined #baserock10:05
*** ChanServ sets mode: +v ssam210:05
SotKDoes anyone have opinions on how my OSTree artifact cache should deal with stratum artifacts, which are just a file containing a list of chunks?10:06
SotKit seems pointless to store them using OSTree, so I'm planning to just store them in the cachedir at the moment10:07
ssam2yeah, they are basically just metadata10:07
ssam2so store them however you store the other meta files10:07
ssam2I guess you could create actual trees for them in the OSTree store, since they wouldn't take up any extra disk space10:08
ssam2and that would maybe make adding stratum-integration-commands easier at a later date...10:08
ssam2but do whatever's easiest for now I think10:08
persiaIF we move to a strataless model, where we have arbitrary named entities with contents, how would we deal with this?10:08
SotKpersia: consideration of this is part of why I asked for opinions, in case someone had already thought about that10:09
persiaI think I agree with ssam2, that doing whatever is easiest/fastest is the right answer for now.  I mostly wanted to make sure that we didn't paint ourselves in a corner to not be able to handle differently organised constructs.10:10
persiaI haven't thought about it in depth, but I suspect that the easiest way is to have metadata associated with each cache object that can identify other cache objects that should be overlaid (and the order) before this cache object is overlaid for the cache object under consideration to be complete.10:11
SotKthat is my thinking too10:11
ssam2there are two approaches: either use metadata to describe components-that-contain-other-components, or create the union of the components at build time and store that10:12
ssam2we currently use the first approach for strata and the second approach for systems10:12
persiaMy worry with the union-of-components model is the duplication of data.10:12
ssam2sure, but OSTree deduplicates all objects10:12
persiaSo If I have two cache keys, one of which objects contains the other, OSTree will never download things twice?10:13
ssam2it deduplicates at the file level10:13
ssam2so if one contains the other, both contain the same files10:13
ssam2and it'll only download those files ones10:13
ssam2at least, in theory. I think that the `ostree pull` command still has quite a lot of inefficiencies in practice10:14
*** Krin has joined #baserock10:15
persiaIn that case, I'd be happy storing the entire object for strata, as that makes it more consistent.10:15
persiaAnd, in practice, doesn't mean any differences in download.10:15
persiaThat said, downloading many small files is *much* slower than downloading an archive of those files (even ignoring compression), because of how TCP works.10:15
ssam2yes. I think there's quite a bit of thought going on right now as to how to make `ostree pull` fast enough10:16
*** zoli__ has quit IRC10:16
ssam2walters is keen to not require anything special server-side, which makes it more complex10:16
ssam2i.e. a dumb http server should be able to serve an OStree repo10:16
ssam2i'm not sure if that restriction makes sense for us or not, though10:17
persiaI think it does.10:19
persiaI'd rather not impose strong software requirements server side for artifact caches.  Anything that can provide a filestore should be sufficient.10:19
SotKIf we're worried about download speed, surely we could keep the server-side cache as it is and only use OSTree locally?10:22
*** lachlanmackenzie has joined #baserock10:24
ssam2SotK: I think OSTree should be faster in theory than downloading each artifact as a separate tarball10:26
ssam2I just don't want to claim that it is faster, when I don't actually know10:26
ssam2pedroalvarez: how did you disable artifact uploading on ?10:29
ssam2I want to do the same for but i can't work out how10:29
persiaSotK: My revised opinion: test and benchmark a couple different things, and choose the fast one, unless doing so is onerous, in which case do it the way that seems possible :)10:31
pedroalvarezssam2: I've already done it, sorry I didn't say anything10:37
pedroalvarezssam2: I just removed a few lines of the /usr/lib/mason/ script10:37
ssam2oh, thanks10:37
ssam2better to comment them in future so it's clear that you did it :)10:38
pedroalvareztrue :/10:40
pedroalvarezI guess I like pressing 'd' in vim10:40
ssam2it's only a few more keypresses...10:45
straycatwhat is that magic10:53
ssam2block select, move down a bit, insert # at start of each line you selected10:53
straycati usually x,ys/^/#/ for that sort of thing10:54
ssam2i never thought of that. might be easier10:54
straycatmmm but the visual selection seems more intuitive somehow10:54
rjekOh, I thought ssam2 had a face/keyboard interface10:59
rjekBut it's a vi command :)10:59
straycatoh i finally get how this works11:00
straycatokay i find the sub easier in this case11:00
ssam2yeah I think it's better too. thanks for the tip!11:02
straycatheh np :)11:06
*** rdale_ has joined #baserock11:44
*** rdale has quit IRC11:47
franredcould I have a quick review of a tarball lorry?
* pedroalvarez can't open the url and hopes that nothing is going wrong with paste.baserock.org12:10
straycati'm still waiting12:11
* jjardon neither12:11
Kinnisonit opens for me12:11
franredumm, I can not open ...12:11
Kinnisonyou lot are all on one site, yes?  Maybe a routing hiccough12:11
radiofreepbo working for me (at home)12:12
straycatyeah i can wget it from tardis just fine12:12
pedroalvarezI couldn't access any of the machines of DC, but everything looks fine again12:12
ssam2franred: I'm pretty sure that's lorried already12:13
* pedroalvarez stops panicking12:13
ssam2from bazaar12:13
ssam2do you need the tarball instead of the bzr repo for some reason?12:13
franredssam2, yes, I know, but we are having troubles to make horizon running from git and none from the tarball12:13
franredKrin, can give you more information about this ^^12:15
ssam2ok... if there's some special magic in the tarball that's missing from the bzr repo (i.e. you can't just generate the tarball from the bzr repo) then +112:15
straycatssam2, i think it's because the maintainer of pytz isn't expecting anyone to use his source repo, so he just releases tars and puts them on pypi12:16
straycatso the source repo isn't packaged as a python distribution12:16
straycati've seen a couple of other packages like this12:17
ssam2yeah, some maintainers of rubygems like to work this way too12:18
ssam2I guess the only solution is build a cool OS that's all built from git except for their components, so they can see what they're missing out on :)12:19
*** petefoth has left #baserock12:19
*** petefoth has joined #baserock12:20
franredany other quick review of the previous patch? or ssam2, can I take your review as a +2?12:20
Krinthe bzr repo expects to be built and installed in an environment that we do not currently have in baserock, ripsum explained it but i did not understand his explanation. the long and short of it was that it was not possible to leave the pytz btz as it was because although it builds well enough to work as a dependanciy, actual use of the pytz module fails.12:20
straycat150224 12:32:36 [Warning] Unknown character set: 'uft8'12:35
straycat150224 12:32:36 [Warning] Aborted connection 66 to db: 'foom' user: 'foom' host: 'localhost' (init_connect command failed)12:35
straycatwell i suck12:35
*** gfinney has quit IRC12:43
tiagogomes_I've a device tree being installed to "boot/dtbs/3.19.0-rc5+/apm/apm-mustang.dtb". Does anyone knows how to change the install location to not include the kernel version dir12:49
*** gfinney has joined #baserock12:50
*** gfinney_ has joined #baserock12:50
radiofreetiagogomes_: can't you just copy it out?12:50
tiagogomes_radiofree not really, we want the armv8 kernel to be generic, so we are installing all armv8 device trees12:51
ssam2anyone seen a btrfs loopback mount giving lots of 'loop: Write error at byte offset xxx, length 4096' errors?12:56
ssam2this is on an SSD attached to a Jetson, with Linux 3.1812:56
ssam2hmm. Also our 'build' systems seem to lack fsck.ext412:57
ssam2eventually I get a kernel backtrace involving btrfs_commit_transation12:58
richard_mawI've seen it happen with a flaky USB pass through for a USB attached device, but that was because the device had completely gone away12:59
nowsterssam2: good luck in recovering that filesystem...13:01
ssam2luckily I don't need to.13:01
jjardonHey! ok to lorry the python3 cairo bindings?
richard_mawI think we put python modules in the python-packages/ subdirectory these days13:15
richard_mawIf we can confirm that this is the case, and that it's not already lorried somewhere I'll +2 the lorry13:16
*** zoli__ has joined #baserock13:17
jjardonrichard_maw: I think it would be a bit strange to have the bindings in different places ( py2cairo and python-packages/pycairo in g.b.o ) , but no problems to me to rework the patch13:19
*** zoli__ has quit IRC13:21
ssam2it's a bit strange to have 2 repos for one set of bindings13:22
ssam2most projects do python 2 and 3 out of one git repo13:23
richard_mawsome do python 2 and 3 out of the same commit, either writing the safe subset, or writing 2to3 compatible code13:25
jjardonagree they seem to have created the repo just for python3 since version 1.8:
ssam2well, maybe put it in the same place as py2cairo then. I agree it seems weird to have 2 related repos in different places13:27
ssam2and we're not going to get to a point where we can move repos around on any time soon13:28
jjardonok, so should I apply my patch as it is?13:28
ssam2+1 from me13:30
richard_maw+1 from me too then13:35
*** zoli__ has joined #baserock13:37
jjardonmerged! thanks!13:37
*** zoli__ has quit IRC13:45
* SotK successfully builds a system using his OSTree artifact cache13:48
SotKbuilding the actual system itself took 3:48, I don't know if that is any quicker than normal13:55
persiaThe quoted numbers yesterday were ~5 minutes to build and ~10 minutes to download, so that's better than either.13:56
* straycat said < 5 minutes13:57
straycati've never timed it though, so don't rely on that figure13:57
straycatprobably best to do your own experiements13:57
* SotK populates an old style cache to try to get a comparison13:57
pedroalvarezI'd like to have a comparison of sizes of the system-artifact14:03
jjardonpedroalvarez: what is the best way to run a command in integration time?, should I add a .configure in morphlib/exts/ in the morph tree?14:04
jjardonSotK: nice14:06
pedroalvarezjjardon: is not well documented I'm afraid, but here there is an example:
jjardonpedroalvarez:  thanks!14:07
pedroalvarezthis example will generate a sh file called, which will have inside the `update-ca-certificates` call.  This script will be included in the ca-certificates-misc  artifact.14:08
* SotK decides something has gone wrong with his test and tries again14:08
pedroalvarezthis script will be called at system-artifact constuction14:08
SotKpedroalvarez: the system artifact built with OSTree is 1.6M, I am only storing the files changed by stuff done when building the system14:09
persiaNote that this doesn't work if you need to build-depend on the results of the script.14:10
pedroalvarezjjardon: indeed, note that ^^14:10
pedroalvarezSotK: awesome14:11
jjardonpedroalvarez: ok, so simply calling "gdk-pixbuf-query-loaders" will put the stuff correctly in /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache? or should I redirect to a specific PATH?14:14
pedroalvarezjjardon: if you do what you said, it will have the same consecuences as if you run it on first boot14:18
jjardonI mean, my understanding is that "gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache" will work14:19
pedroalvarez(these commands are run once the system is built, in a chroot)14:19
jjardonrigth, thanks14:19
*** zoli__ has joined #baserock14:21
*** zoli__ has joined #baserock14:21
ssam2SotK: i was hoping for better than 3-4 minutes... would be interesting to see what's taking the time14:26
*** zoli__ has quit IRC14:26
ssam2although the reduced system size is already a great improvement14:26
*** zoli__ has joined #baserock14:27
SotKssam2: it looks like the system integration commands themselves are taking the time to me14:29
SotK2015-02-24 13:51:31 [Build 218/218] [test-system] Running the system integration commands14:29
SotK2015-02-24 13:54:16 [Build 218/218] [test-system] Removing staging area14:29
*** zoli__ has quit IRC14:30
*** zoli__ has joined #baserock14:30
pedroalvarezI'd like to believe that the system-instegration commands are to not taking all that time14:32
Kinnisonextra debugging will probably point the finger14:33
SotKI may be doing something silly, but its already a decent improvement (my comparison build has taken 10 minutes and is still going)14:35
SotK11:55 for the same system before the OSTree work14:36
ssam2sweet! much progress!14:36
SotKsystem artifact for that is 1.8G14:37
franredSotK, great!!14:39
* SotK adds some print statements to debug what on earth is taking so long, and the build takes 01:4814:56
SotKI love it when print statements make things work better14:57
ssam2echo 2 > /proc/sys/vm/drop_caches and try again :)14:58
SotK05:04 :(15:10
SotK4 minutes of which was the runcmd_unchecked call which runs the system integration command15:12
SotKI wonder if the overlayfs stuff is making its containerisation be slow maybe?15:12
richard_mawpossibly, but my guess is that the CoW overlayfs operation is making writes slow15:13
straycathave folks done any kind of ntp configuration for openstack nodes?15:29
straycatfranred, pedroalvarez ^15:29
franredstraycat, not really, I think ntp is configured by systemd now, but I can be wrong15:30
* straycat nods15:30
* pedroalvarez hasn't15:30
straycatokay we could do that then, it'll be needed for swift15:30
straycatwhen i say we i mean pdar/me15:31
franredssam2, Im testing your morph's branch "sam/sourceresolver-fixes", for now looks promising, and someone has to review it more, because fixes important bugs15:34
franredsomeone, apart from me, I meant15:35
ssam2thanks for testing it15:35
ssam2I suspect not many people are using 'master' of Morph right now. If they were, they'd be hitting the bugs that branch fixes15:35
nowsterah... I'm using "master".15:42
ssam2commit 88e8767791d85aa0d01715f04b7e0f311f54759e ?15:44
*** DavePage_ is now known as DavePage15:45
SotKrichard_maw: I guess its not that, it takes ~1:30 without the OSTree stuff but with overlayfs15:52
SotKas opposed to almost 4 minutes with the OSTree stuff15:53
ssam2that's weird. surely there is no ostree operation happening when running the system-integration commands?15:53
SotKthere shouldn't be15:53
ssam2perhaps the fact that the system is now a hardlink tree is making it slower?15:53
jjardonhi, about versioning of definitions: Do you think it should be a parameter of each morphology file or something that lies in the root directory of definitions?15:54
ssam2jjardon: this was already documented at the meetup I think15:54
jjardongreat, thanks15:55
*** zoli__ has quit IRC16:03
robtaylorrdale_: how's it going with musl?16:20
rdale_ok - i've built build-essential with musl and it's all working, and builds chunks in core.morph16:30
rdale_i'm just working out how to apply the linux headers patched for musl in a 'kernel-headers' repo, to a branch based on the v3.12 in the lorried linux git repo16:32
Zarahi, I'm getting an error when trying to build 'thrift' as part of building openstack-server.morph. I can paste the end of morph.log if this isn't something we've hit before.16:43
* SotK wonders how long deploy normally takes16:47
SotKs/deploy/rawdisk deploy/16:47
franredZara, could you post the error?16:48
Zarahere's a paste, I included some of the earlier stuff in case it was useful, but there was too much to fit:
franredgen-c_glib/shared_service.h:86:69: error: unknown type name 'SharedStruct' --> that's the error. which version of thrift are you using?16:52
franredZara, ^^16:52
* SotK boots a system built and deployed with an OSTree artifact cache :D16:53
franred:D :D16:53
SotKdeployment took ~7 minutes, 5 of which was doing the copy into orig16:55
Zarafranred: I'm trying to build from this branch, so it's the version of thrift here: . Not sure where it's defined.16:55
ssam2SotK: sweet! how does it compare when you use tar.write ?16:56
SotKssam2: dunno yet, I'll do that and see16:57
franredZara, that branch should compile without errors, AFAICT - we haven't modify thrift since long time ago :/16:57
Zarayeah, this is why I'm really confused by it. :S I think it's probablyl something weird on my end, but I don't know what would affect it16:58
Zarathe build failed at step 362 of 40716:58
*** Krin has quit IRC17:03
*** a1exhughe5 has quit IRC17:04
SotKssam2: 7 minutes 20 seconds (rawdisk was 7 minutes 32 seconds)17:09
SotKnow to see how they compare with old deployments17:09
ssam2hmm, that's a bit unfortunate17:10
SotKdid tar used to be faster?17:10
ssam2I doubt it17:10
SotKwhat is unfortunate then?17:10
ssam2that it takes 7 minutes instead of like, 3 seconds :)17:11
SotKthe "Writing to device" step took 6:3317:11
*** zoli__ has joined #baserock17:20
pedroalvarezMaybe what about sysroot.write?17:21
pedroalvarezthat one only puts the rootfs in a folder17:22
SotKI'll try that once this old-cache deploy is done :)17:22
* SotK wonders how we managed for so long with such slow deployments17:25
straycatwith despair17:27
straycatand various hacks to reduce the number of times you needed to deploy17:28
straycatit was even worse when we didn't have liw's xfer-hole thing17:28
pedroalvarezdeploying an installer system (3 systems in one) was really really slow17:29
ssam2i mostly deployed minimal systems with the gcc binaries removed, then ran them as containers17:30
ssam2but that doesn't work for all cases17:30
*** mdizzle has quit IRC17:38
*** lachlanmackenzie has quit IRC17:43
*** bashrc has quit IRC17:58
* richard_maw grumbles that the minimal system currently contains gcc, and adds that to his list of things to look at tomorrow18:02
pedroalvarezrichard_maw: not sure if this branch is relevant:
*** ssam2 has quit IRC18:05
richard_mawI've made a note of it18:06
*** jonathanmaw has quit IRC18:07
*** Guest3380 has quit IRC18:17
*** rdale_ has quit IRC18:36
*** gfinney_ has quit IRC19:17
*** gfinney has quit IRC19:17
*** gary_perkins has quit IRC19:45
*** zoli__ has quit IRC20:10
*** gfinney_ has joined #baserock20:32
*** gfinney has joined #baserock20:32
*** zoli__ has joined #baserock20:45
*** zoli__ has quit IRC20:49
*** zoli__ has joined #baserock20:49
*** gfinney has quit IRC21:36
*** gfinney_ has quit IRC21:36
*** gfinney_ has joined #baserock22:13
*** gfinney has joined #baserock22:13
*** gfinney_ has quit IRC22:14
*** gfinney has quit IRC22:14
*** gfinney has joined #baserock22:14
*** gfinney_ has joined #baserock22:14
*** gfinney has quit IRC23:18
*** gfinney_ has quit IRC23:18
*** zoli__ has quit IRC23:46
*** zoli__ has joined #baserock23:47
*** zoli__ has quit IRC23:52

Generated by 2.14.0 by Marius Gedminas - find it at!