IRC logs for #baserock for Tuesday, 2014-11-04

radiofreeBen has been hardware hacking it01:48
radiofreeBut he's not here01:48
*** zoli_ [~zoli_@0x5e91887a.adsl.cybercity.dk] has joined #baserock04:43
*** zoli_ [~zoli_@0x5e91887a.adsl.cybercity.dk] has quit [Changing host]04:43
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock04:43
*** jmacs_ [~jimmacart@access.ducie-dc1.codethink.co.uk] has joined #baserock06:12
*** paulsher1ood [~paulsherw@access.ducie-dc1.codethink.co.uk] has joined #baserock06:13
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]06:16
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock06:56
*** mike [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock07:38
mike is now known as Guest7693407:38
cosmrobtaylor: yep, I've emailed him because he's the author of the git commit containing the default BCT for jetson08:14
cosmbut apparently someone else wrote the actual config08:14
*** dutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:16
dutch is now known as wdutch08:16
persiacosm: For hardware-type issues, you might also find #tegra useful08:18
*** perryl [~lauren@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:36
cosmthanks persia, I'll give it a try08:56
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:00
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:13
Mode #baserock +v ssam2 by ChanServ09:13
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:20
*** tiagogomes [~tiagogome@213.15.255.100] has joined #baserock09:22
petefothAre there differences between a ‘configuration extension’, a “deployment extension’ and a ‘write extension. Is so, can somone spare the time - maybe FTF - to explain them to me?09:47
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:47
straycatThe write extension, is the bit of code that performs the deployment, so if you're deploying over kvm for example, the kvm write extension handles that.09:49
straycatConfig exts are scripts that configure a system, they take a rootfs and perform some modification on it, e.g. appending some entry to /etc/network/interfaces09:51
petefothstraycat: Thanks09:52
petefothIs there such a thing as a ‘deployment’ extension, or is that just a term which covers both write and config extensions?09:53
straycatI don't know what a deployment extension is, I guess someone might have been referring to a conf ext or a write ext09:54
petefothstraycat: thanks again09:57
ssam2I sometimes use the term deployment extension to cover both write and config extensions09:57
ssam2I don't know if I like that term, but I tend to make up new terms by accident when talking to people09:58
petefothssam2: thanks. I think I have enough information now to have sensible text in the glossary 10:01
*** Guest76934 [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]10:02
* richard_maw uses deployment extension to refer to both configure and write extensions10:04
petefothSo does http://ur1.ca/ioqwm seem OK?10:10
straycatSeems okay to me10:12
*** Krin [~mikesmith@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock10:12
Krinmorning all10:13
*** Guest76934 [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock10:16
Krindoes anyone know if there is a way of telling morph to not clean up certain files/folders after it is done building a system, as zookeeper actually uses the files after installation and 'make install' only does half the job due to cosmic rays being beamed onto the heads of whoever made it?10:17
KinnisonNo, unless you install the files into $DESTDIR during the chunk build, they will not end up in the target system10:17
Krinwould that mean simply moving them to $DESTDIR then? if i move them to $DESTDIR can i build inside $DESTDIR? as moving the folder seems to break things too which means they probebly have some relative path refferances in there ¬.¬10:18
Kinnison$DESTDIR will not be in the same place in the target system anyway10:19
KinnisonI'd suggest looking at how Debian does it10:19
Kinnisonsince they have a similar set of constraints10:19
franredKrin, Kinnison, why not copy the required files in the system and use them later? I did something similar when one repo need another inside to get compiled10:20
* Krin <sarcasm>is feeling all the love and unity for mankind today</sarcasm>, hopes he does not offend anyone who does not deserve it.10:21
Krinhow do you mean franred? you mean move them into the system after build? during build? 10:21
franredumm Kinnison told you before me... I need to drink more coffee10:22
franred<Kinnison> No, unless you install the files into $DESTDIR during the chunk build, they will not end up in the target system10:22
Krinthis calls for reading and experimentation~!10:23
richard_mawpetefoth: I'd say s/VM file/VM disk image/, but other than that, it looks fine to me10:23
Krinand caffeine 10:23
petefothrichard_maw: ta!10:23
persiaDoesn't install-commands: provide a place to put scripting to move any magic bits at build time?10:24
rjekISTR some work to automatically create Baserock chunks from ruby gems.  Am I making that up?  has anybody done anything similar with pip?10:24
richard_mawrjek: ripsum is working on it10:25
richard_mawstraycat may be able to tell you more10:25
straycatrjek, ssam2 has done some stuff with that10:25
persiarjek: There was some work like that discuss ed on the mailing list.  I don't remember the final consensus, sadly.10:25
rjekOK, I'll look it up there.10:25
rjekThanks!10:25
persiaTry September, IIRC10:26
straycatMore specifically we're working on an import tool for rubygems, pip and others10:26
ssam2rjek: for rubygems, it's usable although not yet integrated or documented10:26
franredKrin, Kinnison, something like: http://fpaste.org/147670/50967871/ where rabbitmq-codegen is needed to compile rabbitmq-server10:26
persiastraycat: A generic tool for all the platform-specific systems?10:26
ssam2rjek: please come chat to me if you want to integrate some rubygems, more testing would be great !10:26
Kinnisonfranred: either you or I am missing a point on this, so let's let Krin investigate for himself :-)10:26
rjekssam2: I want to integrate some pip things, I was just wondering if I was making the rubygems thing up because it sounds similar!10:27
straycatpersia, One tool for importing packages from different packaging systems into Baserock yes10:27
franredKinnison, ok :)10:27
KrinKinnison, franred: ... my brain hurts already10:27
ssam2Richard Ipsum is doing some work for PIP, but it's at an earlier stage than the RubyGems integration10:27
persiastraycat: Do be careful.  I remember the work in packagekit, and that was only trying to define common semantics for distro packages.  There's more divergent semantics in platform-specific packaging systems.10:27
persia(but I'd be delighted if this problem can be reasonably solved)10:28
jmacs_For PIP, are you allowing PIP to run on baserock systems, or importing the data from PIP into baserock definitions?10:28
richard_mawthe latter10:28
richard_mawI believe we can already do the former10:28
* richard_maw has previously done so10:28
ssam2the tool as it exists is here: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/import.git/tree/10:28
jmacs_OK, thanks10:28
persiajmacs_: Be warned that as soon as you run pip on a system, your system becomes a snowflake.  Nice for prototyping, but not so nice for maintainability.10:30
jmacs_Not my choice of tool.10:30
straycatpersia, *nod* What we have at the moment amounts to a best effort to generate lorry files from package names, then finding dependency information for the package and generating definitions from that.10:30
persiaCool.  Just do remember that "depends upon" doesn't always have a consistent meaning.10:31
KinnisonKrin: have a look at http://ftp.uk.debian.org/debian/pool/main/z/zookeeper/zookeeper_3.3.5%2bdfsg1-2.debian.tar.gz10:33
Krinthanks Kinnison 10:33
jmacs_ is now known as jmacs10:33
straycatpersia, The way ssam2's designed it, each packaging system provides extensions to map a package name to a lorry file, and map a package name to a set of build dependencies and a set of runtime dependencies, that seems to work for the packaging systems we're interested in.10:35
petefothpersia:  I think I understand what you mean by the word ‘snowflake’ in this context - something with a unique structre, hard - if not impossible - to reproduce. Is that roughly correct? (I copuldn’t find a useful definition with DDG or google)10:39
persiapetefoth: Yes.10:40
petefothpersia: tvm10:41
persiastraycat: Ah, so it's just an abstraction that doesn't provide semantic guarantees, leaving these to be platform-dependent?  That's probably sensible, as users of a given platform can assume their platform semantics.10:41
* straycat nods10:50
Krinthese guys are GENIUSES! simple elegant solution making GENIUSES! *looking at the debian rules in Kinnison's link* and i think most of this can be done easily in baserock too happy Krin is happy now10:51
Kinnison:-)10:51
* petefoth likes to see a happy Krin!10:51
Krinbut i'm not a genius, i forgot to bring my breakfast with me ¬.¬10:51
KrinBISCUITS!10:52
Kinnisondoh10:52
jmacsbaked doh10:53
Kinnisonplay doh10:53
rjekforbidden dohnut10:54
* Kinnison is out of tea :-(10:54
petefothCan anyone give me a short (one or two lines) definition for ‘staging area’? I coulkd write one myself and ask people to check it but I think this way may be quicker)11:01
paulsher1oodpetefoth: http://en.wikipedia.org/wiki/Staging_area11:02
paulsher1oodin baserock context it is an a location where software is assembled prior to release/publication/deployment11:03
petefothpaulsher1ood: ta! Didn’t think to look there as I assumed we had a mpre specific usage.11:03
petefothwould it normally be on the development system or on the idstbuild controller or…?11:04
richard_mawdevelopment system for local builds, worker nodes in distbuild11:05
petefothrichard_maw: thanks11:05
*** madhu [~madhu@202.0.77.198] has quit [Remote host closed the connection]11:06
abdulssam2:  even updated the build-system as cpan in xml-parser morph file. build is getting failure 11:10
ssam2abdul: could you paste the error output somewhere for me to see?11:12
franredif we specify build-system: autotools, and add configure commands ./autogen.sh --foo --blah ... does the chunk expect to add ./configure or it does by default?11:29
ssam2franred: you can't 'add' configure-commands11:30
ssam2you override the default configure-commands11:30
ssam2you can set pre-configure-commands or post-configure-commands, though11:30
KinnisonIf you're changing how autogen.sh is called, you should override configure-commands11:31
straycatssam2, Can we change the import tool so that the chunk morph stage is optional? Morph has instructions for handling python packages built in.11:32
ssam2makes sense, but it might break some assumptions I made in the code11:34
ssam2might be best to always create one for now, and rework the code to be less dumb later on11:34
* straycat nods11:34
straycatOkay11:34
abdulssam2: http://fpaste.org/147691/14151009/11:35
ssam2it seems completely impossible to configure a trove with a proxy.conf :(11:36
straycatI don't know much about gems, what stops us adding a buildsystem for gems as we have done for python packages?11:36
ssam2straycat: it's not impossible, but a bit of a pain11:36
ssam2the build commands need to include the name of the Gem11:37
* franred will spend some time reading morph/README sounds like a nice place to know these magic tricks which people around me do11:37
richard_mawabdul: that's weird11:37
* richard_maw will see if there's something obviouly wrong with the Makefile11:38
ssam2abdul: I think this must be a timestamps issue11:38
ssam2however, Morph tries to reset all timestamps before running a build11:38
ssam2maybe the timestamps are getting messed up by NFS somehow11:39
KinnisonIf your staging area is being constructed over NFS, it could do all sorts of bad things (tm) to timestamps11:39
Kinnisonesp. *during* the build itself11:39
ssam2OK. I knew NFS wasn't ideal, but I believe it's only way that Abdul can get an ARM build going on his target hardware11:41
KinnisonYou *really* want to ensure clock-sync between the NFS server and the ARM board then11:42
Kinnisonif you get good clock-sync you should be safer11:42
ssam2ah, that's a good point11:42
Kinnisonthe issue is that some timestamps come from the client and some from the server and NFS's metadata is, in part, eventual-consistency11:42
straycatssam2, Oh so we'd just need to pass the chunk name to BuildSystem11:42
ssam2straycat: be careful with use of the word 'just' ;) but yeah, that's the main blocker11:43
richard_mawssam2: also, that assumes you're always able to have your chunks have the same name as the gem presumably11:43
ssam2richard_maw: yes it does11:44
ssam2there may be other solutions11:44
straycatparameterise the build system, so that in the case of rubygems buildsystem you also specify the gem name11:45
richard_mawKinnison, abdul: It's definitely the timestamps which are the issue. There's a rule in the generated Makefile that will re-generate the Makefile if the Makefile.PL is newer.11:45
* Kinnison nods11:46
* Kinnison leapt in when he heard NFS, but didn't have the background11:46
Krinhmm, could someone explain $DESTDIR and $PREFIX to me? what do these end up being?what path would they be equivilant to in the resulting build? or have i got completely the wrong idea?11:50
richard_mawPREFIX comes from the stratum, defaulting to /usr11:50
KinnisonBasically $PREFIX is the filesystem basis for the installed and *running* code11:50
Kinnisonas richard_maw says, it's typically /usr11:51
KinnisonDESTDIR is where morph wants you to pretend will be '/' on the installed system11:51
Kinnisonand will typically be CHUNKNAME.inst11:51
richard_mawthis is traditionally used in builds because /usr is vendor installed software, /usr/local is locally installed, and "$HOME/.local" is per-user installed11:51
Kinnisonsorry, /CHUNKNAME.inst11:51
richard_maweverything put in DESTDIR gets cached, so it can be used to satisfy build-time and run-time dependencies11:51
richard_mawit gets unpacked into the staging tree before commands get run in it11:52
Kinnisonso if the chunk builds a program called 'foo', and you told it PREFIX=/usr and DESTDIR=/foo.inst, when you 'make install' you'll get /foo.inst/usr/bin/foo and in the final system that'll be /usr/bin/foo11:52
Krinso $DESTDIR$PREFIX will essentialy lead to /usr but if that is not the default it will lead to the vendor installed software location?11:52
Krinor should i use $DESTDIR for things that will be cleaned up later and $PREFIX for things that ned to remain?11:53
KinnisonNot quite11:53
Kinnisonanything you put into $DESTDIR will end up in the chunk and thus in further builds or the system itself11:54
Kinnison$PREFIX is the path that the stratum is asking your chunk to put its content inside11:54
*** cosm [~Unknown@us2x.mullvad.net] has quit [Ping timeout: 245 seconds]11:54
richard_mawyou pass $PREFIX to any configure scripts, so it knows where it files will end up on the final system. you use `make DESTDIR="$DESTDIR" install` to make it put the files there, and if you need to install any software manually, you need to copy it into "$DESTDIR$PREFIX"11:54
Kinnisonso anything in $DESTDIR$PREFIX will end up in the right place on the target system11:54
KinnisonPREFIX might get baked into the build, DESTDIR is purely about staging the results so morph can pick them up right11:55
Kinnisonhttps://www.gnu.org/prep/standards/html_node/DESTDIR.html might explain DESTDIR11:55
Krini... think.. i think i understadn *goes to read*11:56
KinnisonI can whiteboard it for you if you're still unsure :-)11:56
KinnisonIt's a pain, but once you 'get' it, you'll be much more comfortable11:56
richard_mawsome other build systems refer to it as the "install prefix"11:56
* richard_maw likes the commend on that page11:57
richard_maw"You should not set the value of DESTDIR in your Makefile at all; then the files are installed into their expected locations by default. Also, specifying DESTDIR should not change the operation of the software in any way, so its value should not be included in any file contents. "11:57
* richard_maw has seen SO MANY build systems fail to do this properly11:57
franredssam2, http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/README is very useful, thanks11:58
richard_mawPeople put `DESTDIR = ` in their Makefiles as they think they need to define the variable first11:59
richard_mawDESTDIR is often picked up by the environment, but if you put `DESTDIR =` in, it will be overridden with nothing, so the build fails11:59
richard_mawwhich is why we do `make DESTDIR="$DESTDIR" install`11:59
richard_mawsince variables defined on the make command line override definitions in the Makefile12:00
radiofreecan we upgrade systemd and let it take charge of the network stuff?12:12
Kinnisonwe are working toward that I believe12:12
radiofreefor some reason (dodgy wires?) my board here seems to boot before the link is ready12:12
radiofreewhich means a manual udhcpc, over serial... which isn't great12:13
Kinnisonradiofree: I think jjardon was working toward newer systemd, so you could ask him12:14
radiofreeKinnison: thanks12:14
radiofreereplace wires aswell?12:14
Kinnisonyou could try, but it's more likely to be the PHY or MAC at the other end being slow12:15
radiofreethe link is coming up *after* the kernel dhcp for some reason12:15
Kinnisonboggle12:15
KinnisonHang on, kernel DHCP?12:15
KinnisonAre you doing ip=dhcp in the kernel and then expecting udhcpcd to work?12:15
radiofreeno no12:16
Kinnisonphew12:16
* radiofree might be an idiot, but he's not that much of an idiot12:17
radiofreeis it possible to pass environment variables to a chunk?12:28
ssam2radiofree: not without changing Morph12:32
ssam2what do you need to do ?12:32
*** madhu [~madhu@202.0.77.198] has joined #baserock12:33
radiofreessam2: i want to pass environment variables to a chunk12:34
*** cosm [~Unknown@cspc154.cs.man.ac.uk] has joined #baserock12:34
radiofreeis it possible to do in the build/configure commands?12:34
radiofreeFOO=bar&&./configure?12:34
radiofreethat'll be ok right?12:34
* radiofree tries it12:35
radiofreeFOO=bar ./configure is thumbs up!12:35
ssam2radiofree: oh, that's fine12:36
radiofreeis there any global way of setting FOO though?12:36
ssam2no, chunks are built with a clean environment12:36
radiofreee.g i have several chunks where FOO is important, i want them all to take a global FOO12:36
ssam2see the recent thread on baserock-dev about templating12:36
ssam2I think everyone agrees that they want this, but there's debate about how it should be done12:36
radiofreeok, thanks12:37
ssam2'Conditionals and branching in definitions' is the thread12:37
radiofreeyeah i want it but the "how it should be done" bit is way beyond me12:37
ssam2setting env vars. is one option, but they need to be included in the cache identity of the build artifact12:38
abdulssam2: is there anyway to solve this issue12:38
ssam2abdul: check that the clocks on the NFS server and the build machine are in sync12:38
ssam2ideally get 'ntp' running on both of them12:39
radiofreewould writing the last time you got from ntpd to a file, so you could load that if the network is down, be desirable, or just a massive hack?12:42
straycatIf the network is down how do you know your current time is accurate?12:44
persiaYou compare it to the feed from your USB atomic clock.  Lacking one of those, you can estimate drift over time based on historical ntpd data.  Lacking that, you don't.12:45
ssam2abdul: beware that NTP uses some random port, it's not HTTP traffic, so you might not be able to talk to NTP servers outside your firewall12:45
straycatpersia, I assumed a lack of both atomic clock and historical data12:45
radiofreestraycat: you don't, but i'd be happier being 1,2,3 hours out rather than 44 years12:46
persiaradiofree: There are some hacks to deal with the 44 year problem, typically used for boards that do not have batteries for their clocks, and/or lack actual hardware clocks.  If you haven't booted before, set the clock to the atime of the kernel you booted, and if you have, add an offset based on the last entry in the dmesg ring.  I was told it was a kernel command-line option, but that might have involved a patched kernel.12:48
radiofreethat seems like a decent enough hack :)12:49
persiaIt can lead to poor results if one forgets, and builds a kernel with an atime matching the epoch :)  Another trick I remember being used was the last mount data of specific filesystems, but I think that ended up being dropped because it doesn't work for read-only filesystems so well...12:51
jjardonI have systemd v217 working here, but we need the coreutils patches first13:04
jjardonradiofree: ^13:04
paulsher1oodany idea how i could do 'su - postgres' on a machine with that user setup, and yet still be root afterwards?13:23
paulsher1oodmaybe i'm misunderstanding how 'su' works. 'su postgres' => postgres, 'su - postgres' => root13:24
richard_mawooh, new patches on the mailing list!13:27
richard_mawSotK: I don't recall why the GitDirectory had to search upwards for actual git directories, ssam2 thinks it's no longer needed, so I'd appreciate it if you could chime in on that one.13:28
* SotK tries to remember why13:29
richard_mawI think it was about the time of the git-fat work13:29
abdulssam2: After syncing both nfs server and build machine, xml-parser package is built successfully 13:30
richard_maw\o/13:30
Kinnisonabdul: excellent13:30
paulsher1oodw00t! :)13:30
paulsher1oodthis needs to be on the build errors page if it's not already?13:31
SotKrichard_maw: yup13:31
madhuabdul, cool13:32
madhuabdul, now we may not require external disk13:32
*** jjardon84 [~jjardon@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock13:41
richard_mawpedroalvarez: wow, glibc is taking a while to clone13:46
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]13:53
petefothI’ve pushed an updated version of the Glossary, and would welcome comments or - even better - others adding to, correcting, or otherwise improving it13:54
petefothhttp://wiki.baserock.org/glossary/13:54
* Kinnison takes a break from his thinking to have a read13:55
richard_mawpetefoth: we probably don't need to go into so much detail about how the cache key is calculated in the glossary13:56
Kinnisoncache key is a little confusing.  we calculate it beforehand not by looking at the built outputs13:56
petefothrichard_maw: good point - that was a cpy and pasa - I’ll trim it down13:56
Kinnisonthe first paragraph is possibly superfluous entirely13:57
jjardonpedroalvarez: building a weston system here with your glibc patches, I will let you know13:57
persiaThe distinction between configuration extensions and write extensions can be useful: maybe "Deployment Extension" should reference those, rather than the other way about?13:57
persiaDeployment fails to mention staged deployment: things like deploying to a master ISO, or deploying to glance, etc.13:57
Kinnisonpetefoth: the cache server also provides access to querying the git repositories without having to clone them13:58
persia"Trebuchet" is a fancy name, but it isn't backed by a specific tool.  There is some trebuchet-derived stuff that gets used, but II'm uncertain about this definition (although it is better than the old one).13:58
persiaKinnison: Isn't that a side effect of the cache server happening to be colocated with the git server?13:59
Kinnisonpetefoth: In Chunk/Chunk-artifact it'd be better to say that a chunk is, in general, built from one git repository.13:59
robtaylorpersia: well, there's tb-diff13:59
persiaIt probably also makes sense to talk about artifact splitting somewhere.13:59
Kinnisonpersia: Well, without it being so, morph will have a harder time of things.  Of course, morph allows you to specify a cache server which is next to the git repos, and a separate one which has artifacts13:59
richard_mawpetefoth: deploy is missing the second ` around the `morph deploy`13:59
persiarobtaylor: Yes, but the diff bits don't do what they say on the tin (or didn't last time I looked, which was admittedly months ago).14:00
* petefoth though he had run out of stuff to do. He was wrong :)14:00
robtaylorprobably true..14:00
petefothPlease keep the comments coming!14:00
persiaKinnison: Right.  The morph defaults assume that, but I don't think we want to reinforce it conceptually.14:00
Kinnisonpetefoth: in deploy,deployment you have a missing ` closing off `morph deploy14:00
persiapetefoth: Also, thanks a lot for trying to sort this out.14:00
Kinnisonpersia: then we should have 'artifact cache server' and 'git cache server' as separate glossary entries14:01
persiaYes, we should.14:01
* Kinnison concurs that we should have the write extension and configuration extension explanations separated, and then have deployment extension reference the two14:01
persiaAlthough I admit I have difficulty imagining a git cache server distinct from a local curated mirror.14:01
Kinnisonpetefoth: since we added build systems, the devel system contains the tools *typically used* in developing on Baserock14:02
Kinnisonpersia: a git mirror service can't directly answer morph's questions, hence we wrote a service to do it for us14:02
Kinnisonpetefoth: in development systems you have _definitions- instead of _definitions_14:04
* richard_maw enjoys the comment "Blame the ancient Romans"14:05
* richard_maw objects to the comparison of chunks to strata as lego to duplo14:06
richard_mawduplo is not made out of lego14:06
* richard_maw thinks the following analogy is better14:07
Kinnisonpetefoth: in the `morph` section I'd also have a statement that "Also 'a morph' often refers to a morphology file.  See definitions"14:07
petefoth richard_maw I refuse to take the blam for that - another copy / pasta from ane earlier document14:07
richard_maw:-(14:07
richard_maws/(/)/14:07
jmacsrichard_maw: duplo-compatible bricks can be made out of lego14:07
Kinnisonjmacs: really? I thought the pips were incompatibly sized14:07
persiaKinnison: Thanks for the explanation.  So there is a bit that needs to sit above the git server to make a git cache server?  What is the use case for running that separately from the artifact cache server?  (I'm not opposed, just curious)14:08
jmacsKinnison: Not in all directions, clearly. I'm pretty sure you can put duplo on lego though.14:08
Kinnisonpersia: the use case for separation is fr.ex. mason where you don't necessarily want all the artifacts making their way to the shared artifact cache server until a build succeeds14:09
richard_mawclearly we need a trip to the Lego shop to settle this issue!14:09
Zaramaybe there's a lego conference coming up?14:09
Kinnisonpersia: also distbuild workers use the artifact cache server code to share artifacts back to their shared cache for the build14:09
Kinnisonpetefoth: also, if you say 'Lego bricks' like a good brit, saying 'Duplos' is just vile.14:09
Kinnisonpetefoth: workspace has `morph uses` where I think you mean `morph` uses14:12
* Kinnison can do a more careful review later if you want, once you've made the current fixups you have queued14:13
petefothKinnison: tvm - I’ll let you know when it’s ready14:13
richard_mawpedroalvarez: wow, still cloning glibc14:15
ssam2I've added 'Building on NFS' to http://wiki.baserock.org/guides/build-failures/?updated#index7h214:22
Krinhmm, on a baserock system, is there a reason i would be refused access to my local host when i use 'echo ruok | nc localhost 2181'14:27
Kinnisonssam2: looks good14:27
Krinit is running on a VM if that makes a diference14:27
KinnisonKrin: does the VM listen on 2181 ?14:27
Krini'm unsure and unsure how to tell14:27
richard_mawlocalhost != 127.0.0.114:28
KinnisonWhy do you think 2181 would be listening at all?14:28
richard_mawunfortunately14:28
Krinbecause that is the default port for zookeeper Kinnison 14:28
Krinah, i looked on the ifconfig and it gave that address richard_maw 14:28
KinnisonKrin: erm, is zookeeper running in the VM?14:29
richard_mawthere's a bug in our /etc/nsswitch.conf that allows localhost to be resolved by dns14:29
Krinhow do i discover what the local loopback for a VM is?14:29
pedroalvarezrichard_maw: has it finished? I can't remember how long it took to me the first time :/14:29
KinnisonKrin: loopback for any system is 127.0.0.114:29
KinnisonKrin: or ::1 in IPv6ish14:29
pedroalvarezjjardon: thanks for testing weston! :)14:29
richard_mawpedroalvarez: still cloning14:29
ssam2in future, if you run `morph --verbose` you'll see the progress output from Git :)14:29
richard_mawif I also remember to update my version of morph14:30
*** tiagogomes [~tiagogome@213.15.255.100] has quit [Remote host closed the connection]14:31
*** tiagogomes [~tiagogome@213.15.255.100] has joined #baserock14:34
*** zoli_ [~zoli_@0x5e91887a.adsl.cybercity.dk] has joined #baserock14:34
*** zoli_ [~zoli_@0x5e91887a.adsl.cybercity.dk] has quit [Changing host]14:34
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock14:34
richard_mawpedroalvarez: took nearly an hour (52 minutes) to clone glibc14:45
pedroalvarez:/ 14:45
ssam2how big is the repo ?14:46
pedroalvarezssam2: and how long took to you?14:46
ssam2not sure, didn't seem very long14:46
ssam2I use a mismash of different troves on different machines right now though14:46
richard_mawpedroalvarez: 113.3M14:46
radiofreejjardon: ?14:48
*** perryl [~lauren@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Lost terminal]14:49
ssam2pedroalvarez: I'm willing to believe that the problem on ARM is the bottom of http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/build-essential/stage2-eglibc.morph14:49
pedroalvarezis exactly what your email suggested me14:50
pedroalvarezthat can be an easy fix14:50
radiofreepedroalvarez: did you test your patches on arm? i can start a build now if you haven't/want14:51
*** perryl [~lauren@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock14:51
radiofree(glibc patches)14:51
ssam2radiofree: I tried on a Jetson distbuild network already and it didn't work for me14:52
radiofree:\14:52
pedroalvarezand I have just tried an armv7lhf cross-bootstrap and it has failed14:53
pedroalvarezand stage2-glibc fails in this way: http://pastebin.com/thWpkM8D14:54
richard_mawwithout more information from config.log you can't really say what's gone wrong, but that error usually means that the cross-compiler is broken14:55
pedroalvarezlooking at config.log, looks like that. Trying to paste the log somewhere14:56
pedroalvarezhere it is: http://sprunge.us/hbjA14:57
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]15:00
richard_mawhm, not much new information there15:01
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock15:01
richard_mawjust that the cross-compiler is non-functional15:02
pedroalvarezjust tested with definitions master, and the same error happens15:10
pedroalvarezso that failure is not related with my glibc change15:10
pedroalvarezrichard_maw: 15:21
pedroalvarez2014-11-04 15:21:11 [Build 4/147] [stage2-glibc] Cloning upstream:glibc15:21
pedroalvarez2014-11-04 15:22:07 [Build 4/147] [stage2-glibc] Creating staging area15:21
richard_mawthere's a few things which could cause issue: 1. I'm using macvtap to bridge my VM onto my wired ethernet output, 2. the connection between my laptop and git.baserock.org may be further than you15:23
*** madhu [~madhu@202.0.77.198] has quit [Quit: Leaving]15:24
pedroalvarez3. git.baserock.org was slow when you cloned it15:24
petefothII’ve pushed some updates to the glossary. I believe I have adressed (if not necessarily resolved) all of the issues raised in this channel earlier.15:26
petefothKinnison: if you have time to give it another going over, that would be grand, but I’m going now.15:27
Kinnisonpetefoth: I'll try and go over it before I leave tonight15:27
Kinnisonpetefoth: I'll email you my comments if you're off15:27
petefothKinnison: ta!15:28
franredrichard_maw, looks like in the latest version of libvirt-gest.sh they remove the bash commands or at least remove the for ((blah)) sintax, I may try to update libvirt to the latest stable tag15:43
franreds/latest/last/15:43
*** cosm [~Unknown@cspc154.cs.man.ac.uk] has quit [Ping timeout: 250 seconds]15:51
jjardonis it possible in baserock to have some patches locally and apply them before build a chunk?16:14
straycatyou could morph edit then git apply?16:15
straycatssam2, http://sprunge.us/DNcM :/16:15
jjardonstraycat: I mean permanently, instead having a branch in the git repo16:16
straycatjjardon, Oh, then, no I don't think so16:17
richard_mawyou could in-line the patch content in pre-configure-commands16:17
richard_mawthough I think we're better maintaining a proper delta for important things16:17
*** cosm [~Unknown@cspc154.cs.man.ac.uk] has joined #baserock16:18
KinnisonI'd rather see a branch in the repo unless that's going to cause a lot of issues16:18
Kinnisonthey're way easier for an observer to grok16:18
ssam2straycat: how 20th century16:18
jjardonyeah, I was thinking in little patches like change the .gitmodules file16:18
richard_mawstraycat: you've made me angry at pip16:18
straycaterm16:19
straycat...16:19
richard_mawjjardon: can't do that there, since the .gitmodules are parsed before any build commands16:19
straycatrichard_maw, there there16:19
ssam2jjardon: there may be a better solution for submodules than manually patching .gitmodules16:19
radiofreehow much work would be required to make morph automatically adjust submodule repos?16:20
ssam2I agree that it sucks having to make a branch just to change that16:20
jjardonssam2: any idea already? :)16:20
radiofreeso we can just point at master rather than needing a patch to "adjust submodule location"16:20
ssam2jjardon: my head is too full of other stuff right now to think about it :(16:20
richard_mawmy current thinking for that is to drop parsing of .gitmodules, allow people to specify multiple source repositories, and require people use that to handle submodules16:22
Kinnisonwouldn't that run the risk of not keeping SHAs matching?16:23
richard_mawyeah, but you could have it inspect the .gitmodules to check whether it matches before the build16:24
straycatSo yeah we'll need to write something that keeps track of the constraints and raises an error if there's conflicting dependencies16:24
straycatrichard_maw, Actually I'm pretty irritated by this too :)16:25
Kinnisonrichard_maw: I'd rather have a module => repo mapping in the stratum, and apply that to the .gitmodules before resolving them at build time16:26
richard_mawbut then you get all the way towards build time before knowing that you need to cache another repository16:27
Kinnisonor at least update the cache, true16:27
Kinnisonunless you, at that point, use the git cache to resolve the modules for SHAs16:28
Kinnisonwe could add a method to the git cache server to resolve submodule SHAs and return them16:28
richard_mawwe want to be able to fall back when there isn't a cache server though, so we'd end up having to clone those repositories and poke at them, at which point we're back to the status quo16:38
pedroalvarezdigging in the problem of arm + glibc I found this error: http://pastebin.com/qYmHwC416:40
richard_mawThis paste has been removed! What an error indeed!16:41
pedroalvarezno! :/16:41
* pedroalvarez opens the paste from a different browser and sees a lot of ads...16:42
pedroalvarezI should change to another one16:42
* pedroalvarez tries again16:43
pedroalvarezhttp://fpaste.org/147811/14151193/16:43
ssam2wow, that looks excellent.16:44
pedroalvarez:)16:44
Kinnisonpedroalvarez: bleurgh :-(16:46
pedroalvarezas a side note, I think cross-bootstrap is broken16:46
persiaSpecifically, or generically?16:48
ssam2pedroalvarez: I found https://sourceware.org/bugzilla/show_bug.cgi?id=1645216:50
ssam2it might help a little16:50
Kinnisonpedroalvarez: Yeah, a little more help on what makes you say it's broken would be good :-)16:50
pedroalvarezpersia: not sure yet, since I've used it few weeks ago when doing the armv6 port, and it worked. but today it didn't16:50
persiaI don't really understand the details (even after having looked at the strata a couple times), but is there any way we can share  more between the bootstrap path and the cross-bootstrap path?16:51
persiaI had the impression before that cross-bootstrap had a tendency to bitrot due to lack of exercise.16:52
pedroalvarezKinnison, persia: I tried today cross-bootstrap and stage1-gcc failed16:52
Kinnisonpedroalvarez: so this is constructing the tarball?16:53
Kinnisonpedroalvarez: i.e. before you get to running natively?16:53
pedroalvarezyes16:53
pedroalvareznatively we only run stage316:53
Kinnisonbleh16:54
persiaAre bootstrap and cross-bootstrap using the same toolchain?16:54
Kinnisonthey're separate morphologies16:54
Kinnisonso unless humans are keeping them in sync, no.16:55
persiaThat was the source of failure in a previous iteration.16:55
pedroalvarezKinnison: they use build-essential.morph16:55
ssam2persia: the normal bootstrap does actually function in a very similar way to the cross-bootstrap16:55
persiaWIth the new structure, is there a way to share more between them?16:55
persiassam2: Yes, but with different repos and refs :)16:55
Kinnisonpedroalvarez: oh, cool, I didn't think they did16:55
ssam2I don't think there's a way to share anything between them without allowing templating/conditionals in chunk morphologies16:56
persiaAh, that again.  Oh well.16:56
persiaMaybe it is just a matter of watching more carefully on code reviews to keep them in sync for now.16:56
richard_mawnah, we only split them to reduce the amount of stuff you needed to build to cross-bootstrap, as you don't need all of a devel system16:57
ssam2oh, you're right16:57
ssam2I'm thinking of the cross-toolchain strata we did for the SDK, which is totally separate16:57
richard_mawwe ought to be able to go back to sharing if we have finer grained strata, but without runtime-dependencies this is a headache16:57
ssam2but yes, cross-bootstrap and bootstrap both use the build-essential stratum, so should be pretty much in sync!16:57
persiaDoes some of the work on the "build" system help with this?16:58
ssam2I imagine that it should do16:58
ssam2pedroalvarez: this seems to be the issue breaking x86_32 stage2-gcc: https://gcc.gnu.org/ml/gcc-patches/2012-03/msg01073.html17:04
ssam2there's a fairly simple patch that we can apply to GCC 4.6.3: http://pastebin.com/VkgE27Pd17:04
ssam2(i'll try to find an actual commit for that patch)17:05
pedroalvarezssam2: great! thanks17:05
ssam2i like that thread also contains a brief mention of doing full rebuilds of a distribution on every commit to GCC17:06
persiaA toy distro, or one of the big ones?17:07
ssam2one commenter in the thread suggest OpenEmbedded, another suggests Gentoo17:07
ssam2it's only a tiny discussion of the idea17:08
persiaFirst is toy-sized.  Second gets complicated :)17:08
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 272 seconds]17:21
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Quit]17:35
*** cosm [~Unknown@cspc154.cs.man.ac.uk] has quit [Ping timeout: 265 seconds]17:36
*** Krin [~mikesmith@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]17:48
*** Guest76934 [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:52
jjardoninstead commiting tarball content to the git repos, maybe its better to lorry the tarball instead?: so you can have m4 and m4-tarball for example17:52
Kinnisonjjardon: yes, we could do that17:52
richard_mawI was going to send this is an e-mail, but I wrote up an rfc about stratum runtime dependencies: http://wiki.baserock.org/rfcs/stratum-runtime-depends/17:55
jjardonok to lorry gcc? http://fpaste.org/147838/12388414/17:57
* Kinnison wonders why we don't lorry it already17:58
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Ex-Chat]17:58
KinnisonIs it a particularly uncomfortable repo?17:58
radiofreecan we lorry https://git.fedorahosted.org/cgit/fpaste.git ?17:58
pedroalvarezvoila: http://85.199.252.8717:59
richard_mawgcc was _the_ worse repo we found to lorry17:59
richard_mawI eventually got it to clone, but it took a _long_ time18:00
Kinnisonrichard_maw: worse than gnulib?18:00
richard_mawthey never had overlapping timelines so I can't say18:00
* Kinnison thinks it looks like gcc might be a little better now18:00
Kinnisonperhaps they cleaned up their history18:00
* Kinnison tests18:01
Kinnisonradiofree: why?18:01
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]18:02
jjardonpedroalvarez: did not upgrade m4 fix the build issue with newer glibc18:03
jjardon?18:03
Kinnisonjjardon: I'm test cloning it18:03
*** tiagogomes [~tiagogome@213.15.255.100] has quit [Ping timeout: 272 seconds]18:03
jjardonKinnison: thanks! 18:03
pedroalvarezjjardon: I found problems to do the upgrade, so I only patched it18:04
radiofreepedroalvarez: ok :)18:04
radiofreelet's add "haste" instead :)18:04
pedroalvarezuh?18:06
radiofreeKinnison: so i can do cat mylog | fpaste18:07
radiofreefrom a serial console or something18:07
radiofreebut pedroalvarez has linked a baserock pastebin, that looks nicer, there's a "haste" command for that18:08
KinnisonI see18:08
KinnisonI think sprunge.us works with curl or something18:08
paulsher1oodrichard_maw: while you're on with the runtime depends stuff, what's to stop us dropping strata altogether, and getting to definition has contents, build-depends, runtime-depends. contents are definitions?18:09
pedroalvarezradiofree: ahh! now I follow you.18:10
Kinnisonpaulsher1ood: Richard has left for the day, he may pick up on that later or else tomorrow18:10
pedroalvarezradiofree: yeah, we can add the haste client and configure it to default to a public baserock haste server18:11
pedroalvarezradiofree: but I don't know if that's safe for deverlopers with private suff18:11
radiofreethat's not your problem though?18:12
pedroalvarezradiofree: you can do this for now: `commands | curl -F 'sprunge=<-' http://sprunge.us`18:12
radiofreeif a developer has private stuff i would hope they're not using public fpaste/haste/pastebin instances anyway18:12
radiofreeah, nice!18:12
pedroalvarezI decided to use haste-server because it's nodejs, and I think that it can be done in baserock easily, if we want to move our infra to baserock18:13
pedroalvareznow is running in ubuntu18:13
*** cosm [~Unknown@host-78-150-56-250.as13285.net] has joined #baserock18:20
*** cosm_ [~Unknown@us1x.mullvad.net] has joined #baserock18:37
*** cosm [~Unknown@host-78-150-56-250.as13285.net] has quit [Ping timeout: 244 seconds]18:41
cosm_ is now known as cosm18:41
pedroalvarezwith this python client is possible to paste things in our hastebin server18:54
pedroalvarezhttp://85.199.252.87/hastebin18:54
pedroalvarezsorry: http://85.199.252.87/hastebinit18:55
pedroalvarezradiofree: ^^18:55
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]19:35
*** genii [~quassel@ubuntu/member/genii] has joined #baserock20:04
paulsher1oodnow that ruby and node are in devel, and we have the build-systems, would anyone object to adding the databases stratum to devel?20:19
franredpaulsher1ood, what is the definition of devel-system for you? a) a system which any developer can use for create their applications or b) a system to devel system which you can devel other systems which are going to be your target20:29
franredfor b) I can't see the point to add them, for a) yes, they would be needed20:29
paulsher1oodi'm using devel now for creating appliances - hence i created the websystem initially. now the main role of what was devel (ie building baserock systems) is done by build-system, it seems to me adding more developer tools to devel would be ok20:33
paulsher1ood(one problem previously with adding stuff to devel was that it increased the build/deploy/test cycle for validating core baserock itself)20:35
paulsher1oodfranred: ^^20:35
paulsher1oodnote i can't reallt define what devel is for me now. but on the other hand, anyone using node or ruby in devel is quite likely to find themselves wanting something from the databases stratum at some point20:36
* paulsher1ood would go the whole hog, and add docker too :-)20:37
*** tiagogomes [~tiagogome@host-2-99-200-67.as13285.net] has joined #baserock20:38
franredthen I don't have objections to add databases to devel system - did someone tried to compile a database stratum for an architecture different than x86?20:38
paulsher1oodin effect it seems to me that b) is now solved by build-system, no need for devel. but i may be wrong20:39
paulsher1oodfranred: actually that is a good question20:39
paulsher1oodi didn't. but i can20:39
pedroalvarezAfaik, we added ruby and node to the devel system because is needed for the import tool.21:21
paulsher1oodyup. but now they're there, folks like me will want to use them :)21:22
pedroalvarez also libvirt, xfce, gnome, openstack21:25
* jjardon just built a GNOME system with latest gnome-shell, xserver and systemd21:25
paulsher1oodcool :)21:25
pedroalvarezVideo! 21:26
pedroalvarezOr that never happened! 21:26
jjardongnome-shell still doesnt work, will do when I find out why ;)21:26
pedroalvarezHehe ok21:26
pedroalvarezpaulsher1ood: I was just saying, our tools dont need databases 21:28
jjardonmmm, seems the new systemd has a "update" mode when first boot21:30
* jjardon will investigate that21:30
*** tiagogomes [~tiagogome@host-2-99-200-67.as13285.net] has quit [Ping timeout: 244 seconds]21:31
paulsher1oodpedroalvarez: you may be right. but how would you answer franred's question? what is devel-system for, now?21:32
pedroalvarezA system to develop kissing baserock tooling? 21:37
pedroalvarezS/kissing/using/21:38
pedroalvarezI don't want it to be much bigger than how ir is now21:41
*** tiagogomes [~tiagogome@host-2-99-200-67.as13285.net] has joined #baserock21:43
paulsher1oodpedroalvarez: what's to stop you doign that in build-system?21:48
pedroalvarezpaulsher1ood: I can't use the import tool in the build system, since it doesn't have ruby and node21:51
pedroalvarezI may be wrong 21:52
pedroalvarezI'm considering the import tool  part of the baserock tooling 21:52
paulsher1oodah, ok. i've not been following this. is the import tool documented somewhere?21:53
*** tiagogomes [~tiagogome@host-2-99-200-67.as13285.net] has quit [Quit: Leaving]21:54
pedroalvarezI believe it's not documented yet, but we know that it has to be documented an soon21:55
paulsher1oodok22:02
paulsher1oodin other news, is there any way back if i delete my factory version?22:03
paulsher1oodor rather, any way forward? seems i can no longer deploy updates to self22:04
pedroalvarezI want to fix that bug. But to what system do we have to apply the upgrade? 22:13
pedroalvarezI looked into that a week ago and I couldn't find any suspicious in the code 22:14
paulsher1oodthe current default system i'd say22:16

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!