IRC logs for #baserock for Tuesday, 2015-01-06

*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]02:14
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock02:14
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]02:14
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock02:15
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]02:29
*** zoli_ [] has joined #baserock02:29
*** zoli_ [] has quit [Changing host]02:29
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock02:29
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]02:30
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock02:30
*** zoli_ [~zoli_@linaro/zoli] has quit [Read error: Connection reset by peer]02:36
*** zoli__ [] has joined #baserock02:36
*** zoli__ [] has quit [Remote host closed the connection]03:04
petefoth is an interesting view on something (nix and NixOS),  that tries to solve many of the same problems as Baserock08:05
*** petefoth_ [] has joined #baserock08:13
*** petefoth [] has quit [Ping timeout: 245 seconds]08:14
petefoth_ is now known as petefoth08:14
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock08:15
*** mariaderidder [] has joined #baserock08:37
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]08:44
*** bashrc [] has joined #baserock08:56
* Kinnison remembers nix from the days of writing slinux09:05
paulsherwoodiirc someone from nixos approached lars after his original baserock-related keynote at FOSDEM09:06
paulsherwood(to say nixos already did what he was describing)09:07
*** franred [] has joined #baserock09:07
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock09:07
Kinnisonnixos is somewhat different, but has related approaches in some cases (or at least it did back when I last examined it)09:07
*** tiagogomes [] has joined #baserock09:15
franredgood morning, yesterday I sent a patch to lorry some XStatic-* packages. I said in a previous email that I wouldn't [like to] do it because it is against baserock foundations. I though I could go with them from their github repos but 2 of them are in mercurial not in git, if we don't want to lorry them, do we have a test/clone trove to lorry these 2 repos so we don't lorry these XStatic packages and just use them until we can replace th09:16
franredem. NOTE: Using them will cut the development time to integrate Horizon in Baserock09:16
paulsherwoodfranred: +2 for lorrying them09:22
paulsherwoodi can reply on list if you like09:22
*** mike [] has joined #baserock09:23
franredpaulsherwood, not need it, I will go with this +2, cheers :)09:23
mike is now known as Guest5613709:24
*** jonathanmaw [] has joined #baserock09:29
Kinnisonfranred: please ensure the thread on-list is updated with the merge state though09:37
* Kinnison tracks whether threads are dealt with in that way09:37
*** ssam2 [] has joined #baserock09:40
Mode #baserock +v ssam2 by ChanServ09:40
straycatnixos abstracts all configuration where baserock doesn't and the implementations are pretty different, Baserock doesn't create the fhs by symlinking from some central store09:41
*** CTtpollard [] has joined #baserock09:42
franredKinnison, I've sent the mail to confirm the patch has been merged09:43
Kinnisonfranred: awesomesauce :-)09:45
Kinnisonstraycat: aye, those are some of the fundamental differences :-)09:46
Kinnisonstraycat: that nixos remains "packages" is another :-)09:46
straycatyeah I didn't realise that you can just nix-env -i foo or something to install new stuff on your system, I guess that explains why it's still pretty useable on desktops09:48
KinnisonMmm, tinycore linux is similar IIRC09:50
persiaInstalling new software on a system in baserock isn't that difficult: it just requires a reboot.09:56
KinnisonCurrently :-)09:57
*** locallycompact [] has joined #baserock10:12
*** Krin [] has joined #baserock10:13
pedroalvarezHi guys, I'm looking at the migration of to the Datacentred tenancy. I wanted to start the migration by copying the contents of /home to a volume there, so later we can upload  to Datacentred the disk and attach the volume. Sounds ok to you? Note: I think that to start copying the contents of /home we don't have to stop g.b.o. , only stopy its lorry controller.10:37
persiaWhat happens if people push?10:38
pedroalvarezmy thougths were to save the cpu that lorry controller is using, but yeah, there should be another rsync just before turning the original g.b.o and moving to the new one10:39
pedroalvarezs/turning/turning off/10:39
pedroalvarezare there others ideas about how should I do this?10:40
ssam2pedroalvarez: awesome :)10:40
persiaAnd that one probably needs to be guarded so that nobody can push during the transition.10:40
ssam2I think it's unavoidable to have a 'nobody can push during this time' period while we do the switch over10:41
pedroalvarezyeah, indeed10:41
persiaDoing a few rsync passes before the cutover ought significantly reduce that timeframe.10:42
ssam2doing an initial rsync with the machine live, then a second one later on with pushing disabled, seems to me like a good way of ... yeah10:42
persiaHow is testgerrit testing going?  Is there a chance that we can do both migrations at the same time?10:42
ssam2persia: I've not had any more time to look at doing a proper Gerrit deployment, sadly10:43
DavePage_Remember rsync --delete :)10:43
ssam2I think it'd be better to do the>DataCentred migration separately to anything involving Gerrit10:43
persiassam2: No worries.  I just tend to look for optimisations if possible :)10:43
ssam2persia: worth asking :)10:43
persiaThat's safer.  Fewer moving parts.10:44
franredpedroalvarez, ssam2, that plan looks ok to me10:44
persiaDavePage_: excellent point!10:44
ssam2pedroalvarez: if you think you can get the migration done today, perhaps pick a time to do it and announce it now10:45
pedroalvarez--delete means to remove files in the dest that are not in the original?10:45
pedroalvarezssam2: that depends on the rsync speed, so I don't know yet10:46
ssam2pedroalvarez: fair enough. but I don't *think* anyone will be angry if you announce downtime and then it doesn't actually happen10:47
ssam2less angry than if you don't announce it and then it does happen!10:47
persiapedroalvarez: That is my understanding of --delete10:47
ssam2a couple of other thoughts: (1) how are you thinking of disabling push access?  (2) and how are you thinking of actually switching the machines?10:49
ssam2i ask because I don't really know how I'd do those things ;)10:49
ssam2for (2) I guess the operator of DNS can switch the subdomain. and actually that should update instantly, if I'm not mistaken? maybe I am mistaken.10:50
persiaFor (1), I'd stop the git server process.  I'm not sure how that is accomplished.10:50
ssam2persia: there are a couple of systemd units to be stopped. I was wondering if there was a nice way of making Git 'read-only' rather than just 'dead', though10:51
pedroalvarezyeah, for 1 stop the git daemon, that means "downtime" since you cannot push or pull10:51
persiaFor (2), using netcat (or similar) for port-forwarding would be wise.  Distribution of DNS globally can take three times the max acceptable age in the SOA record.10:51
persiaRefreshing my memory, the normal propagation estimate should be 3*refresh, but may be as long as expiry.10:52
ssam2persia: how would you use netcat to forward the ports to the new machine? I thought netcat could only output to a file, not a TCP socket ...10:52
ssam2oh, I guess on the old machine we run `nc -l -p 23 | nc $NEW_IP 23`10:53
persiae.g. 'netcat -L -p 25000 -vvv'10:53
ssam2this isn't helped by the fact that there are at least 3 implementations of Netcat :(10:54
persiaHeh, true.10:54
persiaYou could also do it with ssh, if you like.10:54
persiaDepends what tools are present on g.b.o10:54
ssam2we have Busybox `nc`, and `ssh`10:54
pedroalvarezI asked the operator of DNS to change the cache times of the DNS so the DNS propagation of the new server happens fast10:54
persiaThen ssh is the tool of choice.10:54
persiae.g. `nohup ssh -N -L80: &`10:56
*** fay_ [] has quit [Ping timeout: 244 seconds]11:19
*** fay_ [] has joined #baserock11:33
pedroalvarezANNOUNCEMENT: Lorry controller will be off in for some hours today to speed up the migration. This migration is expected to conlcude at 18:30 today, and it means that there will be a 30 minutes of g.b.o downtime (18:00 to 18:30)11:36
*** petefoth_ [] has joined #baserock11:46
tiagogomeswhy the native-bootstrap script installs the files to DEST dir which is different than /, and then copies the files to /11:47
*** petefoth [] has quit [Ping timeout: 256 seconds]11:48
petefoth_ is now known as petefoth11:48
pedroalvarezI think my announcement of downtime was too optimistic11:52
ssam2tiagogomes: no idea, sorry ...11:53
perryldoes baserock currently have more than one version of python running?12:04
*** fay_ [] has quit [Ping timeout: 245 seconds]12:05
pedroalvarezperryl: there was a patch to add python 3, but I think it hasn't been merged yet12:05
perrylno worries, i'm trying to run pkg_resources.get_provider to get the location of an installed package, but it isn't working, a bit of digging mentions that multiple versions of python can cause it to fail12:06
perrylpedroalvarez ^^12:06
ssam2perryl: ouch. what error do you get from it? or does it just not return the package?12:08
perrylssam2: NameError: name 'pkg_resources' is not defined12:08
ssam2ah, you need to 'import pkg_resources' first12:09
* perryl quickly checks resources should have a 'u' in it12:09
perrylahh darn it :) i'd missed an import12:09
perrylthanks ssam, works fine now12:10
pedroalvarezso, assuming that I have already a copy of the g.b.o disk running  (with everything except /home), and /home mounted and sync'ed, could I just do the switching? I think I will need to sync  other things as well (like /var?) but I'm not sure if this is needed. Should I literally turn of g.b.o, and not turn it on until is moved to datacentred? (that means that in the downtime I have to copy the image and upload it to DC)12:14
franredpedroalvarez, is everything synchronized? I've just push to g.b.o12:15
pedroalvarezfranred: no no, don't worry. I'm just trying to figure out all what is needed to be done12:16
franredah, ok :)12:20
ssam2pedroalvarez: I don't think there should be any valuable state in /var12:23
ssam2we'd lose some journal entries from the old system12:23
ssam2'I don't think there should be any valuable state in /var' is a pretty inaccurate statement. What I mean is, there should be no valuable NEW state since you first copied the disk12:24
ssam2I don't know of any daemons that use it for persistent state, and there aren't any humans using that machine12:25
pedroalvarezin this case the migration will be easier12:26
persiaThere is really nothing in /var/run?12:27
ssam2I'm assuming pedroalvarez will restart all the daemons on the new system12:31
pedroalvarezsome .sock, .sockets-* .pid and .lock files12:31
pedroalvarezMy idea is to copy the disk when the machine is off.12:31
Zara_I'm confused-- using ssam2's definitions branch as a base, I changed the nodejs stratum and nodejs.morph file, and I pushed my changes to the nodejs stratum to a repo on my github account. I tried git-format-patch so I could send a patch to the mailing list, and I got some files that said 'from Sam', containing his changes from earlier, as well as another file containing my changes. I couldn't find emails containing the 'from Sam' patch12:46
Zara_sorry that's so long!12:47
DavePage_Zara_: Was there anthinhg after 'from Sam' patch?12:47
DavePage_If so then it truncated; try in irssi :)12:48
Zara_es but I don't want to send stuff twice over, and I'm not 12:48
Zara_               sure what to do/if this is normal.12:48
Zara_12:47 < Zara_> sorry that's so long!12:48
Zara_hopefully that's the rest12:48
*** fay [] has joined #baserock12:49
fay is now known as Guest2919412:49
straycatZara_, I would just send your branches but state in the cover letter the branch your branch is based off12:50
ssam2Zara: when you do `git format-patch` you specify the branch you want to merge *to*13:02
ssam2normally you specify 'master'13:02
ssam2but if you specify 'sam/nodejs' instead then you'll get patches for everything in your branch that *isn't* in sam/nodejs13:02
ssam2which I guess is what you want13:02
Zara_ah, okay, so then you'd merge your branch to master later?13:04
Zara_yay, that worked, thanks! :>13:09
*** Guest29194 [] has quit [Ping timeout: 264 seconds]13:32
dabukalam_ is now known as dabukalam13:33
*** Guest29194 [] has joined #baserock13:47
*** Guest29194 [] has quit [Ping timeout: 264 seconds]14:00
*** Guest56137 [] has quit [Ping timeout: 244 seconds]14:04
*** Guest56137 [] has joined #baserock14:19
*** Guest29194 [] has joined #baserock14:26
*** tiagogomes [] has quit [Quit: Leaving]14:46
*** tiagogomes [] has joined #baserock14:46
*** tiagogomes [] has quit [Ping timeout: 245 seconds]15:03
Zara_There's now a skeletal tutorial up on the baserock wiki for importing npm packages. Eagle-eyed observers may be amazed at the similarity to the rubygems tutorial.15:38
paulsherwoodlink? video?15:39
Zara_it's not as tidy as I'd like, and it's missing an example walkthrough, but I think it's better than nothing for now
pedroalvarezZara_: given that is the same tool, I'd expect it to be pretty similar to the rubygems one15:41
Zara_heh, I thought I might as well make a page for it, because importing rubygems seems more complicated and it's not worth going through the whole rubygems tutorial if only a fraction of it is relevant for your npm-importing needs :P15:43
*** tiagogomes [] has joined #baserock15:43
Zara_There's some duplication of content, but since the page is short, I thought a user who just wanted npm packages would prefer to have all the info on one page. Plus the earlier steps should become redundant as the import tool gets merged to master.15:47
tiagogomesis there any documentation on artifact splitting16:03
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]16:06
Guest29194 is now known as fay_16:08
ssam2tiagogomes: not much, sorry16:15
richard_mawtiagogomes: what do you need to do with artifact splitting?16:21
richard_mawthere's a handful of examples in definitions.git, the products rules in strata/build-essential/glibc.morph show how to make a chunk provide more artifacts in the "products" field16:24
richard_mawthe "products" field in strata/build-essential.morph says how to define a new stratum split by matching against chunk artifact names16:25
richard_mawthe glibc entry in the file has an artifacts field which specifies explicit assignment of chunk artifacts produced by glibc to stratum artifacts produced by build-essential16:25
tiagogomesrichard_maw I was just curious16:26
richard_mawsystems/minimal-system-x86_64-generic.morph shows how you can list specific artifacts to include, so you can make a smaller system16:26
pedroalvarezANNOUNCEMENT: Lorry controller is still down to speed up the migration of There will be some g.b.o downtime starting at 18:00 (UK time) to create a copy  of the g.b.o disk. Sadly, the migration won't be competed today. 17:07
DavePage_pedroalvarez: You know, you could shut down the VM, snapshot the LVM partition on the host, restart the VM and then copy the snapshot - that would reduce downtime?17:16
* pedroalvarez googles about LVM snapshots17:18
pedroalvarezI was goint to `cp` the .img file17:18
DavePage_Yeah, you can mount the snapshot as a read-only filesystem and cp the image file from that17:18
tiagogomesIf I updated the gcc-tarbal lorry to point to a new location, how that would work? It would remove the old repo and create a new one? What happens to the build-essential branch?17:24
persiaUnless there is upstream continuity of history, the practice has been to give lorries of different locations different names.17:27
richard_mawtiagogomes: it won't remove and re-create the repo, it will include all the old branches, but it may not be what you want to do17:28
pedroalvarezthis practice will be scary for all those repos where we are using a tag for ref17:29
tiagogomessearching trough the git history, gcc-tarball was updated with a new URL17:29
tiagogomesrichard_maw ah, so it is safe to update the tarball location to point to a new gcc, as we will be using the build-essential branch17:30
persiapedroalvarez: Isn't the use of tags restricted to the unused unpetrify-ref?17:31
persiatiagogomes: If you are updating the location just to add a new version, it is an entirely safe operation.17:31
pedroalvarezassuming that this new version is also a tarball import and not the git repo17:32
persiaRight.  If we want the git repo, we should use a new name.17:32
tiagogomesI know, my patched used a different name17:33
tiagogomesI am still benchmarking cloning from the git repo17:33
*** mariaderidder [] has quit [Quit: Ex-Chat]17:35
straycatWould it not be better to tweak ci so that system artifacts don't get uploaded to the cache? They can be constructed quicker than they can be downloaded.17:36
pedroalvarezcurrently the system artifacts are not being uploaded anywhere, is the shared cache that the distbuild networks of the CI need to work.17:39
straycatradiofree, Think I just observed the bug you thought you might have had a while back, if the stratum name differs from the stratum morph filename then it doesn't get built, morph gives no warning.17:39
straycatso there are system artifacts on cache.baserock.org17:40
persiapedroalvarez: Hrm?  Don't the systems artifacts get cached (just not the deployment artifacts)?17:40
*** franred [] has quit [Quit: Leaving]17:40
radiofreestraycat: yeah that was really annoying!17:41
radiofreeWhy is there a name field anyway?17:41
* straycat wanted to get rid of the name field17:41
straycatit serves no purpose anymore17:41
richard_mawit's either that, or make morph load every definition file it can find, and have a global namespace of morphologies by "name" field17:41
richard_mawI'm leaning towards preferring that these days, over file names, since it means we don't need to fix up file paths everywhere when we move things17:42
radiofreewhy can't it just be the filename?17:42
radiofreebuild-depends: [] is another annoyance for me as well17:43
richard_mawradiofree: because we also allow multiple artifacts to be produced by morphologies defined in one file17:43
richard_mawit's not a 1 to 1 relationship17:43
richard_mawwhich we'd require for filenames to be sufficient17:43
pedroalvarezpersia: chunk, stratum, and system  artifacts get cached, but this happens without needed to upload them to anywhere. They are being created in at the same  that the CI is running.17:44
radiofreewhy can't you just fill in "name" from the filename then?17:44
richard_mawradiofree: we can make `build-depends: []` optional17:44
richard_mawradiofree: because it requires magic to go from the filename to the name. We've done it in the past, but the result was that it required a lot of work to change the code so we could put files in subdirectories, rather than every definition file in the top-level directory of the repository17:45
persiapedroalvarez: Ah, right.  It was the "upload" that confused me.17:45
* paulsherwood sighs17:46
* richard_maw gets on with some work17:46
* paulsherwood too :-)17:46
radiofreerichard_maw: i can't quite understand why "name: foo" is fine and parsing foo.morph into "foo" isn't?17:46
radiofreethe magic is just remove the .morph bit?17:46
Zara_hm, Krin, did you find you got different error messages from those listed in the rubygems import tutorial? (I'm trying to work if I've unwittingly left some noise around from last time, or if the tool has changed since I last used it.)17:47
Zara_*work out17:47
paulsherwoodradiofree: do you mean filename.ext, or relpath/filename.ext?17:48
richard_mawradiofree: because it's actually some/sub/directories/foo.morph, so it would have to strip the directory off it and it requires we end our definition files with .morph17:48
radiofreewell if it's strata/blah/blob/foo/bar.morph the "name" is still just "bar"17:48
radiofreerichard_maw: that's pretty easy though?17:48
paulsherwoodradiofree: yes. so name: bar is the best thing?17:48
Krinyes Zara_ i got some strange error messages, the tutorial should specify that the errors may not be the same as the one pressented if i recall?17:48
richard_mawradiofree: we have at various times considered alternative suffixes, such as .{chunk,stratum,system}, or .def17:48
Krinor i may be losing my mind a little more17:48
Zara_Krin: when I first attempted the tutorial, I got the same error messages, so that suggests the tool has changed17:49
Zara_or we're both losing our minds :P17:49
radiofreepaulsherwood: yes, but that's easy enough to get from the full path17:49
KrinYAY! lost minds :)17:49
paulsherwoodyes. but why require user to fill in full path?17:49
richard_mawradiofree: 17:45 < richard_maw> radiofree: because it's actually… and it requires we end our definition files with .morph17:49
radiofreepaulsherwood: i'm not17:49
* straycat was but doesn't mind either way17:50
radiofreei'm saying there's no difference between having "name: bar" in the morph file, or just parsing name from strata/blah/blob/foo/bar.morph (bar)17:50
radiofreeso you didn't have to put name in it17:50
*** Krin [] has quit [Remote host closed the connection]17:50
straycatThere is a difference, though, where you have two files names bar.morph in the workspace17:50
richard_mawstraycat: indeed, there is ambiguity as to which you meant17:51
radiofreeyou can call things something other than morph?17:51
straycatradiofree, strata/foo/bar.morph strata/baz/bar.morph17:52
radiofreestraycat: i thought that issue has gone away now though?17:52
radiofreestrata/bsp-foo/u-boot.morph strata/bsp-bar/u-boot.morph isn't an issue anymore?17:52
radiofreeit's u-boot@foo and u-boot@bar17:52
richard_mawradiofree: alas, no, we've ended up with something half-way between purely (path, artifact name) based, and globally named, due to being pulled in different directions. The workspace tooling requires unique names(yes, we want to get rid of workspace tooling, I'm aware of this), but the build logic requires (path, artifact name).17:52
radiofreei see17:54
*** bashrc [] has quit [Quit: Lost terminal]17:59
*** locallycompact [] has quit [Ping timeout: 264 seconds]17:59
pedroalvarezANNOUNCEMENT: will go down soon (few seconds/minutes) 17:59
*** jonathanmaw [] has quit [Quit: Leaving]18:02
straycatSince we don't have this figured out yet, adding a warning to morph is probably for the best.18:02
tiagogomeshmm, why the ref for gcc is different from the ref for stage1-gcc and stage2-gcc18:04
richard_mawAIUI it's because stage1 and stage2 didn't need changing to be able to build the new version of gcc18:06
richard_mawso changing them would just be extra work for everyone involved18:06
pedroalvarezANNOUNCEMENT: is up and running again18:07
persiaExcept that there is something nice about using the same gcc for everything, to reduce user confusion.18:07
persiapedroalvarez: That was lots less than the forecast 30 minutes.  Nice work!18:07
pedroalvarezpersia: and I've done lot less that I was expecting to do :( the migration is not done yet18:08
persiaOh :(18:08
tiagogomesrichard_maw, but doesn't that mean that the newer version of gcc was built by an old version of gcc? What guarantees that the newer version of gcc can build the newer version of gcc?18:14
tiagogomesIt couldn't happen that deploying image X, and using image X to build again image X could fail18:16
ssam2it could happen18:17
ssam2there's no way to prove that it can't happen, other than testing18:17
ssam2gcc 4.7 could have a bug whereby it generated a gcc 4.7 that generated bad code, for example18:17
ssam2so you're right that this is a possible problem, but I don't think that using gcc 4.6 to build gcc 4.7 has anything to do with it18:18
persiaIt is probably best practice to update each stage once the later one is working.18:18
tiagogomesbut we used gcc 4.7 in stage2 to build gcc 4.7 we could detect this problem before deploying the image18:20
persiaThat only detects if gcc can build gcc, not if it can build the entire system18:20
*** ssam2 [] has quit [Remote host closed the connection]18:21
tiagogomeswhether it can build the entire system will be verified when the remaining chunks are built18:22
persiaAh, right.18:27
*** Guest56137 [] has quit [Quit: Leaving]18:54
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock19:17
*** zoli_ [~zoli_@linaro/zoli] has quit [Read error: Connection reset by peer]20:01
*** zoli_ [] has joined #baserock20:05
*** zoli_ [] has quit [Changing host]20:05
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock20:05
*** petefoth_ [] has joined #baserock21:19
*** petefoth [] has quit [Ping timeout: 264 seconds]21:20
petefoth_ is now known as petefoth21:20

Generated by 2.14.0 by Marius Gedminas - find it at!