IRC logs for #baserock for Friday, 2014-08-15

*** franred [] has joined #baserock06:47
*** franred [] has quit [Ping timeout: 272 seconds]07:11
*** franred [] has joined #baserock07:14
*** franred [] has quit [Client Quit]07:14
*** fay [] has joined #baserock07:24
fay is now known as Guest3504907:24
*** Guest35049 [] has quit [Client Quit]07:27
*** fay_ [] has joined #baserock07:27
paulsherwoodradiofree: i think others should be able to do that?07:38
pedroalvarezBulding attr  now fails:
pedroalvarezthis is not good :*07:56
*** tiagogomes [~tiagogome@] has joined #baserock08:01
*** tiagogomes [~tiagogome@] has quit [Client Quit]08:01
*** tiagogomes [~tiagogome@] has joined #baserock08:02
paulsherwoodinteresting... on x86?08:02
paulsherwoodpedroalvarez: ^^08:03
paulsherwooddefinitions master, or something else?08:03
pedroalvarezx86, arv7, and ppc6408:08
pedroalvarezdefinitions master08:08
pedroalvarezthis is the change we have to look at:
pedroalvarezWell, I'm building from master with a couple of patches on the top, but I don't think they are the problem.08:10
paulsherwoodmaster same result08:12
paulsherwoodpedroalvarez: ok i confirm reverting that patch fixes it08:16
paulsherwoodmaybe SotK has to wear the rubber chicken and buy donuts? :)08:20
* SotK needs to stop breaking things08:22
paulsherwoodSotK: it's a talent, to be nurtured. i've used my uncanny knack for breaking things to terrify engineers on several continents :)08:23
pedroalvarezpaulsherwood: reverting the patch break more things using master of 'morph'08:23
paulsherwoodnot for me?08:24
pedroalvarezpaulsherwood: using the latest morph?08:24
paulsherwoodunless something changed in morph yesterday?08:24
*** franred [] has joined #baserock08:24
pedroalvarezpaulsherwood: that's it08:24
SotKI updated morph yesterday and forgot about all these broken repos :/08:25
paulsherwoodi'm at b2786c73c0f4d4b76f824e309598ce57065e30ab08:25
*** jonathanmaw [] has joined #baserock08:26
pedroalvarezSotK: can you take a look at what happens with attr now? the log I provided is the failure using master of defnintions and morph08:27
pedroalvarezI can't see any difference betwen the JSON and the YAML08:28
SotKneither can I, it is confusing to me08:31
richard_mawit's probably the \ escapes08:31
richard_mawpreviously JSON would be consuming them, but YAML doesn't need escaping there, so it isn't and they've getting through to the build08:32
* paulsherwood proposes we revert that patch08:32
franredpedroalvarez, it is '\t' in JSON which in YAML fails when load08:33
richard_mawyep, and re-do it starting from json.loading the file, then yaml.dumping it08:33
richard_mawfranred: are you sure? In this case it gets as far as building it, so the yaml loads ok08:33
pedroalvarezhe doesn't have any context08:34
pedroalvarezfranred: I'm talking about this:
pedroalvarezrichard_maw, paulsherwood: should we do that, or wait until we move chunks into definitions08:35
franredpedroalvarez, oh, I've seen - no this is not the case that Im talking about08:36
radiofreepedroalvarez: if master isn't building then surely it should be resolved asap?08:37
paulsherwoodi'm for reverting it now. then separately bleeding edge morph won't build definitions master, so that's technically a morph problem, but it would be resolved if/when fran's script gets run :)08:37
franredpaulsherwood, hold on with this, the new version is failing not sure why (I've rebased the new branch with master :S )08:39
paulsherwoodnew version of what?08:39
franredpaulsherwood, definitions08:40
franrednew version of the script08:40
paulsherwoodthat's a separate issue, imo08:40
pedroalvarezMakes sense to me. 08:46
* SotK will revert the patch then08:47
*** ssam2 [] has joined #baserock09:04
pedroalvarezfranred: your problem could be related to the patch i did to "fix" attr. The new yaml morphology could have problems.09:07
pedroalvarezfranred: can you try your script with master of definitions once SotK has reverted that patch?09:07
pedroalvarezAnd it has already been reverted. Good.09:10
ssam2wow, I'd missed the commit "Avoid creating and pushing temporary build branches when they aren't necessary"09:10
ssam2what an excellent day August 8th was for Morph !09:10
richard_mawFebruary 2013 was better for Morph, a lot of reproducible build stuff landed in that month09:11
franredpedroalvarez, I will try. but minimal build fine before using - and then it fails. So firstly it is changing something that makes minimal to build again (which is undesirable) and then building mini-utils-devel fails :S09:12
pedroalvarezfranred: then I don't think it''s related09:13
*** fay_ [] has quit [Ping timeout: 245 seconds]09:25
*** fay_ [] has joined #baserock09:27
persiarichard_maw: Why doesn't update-ref work for detached HEAD?  It's not gc-safe, but it's presumably only local anyway (and should be tagged on push).09:50
richard_mawpersia: I thought the idea was to update-ref and push, but I guess if you're suggesting that local builds work by changing the repo: field to a file:// URI then you might get away with just putting "HEAD" in the ref field10:09
persiaDepends on whether one is distbuilding: if distbuilding, one needs to push a temporary branch at that ref (making it no longer detached), but that's not different than the general file:/// distbuild case.10:10
persiaThe key thing for me was 1) to have a ref for the tree that worked (rather than having to wonder which random files in the working set made things work), and 2) to always build what was in definitions (to avoid having to wonder what to do to definitions to make it continue to work)10:12
richard_mawwell, the needing to match the ref thing is removed by my patch, replaced by needing to set a config option to say that this repository contains the source you should use for this chunk10:17
richard_mawbut as I understand what you want10:17
richard_mawinstead of making temporary build branches for uncommitted changes, we should refuse to build10:18
persiaI'm rather uncomfortable with your patch doing that.10:18
richard_mawand offer tooling to update the ref to what we committed10:18
richard_mawand only do temporary build branches when distbuilding10:18
persiaI agree about temporary branches and distbuilding, but I'm nervous when you write "what we committed": I don't think morph should commit anything.10:19
richard_mawI meant we as in the user10:19
persiaAha.  Yes.10:20
persiaAnd for the config option you mention: am I also misunderstanding that this could cause the build artifact to use source differing from that specified in the ref in definitions?10:21
richard_mawyes, as we need to do that when we make a temporary build branch10:22
richard_mawI'm not 100% through the thinking of what should happen to the ref field in the morphology10:23
persiaThanks for confirming my misunderstanding, but I don't see how it is related to temporary build branches.10:23
* paulsherwood can't get his head round this... will try richard_maw's patches and see what they do :)10:25
richard_mawWhen morph makes a temporary build branch, it makes branches for all your chunk repositories, and a branch for the definitions repository. To be able to use the temporary build branches of the chunk repositories it has to alter the morphologies to set ref to the temporary build branch for that repository10:25
richard_mawso what ref looks like when you open the file is different to what was used when building using a temporary build branch10:26
ssam2it's a huge hole in our provenance tracking of artifacts. It has been raised on the mailing list before10:28
persiaThat's stunningly overly complex in ways that can only cause user confusion and pain.10:28
ssam2that's why I'm very happy that we at least only make temporary build branches when there are actually local changes, now10:28
persiaWe should probably just remove all of that entirely, and instead, if we have local refs, ensure those are available to distbuilders (at that ref) in some entirely different way.10:29
ssam2I'm not sure that's a constructive suggestion unless "some entirely different way" is fleshed out a bit more10:29
ssam2seems to me that there are many ways to reintroduce exactly the same problem10:30
persiaIndeed, it's not constructive at all.10:30
paulsherwoodrichard_maw: did you just describe what morph master does now, or what your latest patches would make it do?10:30
richard_mawboth, the latest patch just allows you to have a different branch checked out and it still work10:30
* persia needs to think a lot more to come up with a constructive suggestion here, because it's a very hard problem10:30
ssam2the problem is discussed here:
persiaIndeed.  Yet another reason to avoid morph edit, because then morph can't do that :)10:33
paulsherwoodrichard_maw: is there a branch somewhere i can get yout 'Always use current branch...' patch?10:34
richard_mawpaulsherwood: not currently, I'll make one10:34
richard_mawhm, actually, I can't ssh to right now, so I can't push the branch10:37
paulsherwoodno problem, i'll try git magic10:38
* paulsherwood fails10:43
persiagit am?10:43
persiaSometimes when dealing with annoying circumstances, I get patches locally, and then run `git apply` on each one.10:45
persia(this only works if one has less than 5 patches to process: more than that quickly gets very annoying)10:45
persiaSHAs don't match, and you have to rewrite commit messages if you don't squash them, but it can allow testing when other methods fail.10:46
ssam2i find GNU Patch sometimes works where 'git am' has failed10:49
radiofreefranred: did you merge [PATCH 0/3] Organize morphologies in definitions repository yet?10:51
radiofreei have changed bsp-jetson-devel10:51
radiofreeah it's been rebased :D10:52
franredradiofree, no, I didn't. Im doing the final test to submit the patch - but I have to rebuild minimal11:02
radiofreeok, i noticed in v2 it's been rebased to (mostly) current state11:02
franredyep :)11:03
paulsherwood"2014-08-15 10:55:58 Deciding on task order12:08
paulsherwoodERROR: Field comments not allowed in morphology string"12:08
paulsherwoodin which morph, i wonder12:09
SotKattr probably, I think that is the only one with a comments field12:09
pedroalvarezpaulsherwood: using latest version of morph, right?12:09
* SotK fixes the reverted-back-to-JSON version to not have a comments field12:10
*** flatmush1 [] has quit [Ping timeout: 272 seconds]12:11
SotKhmm, should I also try converting it to YAML using json.load, yaml.dump as well?12:11
paulsherwoodpedroalvarez: yes, with richard_maw's patches12:13
persiaThat's going to be safer in terms of confirming the parser, but shouldn't make a difference.12:13
SotKthat should sort out all the \ escapes that richard_maw suggested were causing the earlier problem too12:14
paulsherwoodso morph checkout b:b/d master; morph build base_system => ERROR: Field comments not allowed in morphology string12:16
paulsherwood(with morph master)12:18
paulsherwoodcan we just delete the comment?13:08
persiaI thought that was what SotK was doing, except in a manner that was more history-consistent13:10
persia(and also fixing the escaping issues)13:10
*** radiofree_ [radiofree@unaffiliated/radiofree] has joined #baserock13:13
franredsecond version of the script works with the latest version of morph and 3 patches on top13:18
franredwell at least build minimal without having to rebuild anything13:18
franredrichard_maw, pedroalvarez ^^13:19
pedroalvarezgood news!13:19
* franred is preparing the patches and sending to the mailing list13:20
*** flatmush [] has joined #baserock13:20
radiofree_ is now known as radiofree13:20
persiafranred: Be sure to mention *which* morph patches you need if you depend on some in your post (or retest with the morph in definitions).13:22
radiofreemy quassel has gone bonkers13:23
*** radiofree [radiofree@unaffiliated/radiofree] has quit [Quit: Bye]13:23
franredpersia, I will13:23
persiaIt's an IRC client not usually known for going bonkers.13:29
* paulsherwood confesses he just stole a jetson from a distbuild network13:29
KinnisonOh dear13:29
ssam2um, there are lots of them behind me13:32
ssam2which aren't in use in working build networks ...13:32
paulsherwoodssam2: yup, i know :)13:33
paulsherwoodi'm going to steal one of those too13:34
radiofreewasn't someone going to use the jetson distbuild network13:34
paulsherwooda build network should be able to cope with the disappearance of one minion13:34
radiofreedoes it? I used to see total failure when the controller couldn't connect to one worker at startup13:35
paulsherwoodouch, then13:35
radiofreehence why i put in my documentation "start the workers before the controller"13:35
paulsherwoodaha. this is a different use-case - a worker has disappeared from a running network. i'd hope it can cope with that?13:36
ssam2I don't think hotplug was part of the initial design13:37
KinnisonThis is not a context we designed for13:37
ssam2it was designed for ARM servers, I believe13:37
ssam2where you'd have to power it off to remove a node anyway13:37
rjekSome brands of ARM server are not known for their reliability :)13:38
pedroalvarez:) the controller is sending build requests to the no-longer-available worker13:38
persiaWe should probably rethink this for elastic resources at some point.13:38
KinnisonThe current design is not good for elastic resource, but we have ideas on how to fix that.13:39
persiaA publish and subscribe model, or something else?13:40
Kinnisonmore like a trad req/rep model I think13:41
KinnisonSince the workers ought to be interchangeable fleets13:41
ssam2as a way of solving this today, Pedro has cooked up a couple of Ansible scripts which allow you to administer a whole distbuild network easily13:41
ssam2allowing you to say "remove this worker from the config and restart everything" easily13:41
persiaThat's more of a workaround than a solution, but it's nice to have it.13:42
rjekThe order of parameters to morph branch seems slightly odd.13:42
persiaKinnison: So workers would spin up, find a controller (by preconfig or some other mechanism), and then ask to be told what to do?13:43
rjek"morph branch baserock:baserock/definitions robkendrick/mips64 robkendrick/mips32" did not do as I expected.13:43
rjekI am probably expecting wrong.13:43
persiarjek: Luckily it's a command you only ever need to run once.13:43
pedroalvarezthe main goal of my script was to make the distbuild use the latest morph13:43
rjek(I wanted to branch the mips64 thing into a new thing called mips32, but apparently the parameters go the other way around)13:44
SotKthis confuses me every time I do it too13:44
straycatI think that's because the last arg is optional.13:44
persiaI think the idea was that the destination wasn't optional and the source was.13:44
SotKthat is why13:44
pedroalvarezwhat's the default?13:44
persiaBut this could also be addressed with argument counting, rather than fixed positional arguments.13:45
pedroalvarezobvious :013:45
rjekpersia: +113:45
rjekI am used to saying with basically every VCS and cp/mv/ln tool that you specify what to copy first, and where you want the new thing second.13:45
rjekIt is probably too late to change this for morph, though13:45
persiaNo, morph is flexible.13:46
rjekChanging it will confuse everybody who it doesn't confuse in order to not confuse those it does :)13:46
Kinnisonpersia: indeed, that's my ideal approach13:46
persiaBut before I'd be interested in fixing it, someone would have to convince me that it's more useful to do this than `git checkout -b robkendrick/mips32 robkendrick/mips64`13:46
richard_mawrjek: I doubt this will surprise you or change your mind, but it's the way git does it as well13:46
paulsherwoodso the distbuild network i've sabotaged is at least still alive13:47
persiaKinnison: I suppose the main difference between that and publish/subscribe is just that there is still a SPOF on the controller?13:47
radiofree" ERROR: Field comments not allowed in morphology string " that's a new one13:48
Kinnisonpersia: Aye, and I'm not sure what to do about that beyond having every initiator be a controller and having the SPOF be the pubsub bus13:48
paulsherwoodradiofree: blame SotK i think13:48
KinnisonIt probably means a morphology somewhere has 'comments:' instead of 'description:'13:48
persiaKinnison: And then you do pubsub over MQ, and that goes away too, but this ends up requiring more infrastructure than may be useful.13:49
straycatrjek, It's a similar thing, the last arg (thing to branch from) in git branch is optional.13:49
paulsherwoodi've proposed a fix, here already. but i believe others are working on a real solution13:49
Kinnisonpersia: aye.  There's nanomsg which is by the øMQ guy, but it's not enough to get us going I don't think13:49
paulsherwoodSotK: ^^ ?13:49
persiaZeroMQ is probably a closer fit, although I'm partial to rabbitMQ13:50
Kinnisonnanomsg is an MQish thing but much lighter weight13:50
Kinnisonwith lots of language bindings13:50
pedroalvarezpaulsherwood: as I said before, is alive but the controller is sending build requests to a dead worker13:50
KinnisonBut I fear it would still have SPOFs13:50
rjekOK, now I'm very confused.13:51
paulsherwoodpedroalvarez: that's amazingly frail13:51
Kinnisonpaulsherwood: It's a known frailty13:51
SotKpaulsherwood: this is the issue caused by reverting the commit that broke attr13:51
Kinnisonand known frailties can be guarded against until they can be fixed13:51
rjekSo, I did "morph branch baserock:baserock/definitions robkendrick/mips64"; this gave me robkendrick/mips64/baserock/baserock/definiations/ that claims to be on the robkendrick/mips64 branch according to git branch -v, but none of my changes are there and git log show changes from today.13:52
SotKpaulsherwood: we knew that this would happen with master of morph13:52
SotKit was my understanding we weren't going to update these broken chunks since they should be fixed by chunks-in-definitions soon enough13:52
Kinnisonrjek: You created a new "local-only" (since you've not pushed) branch called the same as something which already existed13:52
persiarjek: I believe it's just created you a *new* local robkendrick/mips64 branch from master.13:53
Kinnisonrjek: If you wanted to get your extant mips64 branch you probably wanted to do "morph checkout baserock:baserock/definitions baserock/robkendrick/mips64"13:53
* persia would like morph branch and morph checkout to go away, as they are confusing13:53
pedroalvarezwrt: chunks in definitons: I was hoping that to happen today, there is a related patch in the ML13:53
* rjek finds the simplist of things confusing.13:54
rjek/src/workspace # morph branch baserock:baserock/definitions baserock/robkendrick/mips6413:55
rjek2014-08-15 13:53:45 Updating git repository baserock:baserock/definitions in cache13:55
persiarjek: Unfortunately, the precise behaviours of `morph branch` and `morph checkout` neither match the semantics from git nor have documentation outside the morph source code, so you aren't able to use your super powers today.13:55
* rjek taps foot13:55
rjekpersia: Sadface.  Still, the situation appears to be getting better.13:56
rjekDocumenting how things should be done.13:56
persiaI don't things should be done that way.  I think the commands are confusing, and the results of having run them more confusing.13:57
* persia is still working on constructive suggestions to replace this13:57
SotKdoes `morph branch --help` or `morph checkout --help` not help?13:57
persiaNot really.  It doesn't explain the difference in a way I understood, at least.13:58
paulsherwoodmeanwhile the sabotaged distbuild network appears to be happily distbuilding, without being affected by the absence of (dead) worker Pris13:59
rjekAnybody have any ideas?14:00
rjek/src/workspace # morph branch baserock:baserock/definitions baserock/robkendrick/mips6414:00
rjek2014-08-15 13:53:45 Updating git repository baserock:baserock/definitions in cache14:00
rjekERROR: Failed to update cached version of repo git://
rjekCommand exited with non-zero status 114:00
rjeksorry for the paste, I should have pastebinned that.14:00
persiaThat's usually a connectivity issue.14:00
ssam2sounds like flaky network to me14:00
paulsherwoodit's been one of those weeks, rjek :)14:01
rjekIt has.14:05
pedroalvarezfranred: re:, try to ron `./check --full` in morph.git14:18
pedroalvarezalso run14:18
*** jonathanmaw [] has quit [Ping timeout: 272 seconds]14:22
* rjek struggles to get this to do the right thing14:38
*** jonathanmaw [] has joined #baserock14:40
rjek/src/workspace/baserock/robkendrick/mips32/baserock/baserock/definitions # morph edit morph14:50
rjek2014-08-15 14:49:39 Loading in all morphologies14:50
rjekERROR: Unknown architecture mips32el in morphology cross-bootstrap-system-mips32el-generic.morph14:50
rjekI suppose I should have made that change /after/ issueing "morph edit morph"14:51
*** tiagogomes [~tiagogome@] has quit [Quit: Leaving]14:54
KinnisonYou'll need to run a local morph too with your ips32el change in it15:02
KinnisonDo you recall how to do that?15:03
rjekI'm sure I will remember, after I've made a load of mistakes :)15:03
*** fay_ [] has quit [Ping timeout: 272 seconds]15:09
*** ssam2 [] has quit [Ping timeout: 255 seconds]15:45
*** fay [] has joined #baserock15:50
fay is now known as Guest298515:50
Kinnisonrjek: If you're having trouble with the uname stuff on mips, be aware we do have provision for looking at the elf data of the python interpreter to get further information15:58
Kinnisonwe have to do that for armv7l vs. armv7lhf15:58
Kinnisonbecause the uname is the same in both cases15:59
*** Guest2985 [] has quit [Quit: Leaving]15:59
*** ssam2 [] has joined #baserock16:24
rjekKinnison: Ta, I'll look at that.16:36
*** jonathanmaw [] has quit [Quit: Leaving]16:58
*** ssam2 [] has quit [Quit: Leaving]17:24
paulsherwoodis there any way to diagnose a distbuild fail?17:25
pedroalvarezpaulsherwood: what's the failure?17:26
paulsherwoodi don't know - it just says Build failed: patch-misc17:27
pedroalvarezI'm having the same result right now17:27
persiaI've heard you can get the build log from the trove17:27
persia(this should be easier)17:27
pedroalvarezyou should be able to do cat build-step-patch-misc.log17:27
paulsherwoodah, ok17:28
persiaOh, cool.  It was made easier :)17:28
pedroalvarezin my case the log says that it has been build successfuly17:29
paulsherwoodon jetson?17:29
paulsherwoodok - maybe i need to catch up to master definitions :)17:30
pedroalvarezI've been talking with Sam, and he says that the problem could be transferring the artifact to the artifact cache17:39
*** franred [] has quit [Quit: Leaving]18:01
*** flatmush [] has quit [Ping timeout: 255 seconds]18:41
*** flatmush [] has joined #baserock18:50
*** tlsa [RayMWA4eTB@gateway/shell/pepperfish/x-xprqbbyrojlbfpkb] has quit [Ping timeout: 250 seconds]23:05
*** tlsa [H2txpBJigK@gateway/shell/pepperfish/x-obnqyodmpnjyatkf] has joined #baserock23:11

Generated by 2.15.3 by Marius Gedminas - find it at!