IRC logs for #baserock for Tuesday, 2014-08-26

*** thecorconian [~thecorcon@] has joined #baserock01:23
thecorconiangood evening. just letting you know that jte has joined the room.01:24
*** thecorconian [~thecorcon@] has quit [Remote host closed the connection]02:57
petefothpersia: Generally I think fixing docs should go hand-in-hand with fixing / simplifying / specifying workflows.06:42
petefothin the csae of the 'relative paths' issue I could trawl through w.b.o. If paulsherwood has fixed some / most / maybe all of them already, thenther may be little benefit. I'll create a Kanban card and we can decide  at the planning meeeting whetjer ti shoyuld get actioned any time soon06:44
*** fay_ [] has joined #baserock07:00
*** franred [] has joined #baserock07:47
*** jonathanmaw [] has joined #baserock08:48
*** ssam2 [] has joined #baserock08:53
straycatIf we ran all extensions by default would we want to move all extensions to definitions to make sure the build doesn't unnecessarily depend on a particular version of morph?09:26
persiaThe build inherently depends on a particular version of morph anyway.09:27
persiaOr rather, a sufficient set of aspects of morph that we don't have a way to track them independently.09:27
persiaFor discovery, I'd rather have all the configuration extensions in one place, but don't have an opinion on what place that ought be.09:28
straycatOkay so in your opinion it doesn'tmatter.09:28
persiaFor the purpose you state, yes (although I like the idea of consolidating for other reasons).09:28
*** liw-orc [] has joined #baserock09:29
persiaNote that if we run all the configuration extensions by default, we also need to consider the impact on reproducibility: if the set of configuration extensions changed between one build and another, the system could change.09:29
straycatWelcome back liw-orc :)09:29
straycatpersia, That sounds like an argument for keeping all the extensions in definitions?09:30
persiaLike I said, I like your proposal for other reasons, just not the one you stated :)09:31
* straycat nods09:32
pedroalvarezbut, the extensions are for deploying, not for building09:33
persiapedroalvarez: Yes, but if you can't deploy the system in the same way twice, you don't have the same result when you test it.09:34
straycatOkay cool, I'll try to write a patch for this this week09:36
persiastraycat: Thanks!09:36
straycatOh, also, can we run Baserock natively yet?09:40
richard_1awstraycat: I can lend you a USB stick which you can boot from, but we don't yet have an installer09:41
* straycat apologises in advance for his dumb questions09:41
richard_1awif you want to run Baserock on a PC09:41
richard_1aw is now known as richard_maw09:41
persiaAnother option is to deploy with initramfs to RAW, and dd the RAW to a fixed disk, install that, and boot (not prettiest, but should work).09:42
straycatokay cool thanks09:43
persiarichard_maw: Out of curiosity: did you ever test upgrade against the USB stick?09:48
richard_mawpersia: no, because I don't have a fast enough USB stick that I'd have the patience to sit through it, but I'm fairly confident it would work.09:49
* ssam2 discovers CirrOS
ssam2seems to be another minimal Linux/Busybox OS intended for cloud use09:50
persiarichard_maw: heh.  Maybe I'll try it overnight sometime.  Is there some trick to initialising the USB?09:50
ssam2built using buildroot09:50
persiassam2: My understanding is that CirrOS is mostly used as a demo environment when testing cloud infrastructure, rather than being expected to carry an application payload, although I may be mistaken.09:51
ssam2persia: ah, OK. I discovered it while testing cloud infrastructure, so that seems plausible :)09:52
persiaLet me guess: you noticed the default image preregistered in glance :)09:52
ssam2how did you know my password ???09:52
petefothredsoxloseagain :(09:53
persiapedroalvarez: Except sometimes the Sox get a pennant.09:54
franredpatch to update definitions sent to the mailing list - just add a change made in cpython.morph09:54
petefothpersia: and the occasional World Series09:55
straycatfranred, that patch you just sent, is that deliberately reverting a change Kinnison made the other week?10:48
* persia read it as moving the change from the chunk repo to the definitions repo10:48
franredstraycat, it is deliverately adding richard_maw's changes to reflect them in definitions repo10:49
persiaBut if it's the other way, that underscores the need to drop the repo morphologies when changing things.10:49
straycat ?10:50
straycatFeel free to ignore me if I'm in error, I'm still not awake >.>10:51
franredstraycat, ugh... I though it was in the other way around - sorry10:51
* persia is now confused and refrains from having an opinion in this area10:51
franredpersia, it was my fault thinking that the chunk in its repo was the one which was updated - so I made the patch in this way. As straycat correctly said, this revert a previous correct patch.10:57
persiaDoes definitions still point at a ref that contains a chunk morphology?10:57
franredat some point we should rid off all definitions in repositories to avoid this confusion10:57
franredpersia, definitions is fine AFAIK10:58
persiaDoes definitions still point at a ref that contains a chunk morphology?10:58
pedroalvarezthey are pointing to the same ref that were pointing before moving them to definitions.git11:03
persiaIn that case, ebfcbd166dfe49c6632d0aeeb6d2050a4d507576 was incomplete: it should have also changed the ref to something not including the chunk morphology.11:04
persiaWe should watch for that when reviewing morphology changes.11:04
pedroalvarezbut we didn't want to trigger a full rebuild11:04
persiaThen something is deeply broken.11:04
persiaIf changing the ref causes a rebuild, but changing the chunk morphology doesn't, then we have no means to confirm that our artifacts match our definitions.11:05
persia(because it's possible to game things in ways that don't cause rebuilds when they should happen)11:05
* persia had thought the entire contents of the post-processed chunk morphology was part of the cache key, and is very disturbed to discover this can be gamed11:06
pedroalvarezpersia: changing the chunk morphology does causes a rebuild, why do you think it won't?11:06
persiaBecause you said you didn't want to trigger a full rebuild.11:06
persiaWhich implies one didn't happen, which means things that depend on the changed chunk have no means of ensuring they still work at all.11:07
pedroalvarezpersia: if we change the ref of the chunk morphology to point to a different place in which there are not chunk morphologies, it will rebuild/.11:07
persiaDoes it also rebuild if you change the text of the chunk morphology in definitions?11:08
persiaAnd do both rebuilds properly cause a rebuild of all reverse build dependencies?11:08
pedroalvarezmoving the chunks morphologies to definitions didn't trigger any rebuild because the chunk morphologies used to build a sha1 of a repo were the same11:08
persiaTo make sure, are we both talking about ebfcbd166dfe49c6632d0aeeb6d2050a4d507576?11:09
pedroalvarez:) let me read backscroll11:10
* pedroalvarez is now researching the status of the cpython repository, and the sha1 we are using11:15
pedroalvarezI was confused because the same commit also changes the sha1 of morph11:22
pedroalvarezok, now I am full of context11:23
persiaRight, but I think it should have also changed the SHA of cpython, and that it didn't was a review failure.11:23
persiaNot a major one, but we should be more careful in future.11:23
pedroalvarezI agree11:23
pedroalvarezsomeone was looking at creating a script to remove all of them at once11:24
persiaYes, paulsherwood mentioned that a couple times.  I don't think we ought wait on that though, but *also* do per-repo updates as they happen.11:24
persiaIf the script is good, it won't break in this situation.11:25
persiafranred: Based on pedroalvarez and my agreement on context, would you want to update your patch to do that, or should one of us send the fix to complete ebfcbd166dfe49c6632d0aeeb6d2050a4d507576  ?11:25
pedroalvarezthe fix shold be: change the ref of python to point to somewhere without chunk morphology11:26
richard_mawcan I get some quick eyes on ?11:27
persiaThis is probably the parent of the current ref, but needs testing, etc.11:27
richard_mawfor context see: [BUG] Morph edit can not edit linux chunk with current morph master11:27
persiarichard_maw: Could we also add a test to catch this situation in the future?11:28
persia(the patch looks sane, but without that, we may encounter the bug again)11:28
richard_mawpersia: the mail discusses how this could be avoided in future11:29
pedroalvarezpersia: probably. This makes me wonder if we care about using a sha1 which is not the HEAD of the unpetrify-ref. If we can use upstream tags as unpetrify-ref, if we should create baserock/foo branches, etc11:29
persiarichard_maw: Sorry: lack of context: I was responding only to your paste.11:29
liw-orcrichard_maw, looks OK to me, but I lack context having been away for two weeks; +2 if you're sure it works11:30
richard_mawpersia: my preferred solution is to spawn a git daemon in each scenario to act as the git server11:30
richard_mawso the tests are behaving closer to reality11:30
persiapedroalvarez: I don't understand the benefit purportedly provided by unpettrify-ref, as it's trivial to cause it to have a useless or incorrect value.  Someone would need to explain this to me before my response would be more complicated than "just drop unpetrify-ref entirely then".11:31
persiarichard_maw: That makes sense: so this is just a temporary fix, and a more complicated one comes later?11:31
richard_mawpersia: this is a permanent fix to the bug, it is not a fix for the coverage hole11:32
persiapedroalvarez: As a thought experiment, consider the case where I check out the unpetrify ref in a bare git repo somewhere, add a commit, and push.11:32
richard_mawpersia: I intend to patch the coverage hole soon11:32
persiarichard_maw: Yes, we're using different words and/or different meanings for the same words to convey the same intent.11:32
persiaI'm happy with this as the beginning of a solution, but do want the coverage hole also fixed.11:33
persia(or to be rendered irrelevant by other changes)11:33
pedroalvarezpersia: we do that every day11:34
franredpersia, my patch is wrong. we need to submit a patch which fix the error which is what pedroalvarez has suggested11:35
persiafranred: Right: do you want to do that, or should someone else?11:36
pedroalvarezfranred: can you?11:36
persiapedroalvarez: If you do that every day, then it's not uncommon that unpetrify-ref HEAD usually doesn't match ref in most definitions repos, so don't worry about it.11:36
franredpedroalvarez, persia, Im quite busy now, but I take a note an do it later11:37
pedroalvarezpersia: but I usually want that sha1 pointing to the HEAD of unpetrify-ref11:38
persiapedroalvarez: If you want that in a consistent way and always up to date, just use the branch ref as ref.11:40
persiaIf you want a particular SHA, you probably don't care if it happens to be HEAD somewhere, because you don't want to consume other folks changes unintentionally.11:40
paulsherwoodrichard_maw: there was a lot of discussion about the git replace here yesterday - did you read the backscroll?11:54
paulsherwoodi contended that the basis of your original patch may be unsafe, and proposed that it be reverted until we fully understand the implications11:55
richard_mawhm, I'll see if it's still in my scroll buffer11:57
* persia is reminded12:00
persiaDo we have any irc log bots that are known to run in Baserock?  Does anyone know how to configure them?12:00
*** ssam2 [] has quit [Ping timeout: 246 seconds]12:01
richard_mawpersia: not to my knowledge12:02
persiapity.  It would be handy to have posted logs in case the buffers are inadequate.  In any case, if yours is not old enough, I can send you a longer one.12:03
richard_mawI've got it in my history, so you don't need to send me it12:03
richard_mawre: possibly breaking previous builds, no upstreams of Baserock are using refs/replace, or have they ever while we were mirroring their repositories, since we don't delete their refs by default12:07
persiaWe've checked that?  In that case, I agree that the patch isn't unsafe.12:08
richard_mawpersia: I did a quick check before I submitted the patch, and I did a more extensive one just now12:09
persiaCool.  Thanks for that.12:10
richard_mawre: supporting upstreams that _do_ use refs/replace: it is possible, without spurious rebuilds, by changing how we check out the working trees to be something we can instrument, so we can list which replacement refs are used12:10
persiaThe plan documented in your commit message?12:11
persiaRight.  I think the main concerns were 1) the issue for which you asked for paste review earlier and 2) that some upstream might be using this already and our disabling it could cause issues.12:11
persiaWIth 1) resolved by the patch, and 2) resolved by investigation, I think it's now safe to not revert, but from a process perspective, I wonder if the change was reviewed properly, that we do fixup today.12:12
* paulsherwood is ok with this12:14
paulsherwoodthank you12:14
richard_mawIt was not reviewed sufficently, I just got a blanket +2 from Daniel, since he understood the implications immediately. I trusted the test suite to have covered everything I changed, but the file:// URI thing was a failing I didn't anticipate.12:17
richard_mawI'm sorry I left master broken12:17
paulsherwoodas persia said, only a few bleeding edge users could have noticed. my main concern was the possibility that we could be starting to deliberately ignore what upstream wants12:18
richard_mawpeople seem to use git replace as a way to splice git histories to look nicer, so it's very unlikely that anyone would use it for changing the trees associated with the commits, but I can think of a couple of uses for it12:21
paulsherwoodi still don't understand the implications properly, tbh. if upstreams *did* start using replace in future, what would happen in baserock workflows/systemss?12:23
richard_maw1.  inserting expunged secret data, so you can only access the keys if you fetch the refs/replace from some other repository that you can access12:26
richard_maw2.  a weird workflow where you only push commits per-release, but work-in-progress is pushed as a refs/replace of HEAD^{tree}12:27
richard_mawpaulsherwood: if they *did* start using replace for valid reasons, we'd have to undo the change to ignore refs/replace and include information about which refs/replace were used in checking out the commit in the cache key12:28
persiaI suspect that with the strategy for this documented in the latest commit, this would be relatively simple.12:29
richard_mawat which point it pretty much works the same as before, but we need to have `morph checkout` include the replacement refs in the local checkout, or we'd get different results at build-time to development time12:30
richard_mawpersia: simple would be to include every ref in refs/replace12:30
persiarichard_maw: And regarding leaving master broken: thank you for not pushing the change to definitions, so only folk hacking on morph noticed.12:30
paulsherwoodrichard_maw: i mean, as a *user* what will happen? will i notice? will my systems still build? will i still be building what upstream intends?12:30
persiarichard_maw: That's what I was thinking, but I lacked confidence in my understanding of git internals.12:30
richard_mawpaulsherwood: it does have implications for reproducible building12:30
richard_mawsince you have to have a way to select which replacement refs to include in your build12:31
*** thecorconian [~thecorcon@] has joined #baserock12:32
richard_mawmy preferred solution would be to explain why `git replace` is a bad idea to any upstream that decides to use them for anything but history splicing12:32
paulsherwoodcould we discuss with git upstream? maybe get replace deprecated?12:33
paulsherwoodor at least publish the concerns, get discussion going?12:34
richard_mawI'd be happy with the `git replace` command refusing to do so for anything that isn't a commit, and tweaking checkout to refuse replacements anywhere except `git log`12:36
persiaThat would probably be a useful discussion to have upstream.12:40
thecorconiangood morning. just letting you know that jte has joined the room.12:43
paulsherwoodthecorconian: long time no see :)12:43
*** ssam2 [] has joined #baserock12:45
*** ssam2_ [] has joined #baserock12:46
richard_mawpersia: I'll make a note to raise it tomorrow, along with making the test suite simulate a git server better12:56
persiarichard_maw: Sounds perfect.  Thank you.13:00
richard_mawhm,'s git-daemon has spawned too many children again.13:16
persiaDo we know what causes this?13:16
richard_mawat a guess: the git upload-pack processes not terminating correctly, I'll give it 5 minutes to see if I can work out why, otherwise I'm going to restart the daemon13:18
richard_mawhm, no logs for anything before sunday13:22
richard_mawbut we've only had failures today13:23
richard_mawhm, that's interesting; all the stuck processes are morph13:28
richard_mawwell, they're all upload-pack processes for the morph repository13:28
persiaHmm.  I wonder if we shouldn't stop doing development directly against g.b.o, and just lorry morph like everything else.13:29
persiaThat said, it's probably useful to fix the actual problem :)13:29
*** fay__ [] has joined #baserock13:37
*** franred_ [] has joined #baserock13:37
*** ssam2_ [] has quit [Ping timeout: 246 seconds]13:38
*** flatmush1 [] has joined #baserock13:39
*** ssam2 [] has quit [Ping timeout: 250 seconds]13:39
*** jonathanmaw [] has quit [Ping timeout: 250 seconds]13:39
*** flatmush [] has quit [Ping timeout: 250 seconds]13:39
*** fay_ [] has quit [Ping timeout: 240 seconds]13:39
*** ssam2 [] has joined #baserock13:39
*** franred [] has quit [Ping timeout: 255 seconds]13:39
*** jonathanmaw [] has joined #baserock13:39
*** ssam2_ [] has joined #baserock13:39
franred_I've seen that we use nss and nspr for ceph systems, could I put those chunks in a shared strata morphology like webtools? or should I create a Network-security-service strata which includes them?13:53
persiaIf we're using webtools for both gerrit and ceph, putting them in a shared stratum makes sense.  I think we should strive for as few strata as possible, but acknowledge that as the number of different sorts of systems being prepared expands we expect the number of strata to grow extensively.13:54
franred_umm, sorry I think I've created confusion, webtools are not in ceph-services.morph . nss and nspr are chunks of ceph-services.morph. Because nss are "Network-Security-Service" and nspr "Netscape Portable Runtime", I wonder what was the correct place to put them to share between strata, so I was thinking on webtools13:59
paulsherwoodit's ok for now14:01
persiaIf you need to share components, and there's no current common stratum, create a new one14:03
franred_persia, yes, I think this is going to be the case14:04
franred_ is now known as franred14:04
*** fay__ [] has quit [Ping timeout: 250 seconds]14:05
richard_mawI failed to debug what happened to's git-daemon, so I restarted it. Perhaps next time it happens will have been recent enough that there'll be logs14:12
liw-orcrichard_maw, did you restart it just now?14:13
persiaIs there a way we can capture the problem before it becomes unusable?14:13
richard_mawpersia: not sure; we can set timeouts, but that could cause spurious failures for clones that would eventually finish, rather than ones that get stuck14:15
richard_mawwe can also change the subprocess limit, so it happens quicker, so we can debug it, or turn off the limit14:15
persiaI was rather thinking of some periodic job that calculated some metric (e.g. how many are stuck), and notifed someone when it got above a threshold (preferably before operational failure)14:16
persiaEssentially, small shell script in a 5-minute cronjob or similar.14:16
*** fay__ [] has joined #baserock14:18
richard_maw`test "$(wc -l </sys/fs/cgroup/systemd/system/git-daemon.service/cgroup.procs)" -lt 16` would be the important check here I think14:22
persiaAnd once it hit 16, we ought be able to troubleshoot the problem while it is active?14:23
richard_mawand give some wiggle room to see if we can introduce more stuck processes to see what causes it14:26
richard_mawbut we could also include output from `journalctl -u git-daemon.service` to see if that enlightens us14:27
persiaDoing both every few minutes sounds like a good plan.14:29
persiaSeems it needs restarting every week or two, which can be annoying.14:29
richard_mawworse than that, I had to give it a kick last friday14:29
persiaOh my!14:30
richard_mawrecent changes that may be related: 1. we've got CI pipelines running, though the only one of those that should be related is the firehose one, which may be testing merges of morph14:33
richard_maw2. we now do a ls-remote before building to see if we have unpushed changes14:34
flatmush1 is now known as flatmush14:45
*** genii [~quassel@ubuntu/member/genii] has joined #baserock15:18
ssam2franred points out that we currently save openstack passwords in plaintext in morph.log :(16:30
ssam2we should fix that soon if we are going to be making more use of OpenStack !16:30
persiaWe should also fix the annoyance that one has to specify the password on the morph deploy command line: the normal OpenStack environment variable is not honoured.16:36
*** jonathanmaw [] has quit [Ping timeout: 245 seconds]16:36
*** franred [] has quit [Ping timeout: 245 seconds]16:36
*** flatmush [] has quit [Ping timeout: 255 seconds]16:36
*** ssam2_ [] has quit [Ping timeout: 250 seconds]16:36
*** ssam2 [] has quit [Ping timeout: 260 seconds]16:36
*** jonathanmaw [] has joined #baserock16:36
*** fay__ [] has quit [Ping timeout: 260 seconds]16:37
liw-orcit's unfortunate that OpenStack requires having passwords in cleartext anywhere16:37
*** fay__ [] has joined #baserock16:37
*** franred [] has joined #baserock16:37
persiaIt doesn't actually, but not doing it that way is annoying.16:37
*** ssam2 [] has joined #baserock16:37
*** ssam2_ [] has joined #baserock16:37
* persia would prefer it if keystone honoured SSL certs for authentication.16:38
*** flatmush [] has joined #baserock16:38
*** fay__ [] has quit [Remote host closed the connection]16:39
ssam2_if anyone spoke in the past few minutes, our rather crappy ADSL probably prevented me from hearing it16:39
ssam2_I was going to point out our current, rather lame solution for filtering out passwords in /baserock/deployment.meta:
ssam2_we could add this to the logging of environment variables, as a quick fix16:40
*** thecorconian [~thecorcon@] has quit [Quit: Leaving.]16:41
*** franred_ [] has joined #baserock16:42
*** jonathanmaw_ [] has joined #baserock16:43
*** flatmush1 [] has joined #baserock16:44
*** flatmush [] has quit [Disconnected by services]16:44
flatmush1 is now known as flatmush16:44
*** jonathanmaw [] has quit [Ping timeout: 246 seconds]16:44
*** ssam2 [] has quit [Ping timeout: 255 seconds]16:44
*** franred [] has quit [Ping timeout: 240 seconds]16:45
*** ssam2_ [] has quit [Ping timeout: 260 seconds]16:45
*** ssam2 [] has joined #baserock16:45
*** ssam2_ [] has joined #baserock16:45
persiassam2: That would help in the short term.  Longer term, we probably want to fix the root issue, so that we're not transmitting passwords insecurely.16:49
*** jonathanmaw_ [] has quit [Quit: Leaving]16:53
*** thecorconian [~thecorcon@] has joined #baserock17:21
*** flatmush [] has quit [Quit: Leaving.]17:35
*** ssam2_ [] has quit [Remote host closed the connection]17:46
*** ssam2 [] has quit [Remote host closed the connection]17:46
*** franred_ [] has quit [Quit: Leaving]17:48
*** thecorconian [~thecorcon@] has quit [Remote host closed the connection]18:08
*** thecorconian [~thecorcon@] has joined #baserock18:09
*** JPohlmann [~jannis@xfce/core-developer/JPohlmann] has quit [Ping timeout: 255 seconds]22:23
*** richard_1aw [] has joined #baserock22:24
*** JPohlmann [] has joined #baserock22:24
*** JPohlmann [] has quit [Changing host]22:24
*** JPohlmann [~jannis@xfce/core-developer/JPohlmann] has joined #baserock22:24
*** radiofree [radiofree@unaffiliated/radiofree] has quit [Ping timeout: 240 seconds]22:44
*** radiofree [radiofree@unaffiliated/radiofree] has joined #baserock22:45
*** genii [~quassel@ubuntu/member/genii] has quit [Read error: Connection reset by peer]23:03
*** thecorconian [~thecorcon@] has quit [Quit: Leaving.]23:16
*** thecorconian [~thecorcon@] has joined #baserock23:25
*** thecorconian [~thecorcon@] has quit [Quit: Leaving.]23:55

Generated by 2.14.0 by Marius Gedminas - find it at!