IRC logs for #baserock for Wednesday, 2014-07-23

*** pedroalvarez_ [] has joined #baserock12:42
Mode #baserock +cnt by verne.freenode.net12:42
radiofreewhat's the correct thing to do for this particular horror;a=blob;f=Makefile;h=962f94eba661c2ef7254ea725af76a0d9cc0fd99;hb=65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf12:42
radiofreeHOME=$PREFIX make && HOME=$PREFIX make install DESTDIR=$DESTDIR ?12:42
radiofreeon line 28 they set PREFIX=$(HOME), i just tried it like that and it seemed to be ok12:43
radiofree(DESTDIR had usr/bin/dtc in it)12:43
richard_mawyou could also do `make PREFIX="$PREFIX"`, since assignments on the command-line override assignments in the Makefile12:44
richard_mawdon't we already have a device tree compiler chunk?12:44
pedroalvarezradiofree: you can use the other device-tree-compiler repo12:44
pedroalvarezwhich already has a morphology12:44
radiofreei was searching for dtc12:45
radiofreefor u-boot i had to do
radiofreenot sure if the ARCH=arm is needed though, but certainly you have to set CROSS_COMPILE there (it will complain about not being able to find arm-linux-gcc if you don't)12:46
*** fay_ [] has quit [Read error: Connection reset by peer]12:53
*** fay__ [] has joined #baserock12:53
*** pedroalvarez [] has quit [Quit: Leaving]12:58
pedroalvarez is now known as pedroalvarez12:58
persiaThinking about releases: I notice that the list of release systems appears to be documented in *three* places: release.morph in definitions, guides/release-process.mdwn on the wiki, and download.mdwn on the wiki.13:12
persiaDoes anyone have any suggestions about how we might reduce this duplication, or cause some of it to be the result of automation?13:12
richard_mawpersia: we could change the wiki page to say that we only build systems in release.morph in definitions.git13:14
persiaThe release-process page?  That sounds sane.13:14
* persia commences editing13:14
persiaWould it be possible to have download.mdwn generated somehow during the release process, do you think?13:14
richard_mawand we could have the script that uploads artifacts generate download.mdwn, or put the files in a place that download.mdwn can generate itself13:15
* persia isn't quite sure how autogeneration should be indicated on the page to discourage manual edits though13:15
persiaI think I like download.mdwn generating itself better, but I don't know the syntax well enough to know how we might do that.13:15
richard_mawI'd like if there were some ikiwiki magic to generate download pages13:15
persiaIs there not?13:15
tlsarelease-process on the wiki already says: "Update release.morph to contain the systems that are being released and no others."13:15
persiatlsa: Yes, which *encourages* duplication.  I'm changing it to work the other way, so that the release team determines which images are to be released *from* release.morph, rather than the other way about.13:16
persia(as we already have a review process for changes to release.morph which ought handle any considerations that apply)13:16
* persia decides to edit harder, finding a number of other oddities13:17
tlsapersia: I think it depends on whether it's a genivi release or not13:18
richard_mawif we can get releases push-button enough, it doesn't matter if we make GENIVI images when we're not doing a GENIVI release13:18
persiaSo, release-process has a list of requirements for stuff needed: if folk add to release.morph in ways that increase hardware/infrastructure requirements, should we encourage wiki update on merge in the review process?13:18
persiatlsa: Ah, good point.  I think it would make more sense to split things in definitions, so have genivi-release.morph and release.morph or similar.13:19
tlsathen you have to do the do-release script twice13:19
SotKis there a reason for genivi releases to be coupled to baserock releases?13:20
persiaI certainly hope not.13:20
pedroalvarezSotK: no13:21
persiaAlthough that's mostly because I think Baserock "releases" are a temporary workaround for lack of cache population infrastructure, but don't think I can convince GENIVI that their releases are similarly useless.13:21
tlsabeyond wanting to avoid having to do 2 releases in a week, I'm not sure13:21
persiaSo, this document claims I need a powerful x86 device for emulation for genivi releases.  Can't I use a versatile layout on an FPGA, or similar?  Does something actually depend on using qemu-on-x86?13:22
pedroalvarezthe document assumes you dont have a versatile hardware13:23
richard_mawI think it's because we provided scripts and instructions for how to do it with qemu. If we had such hardware available to test releases we could do that13:23
richard_mawbut that would require a change in release process anyway13:23
persiaIf there are scripts to assist automation with qemu-on-x86, that's a good enough reason for me.13:24
persiaAnd I don't think I can actually make the change I want to make here, until we sort out the GENIVI/non-GENIVI bit.13:24
persiaBecause otherwise things will be wrong, which is worse than being hard to maintain.13:24
tlsai think it's best if the updates to the release-process document are done by the people doing a release13:26
robtaylora good read for anyone who hasn't seen it already:13:34
*** motters [] has joined #baserock13:36
persiatlsa: if updates to that page are restricted to people doing releases, by what means should people not doing releases adjust the release process?13:37
radiofreeif you merge something in into baserock/morph in some repo, should you update the ref: in definitions?13:42
persiaYou shouldn't merge something into baserock/morph13:43
persiaBecause it enables things to be garbage collected.13:43
persiaOr is messes up history.13:43
persiaCreate a new branch.13:43
persiaWhether you update definitions.git is a matter of whether you think your new branch is a more sensible default.  I think doing so is nicer, because it lets others use your work.13:43
radiofreeoh right13:44
radiofreeso after i merge my changes into tbdiff master13:44
KinnisonIf you update definitions then your commit must be in the history of unpetrify-ref13:44
Kinnisonthat's the dependency13:44
radiofreei should submit a patch to update the ref in definitions?13:44
persiaradiofree: If you think others would benefit from your work, yes.  If you don't think others would benefit from your work, I encourage you to ask someone about it, as they will surely disabuse you :)13:45
persiaKinnison: Could you explain further about this unpetrify-ref dependency?13:46
tlsapersia: I don't know, but when I did a release with Daniel, the process document was well out of sync with what we needed to do, which made the process of releasing take much longer than it should have13:46
radiofreewell updates won't work on a jetson unless you use it13:46
radiofreethat's about it13:46
persiatlsa: Ah, yes, I entirely agree that the process doc needs to be precisely what should be done.  I don't agree that the best way to achieve this is to restrict edits.13:46
tlsaand changes to the release process that aren't done as part of a release presumably aren't tested13:46
persiaradiofree: RIght, so everyone who uses a Jetson probably wants your update, so please submit to definitions.git so Jetson users can be happy.13:47
Kinnisonpersia: read "unpetrify-ref" as "anchor-ref" and it should make more sense13:48
persiatlsa: It depends on the change.  For example, there's a claim that "Baserock maintainers" have a certain level of access, but that term doesn't mean anything, and there's no correlation between folk changing code and folk with the right access, so it's clearly wrong (but trivally so, so I didn't fix it).13:48
persiaI don't beleive that fixing that would be something that wouldn't be tested.13:48
persiaSimilarly, if some set of folk (not necessarily release folk) figured out a clean way to represent the set of release images in only one place, without significantly changing the set of commands run, I'd suggest this would fall into the same category.13:49
persiaKinnison: Not really.  From prior discussion on the matter, I had the impression that any ref that was used needed to be preserved from garbage collection in the soruce repo permanently, and that unpetrify-ref was irrelevant.  I'm not sure I understand how it becomes relevant, even if I call it "anchor-ref".13:50
Kinnisonpersia: I think of unpetrify-ref as somewhere to merge the commits *such that* they are considered preserved against GC13:52
persiaDoesn't that enforce modification of source repos in all cases, rather than using upstream?13:53
KinnisonIMO, if the upstream commit isn't part of the history of an annotated tag (ideally signed) then it needs to be part of the history of a managed ref13:54
Kinnisonand the manged ref should be noted in the morphologies13:54
persiaAh, OK.  That makes sense, although in the last discussion I had on the topic (with robtaylor and richard_maw, if I recall correctly), I understood the conclusion to be that a branch ref wasn't actually a useful candidate for the managed ref.  That said, I agree it's better than nothing, and serves that purpose (but still think anyone performing merge on such an object does a disservice to the git repo in question).13:56
richard_mawpersia, robtaylor: I replied to Emmet's mail about what you were discussing yesterday. I include my arguments as to why I think your conclusion that we're insufficiently cacheable is incorrect.13:57
persiaIf we're incorrect, all is good :)13:58
richard_mawI just wish I could have kept up in real-time, so I could have saved you a day of arguing.13:58
richard_mawre: anchor refs, there's a distinction between the ref the commit comes from, and where we need to push changes to if we need a delta13:59
richard_mawas in we can have an anchor ref, but be unable to push to it13:59
persiaAbsolutely.  Overloading these isn't useful.13:59
persiaBut as long as the user entirely ignores morph edit, this doesn't actually get in the way.14:00
richard_mawexcept that if they do need to make changes, then they need to come up with their own way to determine this14:03
persiaCould you rephrase?  I can't parse that in a way that makes sense to me.14:04
richard_mawif they need to make a change to the source of a chunk in definitions.git i.e. patch the kernel, then they'd need to manually clone the kernel repo, make their own branch, apply the change, push that, change the repo/ref of the chunk in the stratum to be able to even build it14:05
persiaActually, it's just clone, adjust defintions repo&ref, but yes, I agree.14:07
richard_mawah, you use file:/// then14:08
persiaUntil I'm done testing, then I push somewhere, and put that repo.14:08
persiaPushing untested stuff goes against the grain, because I don't like force-pushing, and I like clean commit histories.14:09
richard_mawwell, after you've gotten it to build, you still need to know which branch your changes should be on. If it's been always from an upstream branch before, you need to make something up14:09
richard_mawpersia: re: pushing untested stuff: that's what `morph edit` was for, it just decides to change the ref fields and have `morph build` handle that rather than changing it to a file:// ref14:10
persiaGenerally speaking, I'd want to use the suggested upstream feture branch nomenclature for that, as I'd expect to have sent my change for review.14:10
persiaIt still makes commits that cause my git visiualisation tools to have conniptions.14:11
* persia spent a lot of time trying to clean up a git repo after morph build once, and then had it messed up again on the next invocation and gave up14:11
richard_mawpersia: the temporary build branches give your tooling problems?14:11
persiaErr, Yes.14:12
richard_mawI was not aware of this14:12
richard_mawI'm starting to understand your objections to the workflow better now.14:12
persiaThey show up as extra branches that have interwoven commit histories, and tend to have complicaed enough tree relationships that they stop having calculable anestry.14:12
persiarichard_maw: Carefully reading your mail response, I find I have the opposite feeling to that you expressed yesterday.  I'm going to need to set aside some significant time to really understand your mail to reply, rather than being able to give a response within what I typically consider appropriate response time.14:15
richard_mawhaving re-read what I replied with, I can see why, I used a lot of unclarified morph terminology14:27
persiaBut it's more comprehensive.  Not only did the main participants in the conversation miss critical bits, but some of the other folk who joined in for a bit missed other bits.14:31
persiaSo although some of the terminology needs a bit of thought, the more relevant bit is that the understanding I developed yesterday needs to be almost entirely unlearned before I can understand correctly.14:32
liw-orcI'd like to update release-process.mdwn with the following patch: -- anyone object? (I'm asking for review, since it's a pretty big change)15:16
persialiw-orc: Is it really restricted to Codethink staff, or is that just coincidence?15:17
persiaAlso, how are you resolving the GENIVI/non-GENIVI release sets in definitions.git?15:19
liw-orcpersia, it is currently restricted to CT staff; there's no inherent requirement for that, of course15:19
persiaMy question is really, is it restricted to CT staff by policy, or is it restricted to an opaque set of which all members are coincidentally CT staff today?15:20
liw-orcfor the GENIVI/non-GENIVI: if it's needed, edit release.morph in the release branch15:20
persiaManually for each release?15:20
persiaI suppose that works, although I'm not sure it's better than having described it in release-process (which is why I didn't delete that section earlier today).15:21
persiatlsa: You had some thoughts on GENIVI/non-GENIVI earlier: what do you think?15:22
persiaFor the staffing, might I suggest "Currently, only members of the release team have enough access to complete this process.  The opaque nature of the release team is to be considered a bug that needs fixing.", unless there are policy reasons to limit to CodeThink staff?15:23
liw-orcthere isn't a release team, though15:24
persiaThe release team is the set of folk who do releases.15:24
persiaIt'S an opaque team.  We should fix that.15:25
persiaBut unless there's some reason it's restricted to Codethink (e.g. requires Codethink-only authentication tokens to access some bit of infrastructure), then I think it's not ideal to describe it as "Codethink only".15:29
liw-orcI've dropped that hunk, which is irrelevant to most of the changes, and will let others deal with that wording15:32
persiaTo be fair, I wasn't happy with the prior wording either :)15:32
persiaAnd I do agree that it's a bug that we don't know who releases, and even more of a bug if there's some limitation that restricts it to staff of a specific organisation.15:33
ssam2liw-orc: the release process says "...promote on your local Trove to get it lorried quickly", but it's not obvious how to do that15:36
ssam2it'd be useful to create and link to documentation in the Trove reference manual, or for now just add a note in the release process about how to do it15:36
tlsassam2: when we did the release, Daniel used curl to poke some urls to promote things in trove15:37
ssam2other than that, +1 for the changes15:39
ssam2the "Announcing the release" should remind about updating the -current symlinks, probably, as it's quite important15:40
* persia thought ripsum wrote a nifty script to automation promotion poking15:41
tlsawe tried it, but there was no information on how to use it, and we didn't git it to work15:42
tlsaso we used curl15:42
liw-orcssam2, the symlink bit should be, IMHO, implemented in the release-upload script (isn't there yet, but shouldn't be a big amount of code)15:50
ssam2I fully agree, but so far it isn't, so the process should note that in a place where it won't be missed15:50
ssam2the current wording implies that following the 'upload-release' script you can announce the release15:51
ssam2making it likely that someone dumb like me would forget to update the symlinks, even though it does mention that earlier15:51
liw-orc"Manually create the "current" symlinks pointing at the new release." is there15:51
ssam2sure, I mean in the "announcing the release" part15:51
ssam2anyway, no big deal if you disagree15:51
liw-orcwhy should it be in two places?15:52
ssam2well, why is "announcing the release" separated from the rest of the process?15:53
ssam2if it followed on directly from the list that contains "manually create the "current" symlinks" then it would be clear15:53
liw-orcI don't think setting the symlinks is part of announcing the release15:53
ssam2nor do I15:53
ssam2but it should precede announcing the release15:54
liw-orcit is?15:54
ssam2oh, actually I've been confused by reading a diff15:55
ssam2I thought there was more content between 'release steps' and 'announcing the release'15:55
persiaThat particular diff is fairly comprehensive :)15:55
liw-orcthe GENIVI release section is there15:55
liw-orc is the whole page15:56
ssam2my tiny worry was the line "Once the `release-upload` script completes successfully: [announce the release]", when "Run `scripts/release-upload` to publish release artifacts." actually has 3 steps following it in the "Release steps" section15:56
ssam2so it should probably say "After you have completed all release steps: [announce the release]"15:57
ssam2but like I said, it's tiny and I will now stop talking about it15:57
liw-orcI'll see about fixing that15:57
*** fay__ [] has quit [Remote host closed the connection]16:02
*** jonathanmaw [] has quit [Quit: Leaving]16:07
*** motters [] has quit [Quit: Lost terminal]16:14
pedroalvarezpersia, richard_maw: hey! both of you commented on the thread to improve the OpenStack deployment, but none of you have voted, could you?16:35
persiapedroalvarez: The one about the OS_FOO and OPENSTACK_FOO variables?16:35
persiaOh, if richard_maw is correct that the environment is passed through, then I think there's no point to the patch, and in fact, that the code section being modified should have all the other OS-related stuff removed.16:36
persiaI can -1 it if you like, but think there is scope for useful discussion regarding what should be done.16:37
richard_mawOTOH, if the deployment commands prefer the command line options to the environment variables, then OPENSTACK_FOO provides a disambiguation mechanism for colliding namespaces, so it would have value16:37
persiaIndeed, although whether that value is worth possible user confusion requires some understanding of under what circumstances such a collision might occur.16:42
pedroalvarezdo you think that supporting both would be useful?16:42
persiaNot unless we understand why we are doing it.16:42
liw-orcif I see a cluster morphology that has config  variables I don't know about, I'd like to have an inclination of what they are related to (so I like prefixes like OPENSTACK_); if I see cluster morphologies that assume that non-standard environment variables are set, I'm going to have a harder time using them16:44
persiaThat's the key bit of the discussion.16:46
persiaAre OS_FOO non-standard in an openstack environment?16:47
pedroalvarezthey are16:52
liw-orcthey definitely non-standard in the Unix environment, at least16:52
pedroalvarezif you have an openstack cloud, and you go to your access details, you can download something like this to set up your environment:
persiapedroalvarez: Right.  There are any number of scripts that set that set of things, which is why I think it's safe to consider them standard for an openstack client.16:58
persiaIn large part because attempting to use any of the openstack tools manually without that is a recipe for pain.16:58
persiaHowever, there is the potential that there exist users who want to set OS_FOO to one set of values, and OPENSTACK_FOO to another set of values, so that morph operates on a non-default cloud.16:59
persiaI'm not convinced these users exist, or that they wouldn't be surprised when morph didn't notice if they changed their OS_FOO setttings to point to a different cloud, but it is because they may exist that I'm not arguing more heatedly that the patch be removed, and the other OPENSTACK_FOO stuff be excised from morph.17:00
pedroalvarezwell, I think that my openstack cloud is not going to need soon this patch, but I thought it was good to support these deployment variables 17:03
*** tiagogomes [] has quit [Ping timeout: 260 seconds]17:08
*** ssam2 [] has quit [Remote host closed the connection]17:12
pedroalvarezI wonder if makes sense to add the capability of specifying the protocol used to lorry from the upstream trove in the cluster morphology.21:33
pedroalvarezSomething like: TROVE_LORRY_PROTOCOL, defaulting to http or ssh21:34

Generated by 2.14.0 by Marius Gedminas - find it at!