IRC logs for #baserock for Monday, 2015-01-19

*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Read error: Connection reset by peer]02:23
*** mdunford [~marcdunfo@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Read error: Connection reset by peer]02:23
*** mauricemoss_ [~simonh@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Read error: Connection reset by peer]02:23
*** sambishop [~sambishop@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Read error: Connection reset by peer]02:23
*** fay_ [~fay@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Read error: Connection reset by peer]02:23
*** mauricemoss_ [~simonh@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock02:24
*** mdunford [~marcdunfo@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock02:24
*** sambishop [~sambishop@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock02:24
*** fay_ [~fay@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock02:24
*** flatmush [~flatmush@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock02:24
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock06:35
*** mike [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock07:51
mike is now known as Guest2410007:51
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:21
*** rdale [~quassel@171.Red-80-39-233.dynamicIP.rima-tde.net] has joined #baserock08:47
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:50
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:51
*** bashrc [~motters@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:08
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:19
*** tiagogomes_ [~tiagogome@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:23
*** bashrc [~motters@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Lost terminal]09:30
*** bashrc [~motters@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:37
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:45
Mode #baserock +v ssam2 by ChanServ09:45
franredcan we have 2 different deployment types in the same cluster morphology?09:50
KinnisonIIRC yes09:51
franredi.e., to raw image and to openstack09:51
KinnisonEach deployment is considered separately, but by default we run all deployments when we deploy09:51
franredso when you deploy, you will deploy to both of them?09:51
KinnisonI believe there's a way to select one, but I'm not certain of how09:51
straycatssam2, Did you get around to doing any profiling of morph?09:52
franredKinnison, ok, thanks09:52
*** CTtpollard [~tom@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:56
bashrcwhat's the default login for the current baserock image ?10:03
bashrcbaserock-current-build-system-x86_64.img10:03
richard_mawroot, no password, console only IIRC10:03
paulsherwooddoes anyone know what's happening wrt mason these days?10:04
bashrcok thanks10:04
richard_mawpaulsherwood: I'm behind in keeping up with changes, but AIUI SotK was working on rewriting it to use OpenStack's tools and perryl was doing the "firehose" bit10:05
paulsherwoodyes, i saw good progress on those, but what about actually getting baserock.org to use them? i'm assuming the current live masons are not that10:06
richard_mawAIUI we have a mason populating cache.baserock.org (anyone who knows more please chime in), if this is not what you mean by getting baserock.org to use them, please clarify.10:09
ssam2straycat: http://linux-mm.org/Drop_Caches is useful for speed testing10:10
perrylIIRC i think work needs to be done on integrating gerrit into baserock before firehose can implemented fully, i may be wrong on this10:10
straycatssam2, thanks10:11
franredperryl, there is a gerrit into baserock, but not fully configured if it is what you are looking for10:11
perrylfranred: indeed, i think it needs to be configured first10:14
ssam2paulsherwood: getting Gerrit properly set up is a precursor to having new Mason be useful, I think10:17
ssam2that's a non-trivial chunk of work, and noone is actively working on it as far as I know10:17
paulsherwoodbefore we try to rope someone into doing that, is there any possiblity that patchwork would be a better/faster/easier solution?10:20
paulsherwood(than gerrit)10:20
ssam2i'd need to investigate what patchwork is to answer that10:21
ssam2we'd have to integrate zuul with patchwork, but that might be easy10:21
ssam2to be honest, the hard part is getting gerrit/patchwork to work with Trove10:22
ssam2i have an idea for how to do that, but things always turn out more complex than I expect them to be10:22
ssam2patchwork is not easy to find with a search engine! I'm about to get caught up buying a duvet10:23
ssam2ah, http://jk.ozlabs.org/projects/patchwork/10:24
ssam2looks cool, we should try it out I think10:25
ssam2should indeed be simpler if it works well enough10:25
paulsherwoodi believe it works well enough for the kernel community... but not openstack, android etc10:28
KinnisonIt's not used throughout the kernel community and it appears to need more gardening than I'd like10:29
paulsherwoodah, iok10:29
Kinnisondue to the flexibility it offers meaning people don't always understand how to operate it correctly10:29
paulsherwoodis gerrit better in that respect?10:29
KinnisonSince Gerrit is operated directly (rather than indirectly via a ML) it is at least more consistent in terms of its state10:30
jmacsHow do I get morph build to keep the staging area? I can't see anything in morph build --help-all10:57
franredjmacs, force it to fail10:57
jmacsBut my build does fail, and the staging area it uses has been deleted.10:58
Kinnisonlook in ...tmp/failed10:58
richard_maws/staging/failed/ in the path it shows in the build failure log10:58
jmacsOh, OK.10:58
ssam2i'm sure I fixed that10:58
ssam2are you using latest morph?10:59
jmacsmorph --version says 384dc8d10:59
ssam2hmm, maybe I never actually sent the patch :(11:00
ssam2no, it's fixed in commit 8557427c11:01
ssam2so use latest Morph (see <http://wiki.baserock.org/using-latest-morph>) and you don't have that annoying s/staging/failed/ step any more11:01
Kinnisonpedroalvarez: thanks for the +2, can you please merge that change for me>11:14
ssam2do you no longer have push access?11:26
ssam2I can merge it if so11:26
KinnisonIn theory I have access, in practice I've not exercised it for long enough I'd rather not risk anything11:31
Kinnisonssam2: Are you okay with doing the merge then?11:33
ssam2you've forgotten how to use git??11:33
ssam2sure, ok, but i don't understand why i need to...11:33
bashrcwhat's the latest baserock version?11:54
bashrc(here I have 14.40.1)11:54
jmacs15.02 I think11:54
straycatCan we stop attempting to infer the build system?11:58
KinnisonNot yet, no11:59
jmacs?11:59
KinnisonUntil and unless we have a way to specify the build system for every chunk (without needing a chunk morphology file for each), we have to infer it to reduce clutter in definitions11:59
straycatNo I'm asking, hypothetically, since we have to cache repos to infer that, which slows source pool creation down11:59
Kinnisonstraycat: If we already know the answer for a given tree SHA, we should cache that somewhere.  I'd *hope* that wouldn't be hard to do12:00
straycatOkay let's do that then12:07
ssam2morph.git branch sam/cache-build-systems implements caching of both tree SHAs and build-systems12:10
ssam2the speed up doesn't seem to be as great as I'd hoped (more testing is welcome though)12:10
straycatoh12:10
ssam2i'm starting to think the reason we see build graph calculations taking 30 + seconds is that git is slow when the repo isn't in the kernel's caches already12:11
ssam2with master of morph.git, I find list-artifacts takes 6 seconds, but if I do 'echo 2 > /proc/vm/sys/drop_caches' it takes more like 30 seconds12:12
ssam2with ybd I don't see the same thing12:12
KinnisonIf everything is cached there should be no need to go looking at the repos12:12
ssam2where do we get the morphologies ?12:12
ssam2or do we cache them too?12:12
Kinnisondefinitions12:12
ssam2sure, which is a repo12:12
Kinnisonthe only one you need to read12:12
ssam2remember this needs to work for distbuild 12:12
straycatssam2, Oh I read that series but in a different branch, even with that we're still going to remote repos to fetch stuff to infer build systems afaict12:13
ssam2Kinnison: my untested suspicion is that even just reading from one commit in definitions.git is slow if it's not in the system cache12:13
ssam2straycat: only the 1st time12:13
ssam2i've not sent that patch series for review, it's very rough12:13
ssam2although Adam extracted one part of it and sent that for review12:14
straycatOh, so there's more to it?12:14
ssam2there's more to sam/cache-build-systems yeah12:14
Kinnisonssam2: Testing on my laptop grepping definitions takes 0.008s cached or 0.170s uncached12:14
Kinnisonssam2: My view is that if we cannot build the entire graph from the content of definitions and some lookaside caches, we're doing it wrong12:15
ssam2Kinnison: indeed12:15
ssam2on my system, `git cat-file` of a file in definitions takes 0.007s cached, 3.122s uncached12:16
ssam2using 'echo 3 > /proc/sys/vm/drop_caches' to empty the cache12:17
KinnisonYou said 212:17
* Kinnison tris 312:17
ssam2i did12:17
ssam2faster with 212:17
Kinnisonokay yes, with 3 it takes 3.5s to grep definitions12:17
KinnisonBut, to be fair, if your definitions repo is not in cache, something odd is happening12:18
Kinnison(and distbuild workers shouldn't need it anyway)12:18
ssam2I know virtually nothing about how the system cache works12:18
Kinnison"magic"12:19
ssam2it seems odd indeed, but distbuild still seems to be very slow12:19
KinnisonSadly without reasonable profiling data we're not going to get very far12:19
ssam2no, when Adam is back I shall ask him to do some profiling12:19
paulsherwoodssam2: why does build graph need anything from git at all (other than definitions.git)? sorry if this is a stupid question12:24
ssam2paulsherwood: it could be rearranged so that it doesn't12:28
ssam2it needs to work out how to build stuff at some point, but it's possible that detecting what build system a chunk uses could be moved later12:28
ssam2and we should now be able to remove chunks-in-chunk-repos support, I think12:28
paulsherwoodssam2: the order is implicit in the definitions.12:28
ssam2paulsherwood: agreed, but it needs to know more than just the order to build something12:29
ssam2it needs to know what commands to run12:29
paulsherwoodat the time it builds it, yes.12:29
ssam2i think we are agreeing12:29
paulsherwoodnot when calculating order ;)12:29
ssam2moving 'determine what the morphology is' later on might be a really good way to speed up the feedback time of `morph build` and `morph distbuild`, in fact12:30
jmalkhi all, I'm aiming to deploy to openstack following 12:30
jmalk              http://wiki.baserock.org/devel-with/#index6h2 but don't know all 12:30
paulsherwoodssam2: have you looked at ybd source? it already does this part relatively well12:31
jmalkof the required parameters, specifically location, OPENSTACK_TENANT, and OPENSTACK_IMAGENAME. how do I go about finding these out?12:31
straycatssam2, isn't the morphology used to compute the cache key though?12:32
paulsherwoodstraycat: morph does it that way, but there is enough info in definitions.git i believe (modulo verifying tree)12:34
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock12:34
persiajmalk: Your horizon instance should show the values at the bottom.12:34
persiajmalk: (well, excepting password)12:34
persiajmalk: Also, if you define them in the environment with a user.rc or similar, you should be safe omitting them entirely12:34
jmalkpersia: thanks. horizon instance?12:35
persiaThe UI12:35
persiaErr, the OpenStack web UI12:35
straycatpaulsherwood, if the morph is in definitions then yes, the problem is when it's not, which is what ssam2's branch is solving (which I didn't realise since SotK didn't submit it)12:35
jmalkpersia: ok. in this case I'm ssh-ing into a machine and haven't yet seen a web UI, but I shall sniff around12:36
jmacsBah, rpm5.org have "source tarballs" that contain different things to their CVS repository12:36
ssam2paulsherwood: I've not looked at ybd's source in detail. I did run it, and with a cold cache it seemed to take about 16 seconds to tell me that there was a dependency loop involving Ruby12:36
straycatpaulsherwood, if we put a 'build-system' key into defintions then we'd have enough information there, but we don't seem to want that12:36
ssam2straycat: do we need to know the cache keys of everything before we start building?12:37
straycatyes12:37
ssam2we only need to know a cache key of something at the time we start building it, to see if it's already built12:37
ssam2and all of its deps12:37
persiajmalk: It's the same configuration you need for nova-client, glance-client, etc.  If you don't have that working, my recommendation would be to go back to your OpenStack provider and ask for a bit of guidance.12:37
jmalkpersia: ok, thanks very much12:38
ssam2straycat: so if we are rebuilding from stage1-gcc, we need to only work out the cache key of stage1-gcc before we know that we have to build it12:38
persia(as we can't know enough about how your OpenStack is configured, or what passwords you  need)12:38
ssam2not sure if this would be a useful optimisation or not without profiling, of course12:38
paulsherwoodssam2: and stage1-binutils, on which it depends i believe12:38
ssam2right, yeah12:38
*** lachlanmackenzie [~lachlan@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock12:38
paulsherwoodssam2: so you could try running ybd on (say) buildessential 12:39
paulsherwood(from master definitions - the dep loop no longer exists in master)12:40
ssam2ok, will try later12:41
* paulsherwood is still unable to follow morph's full trickery, but is confident that ybd shows a simpler faster soln for the top level logic12:42
paulsherwood(modulo that definitions still contains some duplicate names)12:43
* paulsherwood stops trolling12:44
straycatssam2, okay :)12:44
ssam2paulsherwood: if you can get ybd doing distbuilds, we can switch :)12:46
paulsherwoodssam2: i believe that is doable12:47
ssam2sure. I guess the question is which is less work12:48
straycatpaulsherwood, the logic is certainly more complicated than it needs to be, mostly left over from having chunk morphs in source repos, I'm going to have a go at simplifying some of it, ssam2 just suggested removing the old chunk in source stuff, since we don't need that anymore12:48
ssam2I think all known users of Morph now have chunks-in-definitions, so we can remove a bit of complexity from Morph's code12:48
ssam2by no longer supporting chunks in chunk repos12:48
ssam2d'oh12:48
paulsherwoodwhat is chunks in chunk repos?12:50
ssam2we used to keep build instructions for a chunk in the chunk repo, alongside its source code12:51
paulsherwoodah, you mean morph file in the chunk repo? i never heard that called chunk in chunk repo before12:51
ssam2that's what I mean12:51
paulsherwoodack12:51
ssam2chunk in chunk repo is a pretty nonsensical phrase, i'll try not to use it12:52
paulsherwood:)12:52
persiassam2: If we drop that support, could we do so at the same time we add support for versioned definitions?  I'd rather a morph that said "This morph is too new to process your definitions" than one that just randomly failed because I hadn't migrated yet.12:58
persiaAlso, with versioned definitions, we have a cleaner roadmap for future feature deprecations.12:58
ssam2hmm, versioned definitions would indeed be nice12:59
ssam2do you have any opinions on how we should implement it?12:59
paulsherwoodhave a VERSION file in definitions.git? i assume we are never going back to the possiblity if defintions spread across more thna one repo13:02
persiaEven if definitions are spread over multiple repos, as long as we define which of the multiple repos contains the VERSION file, we're good.13:04
persiaI'd probably use baserock_version or definitions_version rather than VERSION, but I'm cautious about nomenclature.13:04
paulsherwooddefinitions_version rather than baserock_version. istm that definitions may be used independently of baserock13:07
ssam2sounds like a good approach. we just need a name, and those are easy!13:08
paulsherwoodssam2: as you may recall, i previously surrendered on a crucial name argument and have lived to regret it :)13:08
persiaAlso provides a good introduction to giving morph a sensible release number.13:09
persiaSo morph 1.0.0 can be the one that deals with definitions 1, and morph 2.0.0 with defintions 2, etc.13:09
persiaAnd we can increment the other digits when it seems sensible.13:09
ssam2i like it13:16
ssam2except that we are at definitions 2 already :)13:17
richard_mawit'll be a smaller version jump than systemd did when it unified its version numbering with udev13:19
persiassam2: Why are we at definitions 2 already?13:23
persiaIn the absence of any prior versioning support, I'd argue that "unversioned definitions may have a variety of formats.  We believe that morph 0.99.1 is capable of parsing all known unversioned formats, but for those seeking assurance, migration to versioned definitions is recommended."13:24
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Ex-Chat]13:25
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock13:26
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Client Quit]13:29
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock13:29
paulsherwood:)13:46
paulsherwoodi don't think it's true, though13:46
persiaSince morph 0.99.1 doesn't exist, I agree it isn't true, but presumably it could be made true.13:50
Kinnisonhttps://github.com/jonashaag/klaus might be of interest to some here.13:54
*** grahamfinney [~grahamfin@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]14:07
*** sambishop [~sambishop@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 276 seconds]14:08
paulsherwoodit looks lovely. what could we use it for?14:29
KinnisonS'just another way to visualise a git repo14:31
Kinnisone.g. for looking at what you've done, prior to pushign somewhere14:31
bashrcAny idea what this means: ERROR: Nothing to cross-compile. Only chunks built in 'bootstrap' mode can be cross-compiled.14:45
KinnisonYou're trying to cross-build something with no bootstrap chunks in14:46
Kinnisonwhich implies no build-essential, which is impressive14:46
*** sambishop [~sambishop@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock14:50
paulsherwoodbashrc: is your branch anywhere?14:51
bashrconly local14:51
paulsherwoodif you push it, others could take a look? (unless it's secret)14:52
pedroalvarezindeed, nobody can build without build-essential :)14:59
paulsherwoodactually, that sounds like it may be faulty logic. it should be possible to build definitions, irrespective of whether build-essential is present or not?15:14
ssam2the logic is actually whether there are any chunks in 'bootstrap' mode15:16
ssam2there's no way to build in an isolated staging chroot without having some kind of bootstrap15:16
paulsherwoodthinking about it now, there are a lot of assumptions underlying that logic :)15:32
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]15:33
* Kinnison wonders what assumptions you're thinking about15:34
persiaHow so?  If one has no 'bootstrap' chunks, one doesn't even have install or cp available to the build process.15:34
Kinnisonor 'sh'15:34
Kinnison:-)15:34
persiaDoes morph explicitly call sh for the build-commands blob?15:35
KinnisonI believe morph writes a shell script with the environment setup etc in it which is used during all commands15:35
persiaAh, so yes.15:36
*** zoli__ [~zoli_@0x5e91887a.adsl.cybercity.dk] has joined #baserock15:37
persiaThe set of things that a build-essential stratum must contain should probably be documented.15:37
persiaAs it is surely smaller than the reference build-essential15:37
paulsherwoodwell, one of the use-cases i'm interested in is building non-linux things. the above logic seems to assume that only things built in a linux chroot are of interest? and that the only things to be built without one are for bootstrapping?15:40
DavePageIs there a plan to move git.baserock.org back to DataCentred?15:40
persiapaulsherwood: For non-linux, one would have a non-linux build-essential that provided the necessary bits to build whatever you wanted to build.15:41
paulsherwood(and that no non-bootstrap things can be cross-compiled)15:41
persiapaulsherwood: Creating that depends on having documentation of what morph requires from build-essential.15:41
Kinnisonpaulsherwood: we're not saying only bootstrap stuff can be cross-compiled, we're saying that only bootstrap stuff can be cross-bootstrapped15:42
Kinnisoncunningly the term 'bootstrap' is in both places15:42
paulsherwoodKinnison: 14:46 < bashrc> Any idea what this means: ERROR: Nothing to cross-compile. Only chunks built in 'bootstrap' mode can be cross-compiled.15:42
paulsherwoodmessage needs fixing then :)15:42
persiaThat no non-bootstrap things can be cross-compiled is an artefact of how the native build policy was implemented: in actuality, one can cross-compile anything, if one does it explcitly, even with current morph.15:42
paulsherwoodpersia: even today we have users who believe that to be untrue. i am sure i've contributed to the confusion15:43
persiaIt is exceedingly annoying to cross-compile with morph today.15:44
Kinnisonpaulsherwood: message cleanup might indeed make sense15:44
persiaOne has to replace all the chunk morphologies with ones that do cross-compile and don't violate cross-compilation rules.15:44
persiaWhen, in practice, one typically wants to native-compile or cross-compile an entire system, so the decision to do so should be inherited.15:45
persiaThat said, there are complications about how artifact caching works, and the unanswered question of whether the results of native vs. cross are the same.15:45
paulsherwoodcoming back to another of the assumptions - it's not obvious to me that linux chroot is the only way to provide a reliable staging approach15:47
persiaBe careful.15:47
paulsherwoodheh.15:47
persiaI think you mean linux-user-chroot15:47
persiaAnd yes, that isn't the only way, it is just the one morph uses.15:48
persiaBut even using something entirely different, the base assertion that in order to build things within a staging area, one must have previously built the tools to populate a staging area so it can build things remains true.15:48
robtayloryou also want soemthing so you can reduce the potential outside influence on your build compoentns/staged compoents15:53
robtaylorlinux-user-chroot uses namespaces (i.e. its a kinda of container). you could also use virtualisation.15:54
persiaDepends.  Some folk are happy to build entirely in bootstrap mode.  This doesn't give them reproducibility.15:54
paulsherwoodit's not that simple - morph uses staging areas from the getgo, for example. it just appears to change the rules as it goes, and that seems to be in morph's logic rather than from definitions (and i confess i still don't understand what is really happening there)15:54
robtayloryeah, that's the path that leads you to openembedded style pain15:54
persiapaulsherwood: It's definitions.  Definitions either has "bootstrap" or it doesn't.  Anything in bootstrap is not built in a staging area, and anything not in bootstrap is built in a staging area.15:55
robtaylorpaulsherwood: thats a good point, it should be explicit15:55
persiaErr, bootstrap mode15:55
paulsherwoodpersia: nope. bootstrap chunks get staging areas too.. they're just 'different'15:55
ssam2morph calls the directory used in bootstrap mode a 'staging area ' in some places15:55
ssam2probably because some of the same code paths are used, and we're lazy15:55
persiarobtaylor: Sure.  I wouldn't recommend it, but we should avoid confusing that morph behaves incorrectly with whether smarter people chose to build in staging areas.15:55
ssam2i think that's only a naming issue though15:56
persiaHow much is this confusion exposed to end users?15:56
paulsherwoodafaict morph even uses the containerised commandline stuff from the getgo15:56
ssam2persia: depends how far they dig into the code :)15:56
* paulsherwood considers himself an end user15:56
persiaIn bootstrap mode, there shouldn't be a staging area: it should just use native tooling.15:56
persiaIf it is entirely in the code, it doesn't need to be fixed, but it should be explained in the code.15:57
ssam2bootstrap involves > 1 chunk15:57
paulsherwoodpersia: it uses native tooling, to build tooling that it uses (wtih native tooling) to build the final tooling, afaict15:57
ssam2e.g. we build binutils, then we need to build a GCC that uses *that* binutils15:57
persiaWhat?15:57
ssam2so we need somewhere to put that binutils, so the GCC can find it15:57
ssam2we call that directory the 'staging area', because it happens to work like one in some ways15:57
persiaBut we're still building that gcc with the local host gcc, right?15:57
ssam2it's not a chroot though15:57
ssam2persia: yeah15:58
persiaRight.  I think "staging area" is technically incorrect, but if it is hidden in the code, only a comment ought be enough.15:58
paulsherwoodheh15:58
robtayloractually no15:58
persiaHowever, if anything mixes host tooling and built tooling, morph is doing something very wrong.15:58
robtaylorits still a place you're staging software to...15:59
persiarobtaylor: I'd accept that argument, but in doing so I'd insist on alternate nomenclature for the chroot of built-blobs that are used to build non-bootstrap code.16:00
robtaylorits a three stage bootstrap, but I can't remember the details myself16:00
persiathree stage?16:00
robtayloryep16:00
persiaBut we only have binary nomenclature.16:00
persiaWhich implies something very broken.16:00
robtayloryep, iirc the first two stages are implicatlyt done by morph, rather than explicatly specified in definitions16:01
* robtaylor could be horrbly wrong and out of date mind16:01
robtaylormore details can be found in strata/build-essential.morph16:02
ssam2the first two stages are explicit in definitions16:10
ssam2nothing really ties Morph to building on Linux using GCC, other than our use of linux-user-chroot16:10
ssam2with support for a different means of 'containerisation', appropriate definitions, and appropriate toolchain source and compile instructions, we could bootstrap any compiler on any OS16:11
ssam2it's quite hard to follow what goes on in the build-essential.morph, we should add more comments16:12
ssam2and fix morph so that it doesn't remove them again16:12
persiaAh, so reference build-essential does two boostrap-mode builds (stage1, stage2), and then a clean build using those.  Nothing semantically wrong there.16:13
persiaAnd everything in definitions: no special magic.16:13
paulsherwoodpersia: i wonder how you can conclude that, without looking at morph's code? :)16:16
persiapaulsherwood: From build-essential.morph and the referenced morphologies.16:18
persiaOr at least, there is sufficient documentation therein to cause me to believe that is is possible to do the three-stage build with the declared semantics and no magic.16:18
persiaSince magic would be superflous, Occam's Razor implies there is no magic.16:18
*** grahamfinney [~grahamfin@cpc14-know11-2-0-cust234.know.cable.virginm.net] has joined #baserock16:18
persiaAlthough I may be underestimating the Clarkian potenail of morph16:18
paulsherwood:)16:22
*** nowster [~pm@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock16:35
*** grahamfinney [~grahamfin@cpc14-know11-2-0-cust234.know.cable.virginm.net] has quit [Ping timeout: 240 seconds]16:37
nowsterHi... newbie here. Trying to attempt a cross-bootstrap and wondering how to do the required changes to baserock/build-essential. (I've already updated definitions.)16:39
ssam2are you following http://wiki.baserock.org/guides/how-to-cross-bootstrap/ ?16:41
persiaMy understanding is that the only required change is to modify the "morph-arch" file in the linux repository.16:41
nowsteryes16:41
nowsterpersia: no, it will need more than that16:41
persianowster: What do you think it needs?16:42
nowsterprobably some compiler flags too16:42
ssam2i'm afraid I don't know what changes you need, but i might be able to help if you try a build and post whatever error you get16:42
nowsterssam2: not got that far.16:42
persianowster: Are you going to need different sets of flags for things that would otherwise be detected as the same architecture?16:42
nowsterHow does one change the morph-arch file? Do you "morph branch baserock:baserock/build-essential crossboot" ?16:43
nowsterpersia: I'm not yet at that stage.16:43
persiaI git clone the linux repo, change the file, commit, and then change the definitions to use a file:/// repo and a ref for the SHA1 for my commit.16:43
persiaSome folk use `morph edit`.16:44
ssam2I have a feeling we might have removed the 'morph-arch' files and inlined them in the chunk morphs16:44
ssam2I know we have done so for glibc: there's no `morph-arch` any more, the work it used to do is handled in the chunk morphs16:44
ssam2if you do find a chunk with a `morph-arch` file (you can browse the relevant repos at http://git.baserock.org/ to check), do as persia suggests16:45
nowsterdelta/linux.git ?16:45
*** grahamfinney [~grahamfin@cpc14-know11-2-0-cust234.know.cable.virginm.net] has joined #baserock16:47
persianowster: Yes.16:48
ssam2in the case of Linux, I think morph-arch-config is no longer needed because there is a different set of build instructions for each architecture now16:49
ssam2I believe we used to have one 'linux.morph' that handled all architectures, which required a bit of per-architecture faff16:49
persiaI'm not finding any of the morph-arch stuff under strata/build-essential/16:50
ssam2http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/build-essential/glibc.morph#n43 is an inlined version of 4 lines of shell that used to be in an external script called `morph-arch-config`16:52
ssam2http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/build-essential/gcc.morph seems to still call out to a `morph-arch-config` script, which I guess nowster will have to modify for his target platform16:52
nowsterok... so only definitions need to be updated?16:53
nowsterHold on! the definitions in build-essential refer to morph-arch and morph-arch-config16:56
persiahttp://git.baserock.org/cgi-bin/cgit.cgi/delta/linux.git/tree/morph-arch?h=baserock/v3.8 is the file you need to use to define your architecture.16:57
persiaWe might want to move the build-essential version of linux ahead of 3.8 at some point :)16:58
persiaAnd I suspect it would make sense to move that file somewhere else.16:58
ssam2it makes sense to use an old tag of linux for the linux-api-headers chunk16:59
ssam2you're correct that that morph-arch script needs updating for the new platform17:00
ssam2and that it's a pain having it in the chunk repo17:00
nowsterpersia: where in the definitions would I define the file:/// repo and ref?17:00
persiabuild-essential.morph : in the linux-apt-headers parts.17:02
persiaErr, "linux-api-headers".17:02
nowsterand gcc (for morph-arch-config)?17:03
persiaYou'll want stage-2-linux-api-headers and linux-api-headers.  I don't think there is a stage1-linux-api-headers.17:03
persiaIf you need morph-arch-config, you'd clone gcc, and then reference it for stage1-gcc, stage2-gcc, and gcc.17:04
pedroalvarezwarning: setting local repo (file:///) doing a cross-bootstrap won't work .17:04
persiaBut, as I said before, unless you need to support multiple ABIs for what linux believes is the same architecture, it's better to just set the defaults in the first place, rather than using morph-arch-config17:04
pedroalvarezI have that in my plate of things to look at17:04
persiapedroalvarez: Why not?17:04
paulsherwoodpersia: this morph-arch thing is one example of magic, don't you think?17:05
nowsterpersia: the arch does not currently exist in Baserock.17:05
nowstermorph-arch will fail with:17:06
nowster        echo "Error: unsupported Morph architecture: $MORPH_ARCH" >&217:06
pedroalvarezpersia: I'm being stupid, it will work, yeah17:07
persiapaulsherwood: No: it's explicit.  I'd prefer it to be in definitions, but that's it.17:07
persiapedroalvarez: Ah, good.  Won't work for distbuild, mind you, which is another target for another day :)17:07
pedroalvarezcurrently `morph cross-bootstrap` takes some arguments like repo, ref and morphology name. I would like to change that so you can use it as `morph build`: only one argument.17:08
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Quit]17:09
persiapedroalvarez: That would be *much* better.  In fact, with that, and with the greater granularity we have in strata now, could we just dispense with cross-bootstrap entirely, and let any morphology be native or cross?17:11
pedroalvarezI think nothing stops you from doing a cross-bootstrap of a random system17:12
persiaIndeed, but it would be nice to use `morph build` and `morph deploy` for systems whether native or cross.17:13
pedroalvarezI see17:16
pedroalvarezI'll give it a think :)17:16
persianowster: Sorry.  Yes, you want to make a local commit to linux changing that file to contain a conditional for your new architecture so that it doesn't fail.17:20
persiaThe goal here is to change the string morph uses to identify the architecture to a string linux understands.17:20
pedroalvarezare there opinions against moving that logic (morph-arch script) into the morphology of linux?17:21
persiaI'd actively like to see it there.17:21
persiaFor that matter, I think morph's ideas of architecture should move to definitions.17:22
persiaThat one needs to change morph to add an architecture is annoying.17:22
pedroalvarezhere there is my raspberry-pi port: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/log/?h=baserock/pedroalvarez/rpi17:22
persia(and doesn't help cause one to consider morph stable)17:22
persiapedroalvarez: Cool.  Side note: one can't safely build armv6 binaries on an armv8 host without explcit cross-compilation.17:23
persia(not true for arm7->arm6 or arm8->arm7)17:23
pedroalvarezI stopped developing in that branch a while ago, but I think is a good example to see what needs to be done to add a new architecture17:24
ssam2pedroalvarez: problem is we have 9 morphologies for linux17:25
persiaWhich is why having the morph-arch script there is annoying.17:25
ssam2oh, but we only have 1 for linux-api-headers17:25
persiaRight.17:25
ssam2which is what I think we're actually talking about17:25
persiaYes.17:25
pedroalvarezindeed, I think that morph-arch is only being used in that one17:26
ssam2so, would make sense to inline morph-arch I think17:26
persiaIf we move morph-arch-script into the definition, then we can update the linux used for linux-api-headers without fuss.17:26
persiaWhereas now, it is annoying, because it means merging the build-essential branch.17:26
ssam2agreed17:26
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:27
pedroalvarezI agree that in the gcc case, we have 3 morphologies using morph-arch-config17:30
persiaCould we put that in scripts/, rather than in the target repos?17:31
persiaI don't like not being able to use raw upstream.17:31
ssam2persia: not presently possible17:31
persiaOh well.17:32
ssam2definitions.git isn't available in the staging chroot where gcc is built17:32
persiaAh, so we'd need a chunk.  Same problem for morph knowing things.17:32
ssam2yeah17:32
persiaPerhaps we ought have a special chunk that contained some utility scripts we wanted to be available at build time.17:32
ssam2that's quite a good idea17:32
persiamorph-arch, morph-arch-config, morph-arch-detect, etc.17:32
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Ex-Chat]17:36
*** grahamfinney [~grahamfin@cpc14-know11-2-0-cust234.know.cable.virginm.net] has quit [Ping timeout: 265 seconds]17:41
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:55
*** bashrc [~motters@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Lost terminal]18:02
*** sambishop [~sambishop@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]18:03
*** Guest24100 [~mike@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]18:36
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]19:04
*** lachlanmackenzie [~lachlan@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]19:14
*** rdale_ [~quassel@164.Red-83-47-9.dynamicIP.rima-tde.net] has joined #baserock19:29
*** rdale [~quassel@171.Red-80-39-233.dynamicIP.rima-tde.net] has quit [Ping timeout: 246 seconds]19:31
*** zoli__ [~zoli_@0x5e91887a.adsl.cybercity.dk] has quit [Remote host closed the connection]19:51
*** 17WAAZDA2 [~zoli_@0x5e91887a.adsl.cybercity.dk] has joined #baserock19:51
*** rdale [~quassel@88.20.2.146] has joined #baserock20:38
*** rdale_ [~quassel@164.Red-83-47-9.dynamicIP.rima-tde.net] has quit [Read error: Connection reset by peer]20:38
*** rdale_ [~quassel@113.Red-83-47-22.dynamicIP.rima-tde.net] has joined #baserock20:45
*** rdale__ [~quassel@184.Red-83-47-23.dynamicIP.rima-tde.net] has joined #baserock20:47
*** rdale [~quassel@88.20.2.146] has quit [Ping timeout: 240 seconds]20:48
*** rdale__ [~quassel@184.Red-83-47-23.dynamicIP.rima-tde.net] has quit [Read error: Connection reset by peer]20:49
*** rdale [~quassel@248.Red-83-61-46.dynamicIP.rima-tde.net] has joined #baserock20:49
*** rdale_ [~quassel@113.Red-83-47-22.dynamicIP.rima-tde.net] has quit [Ping timeout: 244 seconds]20:50
*** rdale_ [~quassel@155.Red-83-55-119.dynamicIP.rima-tde.net] has joined #baserock20:50
*** rdale [~quassel@248.Red-83-61-46.dynamicIP.rima-tde.net] has quit [Ping timeout: 244 seconds]20:53
paulsherwoodsorry, i'm confused. if it's morph that needs these things, why are they not in morph?20:54
*** rdale_ [~quassel@155.Red-83-55-119.dynamicIP.rima-tde.net] has quit [Read error: Connection reset by peer]20:57
persiaThat's an interesting historical question.20:57
*** rdale [~quassel@154.Red-95-121-180.dynamicIP.rima-tde.net] has joined #baserock20:58
persiaBut in practice, I'd prefer to move what remains of arch-detect stuff out of morph and into something that is more easily editable.20:58
persiaFor that matter, if any other tools were to read definitions, they would need the same information.20:59
*** rdale_ [~quassel@27.Red-83-52-105.dynamicIP.rima-tde.net] has joined #baserock21:01
*** rdale [~quassel@154.Red-95-121-180.dynamicIP.rima-tde.net] has quit [Ping timeout: 264 seconds]21:03
*** rdale_ [~quassel@27.Red-83-52-105.dynamicIP.rima-tde.net] has quit [Read error: Connection reset by peer]21:03
paulsherwoodpossibly - these are scripts for understanding and possibly the host we are on?21:03
*** rdale [~quassel@101.Red-79-153-63.dynamicIP.rima-tde.net] has joined #baserock21:03
*** rdale_ [~quassel@80.30.110.182] has joined #baserock21:04
*** rdale [~quassel@101.Red-79-153-63.dynamicIP.rima-tde.net] has quit [Read error: Connection reset by peer]21:04
persiaCould you rephrase the question?21:04
paulsherwoods/possibly/possibly configuring/21:06
persiaNo.21:06
persiaThese are scripts for understanding, and possibly configuring our build target.21:07
*** rdale [~quassel@112.Red-88-13-189.dynamicIP.rima-tde.net] has joined #baserock21:07
persiaFor native builds, this happens to match the host, but this should probably be considered a coincidence.21:07
*** rdale__ [~quassel@187.Red-2-138-205.dynamicIP.rima-tde.net] has joined #baserock21:10
persiaMore importantly ,this is how things like default compilation options, processor optimisations, etc. are encoded, so ought be part of the stuff that defines the system, rather than having the tool force those decisions for every system.21:10
*** rdale_ [~quassel@80.30.110.182] has quit [Ping timeout: 252 seconds]21:10
paulsherwoodso they could be in definitions, except they are too long/complicated?21:11
persiaThey aren't that either: it's just that nobody did it that way.21:12
paulsherwoodso let's have them in definitions?21:12
persiaA long time ago, when we had morphs-in-repos, we would put convenience scripts in the repos also.21:12
*** rdale [~quassel@112.Red-88-13-189.dynamicIP.rima-tde.net] has quit [Ping timeout: 252 seconds]21:13
persiaWith morphs-in-definitions, we need to migrate the rest of the convenience scripts also into definitions.21:13
persiaBut in some cases that gets ugly, because the scripts aren't available at build time from definitions.21:13
persiaAs a result, either we need to put them somewhere else, or we need to be able to recursively identify definitions, and build them from there.21:14
persiaThe advantage to being somewhere else is that they would cause a build-essential rebuild, which we don't want for changes in definitions.21:14
persiaBut it is important to not have them in the chunk repos, unless we believe they are suitable for upstream.21:14
* paulsherwood goes to check the current scripts, to understand what the fuss is about21:14
persiahttp://git.baserock.org/cgi-bin/cgit.cgi/delta/gcc-tarball.git/tree/morph-arch-config?h=baserock/build-essential is the biggest offender.21:17
persiaAnd it is only interesting because we're defining an architecture that gcc doesn't understand.21:18
paulsherwoodi don't get why these aren't in definitions21:18
paulsherwood(as build-commands, or configure-commands etc)21:19
persiaBecause we can't execute scripts in definitions during build.21:19
paulsherwood(thus present at build time)21:19
persiaI suspect it's a DRY concern, because we'd be duplicating things when we have multiple morphlogies for the same source.21:20
paulsherwoodheh. 21:20
persiaBut as we've pared it down to just be gcc at this point, perhaps it makes sense to be explicit.21:21
paulsherwoodthe only examples of this so far are gcc and linux-api-headers21:21
persiaThere have been more.21:22
persiaBut the only remaining unresolved example is gcc.21:22
persia(linux is just pending a patch)21:22
paulsherwoodok21:22
persiaAnd I'd argue that linux is fairly critical, as the current model requires a complex merge to support changing the kernel API.21:23
paulsherwoodwell, on the road to being completely declarative, i think this should be in definitions.21:23
persiaIt's just a matter of how.21:23
paulsherwoodno doubt i'm still missing something fundamental21:24
persiaHowever, as we're talking about some special arguments that seem to only apply for two architectures (where everything else seems to work), we probably ought be able to drop it.21:24
persiaFor that matter, in the event of a cross compilation, I'm fairly certain that the current script is broken.21:25
persiaOf course, it only means that it isn't safe to cross-bootstrap *from* ARM, which isn't widely tested, but that's a coincidence.21:25
persia(and only became obvious when I was looking at the chunk definition *and* the script, convincing me that the duplication is worth the avoidance of confusion)21:26
* paulsherwood notes that definitions for glibc *already* has similar stuff in it21:28
paulsherwoodhttp://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/build-essential/glibc.morph21:29
paulsherwoodhttp://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/strata/network-security/nspr.morph21:30
persiaYes, although in support of the DRY argument, stage2-glibc.morph and glibc.morph don't do the same sorts of architecture adjustments.21:31
paulsherwoodmesa also.21:32
persiaWhereas before that was migrated into definitions, they used an external file, which kept it the same.21:32
* paulsherwood doesn't buy the DRY argument if it's going to lead to a whole new repo :)21:32
persiagcc isn't going to be that much worse than glibc, so I suppose that makes sense.21:33
persiaFor that matter, because we're cross-building in stage2, we may not be safe to use the same detection logic.21:34
persia(even for native bootstrap, we use the cross-compilation logic, performing foo->foo cross)21:34
persiaGiven this careful attention to cross-compilation and safety, after reading this, I don't think I understand why there is a separate cross-bootstrap stratum.21:35
persiaBut since I've been told it is obsolete, and can now go away, I'm happy if I never understand this :)21:36
paulsherwood+1 to that. it's inherently fragile21:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!