IRC logs for #baserock for Wednesday, 2016-01-27

* paulsherwood likes ssam2's changes, but wonders how they are done, where the code is. no sign of changes to definitions.git or infrastructure.git06:04
*** gtristan has quit IRC07:34
*** gtristan has joined #baserock07:54
*** fay_ has joined #baserock08:00
franredpaulsherwood, , I think08:31
perrylthe code lives in trove-setup08:32
*** gtristan has quit IRC08:36
*** toscalix has joined #baserock08:37
franredperryl, yes, it does so the link I posted are the changes made by ssam2.08:37
*** CTtpollard has quit IRC08:54
*** CTtpollard has joined #baserock08:59
*** bashrc_ has joined #baserock09:05
*** toscalix has quit IRC09:07
*** toscalix has joined #baserock09:07
*** franred has quit IRC09:09
*** gtristan has joined #baserock09:31
*** ssam2 has joined #baserock09:33
*** ChanServ sets mode: +v ssam209:33
*** franred has joined #baserock09:36
*** jonathanmaw has joined #baserock09:40
*** edcragg has joined #baserock09:50
*** ctbruce has joined #baserock09:54
*** toscalix has quit IRC09:58
*** toscalix has joined #baserock09:58
*** CTtpollard has quit IRC09:59
*** CTtpollard has joined #baserock10:00
*** fay_ is now known as faybrocklebank10:08
*** mwilliams_ct has joined #baserock10:22
*** Lachlan1975 has joined #baserock10:25
*** tiagogomes_ has quit IRC11:06
*** edcragg has quit IRC11:08
*** edcragg has joined #baserock11:09
*** tiagogomes_ has joined #baserock11:19
*** edcragg has quit IRC11:35
*** edcragg has joined #baserock11:37
*** toscalix has quit IRC12:02
*** bashrc_ has quit IRC12:02
*** nowster has quit IRC12:02
*** jonathanmaw has quit IRC12:02
*** Lachlan1975 has quit IRC12:02
*** CTtpollard has quit IRC12:02
*** ctbruce has quit IRC12:02
*** franred has quit IRC12:02
*** edcragg has quit IRC12:02
*** tiagogomes_ has quit IRC12:02
*** ctbruce has joined #baserock12:02
*** CTtpollard has joined #baserock12:02
*** Lachlan1975 has joined #baserock12:02
*** toscalix has joined #baserock12:02
*** tiagogomes__ has joined #baserock12:02
*** nowster has joined #baserock12:02
*** jonathanmaw has joined #baserock12:02
*** franred has joined #baserock12:02
*** bashrc_ has joined #baserock12:02
*** edcragg has joined #baserock12:02
*** gtristan has quit IRC12:12
*** ssam2 has quit IRC12:12
*** toscalix has quit IRC13:32
*** bashrc_ has quit IRC13:32
*** tiagogomes__ has quit IRC13:32
*** jonathanmaw has quit IRC13:32
*** franred has quit IRC13:32
*** edcragg has quit IRC13:32
*** CTtpollard has quit IRC13:32
*** Lachlan1975 has quit IRC13:32
*** ctbruce has quit IRC13:32
*** nowster has quit IRC13:32
*** bashrc_ has joined #baserock13:32
*** jonathanmaw has joined #baserock13:32
*** edcragg has joined #baserock13:32
*** ctbruce has joined #baserock13:32
*** CTtpollard has joined #baserock13:32
*** nowster has joined #baserock13:32
*** toscalix has joined #baserock13:32
*** Lachlan1975 has joined #baserock13:32
*** 16WAAO5A6 has joined #baserock13:32
*** franred has joined #baserock13:33
*** toscalix has quit IRC14:00
paulsherwoodfranred: aha, thanks!14:01
*** toscalix has joined #baserock14:07
*** gtristan has joined #baserock14:14
*** ssam2 has joined #baserock14:14
*** ChanServ sets mode: +v ssam214:14
*** CTtpollard has quit IRC14:19
*** CTtpollard has joined #baserock14:22
*** paulw has joined #baserock14:34
paulsherwoodin another place i've been discussing names....15:15
paulsherwoodplease don't all sigh or cry bikeshed :)15:15
paulsherwoodit's been suggested that 'layer' is a much more useful word than stratum15:15
*** paulw has quit IRC15:15
paulsherwoodwe shied away from this to avoid confusion with bitbake layers...15:15
paulsherwoodbut maybe that's wrong, given there's a clear implication of one thing on top of another15:16
rdale_cthow do they differ from baserock strata then?15:16
paulsherwoodany strong opinions?15:16
paulsherwoodwell, i'm not an expert but bitbake layers are not described in yaml iiuc, and can have more logic15:16
paulsherwoodalso 'deployment' has been suggested instead of 'cluster'15:17
paulsherwood(nb this would initially only be changes in ybd internal representation, to see if it improves legibility)15:17
ssam2deployment is clearly better than cluster to me15:21
paulsherwoodi agree :)15:21
ssam2as for 'layer'... i think it's just as inaccurate as 'stratum'15:21
ssam2'group' would be accurate... but too generic15:21
franreda stratum is technically a layer (so it won't change anything) - no strong opinion on the change. I assume that cluster was referencing that you can deploy multiple systems at the same time, so once again, Im not against it.15:22
paulsherwoodi wanted 'group' for stratum too15:27
paulsherwoodlevel, or tier, maybe :)15:27
rdale_cta 'level of components' doesn't make sense, a 'tier of components' is better, but i still prefer 'layer'15:28
paulsherwoodi still prefer 'group' but fine :)15:29
richard_maw'layer' works better to describe the mechanism, since you can only compose a system by layering layers on top of each other15:31
richard_mawif you start allowing dependencies directly on chunks, then 'group' becomes more appropriate, since then it's just a convenience for grouping chunks together15:32
paulsherwoodi was assuming we would move to dependencies on chunks at some point15:32
paulsherwood(while still supporting depedency on strata/group/whatever)15:33
richard_mawthen I'd go for group15:33
paulsherwoodrdale_ct: i think we have +3 for group here :)15:34
pedroalvarezare we moving to dependencies on chunks then ?? :D15:34
paulsherwoodpedroalvarez: afaik ybd would already support it (or that was my intention) but i've not tested it since early forays into the data structure while coding the definitions parsing in 201415:36
rdale_ct'group' has a linux meaning and using 'layer' avoids confusion with user/groups15:37
paulsherwoodssam2, richard_maw ^^ are you still preferring group ?15:38
paulsherwoodi know this is bikeshedding, but our names have been a problem for a long time, would like this to be a clear step in the right direction if possible15:39
richard_mawyes, group is generic enough that it's obvious that it requires extra context, in filesystems this is user groups, in our definitions they would be chunk/package/whatever groups15:39
ratmicethe other day was reading:
ratmicea bit surprised by how many of the naming related terms in that paper are still in use today15:40
* richard_maw throws "atom" into the mix for replacing chunk names15:41
paulsherwoodratmice: that looks like a good read for the train :)15:42
paulsherwoodpackage is already loaded, even moreso than component?15:43
* rjek thinks the meaning is clearer to newbies15:44
paulsherwoodyou're probably right15:44
richard_mawpackage does conflate input source with output artifact, hence the clumsy term "source package"15:45
richard_mawwon't be a problem for us, since source packages aren't artifacts15:45
rdale_ctchunks can be split, and calling them 'atoms' would imply that they are, er 'atomic' would 'artifact splitting' doesn't make much sense with them15:45
richard_mawbut could be a conceptual model issue15:45
paulsherwoodso far i'm favouring 'deployment', 'system', 'group' and 'component'... last call... any strong objections?15:47
* richard_maw wonders whether systems need to exist15:48
richard_mawlong term anyway15:48
richard_mawshort term I don't think we can come up with anything better15:48
paulsherwoodif they don't, then it has to be possible to apply 'system integration commands' to groups?15:49
CTtpollardis group a package group?15:49
ssam2systems should just be special kinds of groups15:49
paulsherwoodlol. group is about to be ybd's internal representation of the current 'stratum' concept.15:49
paulsherwoodssam2: +115:49
paulsherwood16WAAO5A6: more letters, and less popular in our straw poll :)15:50
* richard_maw thinks systems should not be, and that it's just convention to have a top-level group defined that isn't depended on by anything15:50
ratmicerdale_ct: chunks can be split? does not artifact splitting split 'artifacts', into different chunks?15:50
richard_mawa group is imbued with system-ness by virtue of the fact that you are selecting it as the build target on the command-line, either directly, or via a deployment15:51
paulsherwoodrichard_maw: oh, so just run all implied s-i-commands in those cases?15:52
paulsherwoodworks for me :)15:52
richard_mawwhether a group is a system is just a point of view15:52
richard_mawkind-of philosophical, but cuts out the need for an explicit concept15:53
rdale_ctyou can say 'top layer', but you can't say 'top group'15:53
paulsherwoodsupergroup :)15:53
richard_mawrdale_ct: the top-layer doesn't imply the system as a whole15:53
richard_mawpaulsherwood: a supergroup is just any group that contains/requires other groups15:53
richard_mawhence supergroups are just groups15:54
richard_mawyou could have a convention to call the 'top group' the system if you wanted15:54
ratmiceor the root..15:54
SotK"a group which is not a subgroup"15:54
richard_mawanother issue with calling them layers, is that a layer is divisible from the things it depends on15:55
richard_mawthe implication you could peel a layer off something and put it on top of something else entirely15:55
paulsherwoodrichard_maw: i was joking, while thinking of Cream and other 70s supergroups :)15:55
richard_mawmathematically the name isn't without merit15:56
paulsherwoodok, i think for ybd in this step it should be 'deployment', 'system', 'group' and 'component', with the stretch goal of trying to remove the 'system' as discussed above15:57
* richard_maw is still concerned about the terminology of how components interact with split artifacts, since that was terminology we hadn't got right before either, and it's more difficult than just replacing the name of a concept we were already sure of15:58
paulsherwoodi take your point, richard_maw - that needs more thought16:00
ssam2I had a go at making a generic data model for build+integration instructions by the way16:00
ssam2here is a clumsy attempt at visualising its current state:
paulsherwoodybd has one artifact per component. istm that splitting is an operation that gets performed on artifacts, not components?16:00
ssam2(the "Project" entity is tying in with the existing DOAP data model and the "Package" entity is tying in with SPDX, you can ignore them)16:01
ssam2thinking about it, having components and artifacts is super confusing16:01
ssam2both really generic terms that could mean exactly the same thing16:01
ssam2source-component would be clearest (but probably too long)16:02
paulsherwooderk :)16:02
ratmicei'm kind of partial to calling things more by what they contain e.g. 'source heirarchy' vs hrm, ssam2 got there first :)16:02
ssam2Morph has always called these things Source internally16:02
richard_mawpaulsherwood: I'm thinking split artifacts should be another definition type which go in a different file, refer to a chunk/source/component and select which files that should be part of it16:02
rdale_cti would call chunks 'components' which can be split into 'packages'16:03
pedroalvarezthis bike is too big and has too many wheels .. :)16:04
paulsherwoodpedroalvarez: agreed :)16:04
rdale_ctwell, terrible names are a really important baserock problem16:04
richard_mawand you can depend on a group (includes everything the group "contains"), a "chunk" which contains all files produced by the build, or a "split artifact" which contains a subset of the files produced by the build16:04
ssam2note that in the data model I tried to produce, I also tried to describe what each entity *is*, which was a useful exercise16:05
richard_mawrdale_ct: I'm liking the idea of packages for split artifacts, not sure component is right for that any more, since component could be a generic term that encompasses packages16:05
ssam2e.g. 'artifact': "A file or set of files that is interesting for some reason."16:05
richard_mawssam2: what do you call the combination of build instructions, sources and dependencies that after building, produces something that can be used?16:08
rdale_cta 'build unit'?16:08
ratmicebuild tree?16:09
ssam2richard_maw: definitions?16:10
richard_mawI like package because it implies the selection of which files should be included, but I don't think artifact fits in the conceptual model, as packages are "artifacts" by a definition16:11
ssam2to use established baserock terms16:11
ssam2if I was talking in a context where 'definitions' wasn't established, I think I'd try to spell out explicitly what I meant instead of trying to find a word16:11
richard_mawssam2: I view definitions as the textual representation from the yaml document tree, rather than the internal concept they are parsed into16:12
richard_mawso you can have group definitions, being the text you write to define a group16:13
richard_mawbut what is the equivalent of a chunk definition16:13
ssam2I would call the data structure that a build tool operates on a 'build graph'16:14
ssam2that's a pretty common term I think16:14
richard_mawwe kind of need a word that is indivisible16:14
richard_mawI don't think component necessarily works, since it's common to see subcomponents16:14
ssam2chunk? :-)16:14
richard_mawyou could use component to describe both groups or chunks16:14
richard_mawssam2: aye, for purity I think it fits, but it's not an intuitive name16:15
* paulsherwood hopes for terms that newbies can grok, without need for a comp sci background16:16
richard_mawhm, rdale's "build unit", while possibly verbose, is both correct and more self-describing than [build] chunk16:17
richard_mawthere's a bit of ambiguity of what a "build" is16:17
richard_mawsince build describes both the top-level system build, and the lower-level component build16:18
paulsherwoodrichard_maw: i think of top-level as assembly16:19
paulsherwoodssam2: how easy is it to generate that png? i wonder i'm tempted to suggest that we try for an 'official' resolution of this as one or more pull-requests to your repo16:20
ssam2it's a total pain to generate it16:20
ssam2involves taking a screenshot of a big Java app called Protégé, with lots of babysitting to get a good layout first16:21
ssam2there may be a better way16:21
richard_mawssam2: ugh, can we just use dot?16:21
ssam2Tracker has some stuff in to convert OWL to dot, I think16:21
ssam2it would definitely be good to have the data model formally specified somewhere16:22
ssam2probably not on my github though :-)16:22
persiaA note on the semantics of "package": many folk seem to think packages should support install-time integration scripts.  If this term is used, and one wants to be able to have reproducible or comprehensible images, be very careful.16:22
richard_mawpaulsherwood: if we can emphasise assembly as the operation to convert a system/group selected to be the system into a system artifact, then yes, I think we could use "build" for the low-level bit16:22
richard_mawpersia: hm, that would be staging area construction time integration scripts, which we've managed to avoid so far, by the few things that needed it putting extra stuff in pre-configure-commands16:23
persiaYes.  Also, many people believe you can "install a package" post-deploy.  I'm not saying don't use it, just be careful that it isn't misunderstood due to heavy overloading.16:24
paulsherwoodrichard_maw: currently ybd has assembly as the process of traversing the definitions to create a target, building necessary components along the way, and combining them as the required artifact.16:26
paulsherwoodi guess you mean assmebly should only be the combine step?16:26
richard_mawwhen building lego, you do the assembly part, someone else has built all the blocks and provided instructions on how to put them together16:27
paulsherwoodgood point. so traverse, build, assemble? is there a better verb than traverse?16:27
richard_mawhm, there's parse, but that's more about turning the tree of yaml documents into something you understand, whereas building the graph is a step after that16:28
richard_mawanalyse possibly16:28
richard_mawor use graph as a verb maybe16:28
* paulsherwood is finally admitting, to the assembled internet multitudes, that the ybd code is hard to understand16:28
paulsherwoodthere, i said it. don't all chortle too loudly :)16:28
richard_mawthis is in particular why I found the variables labelled "this" so difficult16:29
richard_mawwe have all these concepts floating around, and if we don't use the proper label, we need an incredible amount of context to know what's happening16:29
ratmicehrm wrt 'build unit' I was thinking 'origin' might work16:30
richard_mawratmice: hm, sounds more related to where the source code came from to me, but that may just be because the default git remote is labelled origin16:30
*** 16WAAO5A6 has quit IRC16:31
paulsherwoodi must admit i'm being tempted by 'unit' over component (it's shorter, for a start)16:32
rdale_cti'm not sure 'unit' works by itself 'build unit' is about the granularity of the build at the bottom level16:34
* richard_maw is favouring 'build'16:36
richard_mawgiven the top-level operation is assembly, if we were using unit or component, it would be build unit or build component16:37
paulsherwoodwhatever chunk moves to, it has more significance than just being a 'build unit' - from an outside perspective, it's the name of a specific project/program16:37
paulsherwoodi would hate to have to rever to every project as a 'build unit' as much as i've hated using 'stratum/strata'16:38
rdale_ctyes, you might be right16:38
richard_mawpaulsherwood: I'm not sure project is right either, since that's more about how upstream organises themselves, rather than what we build16:39
richard_mawa project may produce many things we build16:39
richard_mawand a build may produce many programs or libraries16:39
paulsherwoodtrue - i'm not suggesting project, but that's the common parlance for some aspects of this concept outside baserock16:39
richard_mawaye, and most projects, mostly small ones, are easily mappable to 1 source code repository and 1 build to produce the 1 program16:40
richard_mawbut using it as terminology gets very clunky when this isn't the case16:41
rdale_ctdebian builds 'binary packages' from a 'source package'16:42
richard_mawso we're back to a "build" for the translation from source code into artifacts/packages16:43
*** tiagogomes_ has joined #baserock16:45
richard_mawit's not a clean comparison, since "source packages" are pre-processed to bundle up the build instructions with the source code, whereas we don't have this bundling step16:45
rdale_ctwe have a reference to a git repo, instead of the actual source - but the thing we now call a 'chunk' as build instructions plus source just like a source package to me16:48
*** tiagogomes_ has quit IRC16:48
ssam2well, 'source package' and 'binary package' don't seem so bad. and should meet the criterion of being already established16:48
*** tiagogomes_ has joined #baserock16:48
richard_mawrdale_ct: aye, except that a "source package" is an artifact in debian's build process, while it's an internal detail in ours16:48
ssam2why not 'source artifact' and 'binary artifact' or 'source component' and 'binary component' ?16:49
rdale_cti like the component pair16:49
paulsherwoodassuming we stay with 'artifact' for the result of 'build' on a 'component/chunk/build-unit' - do we have a different name for the result of 'assemble' on a 'group' or is it just another kind of artifact?16:49
richard_mawfor which bits? source artifact implies that at some point we spit out a tarball of source code16:49
richard_mawpaulsherwood: I'm thinking of artifact closer to its dictionary definition by this point, in that it's anything produced, hence cacheable16:50
paulsherwoodoutput of build, tarred.16:50
paulsherwoodbut i'm ok with output of build or assemble, tarred.16:51
richard_mawI wouldn't bother specifying the format necessarily16:51
richard_mawsince it shouldn't matter whether it's a disk image, loose files, or something else entirely16:52
paulsherwoodfair enough, but we continue with artifact for all kinds of outputs (chunk, strata, system)16:52
richard_mawack, whether stratum artifacts need to exist is another matter16:53
rdale_ctwhere have we ended up with 'source xxxx' and 'binary xxxx'?16:57
rdale_ctfor me xxxx: component > package > artifact16:59
paulsherwoodi'm currently assuming 'artifact' is cached. unclear what splitting generates, or where 'package' fits17:04
rdale_cta 'source component' builds a 'binary component' which can be split into 'binary sub components'17:06
richard_mawpaulsherwood: I'm thinking a "package" is the result of a split, but that you can depend on a build directly, which has a default package containing everything, or the package17:06
paulsherwoodwhy wouldn't 'split' just be something done as part of 'assemble'?17:07
richard_mawin debian parlance, split artifacts are "packages" or "binary packages", while "source packages" are packaged up "build" instructions with the sources17:07
paulsherwoodrdale_ct: 'source component' 'binary component'  'binary sub components' are too complicated, imo17:08
richard_mawpaulsherwood: because assemble is the top-level bit17:08
paulsherwoodrichard_maw: ack. but what's the use-case for splitting a component on its own?17:09
*** faybrocklebank has quit IRC17:09
rdale_cta 'source component' builds a 'binary artifact' which can be split into 'packages'17:10
richard_mawpaulsherwood: you mean that you don't have an explicit step for producing a "package", rather that you filter the "build"'s output when using the "package"?17:10
* richard_maw had considered that to be a possible optimisation17:10
paulsherwoodrichard_maw: yes, i think so17:10
richard_mawyeah, no reason you can't do that17:11
richard_mawI take it your model has assembly as both the construction of the top level system artifact, and the construction of the staging areas that builds are run in too?17:11
paulsherwoodcurrently, yes. but i'm happy with the idea of traverse, build, assemble as above - in which case staging probably gets created in traverse17:12
paulsherwood(or maybe not, unclear)17:13
richard_mawcan't, you need the result of some builds before you can assemble the staging area AIUI17:13
richard_mawI have no problem with build and assemble being corecursive17:13
richard_mawbut we made system assembly different from staging area assembly different for optimisation reasons17:14
paulsherwoodyes. traverse could deal with that. and build is actually (check if already built, then check remote cache, then build)17:14
*** ctbruce has quit IRC17:14
richard_mawpaulsherwood: I think we might be hitting an impedence mismatch here, since in my conceptual model traverse builds a data structure that gets executed, while I think yours has traverse causing assemblies and builds17:15
rdale_cti'm not sure that ybd traverses anything because it doesn't create a build graph17:15
paulsherwoodin part i like the traverse/build/assemble idea because the assembly code is currently almost as hard as the definitions code to grok. it has to be multi-instance, idempotent, atomic17:15
richard_mawrdale_ct: we've had this discussion before17:15
richard_mawrdale_ct: ybd has an implied one rather than an explicit one17:16
paulsherwoodyes :-)17:16
rdale_ctin terms of naming functions within ybd, i can't see the word 'traverse' being useful17:16
paulsherwoodin the previous discussion i had to admit i didn't understand what a graph was, because folks seemed to be thinking i was being wilfully obtuse17:16
paulsherwoodrdale_ct: ok17:16
paulsherwoodany alternative suggestions?17:17
rdale_cti think 'assemble' and 'build' are fine, they just need to include what they are building or assemblying, which they don't at the moment - eg 'assemble_system'17:18
rdale_cti don't go a bundle on 'do_' as a prefix either, i would rather just have 'build()' rather than 'do_build()' - it doesn't seem to mean anything to me17:19
paulsherwoodtrouble with that is that we can apply ybd to any level... and previosly we established that a system is only a subset of 'group' really :)17:19
paulsherwoodrdale_ct: agreed - that's a recent hack, to wrap build()17:20
richard_mawthere's a separate question of whether we should be able to do that, previously there was contention because we wanted to have higher-level definitions able to pass down parameters like the architecture17:21
richard_mawnow, I'm happy to be able to build groups directly17:21
rdale_ctcan the system-integration commands be run on a group at any level - or do they only make sense to run them on what we now call 'systems'?17:22
paulsherwoodany level, i think17:22
richard_mawshouldn't be a problem, we want groups to include all the commands they need to be able to do that17:22
*** ssam2 has quit IRC17:23
*** toscalix has quit IRC17:27
*** jonathanmaw has quit IRC17:27
*** edcragg has quit IRC17:27
*** franred has quit IRC17:27
*** tiagogomes_ has quit IRC17:27
*** bashrc_ has quit IRC17:27
*** CTtpollard has quit IRC17:27
*** nowster has quit IRC17:27
*** Lachlan1975 has quit IRC17:27
*** CTtpollard has joined #baserock17:28
*** tiagogomes__ has joined #baserock17:28
*** Lachlan1975 has joined #baserock17:28
*** nowster has joined #baserock17:28
*** bashrc_ has joined #baserock17:28
*** toscalix has joined #baserock17:28
*** jonathanmaw has joined #baserock17:28
*** edcragg has joined #baserock17:28
*** franred has joined #baserock17:28
*** jonathanmaw has quit IRC17:42
rdale_cti'll copy the transcript of this discussion about baserock names and send it to the mailing list17:47
richard_mawcool, if someone has time a chart would be valuable, but I've already been distracted by this topic for too long17:51
*** franred has quit IRC17:56
*** bashrc_ has quit IRC18:00
*** gtristan has quit IRC18:05
*** edcragg has quit IRC18:12
*** toscalix has quit IRC18:17
*** Lachlan1975 has quit IRC18:47
*** rdale_ct has quit IRC19:33
*** gtristan has joined #baserock23:29

Generated by 2.15.3 by Marius Gedminas - find it at!