IRC logs for #baserock for Wednesday, 2016-08-31

*** jjardon_matrix has quit IRC00:50
*** jjardon_matrix has joined #baserock01:01
*** jjardon_matrix has quit IRC01:24
*** jjardon_matrix has joined #baserock01:34
*** gtristan has quit IRC02:15
*** jmacs has quit IRC02:52
*** jmacs has joined #baserock02:53
*** gtristan has joined #baserock03:28
*** rdale has joined #baserock05:26
*** toscalix has joined #baserock07:13
*** CTtpollard has joined #baserock07:27
*** toscalix has quit IRC08:16
*** toscalix has joined #baserock08:20
*** fay has joined #baserock08:38
*** fay is now known as Guest997208:39
rjekYour new nick suits you, Guest997208:42
*** Guest9972 is now known as faybrocklebank08:43
faybrocklebankthis one suits me better I think08:43
*** faybrocklebank has quit IRC08:47
*** fay_ has joined #baserock09:30
*** andrewleeming_ is now known as leeming09:31
*** locallycompact has joined #baserock09:33
*** lachlanmackenzie has joined #baserock09:36
*** gtristan has quit IRC09:50
*** edcragg has joined #baserock09:55
*** gtristan has joined #baserock10:02
*** edcragg has quit IRC10:55
locallycompactpaulsherwood, how feasible might it be to give ybd the option to only run specific build steps for a particular chunk's build instructions10:56
*** edcragg has joined #baserock10:57
locallycompactas a field in the chunk like build-steps: ["install-headers"]10:57
leemingyou mean split up the build instructions somehow? and be able to run on a subset11:14
persiaI don't understand the developer workflow that would involve doing that.11:18
paulsherwoodlocallycompact: i'm not understanding you, can you elaborate please?11:19
locallycompactSo we have the scenario where things are depending on each other circularly11:19
locallycompactbut only for headers11:19
locallycompactnot the actual build11:19
persiaGiven the vast differences in performance between cached builds and sandboxed builds, I would think most developers would be better served by either running the steps manually, or having some means to run all the build steps against the local directory (with no artifact creation) to understand what went wrong.11:19
locallycompactas it stands we have to have a duplicate build morph for each chunk respectively, one is just mkdir headers cp headers destdir11:20
locallycompactthe other is the full build11:20
locallycompactso we have to remember to update both11:20
locallycompactif we had a custom build step that was just install-headers:11:20
locallycompactthen we can use the same build morphology, one defaults to run all steps and one has an override in the chunk specification11:20
persialocallycompact: Rather than a generic solution, would you be happy with a "build-headers" or "build-build-deps" sort of subclass command?11:21
paulsherwoodcould we frab it somehow by having build-system: 'headers-only' ?11:21
locallycompactno because the directory is different each time11:22
locallycompactunless you want to pass parameters into the build morph11:22
locallycompactwhich is also good11:22
persialocallycompact: How is the directory different each time?  For which value of each?11:22
locallycompactfor each chunk11:22
locallycompactmkdir $DESTDIR/opt/foo/CHUNK/ cp headers/*.h $DESTDIR/opt/foo/CHUNK11:23
persiaRight, so in my imagination, the chunk YAML contains a list of what is necessary to build the stuff that is necessary for building other stuff (with lighter build-dependencies), and a list of what is necessary to build it properly.11:23
persiaIt still creates two chunks (like now), but the code is mixed.11:23
leeminghmm yes, I know where you are coming from with this locallycompact11:23
persiaAnd the build tooling has a means to ask for only the headers.11:23
persiaIf you allow the build tool to select what is run, or you allow the build tool to pass parameters, or anything else generic, you end up with dyynamism, which breaks trust in the signatures.11:24
leemingwhen you say update both. do you mean updating the git information for both?11:24
locallycompactyou have to update the git informatio nfor both anyway11:24
persialeeming: Update the SHA1 for two different chunk definitions for one repo.11:24
locallycompactit's a different artifact11:24
leemingif you are happy with updating both chunk refs, then what are you trying to avoid (update wise)?11:25
locallycompacthaving to update common build instructions11:26
leemingisn't the header almost always the precursor to the main build instructions though?11:26
leemingto build foo.morph, it depends on foo-headers.morph11:27
persiaI think that either one file should produce two different types of artifact (with different instructions, possibly merged using YAML), or the user should update two things separately.11:27
* paulsherwood struggles to understand what is precisely the problem... eg in ?11:28
persialeeming: Consider the case where foo.morph depends on bar-headers and bar.morph depends on foo-headers.11:28
locallycompactpaulsherwood, yes, except i don't want to have to maintain two build morphs11:28
paulsherwoodwhy not?11:28
locallycompactbecause it's overhead11:29
locallycompactone is a subset of the other11:29
paulsherwoodso, build-system: headers-only, with some (new) syntax for 'disregard build-steps x, y, z' ?11:31
leemingbasically the build ordering? automating/priority for *-header.morph to be run first?11:31
paulsherwood(in defaults)11:31
leemingintroduce a "co-dependency" field, that requires a *-header.morph chunk? Just trying to think if there is a current way for this, without changing the definitions format11:32
locallycompactI think that's a bit overkill but sure11:32
locallycompactI think overriding and declaring rather than disregard x, y ,z is clearer though11:34
locallycompact- post-install-commands11:34
locallycompact- etc..11:34
rdale'header-only' sounds like re-inventing artifact splitting to me11:36
locallycompactrdale, the full artifact can't be built at that point11:37
locallycompactpersia, the build tool isn't passing parameters, I'd put them in the chunk specification under environment: or variables:11:43
locallycompactFOO: bar11:43
locallycompactso the build-instructions can be parameterised by the stratum declaration11:44
persialocallycompact: I believe that any parameters arriving into the file are untrustworthy, because they mean that I cannot assert that the checksum of the file is the checksum of the commands executed.11:45
persiaAnd losing that assertion ruins most of the value of this class of tooling for me.11:45
locallycompactybd hashes the whole chunk, not jsut the build instructions11:46
persiaThat doesn't matter.11:46
persiaIf the chunk can be built two different ways, depending on how the stratum calls it, then the checksum for the chunk file does not completely specify the artifact.11:47
persiaThe chunk has to be self-contained.11:47
persiaI'm happy to have two different fields, being:11:47
locallycompactWhen I say chunk11:47
locallycompactI'm talking abou tthis11:47
locallycompactno no11:47
persiaand then use YAML to merge the common bits.11:47
persiaBecause that means that I can assert the chunk is self-contained.11:47
locallycompactthe file that contains the build-instructions is not a chunk11:47
locallycompactit doesn't specify and artifact and never has11:48
locallycompactit's a collection of build instructions that has redundant name: and kind: field11:48
locallycompactthe chunk is the repo, ref and build instructions11:48
locallycompactthose "kind: chunk" files are just not chunks11:48
locallycompactthey're instructions with redundant fields that are getting in the way of their being reused effectively11:48
persiaI have always been under the impression that the "name" specified an artifact.11:49
locallycompactthere's a name: in the chunk specification, use that11:50
persiaI'm perfectly happy with disposing of a layer of files that is blocking re-use, putting all that in the level above, and using YAML variables, etc. to reuse common instructions.11:51
persiaI just care *very much* that no variables are being passed around that can cause the build behaviour to differ between two invocations using the same file.11:51
locallycompactyou only need to ensure invocations are always qualified with the stratum11:55
locallycompactwhich is natural11:55
persiaThat depends on the tool, and one of the points of ybd was to discover all the places where the definitions were incomplete/invalid because of tooling assumptions: introducing more doesn't seem right.11:58
locallycompacteven build-depends is a tooling assumption12:02
locallycompactactually, I retract that, build-depends and variables are ontological assumptions12:03
locallycompactif the build-tool interprets the ontology faithfully then it doesn't need to assume much12:04
persiaRight, hence my objection to "qualified within the stratum": that imposes constraints that were not present before.12:05
locallycompactNo these are already there12:06
locallycompactchunks only exist in strata the fact that the build morph says kind: chunk is just garbage nonsense12:06
locallycompactthey exist because they are holistic with respect to their build dependencies in the stratum12:07
persiaI have come to agree with you that the build morph files are not meaningful.12:07
persiaI'm just really uncomfortable with the idea of passing variables *between* files, rather than requiring all variables to be defined and expanded in very simple ways within files.12:08
persiaBecause doing so violates the assumption that the text of a file specifies the behaviour of a file.12:08
persiaAnd I'm not confident that we have a sufficient mechanism to be sure of what is being done without that assumption.12:09
locallycompactI don't see it as that, the onus is on the person writing the chunk spec to pick or write a collection of build instructions that make sense12:09
locallycompactmaybe the build instructions say they expect a collection of parameters12:10
persiaTo me, that is a failure.12:10
persiaI want it to be impossible for the author of the system definitions to do anything that means that someone else can discover that what they did was not determinisitic 20 years later.12:10
locallycompactwhat's not deterministic12:10
persiaIf there is an onus on the author to do something correctly, that implies they can do it incorrectly, which implies that something may differ.12:11
locallycompactthere's still a need to write build instructions that work12:11
persiaSo anything that is asserted as depending on someone behaviing in some way causes me to immediately reject it.12:11
persiaYes.  If the instructions cannot work except by being done in the right way, that satisfies my requirements.12:12
locallycompactthere's a certain assumption that integrators don't simply roll their faces around on the keyboard12:12
locallycompactbut there's no more or less required in either scenario really12:12
locallycompacts/really/at all12:12
persiaI'd rather assume that the definitions were random noise, and that the tooling would only work when it happened to match a set of criteria that allowed one to be confident.12:13
persiaThat some humans would happen to modify the noise before invoking the tool is just an indication that humans pattern match and dislike being told they have failed.12:14
persiaAnyway, I think I've complained enough about this, and that you've understood my point, so I'll stop.12:15
locallycompactin any case having templates doesn't preclude writing bespoke build instructions for each chunk, if that's the policy12:17
locallycompactbut the build-instructions need to be interpreted as a property of the chunk spec, and if the chunk spec contains everything needed to build it deterministically then I wouldn't mind having templates12:18
*** gtristan has quit IRC12:52
* paulsherwood has been at lunch, and will now require some time to digest the above13:35
*** gtristan has joined #baserock13:40
*** CTtpollard has quit IRC13:50
*** CTtpollard has joined #baserock13:51
paulsherwoodjjardon: pls can you approve my request for access to gitlab/baserock?14:24
jjardonpaulsherwood: done, seems you have 2 users in gitlab?14:26
paulsherwoodpossibly :)14:29
paulsherwoodanyway, i've imported ybd to gitlab/baserock/ybd14:30
paulsherwoodi'll write a note on this and send to the list. then i'll add a top-level commit to the github version pointing to the new url14:30
jjardonpaulsherwood: welcome home! :)14:35
paulsherwoodjjardon: note is not actually doing a build - it's got mode: keys-only15:18
jjardonpaulsherwood: I was just about to ask about that15:18
jjardonalso, why artifact_version=1 ?15:20
paulsherwoodjjardon: because - the original ci on github checks that the cache-key algorithm still generates correct values for some known historical definitions/artifact-version15:22
jjardonpaulsherwood: should we use that for a specific stage only, instead for all?15:23
paulsherwoodwhat is a stage?15:23
jjardonsteps in the ci: you have defined - pep815:24
jjardon- cache_keys15:24
jjardon- build15:24
jjardon- test15:24
jjardon- deploy15:24
paulsherwoodthose settings should only be for the cache_keys step. ideally we could/should have at lease one *old, known* build to verify old builds still work, as well as current15:25
jjardonwe can probably verify all of them15:26
jjardonin parallel15:26
leemingok, I don't see to be progressing very much today (sorry). Trying to build 'clusters/weston-system-x86_64-generic-deploy.morph x86_64' but hitting a few issues16:14
leemingI had not looked into the splitting stuff yet, so just placing that here in case it is obvious to experienced eyes16:15
paulsherwoodeek. not obvious to me16:20
leemingfun debugging it is for me then :)16:25
*** anahuelamo has joined #baserock16:31
*** anahuelamo_ has quit IRC16:31
*** CTtpollard has quit IRC17:27
jjardonpaulsherwood: not sure why the ybd ci is trying to rebuild everything: (the artifact should be in kbas already)17:29
locallycompactthat's usually because of artifact version17:35
*** locallycompact has quit IRC17:39
jjardonyep, I was trying to use version 117:41
*** toscalix has quit IRC18:20
*** rdale has quit IRC23:12

Generated by 2.14.0 by Marius Gedminas - find it at!