| *** jjardon_matrix has quit IRC | 00:50 | |
| *** jjardon_matrix has joined #baserock | 01:01 | |
| *** jjardon_matrix has quit IRC | 01:24 | |
| *** jjardon_matrix has joined #baserock | 01:34 | |
| *** gtristan has quit IRC | 02:15 | |
| *** jmacs has quit IRC | 02:52 | |
| *** jmacs has joined #baserock | 02:53 | |
| *** gtristan has joined #baserock | 03:28 | |
| *** rdale has joined #baserock | 05:26 | |
| *** toscalix has joined #baserock | 07:13 | |
| *** CTtpollard has joined #baserock | 07:27 | |
| *** toscalix has quit IRC | 08:16 | |
| *** toscalix has joined #baserock | 08:20 | |
| *** fay has joined #baserock | 08:38 | |
| *** fay is now known as Guest9972 | 08:39 | |
| rjek | Your new nick suits you, Guest9972 | 08:42 |
|---|---|---|
| *** Guest9972 is now known as faybrocklebank | 08:43 | |
| faybrocklebank | this one suits me better I think | 08:43 |
| rjek | :) | 08:43 |
| *** faybrocklebank has quit IRC | 08:47 | |
| *** fay_ has joined #baserock | 09:30 | |
| *** andrewleeming_ is now known as leeming | 09:31 | |
| *** locallycompact has joined #baserock | 09:33 | |
| *** lachlanmackenzie has joined #baserock | 09:36 | |
| *** gtristan has quit IRC | 09:50 | |
| *** edcragg has joined #baserock | 09:55 | |
| *** gtristan has joined #baserock | 10:02 | |
| *** edcragg has quit IRC | 10:55 | |
| locallycompact | paulsherwood, how feasible might it be to give ybd the option to only run specific build steps for a particular chunk's build instructions | 10:56 |
| *** edcragg has joined #baserock | 10:57 | |
| locallycompact | as a field in the chunk like build-steps: ["install-headers"] | 10:57 |
| leeming | you mean split up the build instructions somehow? and be able to run on a subset | 11:14 |
| persia | I don't understand the developer workflow that would involve doing that. | 11:18 |
| paulsherwood | locallycompact: i'm not understanding you, can you elaborate please? | 11:19 |
| locallycompact | So we have the scenario where things are depending on each other circularly | 11:19 |
| locallycompact | but only for headers | 11:19 |
| locallycompact | not the actual build | 11:19 |
| persia | Given the vast differences in performance between cached builds and sandboxed builds, I would think most developers would be better served by either running the steps manually, or having some means to run all the build steps against the local directory (with no artifact creation) to understand what went wrong. | 11:19 |
| locallycompact | as it stands we have to have a duplicate build morph for each chunk respectively, one is just mkdir headers cp headers destdir | 11:20 |
| locallycompact | the other is the full build | 11:20 |
| locallycompact | so we have to remember to update both | 11:20 |
| locallycompact | if we had a custom build step that was just install-headers: | 11:20 |
| locallycompact | then we can use the same build morphology, one defaults to run all steps and one has an override in the chunk specification | 11:20 |
| persia | locallycompact: Rather than a generic solution, would you be happy with a "build-headers" or "build-build-deps" sort of subclass command? | 11:21 |
| paulsherwood | could we frab it somehow by having build-system: 'headers-only' ? | 11:21 |
| locallycompact | no because the directory is different each time | 11:22 |
| locallycompact | unless you want to pass parameters into the build morph | 11:22 |
| locallycompact | which is also good | 11:22 |
| paulsherwood | heh | 11:22 |
| persia | locallycompact: How is the directory different each time? For which value of each? | 11:22 |
| locallycompact | for each chunk | 11:22 |
| locallycompact | mkdir $DESTDIR/opt/foo/CHUNK/ cp headers/*.h $DESTDIR/opt/foo/CHUNK | 11:23 |
| persia | Right, so in my imagination, the chunk YAML contains a list of what is necessary to build the stuff that is necessary for building other stuff (with lighter build-dependencies), and a list of what is necessary to build it properly. | 11:23 |
| persia | It still creates two chunks (like now), but the code is mixed. | 11:23 |
| leeming | hmm yes, I know where you are coming from with this locallycompact | 11:23 |
| persia | And the build tooling has a means to ask for only the headers. | 11:23 |
| persia | If you allow the build tool to select what is run, or you allow the build tool to pass parameters, or anything else generic, you end up with dyynamism, which breaks trust in the signatures. | 11:24 |
| leeming | when you say update both. do you mean updating the git information for both? | 11:24 |
| locallycompact | what? | 11:24 |
| locallycompact | you have to update the git informatio nfor both anyway | 11:24 |
| persia | leeming: Update the SHA1 for two different chunk definitions for one repo. | 11:24 |
| locallycompact | it's a different artifact | 11:24 |
| leeming | if you are happy with updating both chunk refs, then what are you trying to avoid (update wise)? | 11:25 |
| locallycompact | having to update common build instructions | 11:26 |
| leeming | isn't the header almost always the precursor to the main build instructions though? | 11:26 |
| locallycompact | what? | 11:26 |
| leeming | to build foo.morph, it depends on foo-headers.morph | 11:27 |
| persia | I think that either one file should produce two different types of artifact (with different instructions, possibly merged using YAML), or the user should update two things separately. | 11:27 |
| * paulsherwood struggles to understand what is precisely the problem... eg in http://paste.baserock.org/vameguhili ? | 11:28 | |
| persia | leeming: Consider the case where foo.morph depends on bar-headers and bar.morph depends on foo-headers. | 11:28 |
| locallycompact | paulsherwood, yes, except i don't want to have to maintain two build morphs | 11:28 |
| paulsherwood | why not? | 11:28 |
| locallycompact | because it's overhead | 11:29 |
| locallycompact | one is a subset of the other | 11:29 |
| paulsherwood | so, build-system: headers-only, with some (new) syntax for 'disregard build-steps x, y, z' ? | 11:31 |
| leeming | basically the build ordering? automating/priority for *-header.morph to be run first? | 11:31 |
| paulsherwood | (in defaults) | 11:31 |
| leeming | introduce a "co-dependency" field, that requires a *-header.morph chunk? Just trying to think if there is a current way for this, without changing the definitions format | 11:32 |
| locallycompact | I think that's a bit overkill but sure | 11:32 |
| locallycompact | I think overriding and declaring rather than disregard x, y ,z is clearer though | 11:34 |
| locallycompact | build-steps: | 11:34 |
| locallycompact | - post-install-commands | 11:34 |
| locallycompact | - etc.. | 11:34 |
| rdale | 'header-only' sounds like re-inventing artifact splitting to me | 11:36 |
| locallycompact | rdale, the full artifact can't be built at that point | 11:37 |
| locallycompact | persia, the build tool isn't passing parameters, I'd put them in the chunk specification under environment: or variables: | 11:43 |
| locallycompact | FOO: bar | 11:43 |
| locallycompact | so the build-instructions can be parameterised by the stratum declaration | 11:44 |
| persia | locallycompact: I believe that any parameters arriving into the file are untrustworthy, because they mean that I cannot assert that the checksum of the file is the checksum of the commands executed. | 11:45 |
| persia | And losing that assertion ruins most of the value of this class of tooling for me. | 11:45 |
| locallycompact | ybd hashes the whole chunk, not jsut the build instructions | 11:46 |
| persia | That doesn't matter. | 11:46 |
| locallycompact | why | 11:46 |
| persia | If the chunk can be built two different ways, depending on how the stratum calls it, then the checksum for the chunk file does not completely specify the artifact. | 11:47 |
| persia | The chunk has to be self-contained. | 11:47 |
| persia | I'm happy to have two different fields, being: | 11:47 |
| locallycompact | When I say chunk | 11:47 |
| locallycompact | I'm talking abou tthis | 11:47 |
| persia | build-headers: | 11:47 |
| locallycompact | no | 11:47 |
| persia | build-chunk: | 11:47 |
| locallycompact | no no | 11:47 |
| persia | and then use YAML to merge the common bits. | 11:47 |
| persia | Because that means that I can assert the chunk is self-contained. | 11:47 |
| locallycompact | the file that contains the build-instructions is not a chunk | 11:47 |
| locallycompact | it doesn't specify and artifact and never has | 11:48 |
| locallycompact | it's a collection of build instructions that has redundant name: and kind: field | 11:48 |
| locallycompact | the chunk is the repo, ref and build instructions | 11:48 |
| locallycompact | those "kind: chunk" files are just not chunks | 11:48 |
| locallycompact | they're instructions with redundant fields that are getting in the way of their being reused effectively | 11:48 |
| persia | I have always been under the impression that the "name" specified an artifact. | 11:49 |
| locallycompact | there's a name: in the chunk specification, use that | 11:50 |
| persia | I'm perfectly happy with disposing of a layer of files that is blocking re-use, putting all that in the level above, and using YAML variables, etc. to reuse common instructions. | 11:51 |
| persia | I just care *very much* that no variables are being passed around that can cause the build behaviour to differ between two invocations using the same file. | 11:51 |
| locallycompact | sure | 11:51 |
| locallycompact | you only need to ensure invocations are always qualified with the stratum | 11:55 |
| locallycompact | which is natural | 11:55 |
| persia | That depends on the tool, and one of the points of ybd was to discover all the places where the definitions were incomplete/invalid because of tooling assumptions: introducing more doesn't seem right. | 11:58 |
| locallycompact | even build-depends is a tooling assumption | 12:02 |
| locallycompact | actually, I retract that, build-depends and variables are ontological assumptions | 12:03 |
| locallycompact | if the build-tool interprets the ontology faithfully then it doesn't need to assume much | 12:04 |
| persia | Right, hence my objection to "qualified within the stratum": that imposes constraints that were not present before. | 12:05 |
| locallycompact | No these are already there | 12:06 |
| locallycompact | chunks only exist in strata the fact that the build morph says kind: chunk is just garbage nonsense | 12:06 |
| locallycompact | they exist because they are holistic with respect to their build dependencies in the stratum | 12:07 |
| persia | I have come to agree with you that the build morph files are not meaningful. | 12:07 |
| persia | I'm just really uncomfortable with the idea of passing variables *between* files, rather than requiring all variables to be defined and expanded in very simple ways within files. | 12:08 |
| persia | Because doing so violates the assumption that the text of a file specifies the behaviour of a file. | 12:08 |
| persia | And I'm not confident that we have a sufficient mechanism to be sure of what is being done without that assumption. | 12:09 |
| locallycompact | I don't see it as that, the onus is on the person writing the chunk spec to pick or write a collection of build instructions that make sense | 12:09 |
| locallycompact | maybe the build instructions say they expect a collection of parameters | 12:10 |
| persia | To me, that is a failure. | 12:10 |
| persia | I want it to be impossible for the author of the system definitions to do anything that means that someone else can discover that what they did was not determinisitic 20 years later. | 12:10 |
| locallycompact | what's not deterministic | 12:10 |
| persia | If there is an onus on the author to do something correctly, that implies they can do it incorrectly, which implies that something may differ. | 12:11 |
| locallycompact | there's still a need to write build instructions that work | 12:11 |
| persia | So anything that is asserted as depending on someone behaviing in some way causes me to immediately reject it. | 12:11 |
| persia | Yes. If the instructions cannot work except by being done in the right way, that satisfies my requirements. | 12:12 |
| locallycompact | there's a certain assumption that integrators don't simply roll their faces around on the keyboard | 12:12 |
| locallycompact | but there's no more or less required in either scenario really | 12:12 |
| locallycompact | s/really/at all | 12:12 |
| persia | I'd rather assume that the definitions were random noise, and that the tooling would only work when it happened to match a set of criteria that allowed one to be confident. | 12:13 |
| persia | That some humans would happen to modify the noise before invoking the tool is just an indication that humans pattern match and dislike being told they have failed. | 12:14 |
| persia | Anyway, I think I've complained enough about this, and that you've understood my point, so I'll stop. | 12:15 |
| locallycompact | in any case having templates doesn't preclude writing bespoke build instructions for each chunk, if that's the policy | 12:17 |
| locallycompact | but the build-instructions need to be interpreted as a property of the chunk spec, and if the chunk spec contains everything needed to build it deterministically then I wouldn't mind having templates | 12:18 |
| persia | Agreed | 12:24 |
| *** gtristan has quit IRC | 12:52 | |
| * paulsherwood has been at lunch, and will now require some time to digest the above | 13:35 | |
| *** gtristan has joined #baserock | 13:40 | |
| *** CTtpollard has quit IRC | 13:50 | |
| *** CTtpollard has joined #baserock | 13:51 | |
| paulsherwood | jjardon: pls can you approve my request for access to gitlab/baserock? | 14:24 |
| jjardon | paulsherwood: done, seems you have 2 users in gitlab? | 14:26 |
| paulsherwood | possibly :) | 14:29 |
| paulsherwood | anyway, i've imported ybd to gitlab/baserock/ybd | 14:30 |
| paulsherwood | i'll write a note on this and send to the list. then i'll add a top-level commit to the github version pointing to the new url | 14:30 |
| jjardon | paulsherwood: welcome home! :) | 14:35 |
| paulsherwood | :-) | 14:35 |
| paulsherwood | jjardon: note https://gitlab.com/baserock/ybd/builds/3610567 is not actually doing a build - it's got mode: keys-only | 15:18 |
| jjardon | paulsherwood: I was just about to ask about that | 15:18 |
| jjardon | also, why artifact_version=1 ? | 15:20 |
| paulsherwood | jjardon: because https://gitlab.com/baserock/ybd/blob/master/.travis.yml#L14 - the original ci on github checks that the cache-key algorithm still generates correct values for some known historical definitions/artifact-version | 15:22 |
| jjardon | paulsherwood: should we use that for a specific stage only, instead for all? | 15:23 |
| paulsherwood | what is a stage? | 15:23 |
| jjardon | steps in the ci: you have defined - pep8 | 15:24 |
| jjardon | - cache_keys | 15:24 |
| jjardon | - build | 15:24 |
| jjardon | - test | 15:24 |
| jjardon | - deploy | 15:24 |
| paulsherwood | those settings should only be for the cache_keys step. ideally we could/should have at lease one *old, known* build to verify old builds still work, as well as current | 15:25 |
| jjardon | yes | 15:26 |
| jjardon | we can probably verify all of them | 15:26 |
| jjardon | in parallel | 15:26 |
| paulsherwood | ack | 15:27 |
| leeming | ok, I don't see to be progressing very much today (sorry). Trying to build 'clusters/weston-system-x86_64-generic-deploy.morph x86_64' but hitting a few issues | 16:14 |
| leeming | http://paste.baserock.org/tolurobove | 16:14 |
| leeming | I had not looked into the splitting stuff yet, so just placing that here in case it is obvious to experienced eyes | 16:15 |
| paulsherwood | eek. not obvious to me | 16:20 |
| leeming | fun debugging it is for me then :) | 16:25 |
| paulsherwood | heh | 16:26 |
| *** anahuelamo has joined #baserock | 16:31 | |
| *** anahuelamo_ has quit IRC | 16:31 | |
| *** CTtpollard has quit IRC | 17:27 | |
| jjardon | paulsherwood: not sure why the ybd ci is trying to rebuild everything: https://gitlab.com/baserock/ybd/builds/3616698#down-build-trace (the artifact should be in kbas already) | 17:29 |
| locallycompact | that's usually because of artifact version | 17:35 |
| *** locallycompact has quit IRC | 17:39 | |
| jjardon | yep, I was trying to use version 1 | 17:41 |
| *** toscalix has quit IRC | 18:20 | |
| *** rdale has quit IRC | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!