IRC logs for #baserock for Monday, 2016-09-12

*** paulwaters_ has joined #baserock07:00
*** toscalix has joined #baserock07:44
*** rdale has joined #baserock07:55
*** jonathanmaw has joined #baserock08:12
*** jonathanmaw has left #baserock08:12
*** tiagogomes has joined #baserock09:03
*** edcragg has joined #baserock09:27
*** locallycompact has joined #baserock09:34
jjardonHi, ok to lorry this 2 packages? https://gerrit.baserock.org/#/q/status:open+project:baserock/local-config/lorries+branch:master+topic:lorry-gnu-utilities09:56
paulsherwoodjjardon: ok by me +109:58
*** baserockgerrit has quit IRC10:02
*** gtristan has joined #baserock10:04
*** baserockgerrit has joined #baserock10:07
*** baserockgerrit_ has joined #baserock10:11
*** baserockgerrit_ is now known as baserockgerrit__10:11
*** baserockgerrit has quit IRC10:12
*** baserockgerrit__ has quit IRC10:12
*** baserockgerrit has joined #baserock10:14
*** baserockgerrit has quit IRC10:19
radiofreehello everyone!10:58
radiofreewhat are partial deployments and do they still work?10:58
paulsherwoodSotK: ^^10:58
paulsherwoodradiofree: iirc it was just providing an untar of a single chunk as an upgrade deployment11:07
paulsherwood(unsafe, since no consideration of dependencies)11:07
radiofreeok11:07
* pedroalvarez finds: http://wiki.baserock.org/guides/arm-howtos/#index4h211:09
paulsherwoodpedroalvarez: good spot!11:10
pedroalvarezI've never used it, but it looks like you can specify more than one chunk/strata to deploy11:10
* paulsherwood thinks ybd *might* be ok with the same mechanism11:10
pedroalvarezand also looks like you can deploy an upgrade, or a tar, or any kind of deployment11:11
pedroalvarezalthough deploying upgrades may need  to include some chunks from foundation.morph that includes the upgrade scripts11:12
radiofreeok i'll give that a whirl, thanks pedroalvarez11:14
*** rdale has quit IRC11:20
* SotK reiterates the "you shouldn't use this in production deployment" warning in that documentation12:00
*** rdale has joined #baserock12:43
paulsherwood:)13:22
jmacsShould be a metadata flag for all source, that13:24
paulsherwoodFIXME: make this safe for production :)13:25
radiofreepaulsherwood: it works13:26
radiofreehowever i think it's just easier to ybd the/strata.morph and deploy that manually13:27
paulsherwoodradiofree: cool13:27
radiofreesince the partial-deploy tar ball is everything (with just the partially deployed thing built)13:27
* paulsherwood would like to see ostree here, rather than all the weirdness13:27
pedroalvarezwhat is "all the weirdness"?13:28
pedroalvarezthe "non safe for production" bit?13:28
paulsherwoodpedroalvarez: i may be wrong, but the deployment extensions struck me as weird in general13:30
* paulsherwood is probably conflating things as usual13:31
radiofreefor upgrades, it still requires the system is btrfs right?13:31
paulsherwoodyes13:31
radiofreedeploying to tarball is useful and works, deploy to rawdisk isn't so useful if you can't use btrfs13:31
radiofreeso that leaves us with no meaningful upgrade mechanism13:32
jjardonpedroalvarez: yes, and hard depend on btrfs. Also, ostree is a mature tool to manage updates of systems13:32
paulsherwoodto be fair, ostree didn't exist when we first started with the btrfs upgrades :)13:32
jjardontrue13:33
pedroalvarezdoes ostree include dependencies too?13:33
pedroalvarezerm..13:33
jjardonpedroalvarez: what do you mean?13:33
pedroalvarezwill ostree include dependencies too? as btrfs does?13:34
paulsherwoodyes, it depends on some things13:34
paulsherwoodit's already integrated in some baserock systems iiuc13:35
pedroalvarezas is btrfs, (integrated) but still causing pain13:35
paulsherwoodack13:36
pedroalvarezjust asking questions because I'm curious, not criticizing13:36
paulsherwoodpedroalvarez: aiui some folks here are working on a non-baserock baserock system... where btrfs will never be feasible... but upgrades are required13:38
SotKusing ostree for updates would be nice13:38
radiofreeSotK: did you and sam do some work on that in the past?13:39
pedroalvarezthat work was more complicated13:39
pedroalvarezit was to use ostree in the whole building process13:39
SotKthe previous ostree work was about using ostree in the artifact cache rather than a bunch of tarballs in order to speed up the building and deployment process13:40
paulsherwooddid it succeed?13:40
pedroalvarezwe found a big blocker13:40
paulsherwoodoh?13:40
pedroalvarezthis was a long time ago, and I didn't do the job, nor identify the blocker13:41
richard_mawpaulsherwood: it did build faster, but Baserock filesystem images weren't able to cope with the deduplication of files, since we assume that files with the same contents may not be hardlinks of each other13:41
richard_mawpaulsherwood: the result was that the systems didn't boot properly, because the first thing systemd did was write the machine-id to /etc/machine-id, which was an empty file that was deduplicated with all the other empty files on the system13:42
paulsherwoodargh :)13:42
richard_mawpaulsherwood: so for stuff where an empty file is a logically consistent configuration, they suddenly had an improperly formatted configuration file13:43
pedroalvarezrichard_maw: I remember something about "usr merge" being a fix for some of the blockers of the ostree work13:43
paulsherwoodstrikes me this has to be either a corner case worth fixing in ostree, or a bug in our systems?13:43
richard_mawpaulsherwood: bug in our systems, ostree is designed for read-only /usr and empty /etc13:44
richard_maweasiest fix for systemd-based systems would be to have something move the contents of /etc into /usr/lib/factory/etc and have systemd copy it out of there into /etc on first boot13:45
richard_mawotherwise you might get away with some other "unsharing" of files in /etc13:45
locallycompactpaulsherwood, where does the megayaml thing you mentioned end up13:50
paulsherwooddefinitions.yml in the definitions dir13:57
paulsherwoodrichard_maw: can you refresh my memory about the loom project, please? did it see the light of day?13:58
richard_mawit did not13:59
paulsherwoodnothing in public anywhere?13:59
jjardonthere are some issues opened about that: https://storyboard.baserock.org/#!/story/11 , https://storyboard.baserock.org/#!/story/36 and https://storyboard.baserock.org/#!/story/4814:00
richard_mawthe patches to yarn to make its library consumable for it went public, but weren't merged14:00
paulsherwoodrichard_maw: was there any discussion anywhere about the concepts?14:01
richard_mawnot that I recall14:17
richard_mawI did a friday talk at Codethink about it, but don't recall if it was recorded. persia_ might know more since he was involved in drafting the idea.14:18
*** jjardon_matrix has quit IRC14:18
*** gtristan has quit IRC14:20
mwilliams_ctHey folks! I am making steady headway on getting a trove running on Baserock, thanks to pedroalvarez! One thing we are confused by, where/when/how is git-daemon-export-ok created for repos in /home/git/repos?14:40
mwilliams_cts/on Baserock/on Debian14:40
richard_mawmwilliams_ct: git-daemon-export-ok is created by gitano14:41
mwilliams_ctrichard_maw: ack, that's what pedroalvarez was suspecting. something to do with the ACLs presumably, do I need to play about with the gitano-admin repo (subbing gitano-admin for whatever the repo is called)?14:42
richard_mawmwilliams_ct: https://git.gitano.org.uk/gitano.git/tree/doc/admin/000.mdwn?h=liw/admindoc might be handy if you need a primer into how rules work14:43
richard_mawit's WIP at this point, but feedback would probably be graciously accepted in #gitano14:43
mwilliams_ctthanks richard_maw, will have a look through14:44
persiarichard_maw: I never saw such a video recording.  I may have an older version of the code somewhere.  My memory is that after the initial implementation, Kinnison was drafting some guidelines for integration with Mustard, but I haven't heard since, and that was a long time ago now.14:46
persiapaulsherwood: My memory of the loom tooling was that it parsed YAML containing requirements, and then used that to process tests, where the requirements were expressed as a user story, with tags to identify verification criteria as scenarios, and scenarions.  Looking through my documentation, I appear to have the loom requirements, in the loom format, but not the code to parse them.14:51
paulsherwoodpersia: any reason the requirements can not be made public?14:52
paulsherwoodreason is i'm trying to flush out any prior art on traceability from requirements to tests, at https://lists.veristac.io/pipermail/trustable-software/2016-September/000044.html14:53
* richard_maw has found the code for loom, and the mustard for loom that was its test suite14:54
persiapaulsherwood: I don't know of any, although I don't have a license to redistribute, and some copyright is claimed ("Copyright 2014 Codethink").14:54
persiarichard_maw: As you're the author of most of what I have, you're probably best placed to license it so it can be distributed :)14:54
* richard_maw is looking for a branch of the branch of cmdtest that has the yarn changes in14:55
richard_mawpaulsherwood: the branch of cmdtest that has the changes required for loom is http://git.baserock.org/cgit/delta/cmdtest.git/log/?h=baserock/richardmaw/S10275/factor-out-modules-v3 the loom code is in the codethink git server in ct098/loom14:56
paulsherwoodrichard_maw: ack, thanks. i'll need to get someone to push that somewhere public14:57
*** jjardon_matrix has joined #baserock14:59
persiapaulsherwood: Some other prior art on tracing tests to requirements: http://pythonhosted.org/behave/ ,  https://cucumber.io/ , http://jbehave.org/ , http://behat.org/en/v3.0/ , http://jasmine.github.io/ , and http://blog.liw.fi/posts/yarn/15:08
persiaNote that all of those suffer from compliance with the given-when-then format: none contain any means of tracking that to user stories directly.15:09
persia(which tends to lead to the degenerate format of  user stories commonly found in agile (e.g. "As a user, I want to submit identifying characteristics of my device for cross-site tracking.")15:10
*** locallycompact has quit IRC17:27
radiofreeshould changing the DEFAULTS (adding a splitting rule) force a rebuild of everything?17:33
radiofreewe don't actually have a rule for debug symbols, so they end up in misc17:33
persiaIf you change the split rules, you change the content of all artifacts, so yes, I think it should force a rebuild of everything.17:35
persiaradiofree: Are these debug files that are stripped with --only-keep-debug for later troubleshooting of executables?17:36
* persia would be very excited if that had been implemented17:36
radiofreeyes, we (correctly) strip debug symbols into /usr/lib/debug/17:37
radiofreehowever there's actually no rule for debug splitting in a chunk17:37
radiofreei think that may have been an omission since the stratum level devel refers to -debug17:37
persiaYes, that does seem to be an omission17:43
*** toscalix has quit IRC17:44
*** tiagogomes has quit IRC17:53
paulsherwoodradiofree: no, it shouldn't force a rebuild17:54
paulsherwoodin ybd artifacts are created with everything17:54
paulsherwoodthe splitting only applies on creation of strata (which are just manifests) and image assembly17:54
radiofreeBut the manifest is part of the chunk?18:00
radiofreeOr does that get overwritten when you deploy?18:00
paulsherwoodradiofree: manifest for a chunk contains all of the splits. stratum manifest identifies specific splits (eg foo-minimal)18:01
radiofreeBut I added a rule for chunk splitting?18:06
radiofreeAt what point will they be applied?18:06
paulsherwoodstratum creation applies the rules to decide what shoudl be in the stratum (just generate the lists). system creation applies the lists from the strata to create the actual image18:07
paulsherwood(iirc)18:08
radiofreeOK, so when will my chunks be split according to the new rules?18:09
* radiofree will test this with. deploy at somepoint18:11
radiofreeSo why is the chunk manifest in the artifact for a chunk?18:11
radiofreeIf I create a new splitting rule for a chunk surely something will have to add that to the manifest at some point?18:12
* paulsherwood starts to wonder if he's wrong18:12
radiofreeWell before my rule, you'd have /usr/lib/debug/foo.so in "misc" for the chunk18:13
radiofreeI add a splitting rule in defaults to move that to debug instead, and build the system again18:13
radiofreeBut it didn't rebuild anything, and the manifest that says foo.so is in misc is still part of the chunk aftifact18:14
radiofreeSo at what point is that go to debug (the statum splitting has devel which contains debug)18:15
paulsherwoodi think you've found a hole/bug18:17
paulsherwoodlet me do some investigation18:17
paulsherwoodsadly istm that chunks shouldn't care about splits at all, and all splitting should be done at stratum and system18:53
paulsherwoodcurrently the stratum splitting is using split info from the chunk metadata, which means the current scheme can't support radiofree's usecase :/18:54
paulsherwoodhttps://gitlab.com/baserock/ybd/issues/24218:58
paulsherwoodradiofree: can you show me your new defaults please?19:01
* paulsherwood is wondering if it's possible just to ignore the chunk metadata19:02
radiofreepaulsherwood: not currently, I'm in the pub, I'll pop back on the way home19:21
radiofreepaulsherwood: maybe write the metadata at deploy stage?19:22
radiofreeEntire chunk is the artifact, splitting should happen on that19:24
radiofreeI don't see why the manifest needs to be in the artifact19:24
paulsherwoodi agree19:29
paulsherwoodthis can wait, enjoy your night19:29
*** gtristan has joined #baserock19:54
radiofreepaulsherwood: is there a default kbas server?21:07
* radiofree can come up with a quick example if there's cache for build essential21:07
paulsherwoodhttp://artifacts1.baserock.org:800021:32
radiofreeta21:35
radiofreepaulsherwood: does that build master?21:35
radiofreewell it wants to build stage1 binutils so i guess it does21:36
radiofrees/does/doesn't21:37
radiofreeagain with the latest tag, i think https://gitlab.com/jamesthomas/simpleproject/tree/master may have to wait until tomorrow21:37
radiofreekbas-url: 'http://artifacts1.baserock.org:8000' is all i need in my config?21:37
radiofreeunless you don't build 32-bit?21:38
* radiofree try's a 64-bit chroot21:42
radiofreeno cache in 64-bit either...21:55
radiofreei get a 500 error with http://artifacts1.baserock.org:8000/21:55
radiofreeso i guess it's not actually working21:55
* radiofree gives up for the evening21:56
radiofreehowever, there's a very simple set of rule fixes that should be able to save the project i'm working on quite a significant amount of deploy and testing time21:56
radiofreesimple by virtue of not having to extract such a fat system to a usb21:57
*** gtristan has quit IRC22:07

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!