IRC logs for #baserock for Tuesday, 2016-01-19

*** gtristan has quit IRC03:38
*** gtristan has joined #baserock04:10
*** toscalix has joined #baserock08:22
*** rdale has joined #baserock08:29
*** ctbruce has joined #baserock08:49
*** gtristan has quit IRC08:51
*** bashrc has joined #baserock09:05
*** edcragg has joined #baserock09:10
*** CTtpollard has quit IRC09:27
*** jonathanmaw has joined #baserock09:27
*** gtristan has joined #baserock09:35
*** gtristan has quit IRC09:35
*** gtristan_ has joined #baserock09:48
*** CTtpollard has joined #baserock09:57
*** ctbruce has quit IRC10:03
*** ctbruce has joined #baserock10:03
*** ssam2 has joined #baserock10:06
*** ChanServ sets mode: +v ssam210:06
*** ssam2 has quit IRC10:13
*** gtristan_ has quit IRC10:13
*** gtristan_ has joined #baserock10:14
*** gtristan_ is now known as gtristan10:14
paulsherwoodnot much, i guess :)10:21
mwilliams_cthi! I'm currently working on trying to use concourse to build a base-system. One of the problems we've had is that concourse only accepts branches as a reference to where work is, not tags or specific commits10:25
mwilliams_ctI tried to approximate this by using unpetrify-ref to equal branches, but this has failed as some chunks use tags as their unpetrify-ref10:25
*** ssam2 has joined #baserock10:25
*** ChanServ sets mode: +v ssam210:25
mwilliams_ctwould there be interest here in making a branch for all currently tagged only chunks? My suspicion is not as it may not be practical to branch for everything gbo hosts10:26
radiofreefix concourse to use sha's/tags would be the better option10:26
paulsherwooddo you have any idea how many unpetrify-refs are affected in the whole of master definitions?10:26
paulsherwood+1 to radiofree's suggestion, if concourse upstream will take it. seems like a bug there10:27
mwilliams_ctthere are 15 in base-system alone so I'm guessing it's a large number though I dont know the exact figure10:27
mwilliams_ctI agree, we should try and get this upstream to concourse10:27
*** Lachlan1975 has joined #baserock10:29
pedroalvarezconcourse maintainers may think: Why on earth do you want to track a sha1 on a CI pipeline? :)11:00
mwilliams_ctpedroalvarez: I've phrased it to them as that unlike smaller projects, always tracking HEAD can cause things to break quite easily11:01
pedroalvarezthing is.. if you have a pipeline configured to track a (or multiple) sha1, the pipeline is not going to be triggered ever by them.11:02
pedroalvarezso, why would you?11:02
pedroalvarezs/by them/by changes in them/11:03
pedroalvarezI guess something in the pipeline tracks a real branch of definitions. In that case, is the concourse pipeline going to be re-configured for every change on this branch?11:04
mwilliams_ctpedroalvarez: you saw what I had yesterday right, the pipeline for base-system? The idea will be to have definitions feeding (with trigger: true) into a first job "parse-definitions" which will generate and deploy a pipeline11:06
pedroalvarezmakes sense11:07
paulsherwoodpedroalvarez: 'the pipeline is not going to be triggered ever by them.' no. but things (dependencies) before them in the pipeline wil cause them to rebuild11:15
mwilliams_ctpaulsherwood: what do you mean by "things (dependencies)" and "them"?11:16
mwilliams_ct(in Baserock terms)11:16
pedroalvarezI think they are concourse terms11:17
paulsherwoodchanges in, say, build-essential will cause foo to be rebuilt even if foo is locked to a tag11:18
pedroalvarezand is exactly what you have said, and what I have guessed11:18
pedroalvarezaah.. no, I was wrong then11:18
paulsherwoodi was trying to clarify vs 11:02 <+pedroalvarez> thing is.. if you have a pipeline configured to track a (or multiple) sha1, the pipeline is not going to be triggered ever by them.11:18
paulsherwoodi don't know if i've achieved that, though :)11:19
pedroalvarezyes and no. This is a very complex problem, and it's going to be full of implementation details11:19
paulsherwoodpedroalvarez: i heard elsewhere that storyboard kanban and email notifications are now working... is s.b.o updated with that functionality?11:27
pedroalvarezpaulsherwood: of course :)11:28
paulsherwoodoh, ok! so if i login i should be able to see it working?11:29
paulsherwoodpedroalvarez: maybe worth announcing this onlist?11:29
pedroalvarezThat's a good idea11:31
jjardonHi, can I have another review of to save rebuilding the whole gnome stratum every time something changes in the x-generic stratum?11:31
paulsherwoodpedroalvarez: lovely new colours :)11:32
pedroalvarezpaulsherwood: I was going crazy when using openstack's and baserock's at the same time.. a different color was mandatory.. :)11:33
CTtpollardcd ..11:34
Zarapedroalvarez: hahaha, that's why my dev instance is yellow! it got bad when I had four different instances open, all the same colour (mine, SotK's, s.o.o and baserock's)...11:35
radiofreeis there a "clean up unpacked folders" after build option for ybd?11:44
* radiofree only has a 50G ssd and the unpacked folders take up quite a bit of space11:45
paulsherwoodradiofree: rm -fr $artifacts/*/*-unpacked ?11:59
radiofreei have a script to do it but i was wondering if there was anything i could specify in the config12:00
paulsherwoodi'd be delighted to accept a patch, or an issue :)12:07
* radiofree might also patch checking the .unpacked folder to make sure everything in the manifest is actually in there before using it12:08
*** edcragg has quit IRC12:33
pedroalvarezright, StoryBoard annouced  in baserock-dev :)13:01
*** edcragg has joined #baserock13:08
paulsherwoodpedroalvarez: please can i be subscribed to that board?13:17
pedroalvarezpaulsherwood: sure, you should be able to see it now in your Boards Dashboard:!/dashboard/boards13:20
paulsherwoodpedroalvarez: thanks!13:21
paulsherwoodinteresting... so "morph: allow to not have build-depends" is in 'bugs', yet merged13:22
pedroalvarezpaulsherwood: I just populated the board a bit with random tasks before sending the email13:23
paulsherwoodaha :)13:23
paulsherwoodmakes sense now13:23
pedroalvarezI noticed when doing so that our storyboard could do with a bit of gardening13:24
paulsherwoodmaybe with the boards, we can do that better/easier13:25
paulsherwoodrdale: i've been wondering about this splitting thing, wrt systems and strata13:36
paulsherwoodcurrently ybd doesn't actually create a stratum artifact, because it's just a set of chunk artifacts untarred13:36
paulsherwoodi wonder if we could *only* do splitting at deploy time?13:37
paulsherwoodand/or system creation time13:37
paulsherwoodie not bother doing anything for indivual strata?13:38
rdaleno, i think we should do a copy of the stratum as it is now but filter the built chunks according to the splitting rules, and then tar up the filtered version of the stratum13:39
paulsherwoodoh ok, am i misremembering? does ybd actually create stratum artifacts?13:39
rdalestep 1: create the stratum as now, step2: generate a set of meta file in the baserock dir of the stratum with a sub set of the chunk's artifacts, step 3: copy the stratum according to the lists of files for the chunks from the .meta files13:40
rdalei think ybd creates stratum artifacts, as they get cached13:41
rdaleso i think the chunks should be filtered via the splitting rules before caching - i don't like the idea of doing everytime you deploy13:42
paulsherwoodok i'm interested to see it working :)13:43
rdalehere is a .meta file example:
rdalenot a good one as it's from stage1, so there is only one big list of files for the '-misc' artifact13:47
rdalewhen a file like that is copied to the /baserock dir for a stratum, the products whose artifact name don't match the splitting rules are dropped13:48
*** jonathanmaw_ has joined #baserock13:56
*** jonathanmaw has quit IRC14:00
*** ssam2 has quit IRC14:13
*** ssam2 has joined #baserock14:20
*** ChanServ sets mode: +v ssam214:20
*** jonathanmaw has joined #baserock14:21
*** jonathanmaw_ has quit IRC14:25
jjardonmason is failing: 2016-01-19 11:11:45 Progress: Transferring linux-x86-64-generic-misc to shared artifact cache14:30
jjardon2016-01-19 11:11:56 Build of linux-x86-64-generic failed.14:30
jjardonout of space again?14:30
*** gtristan has quit IRC14:45
*** ctbruce has quit IRC14:49
ssam2maybe, thanks for the heads up14:53
pedroalvarezI've already created some space14:55
pedroalvarezcome on jjardon, I put a PROGRESS row in mason so that you can see what's going on right now :)14:55
jjardoncan we have a timer that removes the oldest artifacts but the ones from releases, so we avoid this problem?14:59
ssam2one day15:01
ssam2it's not trivial, i did it once but it didn't get merged for some reason15:01
ssam2i forget the details15:01
pedroalvarezremoving chunk artifacts was causing some build fails, because the build tool was assuming that the existence of the stratum artifact meant all the chunk artifacts of that stratum existed15:04
pedroalvarezwell, I think that was one of the reasons15:04
paulsherwoodybd copes with that :)15:04
paulsherwoodand culls old artifacts :)15:05
mwilliams_cthistoric question: at some point was unpetrify-ref required to be a branch?15:05
mwilliams_ct(a propos of nothing, was just curious)15:05
paulsherwoodi think by convention it was originally, but users started setting tags because it made more sense in many cases15:06
mwilliams_ctthat makes sense. lots of "baserock/morph" branches that look like they were older tags15:06
jjardonmwilliams_ct: I think so, yes15:06
*** ratmice______ has quit IRC15:07
*** ratmice______ has joined #baserock15:07
jjardonmwilliams_ct: morph files used to be in the chunk repo, so a branch was needed15:07
mwilliams_ctjjardon: oh, did not know that. thanks for the info15:10
tiagogomeswe also use a baserock/morph branch when we needed to have some delta against upstream15:13
pedroalvarezs/use/used/ .. because now we prefer something like baserock/{{ nearest-tag || branch-name || something-else}}15:20
ssam2baserock/upstream-version+description-of-what-the-change-was    (which is *infinitely* better than naming everything baserock/morph !!)15:21
pedroalvarezthat makes unpetrify-ref less useless15:22
paulsherwoodpedroalvarez: actually, i believe mwilliams_ct and others are using it :)15:23
pedroalvarezgood it's still present :)15:24
jjardonnot sure they should: I've been told several times that parameter is only there for information purposes; also, there is not any mechanism to check the "ref:" and "unpetrify-ref:" are in sync15:29
paulsherwoodjjardon: agreed. i believe they are only experimenting at this point15:30
mwilliams_ctjjardon: I discussed this a bit earlier. The problem is that Concourse doesnt support SHAs (or tags). I've spoken to their upstream about this (not had a reply due to timezones) and benbrown_ is looking into changes. for now, unpetrify-ref as an approximation for branch does what I need15:31
mwilliams_ctas an aside, my deployment of concourse is giving me 500s on every resource it tries to fetch. have I upset g.b.o by trying to pull too much too quickly?15:33
pedroalvarezwhat if the build tool could build from unpetrify-ref if ref not present, and we had a command that would generate the ref field?15:34
radiofreewhy do we need that15:35
radiofreeit would be more useful to re-add "petrify-ref"15:35
pedroalvarezyou mean the `morph petrify` subcommand, right?15:35
radiofreethat's the one15:36
pedroalvarezwell, haven't used it in a while, but the petrify command will need some improvements to make it look right15:38
pedroalvarezisn't it going to move ref fields to unpetrify-refs, and then resolve the commit of whatever is in ref and rewrite ref?15:39
pedroalvarezputting a lot of sha1s in unpetrify-refs...15:39
pedroalvarezhas anyone used it recently?15:39
radiofreei will need to at some point15:40
paulsherwoodradiofree: why?15:40
radiofreei'm not going to manually change all the qt5 components from "ref: v5.5.1" to "ref: sha" manually15:40
* paulsherwood still, after all these years, is not clear about the need for petrify15:40
pedroalvarezI normally try to use sha1s directly when doing integration work, but I understand some people prefer not to15:41
paulsherwoodbut why do you/we need to do that, at all?15:41
pedroalvarezI guess that: it's easier; tags are for humans, sha1s for machines15:41
jjardonpaulsherwood: you write in your definitions: I want to build tag "5.5.1" morph petrify will convert that tag in the corresponding sha and put the result in the "ref:" parameter15:42
radiofreepedroalvarez: paulsherwood: if you'd like to do that for to point the ref at the v5.5.1 please feel free to do so15:42
radiofreeto upgrade to qt 5.5.1, i change the ref to the v5.5.1 tag, build, test, fix, build, test fix, to submit the patches i need to change that ref to a sha15:42
paulsherwoodno, i mean why do we need to force the ref: to be a sha... don't builds log the refs, and stick them in the .meta files? isn't that enough?15:43
jjardonthe reason why baserock doesn't use tags directly is because reproducibility problems15:43
radiofreei'll use sams branch to do that, which takes seconds instead of god-knows how long to manually go through a dozen repos to manually get the sha15:43
jjardon(I've been told)15:44
radiofreepaulsherwood: i'd be happy to leave it as a tag15:45
radiofreebut i'm unsure about the default behaviour of trove/lorry15:45
radiofreeimo it should never replace branches/tags that have been force pushed15:45
radiofreeit should put them in a queue somewhere for human input ("what would you like to do here?")...15:45
*** gtristan has joined #baserock15:47
pedroalvarezwe are having a lot of performance issues with :/15:47
pedroalvarezfor example, the lc queue ( shouldn't be full of "now"s15:48
radiofreeit was really slow this morning15:48
pedroalvarezit's using 15G out of 15G of ram15:54
pedroalvarezlighttpd process on 100% of cpu15:55
pedroalvarezvarious cgit.cgi processes... :/15:56
*** ctbruce has joined #baserock15:58
pedroalvarezI feel like this is going to be the host provider, but haven't seen any performance issues in other VMs15:59
mwilliams_ctbenbrown_ and I have noticed something interesting on gbo. seems that using the --branch <tag> command over http doesn't work whereas calling using git:// works fine. not a massive deal, I'll just change protocols, but just wondered why16:15
mwilliams_ct(to clone a repository at a given tag)16:16
franredmwilliams_ct, and https?16:19
pedroalvarezfranred: https needs authentication16:19
mwilliams_ctfranred: not tested. not for the first time today, I think the bug is actually in my code anyway16:19
mwilliams_ctstand down sysadmins, just mwilliams messing up again16:20
CTtpollardI don't think it's a specific g.b.o issue16:21
*** ssam2 has quit IRC17:59
*** bashrc has quit IRC18:02
*** jonathanmaw has quit IRC18:14
*** edcragg has quit IRC18:18
*** Lachlan1975 has quit IRC18:35
*** ctbruce has quit IRC22:00
*** edcragg has joined #baserock22:41
*** toscalix has quit IRC22:54

Generated by 2.15.3 by Marius Gedminas - find it at!