IRC logs for #baserock for Tuesday, 2015-08-25

paulsherwoodwrt ybd cached artifact keys not being the same as morph's i think that does not matter...07:35
paulsherwooda) morph has gone through various iterations/flag-days of its algorithm anyway07:36
paulsherwoodb) i expect that there are still holes07:36
paulsherwoodc) it's only a cache. the tooling needs to be 100% able to reproduce a given artifact from scratch. caching is a convenience.07:37
*** CTtpollard has joined #baserock07:37
paulsherwoodmaybe ybd could be used for fast ci (and as cache-publication for ybd users) alongside what we have, and the current morph ci could be used for releases (and as cache-publicaton for morph users)07:43
*** bashrc has joined #baserock08:01
*** mariaderidder has joined #baserock08:01
*** zoli__ has joined #baserock08:19
*** jonathanmaw has joined #baserock08:45
SotKpaulsherwood: its an extremely helpful convenience though :)08:53
SotKI think I'd like more frequent releases if we were to only publish morph artifacts at release time08:54
SotKPerhaps using ybd to get fast feedback + ybd artifacts alongside what we have, whilst keeping the current slow morph system running to provide artifacts for morph users too?08:55
paulsherwoodfair enough08:57
SotKDid the work on a generic cache server ever get anywhere?08:57
paulsherwoodit ended up in a twisty maze of 'what's the right way to do this'08:58
SotKah :(08:59
paulsherwoodbut i'm hoping we can thrash that out here, today if folks are interested09:01
paulsherwoodcan't remember if i shared this here...09:01
paulsherwoodhttp://i.imgur.com/2oapEDB.png09:01
paulsherwoodwin 1209:01
paulsherwoodffs :)09:01
*** ssam2 has joined #baserock09:07
*** ChanServ sets mode: +v ssam209:07
paulsherwoodssam2: i guess you missed my comments above? https://irclogs.baserock.org/%23baserock.2015-08-25.log.html#t2015-08-25T07:35:3809:20
*** lmackenzie_2015 has joined #baserock09:26
ssam2paulsherwood: on the idea of having 2 different things doing CI.. the more the merrier, in my opinion09:36
ssam2in the short term09:36
ssam2we could get a YBD CI worker uploading artifacts to cache.baserock.org pretty quickly, if you don't mind it being a bit of a hack09:39
ssam2I mean, it just needs to rsync the built artifacts into /home/cache/artifacts on cache.baserock.org with the right permissions, really09:39
ssam2the existing morph-cache-server will happily serve them09:40
paulsherwoodis rsync a one-shot operation? ie we'd run it after build finishes? or could it happen for each artifact?10:09
ssam2rsync can do whatever you want10:10
ssam2so could run it for each artifact10:11
ssam2it would need some hook in YBD or whatever 'wraps' it inside the CI worker so that it knew when an artifact had been added10:11
ssam2rsync is clever enough to not copy thing that are already on the server, but I think it would be inefficient to just rsync the entire artifact cache every time10:11
ssam2might work though10:11
ssam2*things10:12
ssam2s/server/target/10:12
paulsherwoodi was hoping for an http server to accept post requests, but this is the maze i was in10:12
paulsherwoodthe use-cases i can think of are both local (n folks working on their machines, sharing a server) and internet10:13
ssam2right. above, I was thinking just in turns of how we could get a YBD CI worker running at baserock.org this week10:13
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/cache.py#L11410:13
paulsherwoodssam2: wouldn't it be 'trivial' to make m-c-s support posts from 'trusted' machines?10:14
ssam2I think so, yeah10:14
paulsherwoodand is it 'trivial' to update m-c-s on g.b.o?10:15
ssam2if by "trusted" you mean "on the same private network"10:15
paulsherwoodyup10:15
ssam2git.baserock.org or cache.baserock.org ?10:15
paulsherwoodalthough, if it's the same, it won't be  aws10:15
paulsherwoodwhat's the distinction?10:15
ssam2they are two different machines10:15
ssam2and two different hostnames10:15
* paulsherwood thought mcs ran on g.b.o10:15
ssam2it runs on both10:15
ssam2so the answer to both quiest10:15
ssam2I mean, updating morph-cache-server on either is the same process, I just want to make clear that we still have this situation where we have two caches10:16
paulsherwoodoh, so on gbo mcs serves the trees and bundles, and on cbo it serves artifacts?10:16
ssam2yeah10:16
paulsherwoodno artfiacts on gbo?10:16
ssam2we put artifacts for tagged release on git.baserock.org10:17
ssam2manually10:17
ssam2and morph still defaults to fetching artifacts from cache.baserock.org10:17
ssam2I think the main reason is lack of an easy way to clean up cache.baserock.org while preserving the release artifacts10:17
ssam2if we moved all artifacts to cache.baserock.org, I mean10:17
ssam2The way to solve this is definitely to put all our artifacts in cache.baserock.org and not serve any artifacts from git.baserock.org10:18
paulsherwoodright, modulo running out of space on cbo10:18
SotKssam2: I wonder if the work you did with storing information about artifacts would be useful for preserving release artifacts10:18
ssam2oh, the database thing? yeah, probably10:19
SotKi.e. You give the artifact to m-c-s, with some metadata such as "release: true, hash: 1230981"10:19
* SotK envisions the "generic cache server" doing something like that10:20
ssam2makes sense. the artifact .meta file itself could contain that, even10:20
SotKsounds good10:23
paulsherwoodmaybe... but that means we have first-class citizen artifacts vs the rest? and we can still fill up with those...10:26
paulsherwoodyou may be right, though10:26
SotKwe can just remove the oldest non-release artifacts to clear space, like we do now10:26
paulsherwoodi did wonder about just throwing everything into S3, but it might get expensive10:26
ssam2state of the art cleanup technology at the moment is something like `ls -1rt | head -n 1000 | xargs rm`10:33
ssam2i mean, it's a cache, so we don't need to be too clever10:33
ssam2I guess another option (probably not the best) is to tell the CI to build specific tags10:34
paulsherwooddoesn't morph gc do something a lot smarter?10:34
ssam2so if we delete the release artifacts, the CI just builds them again10:34
ssam2I forget how `morph gc` works exactly10:34
ssam2I think it's based on atime, so it's not that much smarter10:34
ssam2or mtime, or something10:34
paulsherwoodssam2: in other news, jjardon pointed out that autotools projects may have a bootstrap script... https://github.com/jjardon/ybd/commit/7073fbf5f3e16c70681603867ebe0ea929452fe710:35
paulsherwoodwould it make sense to add that to defaults?10:35
ssam2makes sense to me10:36
ssam2let's do it after https://gerrit.baserock.org/#/c/1012/ (adding migration for version 7) is done though10:37
ssam2I think it'll be helpful if the initial DEFAULTS file is exactly like the old buildsystem.py file from Morph and YBD10:37
paulsherwoodok10:38
paulsherwood(except ybd has indicators)10:38
paulsherwood(and has that autogen mod already) :)10:38
* paulsherwood has +110:38
ssam2ta! I want to do a bit more testing before merge, but should be done today10:39
*** wschaller has joined #baserock10:41
*** franred has joined #baserock10:44
paulsherwoodssam2: it now has two +111:38
*** paulw has joined #baserock12:12
*** wschaller has quit IRC13:27
*** zoli__ has quit IRC13:34
*** mdunford has quit IRC14:50
*** mdunford has joined #baserock15:05
*** mdunford has quit IRC15:10
*** ssam2 has quit IRC15:21
*** mdunford has joined #baserock15:23
*** ssam2 has joined #baserock15:23
*** ChanServ sets mode: +v ssam215:23
*** mariaderidder has quit IRC15:53
paulsherwoodbashrc: can you remember where your last firehose work got to?15:58
paulsherwoodi think tlsa may be intertested :)15:59
tlsaI've discussed it with him :)16:01
bashrcpaulsherwood: it was all submitted on gerrit, and I think it was eventually merged16:01
bashrcI had implemented the various operation modes, but they needed more testing16:01
*** mdunford has quit IRC16:02
bashrchttp://wiki.baserock.org/firehose-notes/#index8h216:03
*** mariaderidder has joined #baserock16:07
paulsherwoodthanks folks :)16:15
*** paulw has quit IRC16:17
*** mdunford has joined #baserock16:21
*** mdunford has quit IRC16:25
*** mdunford has joined #baserock16:26
*** mdunford has quit IRC16:38
ssam2CI is failing, this patch should fix it:16:42
ssam2https://gerrit.baserock.org/#/c/1084/16:42
ssam2jjardon it's your fault :-)16:43
ssam2I'm also hitting a build failure in 'bison': configure.ac:26: error: version 'UNKNOWN' doesn't follow Gnits standards16:43
*** jonathanmaw has quit IRC16:44
jjardonssam2: it worked in my laptop! :)16:45
ssam2deploying the whole ci.morph ? weird16:45
jjardonbison? I think bison is being build in the ci for a while16:45
jjardonI used the same as the xfce cluster, didnt know the ci needs a special location, sorry16:46
ssam2ah, the error was from scripts/release-build in definitions.git16:46
ssam2that's why on your laptop it was fine16:46
tlsaoh, there's a 5th x86-64 system added to ci.morph?  The mason distbuild really needs to be fixed not to do everything 5 times then, cos its already slow16:47
ssam2indeed16:47
jjardonssam2: you need git  yo build bison, maybe thats the problem?16:47
ssam2jjardon: i thought that, but git seems to be a build-dep of bison16:47
ssam2it also used to have bash as a dep16:48
ssam2i'll try adding that16:48
*** mariaderidder has quit IRC16:49
ssam2nope16:51
*** mdunford has joined #baserock16:52
ssam2ah.. the --no-git blag to bison 'bootstrap' might not help16:59
ssam2*flag16:59
*** bashrc has quit IRC17:01
ssam2nope, not that either17:01
*** mdunford has quit IRC17:19
*** mdunford has joined #baserock17:30
*** lmackenzie_2015 has quit IRC17:47
*** ssam2 has quit IRC17:51
*** mdunford has quit IRC18:35

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!