| paulsherwood | wrt ybd cached artifact keys not being the same as morph's i think that does not matter... | 07:35 |
|---|---|---|
| paulsherwood | a) morph has gone through various iterations/flag-days of its algorithm anyway | 07:36 |
| paulsherwood | b) i expect that there are still holes | 07:36 |
| paulsherwood | c) it's only a cache. the tooling needs to be 100% able to reproduce a given artifact from scratch. caching is a convenience. | 07:37 |
| *** CTtpollard has joined #baserock | 07:37 | |
| paulsherwood | maybe ybd could be used for fast ci (and as cache-publication for ybd users) alongside what we have, and the current morph ci could be used for releases (and as cache-publicaton for morph users) | 07:43 |
| *** bashrc has joined #baserock | 08:01 | |
| *** mariaderidder has joined #baserock | 08:01 | |
| *** zoli__ has joined #baserock | 08:19 | |
| *** jonathanmaw has joined #baserock | 08:45 | |
| SotK | paulsherwood: its an extremely helpful convenience though :) | 08:53 |
| SotK | I think I'd like more frequent releases if we were to only publish morph artifacts at release time | 08:54 |
| SotK | Perhaps using ybd to get fast feedback + ybd artifacts alongside what we have, whilst keeping the current slow morph system running to provide artifacts for morph users too? | 08:55 |
| paulsherwood | fair enough | 08:57 |
| SotK | Did the work on a generic cache server ever get anywhere? | 08:57 |
| paulsherwood | it ended up in a twisty maze of 'what's the right way to do this' | 08:58 |
| SotK | ah :( | 08:59 |
| paulsherwood | but i'm hoping we can thrash that out here, today if folks are interested | 09:01 |
| paulsherwood | can't remember if i shared this here... | 09:01 |
| paulsherwood | http://i.imgur.com/2oapEDB.png | 09:01 |
| paulsherwood | win 12 | 09:01 |
| paulsherwood | ffs :) | 09:01 |
| *** ssam2 has joined #baserock | 09:07 | |
| *** ChanServ sets mode: +v ssam2 | 09:07 | |
| paulsherwood | ssam2: i guess you missed my comments above? https://irclogs.baserock.org/%23baserock.2015-08-25.log.html#t2015-08-25T07:35:38 | 09:20 |
| *** lmackenzie_2015 has joined #baserock | 09:26 | |
| ssam2 | paulsherwood: on the idea of having 2 different things doing CI.. the more the merrier, in my opinion | 09:36 |
| ssam2 | in the short term | 09:36 |
| ssam2 | we could get a YBD CI worker uploading artifacts to cache.baserock.org pretty quickly, if you don't mind it being a bit of a hack | 09:39 |
| ssam2 | I mean, it just needs to rsync the built artifacts into /home/cache/artifacts on cache.baserock.org with the right permissions, really | 09:39 |
| ssam2 | the existing morph-cache-server will happily serve them | 09:40 |
| paulsherwood | is rsync a one-shot operation? ie we'd run it after build finishes? or could it happen for each artifact? | 10:09 |
| ssam2 | rsync can do whatever you want | 10:10 |
| ssam2 | so could run it for each artifact | 10:11 |
| ssam2 | it would need some hook in YBD or whatever 'wraps' it inside the CI worker so that it knew when an artifact had been added | 10:11 |
| ssam2 | rsync is clever enough to not copy thing that are already on the server, but I think it would be inefficient to just rsync the entire artifact cache every time | 10:11 |
| ssam2 | might work though | 10:11 |
| ssam2 | *things | 10:12 |
| ssam2 | s/server/target/ | 10:12 |
| paulsherwood | i was hoping for an http server to accept post requests, but this is the maze i was in | 10:12 |
| paulsherwood | the use-cases i can think of are both local (n folks working on their machines, sharing a server) and internet | 10:13 |
| ssam2 | right. above, I was thinking just in turns of how we could get a YBD CI worker running at baserock.org this week | 10:13 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/cache.py#L114 | 10:13 |
| paulsherwood | ssam2: wouldn't it be 'trivial' to make m-c-s support posts from 'trusted' machines? | 10:14 |
| ssam2 | I think so, yeah | 10:14 |
| paulsherwood | and is it 'trivial' to update m-c-s on g.b.o? | 10:15 |
| ssam2 | if by "trusted" you mean "on the same private network" | 10:15 |
| paulsherwood | yup | 10:15 |
| ssam2 | git.baserock.org or cache.baserock.org ? | 10:15 |
| paulsherwood | although, if it's the same, it won't be aws | 10:15 |
| paulsherwood | what's the distinction? | 10:15 |
| ssam2 | they are two different machines | 10:15 |
| ssam2 | and two different hostnames | 10:15 |
| * paulsherwood thought mcs ran on g.b.o | 10:15 | |
| ssam2 | it runs on both | 10:15 |
| ssam2 | so the answer to both quiest | 10:15 |
| ssam2 | I mean, updating morph-cache-server on either is the same process, I just want to make clear that we still have this situation where we have two caches | 10:16 |
| paulsherwood | oh, so on gbo mcs serves the trees and bundles, and on cbo it serves artifacts? | 10:16 |
| ssam2 | yeah | 10:16 |
| paulsherwood | no artfiacts on gbo? | 10:16 |
| ssam2 | we put artifacts for tagged release on git.baserock.org | 10:17 |
| ssam2 | manually | 10:17 |
| ssam2 | and morph still defaults to fetching artifacts from cache.baserock.org | 10:17 |
| ssam2 | I think the main reason is lack of an easy way to clean up cache.baserock.org while preserving the release artifacts | 10:17 |
| ssam2 | if we moved all artifacts to cache.baserock.org, I mean | 10:17 |
| ssam2 | The way to solve this is definitely to put all our artifacts in cache.baserock.org and not serve any artifacts from git.baserock.org | 10:18 |
| paulsherwood | right, modulo running out of space on cbo | 10:18 |
| SotK | ssam2: I wonder if the work you did with storing information about artifacts would be useful for preserving release artifacts | 10:18 |
| ssam2 | oh, the database thing? yeah, probably | 10:19 |
| SotK | i.e. You give the artifact to m-c-s, with some metadata such as "release: true, hash: 1230981" | 10:19 |
| * SotK envisions the "generic cache server" doing something like that | 10:20 | |
| ssam2 | makes sense. the artifact .meta file itself could contain that, even | 10:20 |
| SotK | sounds good | 10:23 |
| paulsherwood | maybe... but that means we have first-class citizen artifacts vs the rest? and we can still fill up with those... | 10:26 |
| paulsherwood | you may be right, though | 10:26 |
| SotK | we can just remove the oldest non-release artifacts to clear space, like we do now | 10:26 |
| paulsherwood | i did wonder about just throwing everything into S3, but it might get expensive | 10:26 |
| ssam2 | state of the art cleanup technology at the moment is something like `ls -1rt | head -n 1000 | xargs rm` | 10:33 |
| ssam2 | i mean, it's a cache, so we don't need to be too clever | 10:33 |
| ssam2 | I guess another option (probably not the best) is to tell the CI to build specific tags | 10:34 |
| paulsherwood | doesn't morph gc do something a lot smarter? | 10:34 |
| ssam2 | so if we delete the release artifacts, the CI just builds them again | 10:34 |
| ssam2 | I forget how `morph gc` works exactly | 10:34 |
| ssam2 | I think it's based on atime, so it's not that much smarter | 10:34 |
| ssam2 | or mtime, or something | 10:34 |
| paulsherwood | ssam2: in other news, jjardon pointed out that autotools projects may have a bootstrap script... https://github.com/jjardon/ybd/commit/7073fbf5f3e16c70681603867ebe0ea929452fe7 | 10:35 |
| paulsherwood | would it make sense to add that to defaults? | 10:35 |
| ssam2 | makes sense to me | 10:36 |
| ssam2 | let's do it after https://gerrit.baserock.org/#/c/1012/ (adding migration for version 7) is done though | 10:37 |
| ssam2 | I think it'll be helpful if the initial DEFAULTS file is exactly like the old buildsystem.py file from Morph and YBD | 10:37 |
| paulsherwood | ok | 10:38 |
| paulsherwood | (except ybd has indicators) | 10:38 |
| paulsherwood | (and has that autogen mod already) :) | 10:38 |
| * paulsherwood has +1 | 10:38 | |
| ssam2 | ta! I want to do a bit more testing before merge, but should be done today | 10:39 |
| *** wschaller has joined #baserock | 10:41 | |
| *** franred has joined #baserock | 10:44 | |
| paulsherwood | ssam2: it now has two +1 | 11:38 |
| *** paulw has joined #baserock | 12:12 | |
| *** wschaller has quit IRC | 13:27 | |
| *** zoli__ has quit IRC | 13:34 | |
| *** mdunford has quit IRC | 14:50 | |
| *** mdunford has joined #baserock | 15:05 | |
| *** mdunford has quit IRC | 15:10 | |
| *** ssam2 has quit IRC | 15:21 | |
| *** mdunford has joined #baserock | 15:23 | |
| *** ssam2 has joined #baserock | 15:23 | |
| *** ChanServ sets mode: +v ssam2 | 15:23 | |
| *** mariaderidder has quit IRC | 15:53 | |
| paulsherwood | bashrc: can you remember where your last firehose work got to? | 15:58 |
| paulsherwood | i think tlsa may be intertested :) | 15:59 |
| tlsa | I've discussed it with him :) | 16:01 |
| bashrc | paulsherwood: it was all submitted on gerrit, and I think it was eventually merged | 16:01 |
| bashrc | I had implemented the various operation modes, but they needed more testing | 16:01 |
| *** mdunford has quit IRC | 16:02 | |
| bashrc | http://wiki.baserock.org/firehose-notes/#index8h2 | 16:03 |
| *** mariaderidder has joined #baserock | 16:07 | |
| paulsherwood | thanks folks :) | 16:15 |
| *** paulw has quit IRC | 16:17 | |
| *** mdunford has joined #baserock | 16:21 | |
| *** mdunford has quit IRC | 16:25 | |
| *** mdunford has joined #baserock | 16:26 | |
| *** mdunford has quit IRC | 16:38 | |
| ssam2 | CI is failing, this patch should fix it: | 16:42 |
| ssam2 | https://gerrit.baserock.org/#/c/1084/ | 16:42 |
| ssam2 | jjardon it's your fault :-) | 16:43 |
| ssam2 | I'm also hitting a build failure in 'bison': configure.ac:26: error: version 'UNKNOWN' doesn't follow Gnits standards | 16:43 |
| *** jonathanmaw has quit IRC | 16:44 | |
| jjardon | ssam2: it worked in my laptop! :) | 16:45 |
| ssam2 | deploying the whole ci.morph ? weird | 16:45 |
| jjardon | bison? I think bison is being build in the ci for a while | 16:45 |
| jjardon | I used the same as the xfce cluster, didnt know the ci needs a special location, sorry | 16:46 |
| ssam2 | ah, the error was from scripts/release-build in definitions.git | 16:46 |
| ssam2 | that's why on your laptop it was fine | 16:46 |
| tlsa | oh, there's a 5th x86-64 system added to ci.morph? The mason distbuild really needs to be fixed not to do everything 5 times then, cos its already slow | 16:47 |
| ssam2 | indeed | 16:47 |
| jjardon | ssam2: you need git yo build bison, maybe thats the problem? | 16:47 |
| ssam2 | jjardon: i thought that, but git seems to be a build-dep of bison | 16:47 |
| ssam2 | it also used to have bash as a dep | 16:48 |
| ssam2 | i'll try adding that | 16:48 |
| *** mariaderidder has quit IRC | 16:49 | |
| ssam2 | nope | 16:51 |
| *** mdunford has joined #baserock | 16:52 | |
| ssam2 | ah.. the --no-git blag to bison 'bootstrap' might not help | 16:59 |
| ssam2 | *flag | 16:59 |
| *** bashrc has quit IRC | 17:01 | |
| ssam2 | nope, not that either | 17:01 |
| *** mdunford has quit IRC | 17:19 | |
| *** mdunford has joined #baserock | 17:30 | |
| *** lmackenzie_2015 has quit IRC | 17:47 | |
| *** ssam2 has quit IRC | 17:51 | |
| *** mdunford has quit IRC | 18:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
