| *** locallycompact has quit IRC | 00:18 | |
| *** rdale has quit IRC | 00:37 | |
| *** gtristan has quit IRC | 00:57 | |
| *** gtristan has joined #baserock | 01:15 | |
| *** gtristan has quit IRC | 02:53 | |
| *** gtristan has joined #baserock | 03:04 | |
| *** paulwaters_ has joined #baserock | 06:40 | |
| *** fay_ has joined #baserock | 07:12 | |
| *** toscalix has joined #baserock | 07:54 | |
| *** anahuelamo has joined #baserock | 08:01 | |
| *** anahuelamo_ has joined #baserock | 08:05 | |
| *** anahuelamo_ has joined #baserock | 08:06 | |
| *** anahuelamo has quit IRC | 08:06 | |
| *** anahuelamo has joined #baserock | 08:08 | |
| *** locallycompact has joined #baserock | 08:39 | |
| *** JPohlman1 has joined #baserock | 08:44 | |
| *** JPohlmann has quit IRC | 08:45 | |
| *** thrace has quit IRC | 08:46 | |
| *** persia_ has quit IRC | 08:46 | |
| *** wdutch has quit IRC | 08:46 | |
| *** persia_ has joined #baserock | 08:46 | |
| *** thrace has joined #baserock | 08:46 | |
| *** wdutch has joined #baserock | 08:46 | |
| *** rdale has joined #baserock | 08:47 | |
| *** lachlanmackenzie has quit IRC | 08:48 | |
| *** edcragg has joined #baserock | 08:49 | |
| *** gtristan has quit IRC | 08:54 | |
| *** ctbruce has joined #baserock | 08:54 | |
| *** franred has quit IRC | 08:56 | |
| *** lachlanmackenzie has joined #baserock | 08:57 | |
| *** franred has joined #baserock | 08:57 | |
| *** ssam2 has joined #baserock | 09:20 | |
| *** ChanServ sets mode: +v ssam2 | 09:20 | |
| *** gtristan has joined #baserock | 09:52 | |
| *** franred has quit IRC | 10:52 | |
| *** franred has joined #baserock | 11:01 | |
| *** anahuelamo has quit IRC | 11:06 | |
| *** locallycompact has quit IRC | 11:39 | |
| *** anahuelamo has joined #baserock | 11:48 | |
| *** anahuelamo has quit IRC | 11:50 | |
| *** anahuelamo has joined #baserock | 13:46 | |
| *** leeming has joined #baserock | 13:49 | |
| *** CTtpollard has quit IRC | 14:09 | |
| *** CTtpollard has joined #baserock | 14:12 | |
| *** gtristan has quit IRC | 15:12 | |
| *** gtristan has joined #baserock | 15:33 | |
| *** pedroalvarez has quit IRC | 15:39 | |
| *** pedroalvarez has joined #baserock | 15:39 | |
| *** ChanServ sets mode: +v pedroalvarez | 15:39 | |
| leeming | some basic questions about ybd. 1) do cached artefacts age on trove? 2) If you build an artefact, do you always upload back to trove (or which ever cache server your using)?, 3) are artefacts verified when put into the cache? | 15:57 |
|---|---|---|
| * leeming might not fully understand. feel free to point out any flaws to my logic | 15:58 | |
| *** locallycompact has joined #baserock | 16:02 | |
| edcragg | cache artifacts are identified by their cache key | 16:05 |
| edcragg | if anything significant changes about the build the cache key is likely to change | 16:05 |
| pedroalvarez | leeming: Answering questions for ybd: | 16:06 |
| pedroalvarez | 1) I don't quite understand that question | 16:06 |
| pedroalvarez | 2) with ybd you can configure it to upload the artefacts to a shared cache server, so not always | 16:06 |
| pedroalvarez | 3) ybd's cache server does some sort of verification, ensuring that the compressed file is not corrupt (tar -t $file) | 16:06 |
| leeming | edcragg, cache key is essentially some hash of it's git commit and dependency versions right? | 16:07 |
| leeming | pedroalvarez, so it only checks for errors for the verification? what about the integrity? | 16:08 |
| leeming | also, re Q1 - I was asking as I expected something in my build to have an artefact already. Guessed some artefacts might be removed over time (removing very out of date builds/ other reasons) | 16:10 |
| *** anahuelamo has quit IRC | 16:10 | |
| *** anahuelamo has joined #baserock | 16:10 | |
| pedroalvarez | ybd is known for auto-removing things if you "don't have enough space" | 16:11 |
| pedroalvarez | but these artefacts would be in your filesystem, not in any cache server | 16:11 |
| *** toscalix has quit IRC | 16:11 | |
| leeming | ah, maybe a miss understanding of caches here.. I understand there is a cache server that holds many dependencies? | 16:12 |
| leeming | I have a local cache, yes understand that bit | 16:12 |
| pedroalvarez | there is one, yes (I believe: http://artifacts1.baserock.org:8000/) | 16:13 |
| edcragg | leeming, the algorithm depends on a number of things, but generally if you change the definitions for a chunk in any way it'll need to rebuild | 16:13 |
| leeming | am I mixing up things because I read about morph? :S | 16:13 |
| edcragg | holds many dependencies? | 16:14 |
| edcragg | the concepts are the same between the two, just reimplimented in ybd, in general | 16:14 |
| edcragg | sometimes in an incompatible way, so the cached artifact format is different and not compatible between morph and ybd, for example | 16:15 |
| leeming | ok, so it IS a recursive search of caches then? 1. check local copy on disk, 2. check remote cache, if not found, build | 16:15 |
| leeming | when built, generate cache key and push to remote? | 16:15 |
| pedroalvarez | only push if configured to push to remote | 16:17 |
| leeming | right, but my understanding is correct? | 16:17 |
| pedroalvarez | and "generate cache key" sounds confusing | 16:17 |
| leeming | pedroalvarez, isn't it simply a key similar to a hashmap? i.e. some hash function is applied, then that is used as a key | 16:18 |
| leeming | value being the artefact | 16:18 |
| edcragg | check local cache, if not there look in remote cache and download to local cache, otherwise build and put in local cache, maybe push to remote | 16:19 |
| leeming | edcragg, yes thanks. more verbose than my explanation :) copy to local was implicit | 16:20 |
| leeming | but other verification on push to remote (on the remote side) ? | 16:20 |
| pedroalvarez | nope | 16:21 |
| leeming | isn't that a security issue? I can go into details here... or directly | 16:22 |
| pedroalvarez | it can be, to upload artefacts you need a password though | 16:22 |
| pedroalvarez | so I'd assume that, whoever has the password, are trusted uploaders | 16:22 |
| leeming | ah ok, so only trusted uploads | 16:23 |
| pedroalvarez | maybe CI servers, or some sort | 16:23 |
| *** fay_ has quit IRC | 16:26 | |
| *** locallycompact has quit IRC | 16:34 | |
| *** ctbruce has quit IRC | 16:54 | |
| *** leeming has quit IRC | 17:07 | |
| *** anahuelamo_ has joined #baserock | 17:15 | |
| *** anahuelamo_ has quit IRC | 17:16 | |
| *** edcragg has quit IRC | 17:23 | |
| *** ssam2 has quit IRC | 17:49 | |
| *** gtristan has quit IRC | 18:10 | |
| *** locallycompact has joined #baserock | 18:11 | |
| * paulsherwood is sad to find that leeming asked all these questions, but is not still in channel when i can answer them | 19:21 | |
| paulsherwood | leeming: yes, i believe 'calculate cache_key' is a hashmap... the cache_key is unique for a given set of input 'factors' eg... | 19:24 |
| paulsherwood | git SHA of the version of the thing we are building, plus arch we are building for, plus the hashmaps of all th build-dependencies... (which are hashmaps recursively all the way down) | 19:25 |
| paulsherwood | that is all handled in hash_factors() https://github.com/devcurmudgeon/ybd/blob/master/ybd/cache.py#L75 | 19:26 |
| paulsherwood | 16:57 < leeming> some basic questions about ybd. 1) do cached artefacts age on trove? | 19:28 |
| paulsherwood | i think trove handles source repos, which are *inputs* to build/integration processes (eg morph, ybd) | 19:29 |
| paulsherwood | we normally use 'artifacts' to refer to built and/or integrated *outputs* of these processes | 19:29 |
| paulsherwood | i've not checked recently so maybe we are storing morph-generated artifacts on trove... but that is confusing imo | 19:31 |
| paulsherwood | ybd artifacts are stored locally in an artifact cache where ybd is run, and also can be uploaded to (or served from) a kbas server | 19:31 |
| paulsherwood | keyed binary artifact server | 19:32 |
| paulsherwood | no ybd artifacts are stored on a trove (unless we are running ybd or kbas on the trove machine) | 19:33 |
| paulsherwood | artifacts don't age on a kbas instance currently... ie once they're uploaded, they persist. | 19:33 |
| paulsherwood | however i consider that to be a bug ... because kbas servers will crash with ENOSPACE eventually | 19:35 |
| paulsherwood | hence https://github.com/devcurmudgeon/ybd/issues/172 | 19:35 |
| paulsherwood | ybd does remove oldest artifacts from its local cache store (config['artifacts']) if it needs to (less than config['min-gigabytes'] available) | 19:36 |
| paulsherwood | this has led to a surprising amount of emotional discussion - some folks hate that ybd deletes artifacts. conversely i hate that morph will sometimes fail to run/build because ENOSPACE | 19:38 |
| paulsherwood | 2) If you build an artefact, do you always upload | 19:38 |
| paulsherwood | if config['kbas-password'] is set, ybd will attempt to upload to config['kbas-url'] | 19:39 |
| paulsherwood | 3) are artefacts verified when put into the cache? | 19:40 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/kbas/__main__.py#L138 | 19:40 |
| paulsherwood | actually, let's consider the whole function: https://github.com/devcurmudgeon/ybd/blob/master/kbas/__main__.py#L138 | 19:41 |
| paulsherwood | a) it checks the provided password. but this is woefully poor, really... not even encrypted. i've asked for help to improve this a few times (hint) | 19:42 |
| paulsherwood | b) it checks we're not trying to upload to /etc/passwd or similar | 19:43 |
| paulsherwood | c) it checks we don't already have an artifact at this cache_key | 19:44 |
| paulsherwood | d) it checks the provided artifact is a valid tarball | 19:44 |
| paulsherwood | and that's about it. | 19:45 |
| paulsherwood | leeming: i'd be interested to hear from you or others suggestions for any other checks which would make sense | 19:45 |
| paulsherwood | wrt security, the production implementations of kbas that i'm aware of are working behind firewalls - but still, would be nice to have something better than plaintext password | 19:47 |
| paulsherwood | also note kbas can be run ad-hoc by a developer on their machine, to serve artifacts for a colleague, or to allow population of artifacts onto another kbas server | 19:51 |
| *** radiofree has quit IRC | 20:29 | |
| *** radiofree has joined #baserock | 20:31 | |
| *** locallycompact has quit IRC | 20:35 | |
| *** rdale has quit IRC | 20:49 | |
| *** rdale has joined #baserock | 20:50 | |
| rjek | paulsherwood: You're monologing | 22:19 |
| *** De|ta has quit IRC | 22:21 | |
| *** ctbruce has joined #baserock | 22:22 | |
| *** De|ta has joined #baserock | 22:22 | |
| *** ctbruce has quit IRC | 23:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!