IRC logs for #baserock for Wednesday, 2016-05-18

*** locallycompact has quit IRC00:18
*** rdale has quit IRC00:37
*** gtristan has quit IRC00:57
*** gtristan has joined #baserock01:15
*** gtristan has quit IRC02:53
*** gtristan has joined #baserock03:04
*** paulwaters_ has joined #baserock06:40
*** fay_ has joined #baserock07:12
*** toscalix has joined #baserock07:54
*** anahuelamo has joined #baserock08:01
*** anahuelamo_ has joined #baserock08:05
*** anahuelamo_ has joined #baserock08:06
*** anahuelamo has quit IRC08:06
*** anahuelamo has joined #baserock08:08
*** locallycompact has joined #baserock08:39
*** JPohlman1 has joined #baserock08:44
*** JPohlmann has quit IRC08:45
*** thrace has quit IRC08:46
*** persia_ has quit IRC08:46
*** wdutch has quit IRC08:46
*** persia_ has joined #baserock08:46
*** thrace has joined #baserock08:46
*** wdutch has joined #baserock08:46
*** rdale has joined #baserock08:47
*** lachlanmackenzie has quit IRC08:48
*** edcragg has joined #baserock08:49
*** gtristan has quit IRC08:54
*** ctbruce has joined #baserock08:54
*** franred has quit IRC08:56
*** lachlanmackenzie has joined #baserock08:57
*** franred has joined #baserock08:57
*** ssam2 has joined #baserock09:20
*** ChanServ sets mode: +v ssam209:20
*** gtristan has joined #baserock09:52
*** franred has quit IRC10:52
*** franred has joined #baserock11:01
*** anahuelamo has quit IRC11:06
*** locallycompact has quit IRC11:39
*** anahuelamo has joined #baserock11:48
*** anahuelamo has quit IRC11:50
*** anahuelamo has joined #baserock13:46
*** leeming has joined #baserock13:49
*** CTtpollard has quit IRC14:09
*** CTtpollard has joined #baserock14:12
*** gtristan has quit IRC15:12
*** gtristan has joined #baserock15:33
*** pedroalvarez has quit IRC15:39
*** pedroalvarez has joined #baserock15:39
*** ChanServ sets mode: +v pedroalvarez15:39
leemingsome basic questions about ybd. 1) do cached artefacts age on trove? 2) If you build an artefact, do you always upload back to trove (or which ever cache server your using)?, 3) are artefacts verified when put into the cache?15:57
* leeming might not fully understand. feel free to point out any flaws to my logic15:58
*** locallycompact has joined #baserock16:02
edcraggcache artifacts are identified by their cache key16:05
edcraggif anything significant changes about the build the cache key is likely to change16:05
pedroalvarezleeming: Answering questions for ybd:16:06
pedroalvarez1) I don't quite understand that question16:06
pedroalvarez2) with ybd you can configure it to upload the artefacts to a shared cache server, so not always16:06
pedroalvarez3) ybd's cache server does some sort of verification, ensuring that the compressed file is not corrupt (tar -t $file)16:06
leemingedcragg, cache key is essentially some hash of it's git commit and dependency versions right?16:07
leemingpedroalvarez, so it only checks for errors for the verification? what about the integrity?16:08
leemingalso, re Q1 - I was asking as I expected something in my build to have an artefact already. Guessed some artefacts might be removed over time (removing very out of date builds/ other reasons)16:10
*** anahuelamo has quit IRC16:10
*** anahuelamo has joined #baserock16:10
pedroalvarezybd is known  for auto-removing things if you "don't have enough space"16:11
pedroalvarezbut these artefacts would be in your filesystem, not in any cache server16:11
*** toscalix has quit IRC16:11
leemingah, maybe a miss understanding of caches here.. I understand there is a cache server that holds many dependencies?16:12
leemingI have a local cache, yes understand that bit16:12
pedroalvarezthere is one, yes (I believe: http://artifacts1.baserock.org:8000/)16:13
edcraggleeming, the algorithm depends on a number of things, but generally if you change the definitions for a chunk in any way it'll need to rebuild16:13
leemingam I mixing up things because I read about morph? :S16:13
edcraggholds many dependencies?16:14
edcraggthe concepts are the same between the two, just reimplimented in ybd, in general16:14
edcraggsometimes in an incompatible way, so the cached artifact format is different and not compatible between morph and ybd, for example16:15
leemingok, so it IS a recursive search of caches then? 1. check local copy on disk, 2. check remote cache, if not found, build16:15
leemingwhen built, generate cache key and push to remote?16:15
pedroalvarezonly push if configured to push to remote16:17
leemingright, but my understanding is correct?16:17
pedroalvarezand "generate cache key" sounds confusing16:17
leemingpedroalvarez, isn't it simply a key similar to a hashmap? i.e. some hash function is applied, then that is used as a key16:18
leemingvalue being the artefact16:18
edcraggcheck local cache, if not there look in remote cache and download to local cache, otherwise build and put in local cache, maybe push to remote16:19
leemingedcragg, yes thanks. more verbose than my explanation :) copy to local was implicit16:20
leemingbut other verification on push to remote (on the remote side) ?16:20
pedroalvareznope16:21
leemingisn't that a security issue? I can go into details here... or directly16:22
pedroalvarezit can be, to upload artefacts you need a password though16:22
pedroalvarezso I'd assume that, whoever has the password, are trusted uploaders16:22
leemingah ok, so only trusted uploads16:23
pedroalvarezmaybe CI servers, or some sort16:23
*** fay_ has quit IRC16:26
*** locallycompact has quit IRC16:34
*** ctbruce has quit IRC16:54
*** leeming has quit IRC17:07
*** anahuelamo_ has joined #baserock17:15
*** anahuelamo_ has quit IRC17:16
*** edcragg has quit IRC17:23
*** ssam2 has quit IRC17:49
*** gtristan has quit IRC18:10
*** locallycompact has joined #baserock18:11
* paulsherwood is sad to find that leeming asked all these questions, but is not still in channel when i can answer them19:21
paulsherwoodleeming: yes, i believe 'calculate cache_key' is a hashmap... the cache_key is unique for a given set of input 'factors' eg...19:24
paulsherwoodgit SHA of the version of the thing we are building, plus arch we are building for, plus the hashmaps of all th build-dependencies... (which are hashmaps recursively all the way down)19:25
paulsherwoodthat is all handled in hash_factors() https://github.com/devcurmudgeon/ybd/blob/master/ybd/cache.py#L7519:26
paulsherwood16:57 < leeming> some basic questions about ybd. 1) do cached artefacts age on trove?19:28
paulsherwoodi think trove handles source repos, which are *inputs* to build/integration processes (eg morph, ybd)19:29
paulsherwoodwe normally use 'artifacts' to refer to built and/or integrated *outputs* of these processes19:29
paulsherwoodi've not checked recently so maybe we are storing morph-generated artifacts on trove... but that is confusing imo19:31
paulsherwoodybd artifacts are stored locally in an artifact cache where ybd is run, and also can be uploaded to (or served from) a kbas server19:31
paulsherwoodkeyed binary artifact server19:32
paulsherwoodno ybd artifacts are stored on a trove (unless we are running ybd or kbas on the trove machine)19:33
paulsherwoodartifacts don't age on a kbas instance currently... ie once they're uploaded, they persist.19:33
paulsherwoodhowever i consider that to be a bug ... because kbas servers will crash with ENOSPACE eventually19:35
paulsherwoodhence https://github.com/devcurmudgeon/ybd/issues/17219:35
paulsherwoodybd does remove oldest artifacts from its local cache store (config['artifacts']) if it needs to (less than config['min-gigabytes'] available)19:36
paulsherwoodthis has led to a surprising amount of emotional discussion - some folks hate that ybd deletes artifacts. conversely i hate that morph will sometimes fail to run/build because ENOSPACE19:38
paulsherwood2) If you build an artefact, do you always upload19:38
paulsherwoodif config['kbas-password'] is set, ybd will attempt to upload to config['kbas-url']19:39
paulsherwood3) are artefacts verified when put into the cache?19:40
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/kbas/__main__.py#L13819:40
paulsherwoodactually, let's consider the whole function: https://github.com/devcurmudgeon/ybd/blob/master/kbas/__main__.py#L13819:41
paulsherwooda) it checks the provided password. but this is woefully poor, really... not even encrypted. i've asked for help to improve this a few times (hint)19:42
paulsherwoodb) it checks we're not trying to upload to /etc/passwd or similar19:43
paulsherwoodc) it checks we don't already have an artifact at this cache_key19:44
paulsherwoodd) it checks the provided artifact is a valid tarball19:44
paulsherwoodand that's about it.19:45
paulsherwoodleeming: i'd be interested to hear from you or others suggestions for any other checks which would make sense19:45
paulsherwoodwrt security, the production implementations of kbas that i'm aware of are working behind firewalls - but still, would be nice to have something better than plaintext password19:47
paulsherwoodalso note kbas can be run ad-hoc by a developer on their machine, to serve artifacts for a colleague, or to allow population of artifacts onto another kbas server19:51
*** radiofree has quit IRC20:29
*** radiofree has joined #baserock20:31
*** locallycompact has quit IRC20:35
*** rdale has quit IRC20:49
*** rdale has joined #baserock20:50
rjekpaulsherwood: You're monologing22:19
*** De|ta has quit IRC22:21
*** ctbruce has joined #baserock22:22
*** De|ta has joined #baserock22:22
*** ctbruce has quit IRC23:05

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!