IRC logs for #baserock for Friday, 2016-11-04

*** bjdooks has quit IRC01:40
*** bjdooks has joined #baserock01:41
*** gtristan has joined #baserock05:38
*** franred has joined #baserock08:32
*** toscalix has joined #baserock08:43
*** paulw has joined #baserock09:00
*** ctbruce has joined #baserock09:07
*** fay has joined #baserock09:10
*** fay is now known as Guest9946809:10
*** Guest99468 is now known as faybrocklebank09:10
*** ctbruce has quit IRC10:03
*** ctbruce has joined #baserock10:03
*** ctbruce has quit IRC10:29
*** ctbruce has joined #baserock10:31
*** lchlan has quit IRC10:43
*** faybrocklebank has quit IRC10:44
*** faybrocklebank has joined #baserock10:44
*** lchlan has joined #baserock10:45
pedroalvarezmeh, there was an email from paulsherwood waiting to be moderated since sunday, but I only received the moderator request  yesterday11:36
pedroalvarez(which I missed due other email client problems)11:37
paulsherwoodpedroalvarez: never mind, i re-sent it anyway11:52
paulsherwoodby all means delete that mail11:52
pedroalvarez:/ I approved it before coming here12:02
pedroalvarezjjardon: I left some comments in the list re. migration to gitlab12:03
*** paulw has joined #baserock12:21
*** gtristan has quit IRC13:20
*** gtristan has joined #baserock13:26
rjekHello.  I'm looking into adding hierarchical caching into kbas such that your local kbas can query a shared/internet-facing kbas for things it doesn't have.  I've been thinking about how to get artefacts into an upstream cache.  I see some problems, such as DoSing the upstream because of nonsense uploads, as well as the possibility of pushing secret stuff.13:33
rjekI have two ideas: first make upstreams only pushable to with credentials (as is true now, ish), and some sort of configurability in kbas or signal from ybd about the confidentiality of artefacts (perhaps a list of regexps in a white/black list)13:34
rjekAny thoughts or ideas are will be welcomed.13:34
leemingrjek, does kbas have other information other than the hash? e.g name of the chunk/thing13:39
leemingjust wondering if it has access to software name level things, to compare against the master definitions and only cache things that exist there? as a basic whitelist13:40
leemingunless, there are proprietary patches :\13:41
rjekCurrent protocol includes the chunk name when querying or submitting, that would have to change too13:41
leemingshort of introducing a field in the definitions that states cacheable, im not sure... accidents may happen13:41
leemingjust name?13:42
rjekI'm think about this from a "public ubercache" point of view, which may not be possible to do securely, in which case that idea can be shelved13:42
leemingsurely there are multiple versions of the artifact, based on what filesystem exists below it13:42
rjekBut it'd still be useful to have an office-wide cache server each developer can make use of, for example13:42
leemingyes, i am thinking of a pure public side whitelist too13:42
rjekOr for each runner in a CI system to have a local cache that can query a data centre-wide cache13:42
leemingprivate, clearly you wont have the same security issues for caching proprietary things13:43
rjekNod13:43
leemingi can only think that if the hash was the key, then chunks marked as private, could either a) not be uploaded to the public cache, b) include some key for access13:44
rjekI'm already planning on having credential processing for read as well as write13:45
leemingother than whitelisting on the incoming, whitelisting on the private cache for out going is probs safer?13:46
leemingtherefore making it a responsibility of the kbas local server..13:47
leemingno change to definitions13:47
leemingclient -> tmp cache -> local kbas -> globla kbas13:47
leemingcorrect?13:47
rjekyeah13:49
rjektbh it sounds too fraught to consider having public upstream caches at all13:49
leemingyeah, so a simple 'fix' would be to apply a whitelist on the kbas UPloading13:49
leemingwhitelist = in public definitions13:50
paulsherwoodrjek: and yet, we already do? :)13:50
leemingadditional things to control private patches tho.. not sure without thinking about it13:50
rjekpaulsherwood: We don't have anything that anyone can push to13:50
rjekMy concern is accidental uploading13:50
leemingkbas should have in + out rules then13:51
leemingout the out rules are more important13:51
rjekWhere my aim with this is to improve cache hits among teams rather than just per-developer or per-runner13:51
leemingkbas -> kbas13:51
leemingagreed. caching should happen at multiple levels/ network distance matters13:52
radiofreerjek: i think the main kbas (i.e the one everyone can use) should only be pushed to by the CI, so password protected13:53
radiofreei don't trust random artifacts from random users13:53
radiofreei think that's what we currently do?13:53
rjekWell, you probably won't ever fetch random artefacts from random users13:55
rjekUnless you'd be building the same random thing13:55
rjekIn which case, even if the binary is broken some how, at least you found out sooner13:55
rjekBut yes, one where only CI can push to would be a perfectly sensible approach if that's what a team wanted13:56
rjekI need a cup of tea.13:56
paulsherwoodthe 'only ci can push' makes sense to me too. bonus points if (for any public kbas), the uploads can only happen via i private network with the ci runners13:58
rjekpaulsherwood: Why a private network?14:00
rjekBecause credentials are involved and it's over-the-internet, I was just going to mandate TLS14:00
*** edcragg has quit IRC14:17
*** edcragg has joined #baserock14:31
paulsherwoodrjek: because physical security is better than not? if uploads can only happen via private network, wouldn't that be better than TLS?14:53
rjekTricky14:55
rjekThere's no way to have a private network into AWS, for example.14:55
rjekYou can VPN, but that's just the different wrapper around cryptography and the upshot being the same as using TLS14:55
paulsherwoodrjek: oh? i thought it's possible to have a private network shared between aws machines?14:56
rjekpaulsherwood: That's a VLAN, plus the data still needs to get there over the internet.14:56
paulsherwoodnot if all the runners are already on the vlan?14:58
rjekAny secret, sensitive, or propriatary data that they have has to get there somehow14:58
paulsherwoodtrue... but not via kbas?15:01
paulsherwoodjjardon: is https://gitlab.com/baserock/ybd/issues/244 reproducible with master? is there something special about the branch you're using?15:06
rjekGah now I've got something which is essentially ACLs, which is probably too complicated for this15:43
*** anahuelamo has quit IRC15:56
*** anahuelamo has joined #baserock15:58
*** faybrocklebank has quit IRC16:19
*** franred has quit IRC16:23
*** ctbruce has quit IRC16:23
*** toscalix_ has joined #baserock17:26
*** toscalix has quit IRC17:27
*** toscalix__ has joined #baserock17:27
*** toscalix_ has quit IRC17:31
*** CTtpollard has quit IRC18:10
*** pedroalvarez has quit IRC18:28
*** pedroalvarez has joined #baserock18:29
*** ChanServ sets mode: +v pedroalvarez18:29
*** toscalix__ has quit IRC18:33
*** tiagogomes has quit IRC19:10
*** gtristan has quit IRC20:10
*** bjdooks_ has joined #baserock20:45
*** parag0n has joined #baserock20:48
*** thinkl33t has quit IRC20:48
*** bjdooks has quit IRC20:48
*** SotK has quit IRC20:48
*** SotK has joined #baserock20:48
*** parag0n is now known as thinkl33t20:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!