IRC logs for #baserock for Wednesday, 2016-08-03

*** gtristan has quit IRC03:37
*** gtristan has joined #baserock04:06
*** gtristan has quit IRC06:47
*** gtristan has joined #baserock06:48
*** gtristan has quit IRC07:11
*** toscalix has joined #baserock07:38
*** fay has joined #baserock07:41
*** fay is now known as Guest3113907:41
*** gtristan has joined #baserock07:46
*** jonathanmaw has joined #baserock08:18
*** paulwaters_ has joined #baserock08:25
*** paulwaters_ has quit IRC08:54
*** locallycompact has joined #baserock09:30
*** jjardon has quit IRC09:47
*** jjardon has joined #baserock09:50
gtristanpaulsherwood, I've been re-reading your downstream metadata email10:09
gtristanas the problem blew up in my face today10:09
paulsherwoodack10:09
paulsherwoodwhich way would you prefer us to go?10:09
gtristanI had some idea that this could potentially be handled with submodules (in one version of an idealistic future)10:10
paulsherwoodtoo complicated imo10:10
gtristanthat way we would not have a huge amount of irrelevant deadcode sitting in the same definitions source tree, and we could pull in someone's brewed strata & chunks on demand10:11
pedroalvarezthat would bring all the headaches that yocto people have with layers10:12
gtristanbut I find it complicated because it goes against other opinions that I have (mostly that stratum is broken and that they should be referring to chunks instead of containing them)10:12
gtristanmaybe so10:12
gtristanor maybe the whole idea of trying to maintain all that stuff in the same source tree; is *also* a source of the same headaches that yocto people face10:12
tiagogomesYou could also use git subtrees. They are complicated as well, but less odd and error-prone than submodules. The user wouldn't have to know much about them if instructions to update the downstream fork were provided.10:13
locallycompactwhat is this conversation about10:13
gtristanlocallycompact, "downstream metadata"10:13
gtristanthe email from last week10:13
* paulsherwood still thinks if we enforce uniqueness, we're close to something workable10:13
locallycompactmove build-depends out of the strata10:13
locallycompactuniqueness is the worst possible solution to this10:13
paulsherwoodlocallycompact: you're starting to sound like persia10:14
locallycompactif you modify one strata you have to change every chunk the system10:14
gtristanlong term, I think I clearly agree with locallycompact that build-depends belongs entirely at the chunk level and not in stratum10:14
locallycompactstop hypothesising and actually try to do this, change one stratum in the middle of a downstream system and see how much it hurts10:14
locallycompactyou have to change every name of every chunk everywhere10:15
locallycompactmove stratum depends to the parent type10:15
*** rjek has quit IRC10:16
locallycompactthe system can govern the stratum depends, then if you want to alter a stratum you can just copy the stratum in particular, and copy the system and swap out that one stratum10:16
gtristanshort term however (as in, lets have something that works yesterday already), I'm _really_ tempted to just remove _everything_ from the definitions tree that is not related to said downstream project, and just have a complete fork10:16
gtristanwithout any deadcode lying around to confuse me about what exactly in this definitions repo actually gets built10:17
locallycompactthis I prefer10:17
locallycompactbecause then it will force a dialectic10:17
*** rjek has joined #baserock10:17
paulsherwoodlocallycompact: use easier words, please10:18
paulsherwoodunless you don't *want* folks to understand what you're saying :)10:18
* gtristan googled that10:18
locallycompactI want to be precise10:18
locallycompactThe scenario is: qt updates in either the one or the other repository, upstream or downstream it doesn't matter10:19
locallycompactAnd osmeone says hey we updated all the refs to 5.7 or whatever10:19
locallycompactand either way you just want to cherry-pick that file in a merge10:19
locallycompactthe fact that one person or the other builds qt on top of this or that system is not relevant, they each have their own merge criteria, they just want to pull in the new refs and run the ci to check all the important systems are still functioning10:23
locallycompactso move the build-depends out of the stratum so they are just collections of repo/ref combinations with local ordering10:24
locallycompactthey don't even need names, strictly10:24
locallycompactjust chunks:10:25
locallycompactthen you can cherry-pick that file in any situation10:25
locallycompactso a single stratum is just a set of "stratum build instructions" agnostic of where it might be situated10:27
paulsherwoodworks for me. i still want uniqueness, though :)10:27
gtristanwell, maybe I dont understand this proposal exactly, but I think ordering inside the stratum loses even more dependency information (and we have a lot that is already missing)10:28
locallycompactmaybe it works following this but doing it now is crazy10:28
gtristani.e. I would really prefer that the whole thing worked with only chunks10:28
locallycompactwe want to lose some dependency information10:28
locallycompactwe don't want some chunk to claim "no I only build against gcc-4.6" or whatever10:28
gtristanexcept that the way we have lost dependency information, for instance, just because some corner case chunk has changed in foundation, *all of gnome.morph* needs to build for 4-6 hours10:29
gtristanwhen probably only a few chunks really need to be rebuilt10:29
gtristanlocallycompact, except that we probably want to re-build the entire system when gcc version changes, except for say; installation of some python modules ?10:30
gtristani.e. if we want repeatable builds (which I think is the value-add of strictly rebuilding everything)10:31
locallycompactwhen the gcc version changes is the purview of the system, the chunk is just a repo ref and build instructions10:36
locallycompactso my chunk is somewhere in some stratum10:37
locallycompactand now I swap out underneath for a different base and see if it still builds10:37
locallycompactnone of the strata or chunks need change10:37
*** Guest31139 has quit IRC10:37
locallycompactbuild-essential may change if you don't care about keeping around old combinations of build-essential10:38
locallycompactbut if you are holding build-essential-4.6 or whatever for some practical purpose it is still in lockstep with upstream10:39
locallycompactas is latest10:39
locallycompactthen maybe you can get away with uniqueness after this point10:40
locallycompactbut then I don't even see why chunk morphs or strata need even have name: or kind: anymore "chunk morphs" are not chunks, they're just build instructions10:41
locallycompact"chunk" morphs could be solid markdown for all it matters10:41
pedroalvarezhehe, we are throwing in all the ideas at the same time10:42
* gtristan adds an organic chicken to the stew10:54
pedroalvarezthat won't make this easier to digest :P10:55
paulsherwoodwhile i am very interested to improve the format, forcing uniqueness would be a quick win to reduce user confusion imo10:56
paulsherwoodtangentially, one of the reasons i wanted nesting/generic composite things instead of strata, was the idea of (say) ruby, being ruby and stuff needed for ruby. rather than having ruby stratum contains ruby chunk, ruby composite would have repo+ref, plus other composites10:59
paulsherwoodas opposed to the current 'foo contains foo' warnings11:00
locallycompactI get it but they are completely harmless and I'm concerned about stalling downstream development at this moment11:01
*** Guest31139 has joined #baserock11:01
paulsherwoodack11:02
paulsherwoodi'll shut up, then11:02
pedroalvarezbut moving downstream definitions to a different folder is not a bad idea, right?11:03
pedroalvarezI've considered doing that with genivi definitions, but always forgotten to actually do it11:03
gtristanpedroalvarez, I like that structure, I dislike having the rest of baserock in the same tree though, honestly, unless I'm actually using it11:04
pedroalvarez(even if genivi is not downstream, but hey)11:04
pedroalvarezif not using it, there isn't any point on having a fork, but I guess you mean that you may use "core" but not "openstack" strata11:05
gtristanhence why I thought submodules might be a good idea, but it would only be interesting if we really only had a few stratum that various definitions project trees would use as submodules11:05
pedroalvarezI'll keep screaming every time I read "submodules"11:06
pedroalvarezsorry11:06
locallycompactsubmodules are like sick buckets, good in a pinch but you don't want to leave it around for weeks on end11:06
gtristanyeah I got it, it's not great11:07
franredthe problem is that you need to define a workflow where you generalize a definition from a "submodule" to the general if more than one system/strata uses it... or are we suggesting defining the same thing in multiple places?11:07
gtristanfranred, I think I would prefer a completely separate definitions repo for each project, with variable maintenance levels11:09
gtristanfranred, if some system became a "thing" at some point and then nobody cares about it, just let it die, at least it's not sitting there in definitions adding bloat11:09
gtristansubmodules was just a (weak) idea of how code sharing might work in such a scenario11:10
franredok, sounds sensible approach in one side, but if you need to maintain multiple projects, i.e. update a bug in gcc, you need to add these patch to all the repos/systems11:12
franredand you CI system won't apply per definitions but per project/system in this case as well11:12
gtristanfranred, so if build-essential were a submodule, then maintainers of those separate projects would get to decide when their definitions repos pull in the new commits from the build-essential submodule11:13
gtristanI say this but also note that I do, generally dislike submodules; my experience with them is that they just add hassle11:14
franredit doesn't sounds bad for me. i.e. in that scenario we would have openstack-kilo working with old versions of its components but at least we would be sure it still alive as we configured and not what we have know which is a big unknown11:15
franredas an example of dead systems in definitions at the moment11:15
gtristantrue, right now someone can just walk in and upgrade systemd without checking that every "maintained system" actually still boots to a graphic interface11:16
gtristanwhich is a problem11:16
paulsherwoodfranred: not entirely dead. i build them as part of ci11:16
paulsherwoodbut not deployed, tested11:16
franredpaulsherwood, build doesn't means working11:16
paulsherwoodno, true. but sometimes failed builds in the openstack have flagged a problem elsewhere11:17
paulsherwoodeg the 'too many pies' issue11:17
franredpaulsherwood, on the other hand, if it wouldn't change we would have that system open-running and deployable..11:17
paulsherwoodfranred: ack. that just comes down to cost vs value11:18
franredgtristan, suggestion is something that jjardon and other members suggested in the past. We should think on what is valuable for the actual users of baserock. In that approach the systems will be still reproducible and provenance preserve. We lose the build all things with latest changes and make the maintenance of all the systems dependable on the interest of these systems maintainers.11:25
franredbut also means that we will need to keep more metadata available in our servers? one for each version of the same package used in different definitions/repositories/systems11:26
*** tiagogomes has quit IRC11:34
*** tiagogomes has joined #baserock11:42
*** gtristan has quit IRC12:33
locallycompactthe other monumental advantage to having strata only be a dictionary of chunks: is it would be easy write back changes in jq12:35
locallycompactso you could do single point insertions in shell12:35
franredjq?12:50
locallycompactjson filter language https://stedolan.github.io/jq/tutorial/12:52
locallycompactso you could for example move a chunk up or down a strata and that's basically one line of jq12:57
*** cosm has joined #baserock12:58
edcraggand use chunks in multiple strata?12:59
*** cosm has quit IRC13:02
persiastrata exist for the convenience of system developers, so they don't have to understand chunks (assuming the chunk maintainers are sufficiently capable).13:02
*** cosm has joined #baserock13:02
persiaUse of the same chunk in multiple strata is a common way to manage edge cases whilst developing the correct taxonomy for strata within a wider project.13:02
persiaDownstream efforts are almost always going to be slightly at odds with any upstream taxonomy, meaning that it is hugely advantageous to be able to use chunk in multiple strata to allow separation of chunk maintenance and strata integration within the downstream project.13:03
locallycompactchunks *only* exist in strata, mereologically13:04
persiaYes, but not everyone thinks about it that way.13:06
persiaThe sense of a chunk being an independent thing with which one can do something remains strong, heavily influenced by the use of "packages" in distributions.13:06
persiaAnd many downstream projects are more heavily influenced by those semantics than we.13:07
pedroalvarezpaulsherwood: any findings in the CI after fixing latest bug?13:10
paulsherwoodc.b.o seems to be down for me...13:15
pedroalvarez:S13:18
pedroalvarezpaulsherwood: did you bring it to life?13:28
pedroalvarezit's up again13:28
pedroalvarezsame thing happened a while back, maybe the hosting provider saving resources? idk13:29
* paulsherwood needs to discuss with mwilliams13:48
paulsherwoodpedroalvarez: meanwhile i've kicked off a ci build on master definitions... so far so good 16-08-03 00:30:08 [248/1010/1010] [iptables]13:49
pedroalvarezgood13:50
paulsherwoodwhat's the deep change?13:50
pedroalvarezin anycase, I think a change in the cache-key is the thing to do13:50
paulsherwoodfor which?13:51
pedroalvarezthe `sh -e -c`13:51
pedroalvarezis the right* thing to do13:51
paulsherwoodoh, yes. that's done. i haven't released it yet, but the build i'm on has that13:51
paulsherwoods/on/running/13:51
pedroalvarezcool13:51
paulsherwoodam i right thinking morph has the same issue?13:51
pedroalvarezyse13:52
pedroalvarezvery bad error :(13:52
paulsherwoodamazing we've not noticed before now13:52
paulsherwoodi guess most make installs are well-behaved13:52
locallycompactmost open things have a make install at all13:54
paulsherwoodack. and for any congigure or build steps, the breakage would have been spotted during intgration in most cases13:59
*** mwilliams_ct has joined #baserock14:01
mwilliams_ctconcourse's own website appears to be down at the moment, which is making debugging a tad tricky14:16
mwilliams_ctin fact apparently the whole .ci TLD is having issues14:17
locallycompactO_O14:18
*** Guest31139 has quit IRC15:36
paulsherwoodpedroalvarez: i think we can call it ok  03:00:14 [924/1010/1010] [libgdata]16:20
paulsherwoodhence i've bumped artifact-version: 6, and ybd 16.31 is released16:20
pedroalvarezwe are really great integrators \o/16:21
*** jonathanmaw has quit IRC16:23
paulsherwoodpedroalvarez: what makes you say that, now? :-)16:24
pedroalvarezhehe, i wasn't 100% all the chunks were going to be ok16:26
pedroalvarezin fact, I don't think we have tested all of them after that change16:26
pedroalvarezbut we are great :)16:26
paulsherwood:-)16:26
*** toscalix has quit IRC16:51
paulsherwoodhttps://tech.slashdot.org/story/16/08/03/1635240/project-hosting-service-fosshub-compromised-embedding-malware-inside-hosted-files17:13
paulsherwoodwe need to get to reproducible builds as fast as possible17:13
*** gtristan has joined #baserock17:31
persiaPerhaps with cryptographically verifiable content, just to be sure.17:31
*** tiagogomes has quit IRC17:33
*** cosm_ has joined #baserock22:05
*** cosm has quit IRC22:10
*** cosm has joined #baserock22:14
*** cosm_ has quit IRC22:15
*** cosm_ has joined #baserock22:23
*** cosm has quit IRC22:27
paulsherwoodyup22:40

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!