IRC logs for #baserock for Thursday, 2016-02-11

*** locallycompact has quit IRC00:04
*** JPohlman1 has joined #baserock02:11
*** JPohlmann has quit IRC02:15
*** edcragg has joined #baserock07:14
*** toscalix has joined #baserock08:02
*** fay_ has joined #baserock08:12
*** fay_ is now known as faybrocklebank08:17
*** gtristan has joined #baserock08:20
*** edcragg has quit IRC08:50
*** bfletcher has quit IRC08:52
*** bashrc has joined #baserock08:58
*** CTtpollard has quit IRC08:59
*** CTtpollard has joined #baserock09:01
*** tiagogomes_ has joined #baserock09:07
*** edcragg has joined #baserock09:13
*** bfletcher has joined #baserock09:18
*** ctbruce has joined #baserock09:26
*** jonathanmaw has joined #baserock09:27
*** tiagogomes_ has quit IRC09:33
*** edcragg has quit IRC09:35
*** edcragg has joined #baserock09:38
*** tiagogomes_ has joined #baserock09:40
*** ssam2 has joined #baserock10:17
*** ChanServ sets mode: +v ssam210:17
*** ctbruce has quit IRC10:26
*** paulw has joined #baserock10:27
paulsherwoodgtristan: any chance WebkitGtk could be put lower down the stack, so it starts building earlier?10:38
*** paulw has quit IRC10:38
gtristanpaulsherwood, I had (have ?) a patch for that10:38
paulsherwoodooh, cool! :)10:39
gtristanwhat it comes down to is reorganization of strata10:39
paulsherwoodyup10:39
gtristanmy first attempt was denied because "i was creating too many strata" :)10:39
paulsherwoodwell, i'd like to see gnome stratum split up - it makes a mess of concourse10:39
paulsherwoodhow many strata did you have?10:40
gtristanfirst I had one-per-chunk, then I reorganized it so that geoclue and a few more chunks lived inside a webkitgtk strata10:40
paulsherwood-1 on one-per-chunk10:41
gtristannow that we have webkitgtk and webkitgtk1 (the 2 apis)... we'd need one to hold the deps of both and then 2 separate webkit strata I guess10:41
gtristanI would really love to see one-chunk-one-file with all deps explicitly listed... this would solve the whole layering problem :-/10:42
paulsherwoodthat creates different problems10:42
paulsherwoodlet's make the current scheme work first, optimise later10:43
pedroalvarezI imagine that one-per-chunk will create a really ugly concourse graph10:44
paulsherwoodtrue10:44
ssam2we could still define groups of some sort10:44
paulsherwoodalso true10:44
paulsherwoodssam2: did you get much feedback from your talk?10:44
ssam2not really10:46
paulsherwoodstunned silence :)10:46
paulsherwoodi enjoyed it, anyway10:46
ssam2nobody used the questions to prove it all totally wrong, which is a good sign10:47
paulsherwood:)10:50
*** locallycompact has joined #baserock10:55
*** gtristan has quit IRC11:16
jjardonwebkitgtk1 is not part of the gnome core components, so if we split the apps in another stratum it would make things faster for people only interested on building the gnome core modules11:20
paulsherwood+111:21
jjardonuh, reorganizing this is going to be fun :)11:27
*** CTtpollard has quit IRC11:28
*** CTtpollard has joined #baserock11:28
paulsherwoodlocallycompact: maybe you shouldn't be pushing to a production kbas when you're developing a chunk? :)11:33
paulsherwoodlocallycompact: disable kbas-url, remove the chunk artifact11:34
paulsherwoodlocallycompact: pls could you paste the build log for the artifact?11:34
locallycompactI thought you had kbas-url hardcoded11:34
paulsherwoodpassword isn't :) and anyway you can override it11:35
paulsherwoodi think i reverted default kbas-url recently11:35
locallycompactcan you delete go* from the server so I can get a build log out11:36
pedroalvarezlol11:36
paulsherwoodlocallycompact: you can get a build log out by changing *anything* in your go.morph file11:36
* paulsherwood proposes not to tamper with the production kbas, but may change the password :-)11:37
locallycompactno change to either the chunk or the strata is changing the cache key or triggering a rebuild11:43
locallycompactah, got it11:50
locallycompactpaulsherwood, it doesn't warn if the chunk morph doesn't exist, I had mispelled it and it just built an empty thing11:51
paulsherwoodlocallycompact: good catch!12:02
*** rdale has joined #baserock12:02
*** rdale_ct_ has joined #baserock12:04
*** rdale_ct has quit IRC12:05
*** rdale has quit IRC12:07
jjardonmmm, Im getting this error in ybd:12:19
jjardonhttps://www.irccloud.com/pastebin/EnmM3WOn/12:19
jjardonany idea?12:19
paulsherwoodjjardon: please use master?12:22
jjardonpaulsherwood: Nevermind, I was using morph: strata/NetworkManager/ModemManager.morph instead morph: strata/NetworkManager-common/ModemManager.morph12:22
* paulsherwood is considering tagging again12:23
jjardonpaulsherwood: master error message for this case is not much better:12:24
jjardonhttps://www.irccloud.com/pastebin/n7y18y6I/12:25
paulsherwoodthat's better, imo12:26
paulsherwoodit's saying you've not defined build-system, and it can't autodetect it12:26
paulsherwoodi decided to leave in the stack-trace, but maybe it's inelegant :)12:27
paulsherwoodi'll improve it, while fixing 11:51 < locallycompact> paulsherwood, it doesn't warn if the chunk morph doesn't exist, I had mispelled it and it just built an empty thing12:28
jjardonsure, but the real problem is that I'm referencing the morph file incorrectly; anyway problem solved now :)12:28
*** gtristan has joined #baserock12:28
paulsherwoodjjardon: what would be the ideal error message?12:28
paulsherwood(assuming ybd can't be psychic)12:29
jjardonpaulsherwood: The error message from morph was enough for me: "ERROR: Couldn't find morphology: strata/NetworkManager/ModemManager.morph referenced in strata/NetworkManager-common.morph"12:31
paulsherwoodok12:31
rdale_ct_i think the word 'morphology' is totally redundant12:35
rdale_ct_should be 'definition'12:35
paulsherwood+112:36
rdale_ct_probably 'redundant' is the wrong word, 'confusing' might be better12:38
*** paulw has joined #baserock12:48
jjardonok to move polkit to foundation instead of creating a stratum only for it? seems a common enough component in any Linux system12:50
pedroalvarezsomething that is only currently in gnome>?12:52
jjardonmmm, thinking twice I think Im going to separate it12:53
jjardonpaulsherwood: even systemd can use it if its present in the system12:53
jjardonpedroalvarez: ^12:53
jjardonthere are quite a lot stuff in the gnome stratum that really belongs to other layers12:53
pedroalvarezyeah, i can see that12:53
pedroalvarezbut I'm not sure about including all of that components in foundation12:54
jjardonindeed, does privileges-management look good as a stratum name?12:55
* paulsherwood can't comment on these components specifically12:55
jjardonssam2: Hey! did you see yesterday comments about morph not using DEFAULTS? there is a patch to fix that on gerrit but I guess we have to increase the definitions version as well?13:02
pedroalvarezI created a story for the real problem: https://storyboard.baserock.org/#!/story/7213:02
ssam2oh, dear, so this is a Morph bug that means it ignores the DEFAULTS file even when VERSION = 7 ?13:07
pedroalvarezyup13:07
ssam2i'm not sure we need to treat it as a new definitions version when Morph has a bug13:07
ssam2in this case13:08
pedroalvarezthe patch is not fixing morph for this, the patch includes a new build system13:08
ssam2the definitions format version 7 says that you should honour DEFAULTS13:08
ssam2so if Morph doesn't, that's a bug in morph, not a change in the spec13:08
ssam2oh, Morph still supports definitions V6, which affects https://gerrit.baserock.org/#/c/1788/13:11
ssam2but there is already a patch with +2 to remove support for V6, it just needs rebasing I think13:11
* ssam2 comments in Gerrit13:11
pedroalvarezoh, good to see that there are some patches to fix the problem13:14
pedroalvarezI wonder if we can persuade tiagogomes_ to at least tell us if any of the patches in https://gerrit.baserock.org/#/q/topic:cleanup-morphologyloader fixes this issue13:20
tiagogomes_my patches shouldn't fix anything. It was just some cleanups13:24
pedroalvarezwell, if they remove support for definitions version 6, that means that they should honour DEFINITIONS13:24
pedroalvarezsorry, DEFAULTS13:24
jjardonDoes someone have some  time to give some feedback about https://gerrit.baserock.org/#/q/status:open+project:baserock/baserock/definitions+branch:master+topic:reorganization_ofono_modemmanager ?13:25
ssam2it does look like my patches for DEFAULTS in Morph were totally and utterly broken13:25
ssam2that's embarassing13:25
rdale_ct_why is tiago copying json schemas from definitions into morph?13:26
ssam2I remember discussing this13:27
ssam2it's because i didn't want to make Morph's behaviour rely on having some arbitrary files in any definitions repo that it might ever want to build13:28
richard_mawrdale_ct_: The tool that uses the data needs to be the one that owns the validation schema, not the data.13:28
richard_mawOtherwise data could pass validation, but be completely incomprehensible to the tool.13:28
mwilliams_ctrdale_ct_: https://irclogs.baserock.org/%23baserock.2015-12-04.log.html has some background I think13:28
rdale_ct_thanks for the link - i can see a discussion, but i don't see any agreement13:32
locallycompactMe again. When I do these same commands in a build system it works, but done in the chunk morph it reports Read-only file system. https://paste.fedoraproject.org/321197/45519767/13:38
richard_mawo_O go builds in /var/tmp‽13:38
locallycompactah, I can overwrite I think13:40
locallycompactsec13:40
richard_maw /var/tmp is not in the set of paths morph lets you write to IIRC13:40
*** toscalix has quit IRC13:44
richard_maw:¬/ that error is because go dist assumes that if TMPDIR is not set, it should fall back to using /var/tmp13:45
richard_mawrather than /tmp, which would have been sensible13:45
richard_mawso export TMPDIR=/tmp would fix it13:45
locallycompactyeah13:46
richard_mawffs go!13:46
*** gtristan has quit IRC13:47
*** toscalix has joined #baserock13:52
jjardonmmm, seems ybd is stuck in an infinite loop here:13:57
jjardonhttps://www.irccloud.com/pastebin/QvHHta0a/13:57
jjardonpaulsherwood: known bug? ^13:57
ssam2it's like a 'which build tool is more broken?' contest in here today :-)13:57
jjardon:)13:58
pedroalvarezyocto wins13:58
paulsherwoodunfortunately i don't understand the splitting code.... rdale_ct_ ???13:59
rdale_ct_i don't think the splitting code can cause a loop - does the polkit stratum have splitting in it?13:59
*** toscalix has quit IRC14:01
pedroalvarezlocallycompact: you would help us a lot if you send your patches over gerrit :/14:01
pedroalvarezit's not that bad :(14:01
*** toscalix has joined #baserock14:02
paulsherwoodjjardon: pls could you publish your definitions?14:02
rdale_ct_perhaps something is going wrong with the build of libndp, and an exception gets thrown before it writes out a libndp.meta file - that file is used to tell if something has already been built14:04
paulsherwoodi had hoped exceptions were exposed, now... but that sounds like a reasonable theory14:04
paulsherwoodjjardon: i assume you're using master?14:04
jjardonpaulsherwood: http://git.baserock.org/cgit/baserock/baserock/definitions.git/log/?h=jjardon/reorganization14:08
jjardonpaulsherwood: yes, Im using ybd master14:08
jjardon(I've just tried and it seems to build with morph)14:08
paulsherwoodeek :)14:11
rdale_ct_is there anything in the libndp build log?14:15
pedroalvarezjjardon: you might be interested on this one: https://gerrit.baserock.org/#/c/1794/114:17
paulsherwoodrdale_ct_: it's not being created14:21
paulsherwoodso problem is earlier than that14:21
jjardonpedroalvarez: oh nice! thanks for the quick patch ssam2 !14:22
* jjardon abandon his quick fix14:23
jjardonpedroalvarez: can we remove the -1 in https://gerrit.baserock.org/#/q/status:open+project:baserock/baserock/definitions+branch:master+topic:python3-distutils now? :)14:27
pedroalvarezsure14:28
jjardonpaulsherwood: rdale_ct_ FYI, last tag of ybd fails in a different way:14:29
jjardonhttps://www.irccloud.com/pastebin/Y0RCEka5/14:29
paulsherwoodjjardon: unfortunately i'm fighting a headache, will get to this later14:31
jjardonsure14:32
*** gtristan has joined #baserock14:36
*** locallycompact has quit IRC14:38
*** locallycompact has joined #baserock14:40
*** locallycompact has quit IRC15:46
*** inara has quit IRC15:52
*** inara has joined #baserock16:02
*** CTtpollard has quit IRC17:06
*** jonathanmaw has quit IRC17:19
paulsherwoodjjardon: my brain is not working very well, but removing some redundant build-depends.... http://paste.baserock.org/unefehatul17:25
*** locallycompact has joined #baserock17:28
jjardonpaulsherwood: mmm, I like more explicit dependencies; things start breaking easily when you move components between strata if you use implicit dependencies17:32
*** ssam2 has quit IRC17:32
paulsherwoodwe went through this before17:32
paulsherwoodit became a nightmare17:32
locallycompactIs there any way to make things from npm reproducible in baserock or is that another black hole I should just use ansible for?17:32
*** ssam2 has joined #baserock17:32
*** ChanServ sets mode: +v ssam217:32
paulsherwoodbuild-depends are transitive17:33
paulsherwood(i hoped that this wasn't true, originally, but it was proved to be the case)17:33
rdale_ct_but that still shouldn't cause ybd to loop though17:34
paulsherwoodi agree. but i'm failing completely to figure out what's going on17:35
paulsherwoodand i do know that those depends are redundant17:35
paulsherwood(and once removed, the loop doesn't happen)17:35
rdale_ct_ah17:35
paulsherwoodi'll clean it up so ybd reports it properly, but still i don't think it's right to allow the redundancy, even if morph builds it17:38
paulsherwoodnpm? i thought we already did an importer for that17:39
paulsherwoodlocallycompact: ^^17:40
paulsherwoodi may be wrong17:40
ssam2zara did one, indeed17:40
ssam2as part of http://git.baserock.org/baserock/baserock/import.git17:40
Zarayeah, over a year ago; I'd be very surprised if it worked :/17:40
ZaraI think the npm registry has changed several things significantly since then.17:41
jjardonpaulsherwood: Id be happy if ybd warns me about redundant dependencies then17:41
* paulsherwood too :)17:41
ssam2git.baserock.org seems super overloaded17:41
paulsherwoodjjardon: i hope to do that tonight17:42
* locallycompact prods gbo17:42
paulsherwoodssam2: probably concourse folks hammering it17:42
ssam2i can't even ssh in17:42
benbrown_everything is orange, I don't think it's us?17:43
paulsherwoodouch. maybe ddos?17:43
ssam2oh, in the OpenStack status page it says 'Migrating'17:45
ssam2I presume that is something DataCentred are doing17:45
paulsherwood:/17:45
paulsherwoodwas there any notification?17:45
ssam2didn't see any17:46
Zara(oh, also, I don't think the npm importer guarantees things are reproducible, as npm uses semver for versioning, so the transitive dependencies could differ.)17:47
pedroalvarezit supposed to be a 'live migration'17:47
Zara(it's mainly a tool for generating morphs for an npm package and its dependencies faster than it would be to do it by hand. I *think* it should still work for that, but with more errors than a year ago.)17:49
paulsherwoodi think that's what locallycompact is seeking, actually17:50
locallycompactostensibly, yes I think so17:50
locallycompactplus the lorries17:50
Zarayeah, it should hopefully still work for that for many, though it's covered in cobwebs17:50
Zaradocs: http://wiki.baserock.org/guides/import-tool/npm/ , worth checking against reality17:52
ZaraI believe baserock itself has also changed significantly since.17:52
locallycompactI'll let you know17:52
Zara:)17:52
pedroalvarezI've opened requested help to the host provider. I'll let you know whenever I got any news about git.baserock.org17:55
*** bashrc has quit IRC18:02
*** toscalix has quit IRC18:06
pedroalvarezwell, the host provider is aware of the issues. They are trying to fix it now18:10
*** locallycompact has quit IRC18:18
*** faybrocklebank has quit IRC18:19
*** edcragg has quit IRC18:34
ssam2git.baserock.org is back18:44
paulsherwoodyay :)18:45
*** ssam2 has quit IRC18:45
paulsherwoodrdale_ct_: are you still around?19:09
rdale_ct_yes19:50
paulsherwoodhttp://paste.baserock.org/zagawohasu20:02
paulsherwoodthis is what's causing the loop for jjardon's usecase20:03
paulsherwoodclaim() triggers because of that exception20:03
* paulsherwood is still trying to understand the splitting code20:03
paulsherwoodregexp was never my strong suit20:05
rdale_ct_i not sure why it would need to chdir(), i'll have a look at the code20:05
paulsherwoodat that point, on a rerun, sandbox has: ls /src/tmp/tmp4j_Sn820:07
paulsherwoodbaserock  dev  etc  lib  strata  tmp  usr20:07
rdale_ct_i suppose the chdir() reverts to where the app was before outside the scope of the with clause20:07
rdale_ct_why is it trying to write to /baserock/strata rather than /baserock that seems to be the problem?20:08
paulsherwoodquite :)20:09
paulsherwoodi wonder why strata exists at all?20:09
rdale_ct_in fact this part of the path is wrong too: /src/tmp/tmpPHvHrP/strata/privileges-management.inst/20:09
rdale_ct_with an excess strata20:09
paulsherwoodthat might be me, please hold.20:09
paulsherwoodno, seems it predates my recent messing20:12
paulsherwoodanyway, i think it's dinner time here20:12
paulsherwoodi'll take a deeper look later20:12
rdale_ct_ok, i'll look here again in a bit20:13
rdale_ct_the chunk name and path stuff is all a bit of a mess20:13
rdale_ct_in jjardon's diffs: +- name: strata/privileges-management20:18
rdale_ct_in the systems/gnome-system-x86_64.morph definition20:19
rdale_ct_why do you need both a 'name:' and a 'morph' field to specify a stratum apart from it being error prone in that their name values can conflict20:24
*** edcragg has joined #baserock20:32
*** edcragg has quit IRC21:27
paulsherwoodaha, yes.21:35
paulsherwoodwe've been around this too. i would prefer name only, and insist on uniqueness, but all previous definitions would be broken that way21:36
paulsherwoodin the end, path is the only true way to distinguish foo from bar. but then we have a load of definitions with no path, because they don't need a morph file, being autodetected21:37
paulsherwoodhence one of the most horrible functions ever written in python... https://github.com/devcurmudgeon/ybd/blob/master/ybd/definitions.py#L14321:38
* rjek makes a dwfinintion with a legit path of ERROR21:40
paulsherwoodack21:40
rjek:-)21:40
rdale_ct_'all previous definitions would be broken that way' - that's only a problem if you don't believe in migrations21:51
paulsherwoodrdale_ct_: i believe in making tools backward compatible as far as possible. we have clear evidence that some users are not even upgrading morph, for fear of breaking something22:03
paulsherwoodfor ybd, as new kid on the block, i'm doing my best to make it deal with idiosyncrasies that arose before it existed22:05
paulsherwoodfailing, in many cases... but i'm trying :)22:06
rdale_ct_yes, i'm impressed with the velocity of ybd development, but 'fear of breaking something' is because the migration process in morph hasn't been thougth through and users regard moving onto a new version of the schema is a problem. it shouldn't be imho22:19
paulsherwoodit's not just that, tbh. morph has expressly broken things for users a number of times, because of some fundamental difficulties as a result of early assumptions22:22
paulsherwoodthe technical ground-breaking of morph has been great, but we didn't always take users into account properly22:23
paulsherwood(and i speak as both a sometime contributor to morph, and a longtime user)22:24
paulsherwoodas ssam said in his talk, some of the things we are now confident about are expressly the result of doing the exact opposite previously22:26
paulsherwoodcoming back to this bug...22:26
paulsherwoodideally ybd would run json-schema against user's definitions and scream, long before a build has started22:27
paulsherwood(and further... if we are confident in migrations, that step could notice an old version,  report it, run migrations, build against that...)22:28
rdale_ct_i think we need to attack this on two fronts: firstly make migrations work and stop thinking that the idea of supporting 'everything that has every existed' is a good idea, and secondly decouple the process of defining baserock schemas from the projects consuming them by writing adapters for schema to project so that ybd or morph can use plugins from the latest upstream baserock schema or use a legacy schema with the default22:34
rdale_ct_Definitions parser22:34

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!