IRC logs for #baserock for Friday, 2016-09-16

*** gtristan has quit IRC03:50
*** trn has quit IRC04:04
*** gtristan has joined #baserock04:53
*** trn has joined #baserock06:05
*** paulwaters_ has joined #baserock07:28
*** rdale has joined #baserock07:43
*** ctbruce has joined #baserock07:59
*** toscalix has joined #baserock08:04
*** paulwaters_ has joined #baserock09:04
*** locallycompact has joined #baserock09:28
*** gtristan has quit IRC09:32
*** edcragg has quit IRC10:07
*** edcragg has joined #baserock10:09
*** gtristan has joined #baserock10:22
locallycompactwhat is python11:01
locallycompacttring to tell me11:01
locallycompacthere11:01
locallycompactdef load_yaml(filename):11:02
locallycompact  with open(filename) as f:11:02
locallycompact    text = f.read()11:02
locallycompact  contents = yaml.safe_load(text)11:02
locallycompact  return contents11:02
locallycompactprint(load_yaml(sys.argv[0]))11:02
locallycompactyaml.scanner.ScannerError: mapping values are not allowed here11:02
locallycompact  in "<string>", line 5, column 24:11:02
locallycompact    def load_yaml(filename):11:02
pedroalvarezTry putting spaces after the colons in your yaml11:05
locallycompactno that's not it11:05
pedroalvarezgrr11:05
locallycompactargv[0] -> argv[1]11:05
pedroalvarezhah11:05
locallycompactfs11:05
pedroalvarezthat too11:05
jmacsIt's still a weird error11:06
jmacsOh, I get it now11:06
pedroalvarezit's obvious once you get it11:06
pedroalvarezbut yes, very confusing11:06
pedroalvarezgood catch locallycompact :)11:07
richard_mawo_O it was trying to parse your script as yaml‽11:48
locallycompactyarp11:49
leeminghaha11:49
persiaThe extra confusing part is that, I presume because of all the colons in the python, it reached line 5 before hitting issues.11:59
locallycompacthow do you make python look for files where the program is not where it's run12:26
jmacsYou want the run directory or the directory the script is in?12:27
locallycompactrelative to script12:27
richard_mawthat's normally argv[0] I think12:27
jmacsos.path.realpath(__file__)12:27
locallycompactis that the file file or the main executable12:28
locallycompactbecause they might be in different dirs12:28
jmacsThe file that line occurs in12:28
locallycompacthow do I get the main executable dir12:28
richard_mawuse os.path.dirname(__file__) in the main execuable, unless you mean the python interpreter12:29
locallycompactwell ybd has a main exe, and the includes files in subfolder ybd/ I want to always get the folder the main exe is in12:30
jmacsIn that case I think you want os.path.realpath(sys.argv[0])12:32
locallycompactthanks!12:32
locallycompactpaulsherwood, you about?12:47
*** gtristan has quit IRC13:32
*** gtristan has joined #baserock14:51
paulsherwoodlocallycompact: i am now15:23
*** CTtpollard has quit IRC15:59
*** tiagogomes has quit IRC16:14
locallycompactpaulsherwood, I'm going to bring the spec into the build tool to get rid of that maddening parsing logic. Would you prefer it hard copied or submoduled16:24
locallycompactthis gives all the error tracing you could want https://gitlab.com/baserock/ybd/commits/lc/00916:25
paulsherwoodi don't like submodules in general. what do you mean by hardcoded?16:32
paulsherwoods/submodules/git submodules/16:32
paulsherwoodmaybe you didn't mean that16:33
locallycompacthard copied, as in those commits16:34
locallycompactthe list of valid fields things is confusing, jsonschema is very robust and gives decent errors, and morphs.py is just unintelligable16:35
paulsherwoodagreed for all three points16:36
locallycompactso I can polish the schema for version 9 and we can copy it in and change the code, or add a submodule and update it as the spec channges but then you have to clone recursive16:37
paulsherwoodor have a separate spec-parser python library?16:38
locallycompactmm, sure16:38
paulsherwoodi'm not sure which is best tbh. i wanted to bring in the spec, but ssam argued strongly for splitting it out16:39
locallycompactreally? what did he say?16:39
locallycompactall that's happened is we've makeshifted a poorer version of jsonschema.validate(), reusing the spec for that purpose is an implementation detail of ybd imo16:40
* paulsherwood can't remember the details, but we ended up with a separate repo for it16:40
locallycompactthat was to split it out from definitions16:40
locallycompactit was in definitions and now it isn't, that makes sense16:40
locallycompactbut the build tool should still leverage it16:41
paulsherwoodby duplicating the code from the spec repo?16:42
locallycompactI'm fine with that16:43
paulsherwoodin my dreams the spec is a self-consistent library that can validate any given version of definitions, and present it for direct consumption by ybd or other tools16:43
locallycompactI like that too16:43
locallycompactI'll see about trying to publish a library16:44
paulsherwoodif you copy the code into ybd, we have the usual possibility of getting out-of-step16:44
locallycompactwell, no, ybd still has juresdiction over what version of definitions it is capable of parsing at what point in history16:44
locallycompactit would be no more out of step than it is now16:44
locallycompactwe have to update ybd to be able to parse V9, the schema can already do it16:45
paulsherwoodright. but in part morphs.py is unintelligible because it handles all of the previous versions too16:46
locallycompactok, well we can still aim for that with a library but morphs.py is unsustainable16:47
locallycompactI doubt it could handle this and more changes, with a full library we could technically duck type based on the definitions version16:47
locallycompactso grab the version, you get this interface of parsing implementations in return16:48
paulsherwoodiiuc it would be desirable for the spec-library to read datasets at any/all historic schema versions, write converted data to the current schema version, and present an in-memory representation for consumption by other tools16:48
locallycompactok16:49
persiaI'd argue that it should write converted data to any/all historic schema16:49
paulsherwoodwhy?16:49
persiaIdeally with some mechanism where the read/write aspects are bound, to make it easy to add new ones.16:49
locallycompactthat's not possible, it's not necessarily bijective between adjacent versions16:50
* paulsherwood doesn't see the need. the only direction is forward16:50
persiapaulsherwood: A) Someone might have a hacked version of a build tool that needs version 6, and so wants to write out an old one.  B) it means that the entire output module doesn't have to be rewritten every time a new version is created, just a writer for that version from some consistent internal store.16:50
persiaOh, and C) it validates that the internal store can be usefully mapped to version X, so that one can be more confident about consuming version X, because round-trip is a no-op.16:51
paulsherwoodpersia: in the scenario above, build tools use (a given version of) the library, and work internally with the memory representation it gives them.16:51
persia(if round-trip isn't a no-op, then the tool would be introducing something (probably buggy)16:51
persiaYes.16:52
persiaBut that hasn't always been true.16:52
persiaSo, if anyone is using a build tool in production today, and they have modified it in any way, this would be an epoch if there isn't bidirectional compatibility.16:53
paulsherwoodonly if they continue using that build tool?16:54
persiaYes.16:54
persiaBut there aren't any other tools that meet their custom need, or they wouldn't have modified it.16:54
persiaSo if they are changing tooling and data format, what makes this new state you describe interesting?16:54
persiaUnless they beleive in following mainline, they are much more likely to fork and live with it until it becomes too painful, then evaluate available tooling/systems at that future point in time.16:55
locallycompactno hope for them16:55
paulsherwoodi'm interested in ensuring that new tools can build old definitions. not that old tools can build new definitions16:56
locallycompactI don't know how well that's going to work off the top of my head because I don't remember definitions versions older than 7, but I can start with a library that works on 7+ and work backwards16:56
persiaGiven that there needs to be a mapping between a given definitions format and the internal representation *anyway*, and given that there is an expectation of there being an infinite number of definitions versions, I don't understand why it is especially hard to preserve the writer for each version vs. rewriting one "perfect" writer each time, and making all the old stuff immediatly invalid.16:56
locallycompactit might not be much work, 7 and 8 parse identically fr ex16:57
*** toscalix has quit IRC16:57
paulsherwoodlocallycompact: the library could call morphs.py for everything up to 8, and do your new magic for everything newer. in that way, morphs.py would be (kind of) sustainable16:57
persiaSO, yeah, if it's especially difficult, I could be convinced it isn't necessary.  If it's not done just because people want to chase the future, I imagine that to be a poor choice for a broad ecosystem.16:57
paulsherwoods/8/8 or some earlier cutoff/16:58
locallycompactyeah alright16:58
paulsherwoodpersia: why would preserving the (infinite number of) writers in the current version of the library be better than saying just use an old version of the library?17:07
persiaBecause I don't expect the build tool to work with an old version of the library, so I end up hunting for an old version of the build tool, and have to research the bugs it has, etc.  Given this mess, easier to never upgrade.17:08
persiaSince I think people should upgrade all the time, I think we should make it really easy to do so.17:08
locallycompactIt still reads stuff17:09
persiaand given the need to preserve the (infinite number of) readers, I don't expect it is much more code.17:10
persiaI may be wrong, and if it is lots more code, then I am happy to give up the argument.17:11
locallycompactIt's not possible, there is no functional mapping from 9 -> 817:11
persialocallycompact: Why not, assuming there is no variance in build dependencies for a given strata over the set of systems?17:12
persias/strata/stratum/17:12
locallycompactthat's why it's not functional17:13
persiaIt's functional: the function just sometimes returns "undefined"17:14
persia(depending on the input data)17:14
locallycompactthat's not mathmo functional17:15
* persia fails to parse "mathmo"17:15
locallycompactmathmonk17:16
persiaStill no, and my search engines don't help.17:17
locallycompacthigh priest mathmage17:17
locallycompactI'm going to bed17:17
persiahttps://en.wikipedia.org/wiki/Functional_(mathematics) ?17:18
locallycompactthat too but I was more17:18
locallycompacthttps://en.wikipedia.org/wiki/Functional_dependency17:19
* locallycompact sleep17:20
persiaSleep well.  When you wake, consider that the page you linked doesn't require that the target set contain unique values, as long as there is a value for each member of the set.  For example, VIN->engine_capacity works fine, even for cases where there is no engine, so engine_capacity is undefined.17:21
*** locallycompact has quit IRC17:25
*** rdale has quit IRC18:27
*** ctbruce has quit IRC20:24

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!