IRC logs for #baserock for Tuesday, 2016-02-09

*** gtristan has quit IRC04:19
*** gtristan has joined #baserock04:52
*** toscalix has joined #baserock07:49
*** ctbruce has joined #baserock08:25
* paulsherwood finally tracked down what was causing 500 errors in kbas...08:33
paulsherwoodhttp://stackoverflow.com/questions/21351687/bottle-template-not-found-error-50008:33
paulsherwoodTL; DR - don't change working directory in a bottle app08:34
*** gtristan has quit IRC08:58
*** rdale has joined #baserock09:01
*** bashrc has joined #baserock09:05
*** edcragg has joined #baserock09:12
*** franred has joined #baserock09:24
*** jonathanmaw has joined #baserock09:26
*** tiagogomes has joined #baserock09:44
*** edcragg has quit IRC09:54
*** gtristan has joined #baserock09:54
*** ssam2 has joined #baserock10:12
*** ChanServ sets mode: +v ssam210:12
*** edcragg has joined #baserock10:18
*** cosm has quit IRC10:32
paulsherwoodrdale: i think you mentioned elsewhere you may be interested in automating tests for ybd?10:37
paulsherwoodin any case i've raised an issue https://github.com/devcurmudgeon/ybd/issues/15710:37
rdalei you think it is a good idea10:37
rdalei don't believe in 'excessive tests' - maybe like you - but it would be nice to have some perhaps10:38
paulsherwoodi do think it's worth doing. i just want to avoid creating too many tests, or fragile tests10:38
* persia suggests a BDD DSL for automated tests, as being more transparent and accessible.10:38
paulsherwoodsuch as?10:38
persiaAvoid "functional" or "unit" tests, as they tend to be more fragile.10:38
rdaleplease not yarn10:38
persiaI like yarn and cucumber.  Behave is OK.  There are many others.10:38
rdalei'm not a fan of bdd at all10:39
persiaIf you use anything else, you're only testing that the code does what it was written to do, rather than that the code does what it was required to do.10:39
paulsherwoodwhat's wrong with yarn? it seemed very interesting when i first heard about it, but i was not able to understand it properly when i dove deeper10:39
persiaIf you dislike yarn, and you are writing python, http://pythonhosted.org/behave/ is well-recommended, although I've not used it myself.10:40
*** Lachlan1975 has joined #baserock10:40
rdalei don't believe in shell scripting via writing sentences in all upper case, i thought we'd got over that with COBOL10:40
* paulsherwood thinks bdd is ok for some classes of problem10:40
persiaI generally like yarn because it only uses the CLI interface for the tool, so doesn't expose the implementation language to the test environment, making it easier to reimplement things flexible.10:40
pedroalvarezjust a thought: It might be possible to reuse tests from morph?10:41
persias/ble/bly/10:41
rdalebehave is better than yarn as it is actually python, but i still don't like it10:41
persiapedroalvarez: Some of them: some of the morph tests are about morph implementation, rather than morph behaviour10:41
pedroalvarezof course, not the ones about morph implementation10:41
persiaSo, I like BDD, but I'm not advocating tests-before-code currently, only that I only believe in integrated testing for automated testing.  Anything lower-level tends to fall down when someone changes an implementation, in painful ways.10:42
paulsherwoodhttp://ravimohan.blogspot.co.uk/2007/04/learning-from-sudoku-solvers.html10:42
persiaAnd the BDD DSLs provide a nice way to describe what the system does, regardless of whether one writes the tests or the code first.10:42
paulsherwoodBDD requires tests before development - ok for some classes of problem. not for (eg) sudoku solver or other applicationswith heavy algorithm needs, iiuc10:43
ssam2BDD falls down when someone changes how the tool should work, in painful ways10:43
rdalelol  “Going in Circles Means You Don’t Know What You’re Doing”10:44
paulsherwood:-)10:44
persiapaulsherwood: Yes, but use of BDD DSLs for testing does not require tests before development.  Much like one can use a screwdriver without joining the guild these days.10:44
ssam2in the case of YBD that might not be an issue, since the interface is super simple so there's really only one type of test10:44
ssam2try to build something  -> did it build10:44
persiassam2: Fair, but that is true for any integration test suite :)10:45
persia(and I'd assert that the change is more "what the tool does" than "how the tool works", but these are essentially the same)10:45
rdalebuilding a minimal system to test ybd takes three hours on my x220 - i would like a test with dummy files that does the same thing only much quicker10:46
paulsherwoodrdale: 'no-build: True'10:47
rdaleright - but does it produce outputs in the .inst dir?10:47
paulsherwoodskips the build instructions, so artifacts are empty10:47
persiaAnd note that you aren't actually testing if it builds, if you test that way, only that it can handle that parameter.10:47
persia(this is a useful test, just not a "build test")10:48
paulsherwoodactually it's testing that the logic of parsing definitions, and assembling therefrom, works10:48
gtristanif I were to write tests for ybd I'd probably have a fixed set of definitions for it10:48
gtristanmaybe I would have one that checks the environment variables in the sandbox were sane10:49
paulsherwoodif I were to write tests for ybd I'd probably start with the earliest known buildable official definitions tag10:49
rdaleand ignore migrations?10:49
paulsherwoodmigrations are not part of ybd. ybd so far aims to be backwards compatible, but i haven't tested that for a while10:50
richard_mawrdale: Morph builds a fake shell so it can actually test the build isolation, but tbh this is one of the reasons why I was writing bits as independent components, before house purchasing took all my free time, so it would be easier to test the inidiviual parts.10:50
paulsherwoodrdale: maybe it would be worthwhile to run tests on a) earliest known buildable release b) another representative release c) latest tag d) master10:52
rdalei was discussing this with ssam2 at fosdem, and he said when he tried old defintions don't work at all10:53
rdalemigrations should be part of definitions, not part of morph if they aren't already10:53
paulsherwoodthey are part of definitions iiuc10:54
gtristanwhile using existing definitions as a means of validating ybd "works" is an interesting and valuable thing to do - it presumes that "definitions is correct"; while I think it's also interesting of defining what is correct behavior outside of definitions10:55
gtristanfor instance, we ran into some interesting undefined things in the spec this way10:55
gtristanlike that morph assumed that chunks inside a strata had some kind of expected ordering10:55
* gtristan has bad grammar today... s/of defining/to define/10:56
paulsherwoodack. but i think that's testing of *definitions* not testing of ybd10:57
gtristanits essentially both10:58
paulsherwoodyup... i started down this road ages ago with a simplistic json-schema implementation... i think rdale and ssam2 have thought further about this10:58
paulsherwoodwoo ssam2's talk is up http://mirrors.dotsrc.org/fosdem/2016/k4201/the-story-of-a-declarative-structured-format-for-build-and-integration-instructions.mp410:59
rdalei'm playing with ssam2's owl schema atm10:59
rdaleit was a really good talk11:00
gtristanI wonder if the lightning talks will be posted, or were recorded11:01
persiathey were recorded11:07
rdalethe unix history one was brilliant11:07
rdaleusing git blame to see what ken thompson did11:08
*** jonathanmaw__ has joined #baserock11:10
*** jonathanmaw has quit IRC11:12
ssam2that's https://github.com/dspinellis/unix-history-repo right ?11:12
rdaleyes11:12
*** locallycompact has joined #baserock11:13
locallycompactI'm trying to deploy this cluster with ybd but it isn't doing anything. (ybd 16.01) https://paste.fedoraproject.org/320216/50166631/11:19
pedroalvarezlocallycompact: you are deploying a x86_32 system and setting the arch of ybd to x86_6411:20
locallycompactah, ty11:21
pedroalvareznp11:21
pedroalvarezi take you can't ever deploy systemos of a different architecture using ybd11:22
pedroalvarezs/systemos/systems/11:23
locallycompactdifferent error now11:26
locallycompacthttps://paste.fedoraproject.org/320219/17156145/11:26
pedroalvarezthat might be that the image size is too small to put all the contents11:27
pedroalvarezchange the size in the cluster11:27
pedroalvarezI think this is due ybd (that version) not splitting the artifacts, and making that image not minimal at all11:27
rdalemaybe it should be putting the image into /src rather than /tmp11:29
paulsherwoodlocallycompact: would you mind trying ybd master, now? i fixed the warnings last night... so the splitting part should behave as expected11:29
locallycompactsure I'll do that11:29
pedroalvarezrdale: /tmp/tmpXXX is just a mount point I believe11:30
rdaleare ok11:30
pedroalvarezwhat i think  is full is the disk image that is mounted in that mountpoint11:31
paulsherwoodpedroalvarez: it would be trivial to allow deployment of other arches, but i wonder if the deployment *scripts* would behave predictably in that scenario?11:31
pedroalvarezwe have been always caring about making them work in that scenario11:32
locallycompactpaulsherwood, it didn't attempt to rebuild anything updating to master, expected?11:32
paulsherwoodlocallycompact: given you previously built it with 16.06, i expect that's correct11:32
paulsherwoodbut you could remove the minimal-system artifacts and re-run11:33
paulsherwoods/could/should/11:33
*** jonathanmaw__ has quit IRC11:33
pedroalvarezare the artifacts different since artifact splitting  works in ybd?11:33
paulsherwoodwell, they should be smaller :)11:33
*** jonathanmaw has joined #baserock11:33
paulsherwoodand yes, metafiles are now yaml11:33
paulsherwoodrdale can explain more11:34
paulsherwoodsince 16.06 there is an artifact-version: config11:34
rdaleare the cache keys the same for the splitting version?11:34
rdaleor do we have an artifact version - i did suggest discussing that?11:34
paulsherwood11:34 < paulsherwood> since 16.06 there is an artifact-version: config11:35
rdaleah11:35
pedroalvarezI'm just wondering if ybd is producing different artifacts with the same cache key11:35
paulsherwoodpedroalvarez: not any more :)11:35
paulsherwood16.05 had that bug.... i was trying to sweep it under the carpet11:35
paulsherwood(hence the rapid release)11:35
pedroalvarezhhhah11:37
paulsherwoodseen in #reproducible-builds 10:08 < anthraxx> diffoscope is so great. yesterday i found a bug in my reproducible code by using diffoscope in like 30sec :D11:38
rdaleit has some giant dependencies - about half a gig of mono11:39
*** fay_ is now known as faybrocklebank11:53
perryli'm having trouble building build-essential with YBD; i get a `git clone failed` error when it reaches stage2-linux-api-headers, does anyone know what may be happening?12:35
locallycompactperryl, paste?12:42
locallycompactI have also more strangeness with ybd deploy. https://paste.fedoraproject.org/320239/14550217/12:42
perryllocallycompact: sure, one moment12:43
perrylhttp://paste.baserock.org/olakarokam12:45
paulsherwoodlocallycompact: that's a bug :/12:47
locallycompactperryl, What version ybd?12:50
* paulsherwood should stop coding at night12:50
ratmicei generally just stuff tests in the source files themselves using if __name__ == "__main__", or PIE executables, which is then split into 2 sections example usage, and failing gracefully12:50
paulsherwoodmaybe s/at night//12:51
ratmicei think the last part is important because humans are actively trying to avoid that testing path :)\12:51
perryllocallycompact: latest12:51
locallycompacthm12:51
pedroalvarezpaulsherwood: :)12:51
perryllatest being commit 5fbacd98e12:52
paulsherwoodlocallycompact: do you have a cluster morph i can use to reproduce that?12:52
locallycompactpaulsherwood, it's in the paste12:52
locallycompacthttps://paste.fedoraproject.org/320239/14550217/raw/12:52
locallycompactor just use the default minimal-deploy12:53
*** Lachlan1975 has quit IRC12:58
ratmicealso, stuff like coverage.readthedocs.org is useful, not just for finding untested stuff, but for finding the stuff thats tested 50 million times because they sit in the path of every test13:15
paulsherwoodratmice: thanks!13:15
locallycompactperryl, is that ref present in /root/ybd/gits/git___git_baserock_org_delta_linux13:18
rdaleis there any reason why ybd doesn't have a main() function and a13:19
rdaleif __name__ == "__main__": test?13:19
paulsherwoodrdale: iirc ssam2 did that, in making ybd work as a package13:23
paulsherwood(the 'main' processing i mean). as for the construct you're describing, i don't understand what it would do13:24
locallycompactpaulsherwood, what's this 'sandbox' field supposed to be about?13:24
rdalei'm not a python expert, but my understanding is that would make writing tests easier as the main() function wouldn't get executed in test mode and you could do a bit of monkey patching instead13:25
richard_mawrdale: there's two approaches, one is that yes, if you use the `if __name__ == '__main__'` you can import the script without running it, which means you can call its functions as part of tests13:26
richard_mawanother is that you write your code as modules, and run the test suite in the if __name__ == '__main__' section13:26
ratmiceright, i was talking about the latter13:26
* richard_maw prefers doing that, and having a separate module for the cli interface, which has the __name__ == '__main__' run the program13:27
richard_mawsince it's better to use different tools to test the command-line interface of a program to its modules13:28
rdalei think i try the first approach at first as it's less invasive of the current code13:28
paulsherwoodperryl: i'm guessing your problem is either a corrupted repo - maybe a git wizard can suggest a way to check it? - or not enough space for the clone?13:32
paulsherwoodin any case you maybe could re-run the git clone command to see what the problem is13:33
paulsherwoodhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/repos.py#L18413:33
paulsherwoodgit clone --no-hardlinks /root/ybd/gits/git___git_baserock_org_delta_linux /root/ybd/tmp/foo13:34
paulsherwoodlocallycompact: it's supposed to be the tmpdir in which everything happens13:35
* paulsherwood is failing completely to deploy anything, since his machine has no kvm13:35
paulsherwoodit's a long time since i was able to test deploy - baserock systems stopped playing nicely with vbox on my latest laptop13:36
* paulsherwood is an idiot sometimes.... failed to convert the sandbox.setup() in deployment.py to be a 'with sandbox.setup()'13:44
paulsherwoodperryl: any luck?14:01
mwilliams_ctperryl is away atm, probably gone for lunch14:02
locallycompactsomething strange here14:03
locallycompacthttps://paste.fedoraproject.org/320268/26523145/14:03
locallycompactif I change that java-build-system to minimal-system, it deploys fine14:03
*** gtristan has quit IRC14:05
paulsherwoodIOError: [Errno 28] No space left on device14:07
paulsherwoodhow big is your /tmp volume?14:07
locallycompactsmall14:08
locallycompactI thought it would write to /src/tmp?14:08
locallycompacttmp is 1.9G I think14:08
paulsherwoodif base is set to /src, it should...14:08
locallycompactit is set there14:09
paulsherwoodyup... '/src/tmp is directory for tmp'14:09
locallycompacthave 80G free on /src14:09
paulsherwoodSotK_: any ideas on this, please? iirc you did a lot of improvements on deploy14:10
* SotK has a look14:11
*** Lachlan1975 has joined #baserock14:15
perrylpaulsherwood: i've tried, and got an 'insufficient space left on device' message. i've followed the setup instructions for kvm, but it seems that it's only writing to /dev/sda, oddly14:20
SotKpaulsherwood: looks like its passing `this['sandbox']` to the write extension (which in locallycompact's case appears to be `/tmp/tmpCsZaqt`)14:20
* perryl reboots, tries again, will scrap and create new VM if this fails14:21
locallycompactSotK, where does that happen?14:24
*** gtristan has joined #baserock14:24
SotKlocallycompact: https://github.com/devcurmudgeon/ybd/blob/master/ybd/sandbox.py#L20714:25
locallycompactwhat should it be14:25
SotKI think `this['sandbox']` is the right thing to pass (the write extension expects a path to the unpacked system artifact)14:26
SotKidk why it isn't in your tempdir14:26
SotKwhat does your ybd config look like?14:27
locallycompactjust base: '/src'14:28
paulsherwoodthe earlier sandboxes are correctly in /src, eg Renaming networkd file from systemd chunk:                             /src/tmp/tmpiNsZS7/etc/systemd/network/10-dhcp.network to /src/tmp/tmpiNsZS7/etc/systemd/network/10-dhcp.network.morph14:29
SotKhttps://github.com/devcurmudgeon/ybd/blob/master/ybd/deployment.py#L76 appears to be the only thing that does anything to the sandbox that I can see14:32
franredSotK, doesn't that chmod command fail?  0o755 seems like a typo to me14:35
* SotK doesn't know, its newer than his last interaction with ybd14:35
richard_mawfranred: it's not14:35
franredoh14:36
richard_maw0o7 is a valid integer literal14:36
richard_mawit's like 0xa14:36
richard_mawyou can even do 0b101001014:37
SotKhah, its my own code originally it seems14:37
franredoh, I see. thanks richard_maw14:37
richard_maw14:38
* SotK digs more and shuts up until he finishes digging14:39
SotKok, it isn't my code after all, but it was working in the past, so I doubt that line is to blame14:41
SotKunless its always used /tmp and no-one has noticed14:41
* SotK spots that kvm.write is actually to blame14:45
SotKhttp://git.baserock.org/cgit/baserock/baserock/definitions.git/tree/extensions/kvm.write#n4714:45
richard_mawmkstemp should be using the contents of TMPDIR14:46
SotKdoes ybd set TMPDIR?14:46
richard_mawapparently not14:47
richard_mawmorph set it to a temporary subdirectory of the --tempdir setting, which we usually set to /src/tmp14:47
paulsherwoodso the fix is for ybd to export TMPDIR?14:51
rjekDo ybd and morph sanatise the environments they provide to child processes, OOI?14:51
richard_mawaye, may also be worth adding a context manager to have ybd export TMPDIR as a temporary directory inside --tempdir, which it cleans up after runing the deployment14:52
richard_mawrjek: for builds, yes14:52
paulsherwoodbut not for (for example) git operations14:52
richard_mawnor for deployments in morph at least14:52
* rjek idly wonders if they should14:53
* richard_maw thought so, but Lars did not, since they may require things like the SSH agent environment to connect to the target server etc.14:53
rjekMore from a reproducability and security standpoint; environment variables can have surprising side effects.  I was just thinking perhaps both tools should always clean and only put in what they know they want and need for any subprocess14:55
rjekThere's an interested chapter on this subject in O'Reilly's Secure Coding Cookbook, btw14:55
locallycompactWhat should I put and where to test this fix?14:55
paulsherwood    os.environ('TMPDIR') = app.config['tmp']14:59
paulsherwoodl43 sandbox.py14:59
richard_mawis that copied varbatim?14:59
paulsherwoodit's my proposed patch14:59
richard_mawyou want os.environ['TMPDIR'] at least then15:00
paulsherwoodoops - thanks15:00
rjekWhenever I am writting Python I am constantly bitten by all the different ways you can dereference a thing in exactly that way :)15:00
paulsherwoodlocallycompact: os.environ['TMPDIR'] = app.config['tmp']15:00
richard_mawrjek: same number as lua15:01
rjekfoo['bar'] is identical to foo.bar15:01
richard_mawit's just that lua doesn't have a difference between entries and keys15:01
richard_maws/keys/attributes/15:01
rjekWhere in python they refer to different bars15:01
richard_mawaye, and for python's object model that is preferable, as it allows you to have a set object that is indexable with [] syntax, and also methods for operating on the set15:02
richard_mawpython has foo[bar], foo.get(bar, default) and foo.setdefault(bar, default)15:03
richard_mawbut also a separate `bar in foo` to determine contents, while lua has it return the null value, which makes things difficult if you want to explicitly store null and unset differently15:04
locallycompactsuccess, ty paulsherwood SotK richard_maw15:04
* richard_maw would also prefer morph's approach of having TMPDIR set for each deployment to a tempdir in /src/tmp, so deployments don't accidentally pick up each others' temporary files15:05
rjekShouldn't the tmpdir be cleaned after each step anyway?15:05
paulsherwoodyes, but the above patch is sharing TMPDIR - i'll improve it as richard_maw suggests15:07
paulsherwoodybd runs multi-instance, so it's possible that multiple deploys could run at once (although not in the current code)15:07
richard_mawpaulsherwood: is that TMPDIR set just for the deploys, or the builds too15:08
richard_mawif it's for the builds too things could get messed up, since /src/tmp isn't a writable path inside the sandbox15:08
richard_mawrjek: chrooted builds are15:08
rjeknod15:09
paulsherwoodjust deploys - builds get their own sandboxes15:11
paulsherwoodand so do deploys... so this shold be easy to improve15:12
paulsherwoodlocallycompact: would you mind re-running, with os.environ['TMPDIR'] = this['sandbox']15:12
locallycompactsec15:13
paulsherwoodbenbrown_: thanks for the pr - pls can you explain what is the use-case? when would we encounter a non-bare repo?15:15
locallycompactchanging that to that did this https://paste.fedoraproject.org/320308/31218145/15:20
richard_mawnote that just because the paths say /src/tmp doesn't mean the disk image was created in /src/tmp15:21
richard_mawjust that it was mounted in /src/tmp15:21
richard_mawpaulsherwood: alwo works when the repo is corrupted in another fashion15:23
richard_mawsomething morph didn't handle very well15:24
benbrown_paulsherwood: Ah yes, when using concourse to pull repos into gits they will be non-bare.15:26
benbrown_Not wanting to put anything explicitly concourse specific, this seemed like the best approach.15:26
richard_mawconcourse populates your /src/gits?15:26
locallycompactconcourse fills a container for the build step with git repos15:27
benbrown_richard_maw: We sets 'gits: magic_place_where_concourse_puts_things'15:27
benbrown_s/sets/set/15:27
richard_mawwith appropriate name mangling?15:28
* benbrown_ nods15:28
richard_mawlooks like you also need to do the rest of the repo mangling outside of ybd then15:28
ssam2as an aside, i once wanted something that would expose my Baserock /src/cache/gits directory as a normal Git server15:28
richard_mawsince otherwise you're just throwing the repo away completely15:28
ssam2should be possible to hack up on top of git-daemon, but i never got it working15:28
ssam2it would mean scripts could clone 'git://localhost/delta/gcc' for example and actually clone /src/cache/gits/git___git_baserock_org_delta_gcc or whatever15:29
paulsherwood+115:30
paulsherwoodwhy do we need 'bare'?15:30
ssam2bare means 'no working tree', which is sensible if the repo is only there for people to clone it15:30
richard_mawpaulsherwood: when you fetch git complains that the refs are in a different layout15:30
paulsherwoodbenbrown_: it sounds like the short-term soln would be better doing the mangling in whatever script is driving concourse?15:31
richard_mawpaulsherwood: for what purpose would you actually use the ability to clone git://localhost/delta/gcc? you'd be better off with a git wrapper that called git clone with --local from your cache15:31
richard_mawparticularly since the change will completely throw away what concourse put in your repo anyway15:32
richard_mawbetter off mangling it to be a bare repo externally, so it will at least save having to fetch the branch concourse already fetched for you15:32
paulsherwoodrichard_maw: i guess the functionality i'm interested in really is the equivalent of kbas for gits - ability for one ybd user to act as a local server of gits to other users on a local network, save them hammering internet/gbo15:33
paulsherwoodbenbrown_: back to the drawing board, i fear? :)15:33
benbrown_richard_maw: Aye, that might be best tbh, though the reason I went with this is so that it only processed repos as necessary.15:33
ssam2richard_maw: I wanted that daemon thing so that I could write scripts that use Morph/YBD's git cache without having to teach them about the special way in which it works15:37
ssam2richard_maw: it's a step towards making the caching a commodity. You have the script just do `git clone git://$MIRROR/delta/gcc`, and then if you set MIRROR=localhost and run this magical git-daemon tool, it "just works"15:38
ssam2richard_maw: whereas right now you have to have an `if repo is cached locally then use that else clone the remote one and cache it locally and then use that one, or don't cache it, or avoid doing networking connectivity and fail depending on what options the user passed in'15:39
richard_mawssam2: sounds like it would be better as a separate project that provided something like git-remote-localcache so you could do `git clone localcache::git://git.baserock.org/delta/gcc.git`, which knows how to maintain a cache locally15:39
richard_mawthe baserock build tool could then make use of it without needing to have its own cache logic15:40
ssam2yes, that's the idea15:40
* richard_maw was thinking of adding a wrapper around https://github.com/fishface60/python-gitcache that would do that15:41
richard_mawso you wouldn't need to invoke a different command to clone it15:41
richard_mawafter I've settled down again I was planning to go back to it and finish it up, but I lost my free time15:42
ssam2yes, that sounds ideal15:56
ssam2looking at python-gitcache seems you've come up with the concept of a 'coarse' git cache.. what would a less 'course' git cache be? one that shared all the objects between all the repos?16:00
ssam2less 'coarse' I mean16:00
richard_mawcoarse vs fine-grained was named about the implementation detail of how I wanted to do the locking16:02
richard_mawbut yes, a fine-grained one could have all the objects in the same object store, and have the refs of all the repositories available as git namespaces16:02
richard_mawyou'd want to be very careful about when you decided to repack a cache like that16:03
locallycompactHas there been any thought given as to what would be required to shoehorn^Btranslate maven dependencies into definitions?16:06
ssam2you mean, writing an import tool that would create a .morph file listing all the dependencies from a Maven POM file?16:07
ssam2I think the only tricky thing there is how to go from a Maven version string to a Git commit16:08
ssam2for released versions I guess the version string should correspond to a tag16:08
locallycompactIt's more that maven expect to be able to contact the internet16:08
locallycompactunless I'm misunderstanding maven16:08
ssam2oh, the question is how to *run* `mvn` inside a Baserock sandbox?16:08
ssam2yeah that's a bit tricky16:08
ssam2I guess the only "clean" way to do this is include a Maven repo that contains everything you need, inside the sandbox16:09
ssam2building all that from scratch from a C compiler with only Baserock tools sounds like a multi-year project16:10
ssam2but personally I think it makes sense to embrace binary delivery for Java artifacts, that is the whole point of Java existing after all16:10
ssam2so you'd just need a way of injecting the various .jars you need, plus a Java runtime, into the staging area, and setting up Maven to look in the right place for all the jars... I guess16:11
ssam2some kind of 'maven-bootstrap' git repo that contained all those binary .jar files would be the quickest route to 'something' I guess16:11
paulsherwoodi think i remain philosophically opposed to binaries in this context, but i may be alone :)16:11
locallycompactI have no strong opinion on binaries, but in that case I can skip maven entirely and just import riemann16:12
paulsherwoodlol16:13
locallycompactbuilding riemann requires clojure, building clojure requires maven and maven requires maven16:13
ssam2makes sense16:15
ssam2i think getting strong reproducibility in Java isn't something we can do as a side effect of other work, it'd have to be a major focus for us to have any kind of success16:17
jmacsI agree16:20
*** ctbruce has quit IRC16:23
locallycompacthmm, it doesn't seem like java made it into this java-build-system16:24
paulsherwoodlocallycompact: you could list /src/artifacts/*/*.unpacked for clues16:25
locallycompactnvm, it did my brain just rebelled16:25
* jmacs notes that java.morph still points to his github account :S16:26
rjekA definition that points at github is not reproducable16:27
rjek:)16:27
jmacsWhy not?16:27
rjekIn the general sense; I often find links to older projects on github are dead16:28
ssam2yes, it's done that way as a cheat so we don't have to commit it to a repo on git.baserock.org16:29
ssam2it's not at all a good foundation for building real systems that use Java16:29
ssam2for the Gerrit system, I deployed it with Baserock but just injected a binary Java runtime environment at deploy time16:30
ssam2I think that's a more sensible approach than trying to stick binary artifacts into source control16:30
ssam2i mean, I built it with Baserock, I deployed it with Ansible :-)16:30
locallycompactssam2, is that somewhere?16:31
jmacsBut it breaks traceability somewhat. You'd need to keep track of which version of the JRE you injected and ideally store that version somewhere16:31
ssam2http://git.baserock.org/cgit/baserock/baserock/infrastructure.git/tree/baserock_gerrit/instance-config.yml16:31
ssam2the Ansible script is a bit of a mess, I'm not sure that's the best way to do this either, but I think the principle is better16:32
locallycompactis that ansible?16:32
ssam2that link points to an ansible playbook16:33
ssam2which basically installs the JRE and gerrit onto a running system16:33
ssam2and does other setup16:33
ssam2so, it does record exactly what version of the JRE is injected16:34
locallycompacthow does that know where to download16:34
ssam2it depends on Oracle hosting the JRE, but I think that if they mess that up then it's a problem for the whole JAva world16:34
ssam2ah16:34
ssam2i forgot something16:34
ssam2it doesn't download the JRE because of the stupid Javascript license checks Oracle do16:34
ssam2so you have to download it yourself and put it locally16:35
ssam2I'm not sure if the JRE license allows publically hosting the binaries somewhere else (redistributing them) or not16:35
locallycompactput it locally when/where?16:36
ssam2when you run the Ansible script16:36
ssam2on whatever machine you run the Ansible script on (probably the same one you used for the rest of the build and deploy process)16:37
locallycompactoh I thought ansible ran on the target16:37
locallycompactok16:37
ssam2no, it just needs ssh access to the target16:37
jmacsIt'd be good to check the sha1 of the jre tarball16:37
ssam2yes, that'd be good16:37
*** toscalix has quit IRC17:50
*** edcragg has quit IRC17:58
*** jonathanmaw has quit IRC18:00
*** bashrc has quit IRC18:01
*** ssam2 has quit IRC18:16
*** Lachlan1975 has quit IRC18:17
*** locallycompact has quit IRC18:18
*** rdale has quit IRC18:53
*** cosm has joined #baserock19:37
*** cosm has quit IRC22:32
*** cosm has joined #baserock22:34

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!