| *** gtristan has quit IRC | 04:19 | |
| *** gtristan has joined #baserock | 04:52 | |
| *** toscalix has joined #baserock | 07:49 | |
| *** ctbruce has joined #baserock | 08:25 | |
| * paulsherwood finally tracked down what was causing 500 errors in kbas... | 08:33 | |
| paulsherwood | http://stackoverflow.com/questions/21351687/bottle-template-not-found-error-500 | 08:33 |
|---|---|---|
| paulsherwood | TL; DR - don't change working directory in a bottle app | 08:34 |
| *** gtristan has quit IRC | 08:58 | |
| *** rdale has joined #baserock | 09:01 | |
| *** bashrc has joined #baserock | 09:05 | |
| *** edcragg has joined #baserock | 09:12 | |
| *** franred has joined #baserock | 09:24 | |
| *** jonathanmaw has joined #baserock | 09:26 | |
| *** tiagogomes has joined #baserock | 09:44 | |
| *** edcragg has quit IRC | 09:54 | |
| *** gtristan has joined #baserock | 09:54 | |
| *** ssam2 has joined #baserock | 10:12 | |
| *** ChanServ sets mode: +v ssam2 | 10:12 | |
| *** edcragg has joined #baserock | 10:18 | |
| *** cosm has quit IRC | 10:32 | |
| paulsherwood | rdale: i think you mentioned elsewhere you may be interested in automating tests for ybd? | 10:37 |
| paulsherwood | in any case i've raised an issue https://github.com/devcurmudgeon/ybd/issues/157 | 10:37 |
| rdale | i you think it is a good idea | 10:37 |
| rdale | i don't believe in 'excessive tests' - maybe like you - but it would be nice to have some perhaps | 10:38 |
| paulsherwood | i do think it's worth doing. i just want to avoid creating too many tests, or fragile tests | 10:38 |
| * persia suggests a BDD DSL for automated tests, as being more transparent and accessible. | 10:38 | |
| paulsherwood | such as? | 10:38 |
| persia | Avoid "functional" or "unit" tests, as they tend to be more fragile. | 10:38 |
| rdale | please not yarn | 10:38 |
| persia | I like yarn and cucumber. Behave is OK. There are many others. | 10:38 |
| rdale | i'm not a fan of bdd at all | 10:39 |
| persia | If you use anything else, you're only testing that the code does what it was written to do, rather than that the code does what it was required to do. | 10:39 |
| paulsherwood | what's wrong with yarn? it seemed very interesting when i first heard about it, but i was not able to understand it properly when i dove deeper | 10:39 |
| persia | If you dislike yarn, and you are writing python, http://pythonhosted.org/behave/ is well-recommended, although I've not used it myself. | 10:40 |
| *** Lachlan1975 has joined #baserock | 10:40 | |
| rdale | i don't believe in shell scripting via writing sentences in all upper case, i thought we'd got over that with COBOL | 10:40 |
| * paulsherwood thinks bdd is ok for some classes of problem | 10:40 | |
| persia | I generally like yarn because it only uses the CLI interface for the tool, so doesn't expose the implementation language to the test environment, making it easier to reimplement things flexible. | 10:40 |
| pedroalvarez | just a thought: It might be possible to reuse tests from morph? | 10:41 |
| persia | s/ble/bly/ | 10:41 |
| rdale | behave is better than yarn as it is actually python, but i still don't like it | 10:41 |
| persia | pedroalvarez: Some of them: some of the morph tests are about morph implementation, rather than morph behaviour | 10:41 |
| pedroalvarez | of course, not the ones about morph implementation | 10:41 |
| persia | So, I like BDD, but I'm not advocating tests-before-code currently, only that I only believe in integrated testing for automated testing. Anything lower-level tends to fall down when someone changes an implementation, in painful ways. | 10:42 |
| paulsherwood | http://ravimohan.blogspot.co.uk/2007/04/learning-from-sudoku-solvers.html | 10:42 |
| persia | And the BDD DSLs provide a nice way to describe what the system does, regardless of whether one writes the tests or the code first. | 10:42 |
| paulsherwood | BDD requires tests before development - ok for some classes of problem. not for (eg) sudoku solver or other applicationswith heavy algorithm needs, iiuc | 10:43 |
| ssam2 | BDD falls down when someone changes how the tool should work, in painful ways | 10:43 |
| rdale | lol “Going in Circles Means You Don’t Know What You’re Doing” | 10:44 |
| paulsherwood | :-) | 10:44 |
| persia | paulsherwood: Yes, but use of BDD DSLs for testing does not require tests before development. Much like one can use a screwdriver without joining the guild these days. | 10:44 |
| ssam2 | in the case of YBD that might not be an issue, since the interface is super simple so there's really only one type of test | 10:44 |
| ssam2 | try to build something -> did it build | 10:44 |
| persia | ssam2: Fair, but that is true for any integration test suite :) | 10:45 |
| persia | (and I'd assert that the change is more "what the tool does" than "how the tool works", but these are essentially the same) | 10:45 |
| rdale | building a minimal system to test ybd takes three hours on my x220 - i would like a test with dummy files that does the same thing only much quicker | 10:46 |
| paulsherwood | rdale: 'no-build: True' | 10:47 |
| rdale | right - but does it produce outputs in the .inst dir? | 10:47 |
| paulsherwood | skips the build instructions, so artifacts are empty | 10:47 |
| persia | And note that you aren't actually testing if it builds, if you test that way, only that it can handle that parameter. | 10:47 |
| persia | (this is a useful test, just not a "build test") | 10:48 |
| paulsherwood | actually it's testing that the logic of parsing definitions, and assembling therefrom, works | 10:48 |
| gtristan | if I were to write tests for ybd I'd probably have a fixed set of definitions for it | 10:48 |
| gtristan | maybe I would have one that checks the environment variables in the sandbox were sane | 10:49 |
| paulsherwood | if I were to write tests for ybd I'd probably start with the earliest known buildable official definitions tag | 10:49 |
| rdale | and ignore migrations? | 10:49 |
| paulsherwood | migrations are not part of ybd. ybd so far aims to be backwards compatible, but i haven't tested that for a while | 10:50 |
| richard_maw | rdale: Morph builds a fake shell so it can actually test the build isolation, but tbh this is one of the reasons why I was writing bits as independent components, before house purchasing took all my free time, so it would be easier to test the inidiviual parts. | 10:50 |
| paulsherwood | rdale: maybe it would be worthwhile to run tests on a) earliest known buildable release b) another representative release c) latest tag d) master | 10:52 |
| rdale | i was discussing this with ssam2 at fosdem, and he said when he tried old defintions don't work at all | 10:53 |
| rdale | migrations should be part of definitions, not part of morph if they aren't already | 10:53 |
| paulsherwood | they are part of definitions iiuc | 10:54 |
| gtristan | while using existing definitions as a means of validating ybd "works" is an interesting and valuable thing to do - it presumes that "definitions is correct"; while I think it's also interesting of defining what is correct behavior outside of definitions | 10:55 |
| gtristan | for instance, we ran into some interesting undefined things in the spec this way | 10:55 |
| gtristan | like that morph assumed that chunks inside a strata had some kind of expected ordering | 10:55 |
| * gtristan has bad grammar today... s/of defining/to define/ | 10:56 | |
| paulsherwood | ack. but i think that's testing of *definitions* not testing of ybd | 10:57 |
| gtristan | its essentially both | 10:58 |
| paulsherwood | yup... i started down this road ages ago with a simplistic json-schema implementation... i think rdale and ssam2 have thought further about this | 10:58 |
| paulsherwood | woo ssam2's talk is up http://mirrors.dotsrc.org/fosdem/2016/k4201/the-story-of-a-declarative-structured-format-for-build-and-integration-instructions.mp4 | 10:59 |
| rdale | i'm playing with ssam2's owl schema atm | 10:59 |
| rdale | it was a really good talk | 11:00 |
| gtristan | I wonder if the lightning talks will be posted, or were recorded | 11:01 |
| persia | they were recorded | 11:07 |
| rdale | the unix history one was brilliant | 11:07 |
| rdale | using git blame to see what ken thompson did | 11:08 |
| *** jonathanmaw__ has joined #baserock | 11:10 | |
| *** jonathanmaw has quit IRC | 11:12 | |
| ssam2 | that's https://github.com/dspinellis/unix-history-repo right ? | 11:12 |
| rdale | yes | 11:12 |
| *** locallycompact has joined #baserock | 11:13 | |
| locallycompact | I'm trying to deploy this cluster with ybd but it isn't doing anything. (ybd 16.01) https://paste.fedoraproject.org/320216/50166631/ | 11:19 |
| pedroalvarez | locallycompact: you are deploying a x86_32 system and setting the arch of ybd to x86_64 | 11:20 |
| locallycompact | ah, ty | 11:21 |
| pedroalvarez | np | 11:21 |
| pedroalvarez | i take you can't ever deploy systemos of a different architecture using ybd | 11:22 |
| pedroalvarez | s/systemos/systems/ | 11:23 |
| locallycompact | different error now | 11:26 |
| locallycompact | https://paste.fedoraproject.org/320219/17156145/ | 11:26 |
| pedroalvarez | that might be that the image size is too small to put all the contents | 11:27 |
| pedroalvarez | change the size in the cluster | 11:27 |
| pedroalvarez | I think this is due ybd (that version) not splitting the artifacts, and making that image not minimal at all | 11:27 |
| rdale | maybe it should be putting the image into /src rather than /tmp | 11:29 |
| paulsherwood | locallycompact: would you mind trying ybd master, now? i fixed the warnings last night... so the splitting part should behave as expected | 11:29 |
| locallycompact | sure I'll do that | 11:29 |
| pedroalvarez | rdale: /tmp/tmpXXX is just a mount point I believe | 11:30 |
| rdale | are ok | 11:30 |
| pedroalvarez | what i think is full is the disk image that is mounted in that mountpoint | 11:31 |
| paulsherwood | pedroalvarez: it would be trivial to allow deployment of other arches, but i wonder if the deployment *scripts* would behave predictably in that scenario? | 11:31 |
| pedroalvarez | we have been always caring about making them work in that scenario | 11:32 |
| locallycompact | paulsherwood, it didn't attempt to rebuild anything updating to master, expected? | 11:32 |
| paulsherwood | locallycompact: given you previously built it with 16.06, i expect that's correct | 11:32 |
| paulsherwood | but you could remove the minimal-system artifacts and re-run | 11:33 |
| paulsherwood | s/could/should/ | 11:33 |
| *** jonathanmaw__ has quit IRC | 11:33 | |
| pedroalvarez | are the artifacts different since artifact splitting works in ybd? | 11:33 |
| paulsherwood | well, they should be smaller :) | 11:33 |
| *** jonathanmaw has joined #baserock | 11:33 | |
| paulsherwood | and yes, metafiles are now yaml | 11:33 |
| paulsherwood | rdale can explain more | 11:34 |
| paulsherwood | since 16.06 there is an artifact-version: config | 11:34 |
| rdale | are the cache keys the same for the splitting version? | 11:34 |
| rdale | or do we have an artifact version - i did suggest discussing that? | 11:34 |
| paulsherwood | 11:34 < paulsherwood> since 16.06 there is an artifact-version: config | 11:35 |
| rdale | ah | 11:35 |
| pedroalvarez | I'm just wondering if ybd is producing different artifacts with the same cache key | 11:35 |
| paulsherwood | pedroalvarez: not any more :) | 11:35 |
| paulsherwood | 16.05 had that bug.... i was trying to sweep it under the carpet | 11:35 |
| paulsherwood | (hence the rapid release) | 11:35 |
| pedroalvarez | hhhah | 11:37 |
| paulsherwood | seen in #reproducible-builds 10:08 < anthraxx> diffoscope is so great. yesterday i found a bug in my reproducible code by using diffoscope in like 30sec :D | 11:38 |
| rdale | it has some giant dependencies - about half a gig of mono | 11:39 |
| *** fay_ is now known as faybrocklebank | 11:53 | |
| perryl | i'm having trouble building build-essential with YBD; i get a `git clone failed` error when it reaches stage2-linux-api-headers, does anyone know what may be happening? | 12:35 |
| locallycompact | perryl, paste? | 12:42 |
| locallycompact | I have also more strangeness with ybd deploy. https://paste.fedoraproject.org/320239/14550217/ | 12:42 |
| perryl | locallycompact: sure, one moment | 12:43 |
| perryl | http://paste.baserock.org/olakarokam | 12:45 |
| paulsherwood | locallycompact: that's a bug :/ | 12:47 |
| locallycompact | perryl, What version ybd? | 12:50 |
| * paulsherwood should stop coding at night | 12:50 | |
| ratmice | i generally just stuff tests in the source files themselves using if __name__ == "__main__", or PIE executables, which is then split into 2 sections example usage, and failing gracefully | 12:50 |
| paulsherwood | maybe s/at night// | 12:51 |
| ratmice | i think the last part is important because humans are actively trying to avoid that testing path :)\ | 12:51 |
| perryl | locallycompact: latest | 12:51 |
| locallycompact | hm | 12:51 |
| pedroalvarez | paulsherwood: :) | 12:51 |
| perryl | latest being commit 5fbacd98e | 12:52 |
| paulsherwood | locallycompact: do you have a cluster morph i can use to reproduce that? | 12:52 |
| locallycompact | paulsherwood, it's in the paste | 12:52 |
| locallycompact | https://paste.fedoraproject.org/320239/14550217/raw/ | 12:52 |
| locallycompact | or just use the default minimal-deploy | 12:53 |
| *** Lachlan1975 has quit IRC | 12:58 | |
| ratmice | also, stuff like coverage.readthedocs.org is useful, not just for finding untested stuff, but for finding the stuff thats tested 50 million times because they sit in the path of every test | 13:15 |
| paulsherwood | ratmice: thanks! | 13:15 |
| locallycompact | perryl, is that ref present in /root/ybd/gits/git___git_baserock_org_delta_linux | 13:18 |
| rdale | is there any reason why ybd doesn't have a main() function and a | 13:19 |
| rdale | if __name__ == "__main__": test? | 13:19 |
| paulsherwood | rdale: iirc ssam2 did that, in making ybd work as a package | 13:23 |
| paulsherwood | (the 'main' processing i mean). as for the construct you're describing, i don't understand what it would do | 13:24 |
| locallycompact | paulsherwood, what's this 'sandbox' field supposed to be about? | 13:24 |
| rdale | i'm not a python expert, but my understanding is that would make writing tests easier as the main() function wouldn't get executed in test mode and you could do a bit of monkey patching instead | 13:25 |
| richard_maw | rdale: there's two approaches, one is that yes, if you use the `if __name__ == '__main__'` you can import the script without running it, which means you can call its functions as part of tests | 13:26 |
| richard_maw | another is that you write your code as modules, and run the test suite in the if __name__ == '__main__' section | 13:26 |
| ratmice | right, i was talking about the latter | 13:26 |
| * richard_maw prefers doing that, and having a separate module for the cli interface, which has the __name__ == '__main__' run the program | 13:27 | |
| richard_maw | since it's better to use different tools to test the command-line interface of a program to its modules | 13:28 |
| rdale | i think i try the first approach at first as it's less invasive of the current code | 13:28 |
| paulsherwood | perryl: i'm guessing your problem is either a corrupted repo - maybe a git wizard can suggest a way to check it? - or not enough space for the clone? | 13:32 |
| paulsherwood | in any case you maybe could re-run the git clone command to see what the problem is | 13:33 |
| paulsherwood | https://github.com/devcurmudgeon/ybd/blob/master/ybd/repos.py#L184 | 13:33 |
| paulsherwood | git clone --no-hardlinks /root/ybd/gits/git___git_baserock_org_delta_linux /root/ybd/tmp/foo | 13:34 |
| paulsherwood | locallycompact: it's supposed to be the tmpdir in which everything happens | 13:35 |
| * paulsherwood is failing completely to deploy anything, since his machine has no kvm | 13:35 | |
| paulsherwood | it's a long time since i was able to test deploy - baserock systems stopped playing nicely with vbox on my latest laptop | 13:36 |
| * paulsherwood is an idiot sometimes.... failed to convert the sandbox.setup() in deployment.py to be a 'with sandbox.setup()' | 13:44 | |
| paulsherwood | perryl: any luck? | 14:01 |
| mwilliams_ct | perryl is away atm, probably gone for lunch | 14:02 |
| locallycompact | something strange here | 14:03 |
| locallycompact | https://paste.fedoraproject.org/320268/26523145/ | 14:03 |
| locallycompact | if I change that java-build-system to minimal-system, it deploys fine | 14:03 |
| *** gtristan has quit IRC | 14:05 | |
| paulsherwood | IOError: [Errno 28] No space left on device | 14:07 |
| paulsherwood | how big is your /tmp volume? | 14:07 |
| locallycompact | small | 14:08 |
| locallycompact | I thought it would write to /src/tmp? | 14:08 |
| locallycompact | tmp is 1.9G I think | 14:08 |
| paulsherwood | if base is set to /src, it should... | 14:08 |
| locallycompact | it is set there | 14:09 |
| paulsherwood | yup... '/src/tmp is directory for tmp' | 14:09 |
| locallycompact | have 80G free on /src | 14:09 |
| paulsherwood | SotK_: any ideas on this, please? iirc you did a lot of improvements on deploy | 14:10 |
| * SotK has a look | 14:11 | |
| *** Lachlan1975 has joined #baserock | 14:15 | |
| perryl | paulsherwood: i've tried, and got an 'insufficient space left on device' message. i've followed the setup instructions for kvm, but it seems that it's only writing to /dev/sda, oddly | 14:20 |
| SotK | paulsherwood: looks like its passing `this['sandbox']` to the write extension (which in locallycompact's case appears to be `/tmp/tmpCsZaqt`) | 14:20 |
| * perryl reboots, tries again, will scrap and create new VM if this fails | 14:21 | |
| locallycompact | SotK, where does that happen? | 14:24 |
| *** gtristan has joined #baserock | 14:24 | |
| SotK | locallycompact: https://github.com/devcurmudgeon/ybd/blob/master/ybd/sandbox.py#L207 | 14:25 |
| locallycompact | what should it be | 14:25 |
| SotK | I think `this['sandbox']` is the right thing to pass (the write extension expects a path to the unpacked system artifact) | 14:26 |
| SotK | idk why it isn't in your tempdir | 14:26 |
| SotK | what does your ybd config look like? | 14:27 |
| locallycompact | just base: '/src' | 14:28 |
| paulsherwood | the earlier sandboxes are correctly in /src, eg Renaming networkd file from systemd chunk: /src/tmp/tmpiNsZS7/etc/systemd/network/10-dhcp.network to /src/tmp/tmpiNsZS7/etc/systemd/network/10-dhcp.network.morph | 14:29 |
| SotK | https://github.com/devcurmudgeon/ybd/blob/master/ybd/deployment.py#L76 appears to be the only thing that does anything to the sandbox that I can see | 14:32 |
| franred | SotK, doesn't that chmod command fail? 0o755 seems like a typo to me | 14:35 |
| * SotK doesn't know, its newer than his last interaction with ybd | 14:35 | |
| richard_maw | franred: it's not | 14:35 |
| franred | oh | 14:36 |
| richard_maw | 0o7 is a valid integer literal | 14:36 |
| richard_maw | it's like 0xa | 14:36 |
| richard_maw | you can even do 0b1010010 | 14:37 |
| SotK | hah, its my own code originally it seems | 14:37 |
| franred | oh, I see. thanks richard_maw | 14:37 |
| richard_maw | ☺ | 14:38 |
| * SotK digs more and shuts up until he finishes digging | 14:39 | |
| SotK | ok, it isn't my code after all, but it was working in the past, so I doubt that line is to blame | 14:41 |
| SotK | unless its always used /tmp and no-one has noticed | 14:41 |
| * SotK spots that kvm.write is actually to blame | 14:45 | |
| SotK | http://git.baserock.org/cgit/baserock/baserock/definitions.git/tree/extensions/kvm.write#n47 | 14:45 |
| richard_maw | mkstemp should be using the contents of TMPDIR | 14:46 |
| SotK | does ybd set TMPDIR? | 14:46 |
| richard_maw | apparently not | 14:47 |
| richard_maw | morph set it to a temporary subdirectory of the --tempdir setting, which we usually set to /src/tmp | 14:47 |
| paulsherwood | so the fix is for ybd to export TMPDIR? | 14:51 |
| rjek | Do ybd and morph sanatise the environments they provide to child processes, OOI? | 14:51 |
| richard_maw | aye, may also be worth adding a context manager to have ybd export TMPDIR as a temporary directory inside --tempdir, which it cleans up after runing the deployment | 14:52 |
| richard_maw | rjek: for builds, yes | 14:52 |
| paulsherwood | but not for (for example) git operations | 14:52 |
| richard_maw | nor for deployments in morph at least | 14:52 |
| * rjek idly wonders if they should | 14:53 | |
| * richard_maw thought so, but Lars did not, since they may require things like the SSH agent environment to connect to the target server etc. | 14:53 | |
| rjek | More from a reproducability and security standpoint; environment variables can have surprising side effects. I was just thinking perhaps both tools should always clean and only put in what they know they want and need for any subprocess | 14:55 |
| rjek | There's an interested chapter on this subject in O'Reilly's Secure Coding Cookbook, btw | 14:55 |
| locallycompact | What should I put and where to test this fix? | 14:55 |
| paulsherwood | os.environ('TMPDIR') = app.config['tmp'] | 14:59 |
| paulsherwood | l43 sandbox.py | 14:59 |
| richard_maw | is that copied varbatim? | 14:59 |
| paulsherwood | it's my proposed patch | 14:59 |
| richard_maw | you want os.environ['TMPDIR'] at least then | 15:00 |
| paulsherwood | oops - thanks | 15:00 |
| rjek | Whenever I am writting Python I am constantly bitten by all the different ways you can dereference a thing in exactly that way :) | 15:00 |
| paulsherwood | locallycompact: os.environ['TMPDIR'] = app.config['tmp'] | 15:00 |
| richard_maw | rjek: same number as lua | 15:01 |
| rjek | foo['bar'] is identical to foo.bar | 15:01 |
| richard_maw | it's just that lua doesn't have a difference between entries and keys | 15:01 |
| richard_maw | s/keys/attributes/ | 15:01 |
| rjek | Where in python they refer to different bars | 15:01 |
| richard_maw | aye, and for python's object model that is preferable, as it allows you to have a set object that is indexable with [] syntax, and also methods for operating on the set | 15:02 |
| richard_maw | python has foo[bar], foo.get(bar, default) and foo.setdefault(bar, default) | 15:03 |
| richard_maw | but also a separate `bar in foo` to determine contents, while lua has it return the null value, which makes things difficult if you want to explicitly store null and unset differently | 15:04 |
| locallycompact | success, ty paulsherwood SotK richard_maw | 15:04 |
| * richard_maw would also prefer morph's approach of having TMPDIR set for each deployment to a tempdir in /src/tmp, so deployments don't accidentally pick up each others' temporary files | 15:05 | |
| rjek | Shouldn't the tmpdir be cleaned after each step anyway? | 15:05 |
| paulsherwood | yes, but the above patch is sharing TMPDIR - i'll improve it as richard_maw suggests | 15:07 |
| paulsherwood | ybd runs multi-instance, so it's possible that multiple deploys could run at once (although not in the current code) | 15:07 |
| richard_maw | paulsherwood: is that TMPDIR set just for the deploys, or the builds too | 15:08 |
| richard_maw | if it's for the builds too things could get messed up, since /src/tmp isn't a writable path inside the sandbox | 15:08 |
| richard_maw | rjek: chrooted builds are | 15:08 |
| rjek | nod | 15:09 |
| paulsherwood | just deploys - builds get their own sandboxes | 15:11 |
| paulsherwood | and so do deploys... so this shold be easy to improve | 15:12 |
| paulsherwood | locallycompact: would you mind re-running, with os.environ['TMPDIR'] = this['sandbox'] | 15:12 |
| locallycompact | sec | 15:13 |
| paulsherwood | benbrown_: thanks for the pr - pls can you explain what is the use-case? when would we encounter a non-bare repo? | 15:15 |
| locallycompact | changing that to that did this https://paste.fedoraproject.org/320308/31218145/ | 15:20 |
| richard_maw | note that just because the paths say /src/tmp doesn't mean the disk image was created in /src/tmp | 15:21 |
| richard_maw | just that it was mounted in /src/tmp | 15:21 |
| richard_maw | paulsherwood: alwo works when the repo is corrupted in another fashion | 15:23 |
| richard_maw | something morph didn't handle very well | 15:24 |
| benbrown_ | paulsherwood: Ah yes, when using concourse to pull repos into gits they will be non-bare. | 15:26 |
| benbrown_ | Not wanting to put anything explicitly concourse specific, this seemed like the best approach. | 15:26 |
| richard_maw | concourse populates your /src/gits? | 15:26 |
| locallycompact | concourse fills a container for the build step with git repos | 15:27 |
| benbrown_ | richard_maw: We sets 'gits: magic_place_where_concourse_puts_things' | 15:27 |
| benbrown_ | s/sets/set/ | 15:27 |
| richard_maw | with appropriate name mangling? | 15:28 |
| * benbrown_ nods | 15:28 | |
| richard_maw | looks like you also need to do the rest of the repo mangling outside of ybd then | 15:28 |
| ssam2 | as an aside, i once wanted something that would expose my Baserock /src/cache/gits directory as a normal Git server | 15:28 |
| richard_maw | since otherwise you're just throwing the repo away completely | 15:28 |
| ssam2 | should be possible to hack up on top of git-daemon, but i never got it working | 15:28 |
| ssam2 | it would mean scripts could clone 'git://localhost/delta/gcc' for example and actually clone /src/cache/gits/git___git_baserock_org_delta_gcc or whatever | 15:29 |
| paulsherwood | +1 | 15:30 |
| paulsherwood | why do we need 'bare'? | 15:30 |
| ssam2 | bare means 'no working tree', which is sensible if the repo is only there for people to clone it | 15:30 |
| richard_maw | paulsherwood: when you fetch git complains that the refs are in a different layout | 15:30 |
| paulsherwood | benbrown_: it sounds like the short-term soln would be better doing the mangling in whatever script is driving concourse? | 15:31 |
| richard_maw | paulsherwood: for what purpose would you actually use the ability to clone git://localhost/delta/gcc? you'd be better off with a git wrapper that called git clone with --local from your cache | 15:31 |
| richard_maw | particularly since the change will completely throw away what concourse put in your repo anyway | 15:32 |
| richard_maw | better off mangling it to be a bare repo externally, so it will at least save having to fetch the branch concourse already fetched for you | 15:32 |
| paulsherwood | richard_maw: i guess the functionality i'm interested in really is the equivalent of kbas for gits - ability for one ybd user to act as a local server of gits to other users on a local network, save them hammering internet/gbo | 15:33 |
| paulsherwood | benbrown_: back to the drawing board, i fear? :) | 15:33 |
| benbrown_ | richard_maw: Aye, that might be best tbh, though the reason I went with this is so that it only processed repos as necessary. | 15:33 |
| ssam2 | richard_maw: I wanted that daemon thing so that I could write scripts that use Morph/YBD's git cache without having to teach them about the special way in which it works | 15:37 |
| ssam2 | richard_maw: it's a step towards making the caching a commodity. You have the script just do `git clone git://$MIRROR/delta/gcc`, and then if you set MIRROR=localhost and run this magical git-daemon tool, it "just works" | 15:38 |
| ssam2 | richard_maw: whereas right now you have to have an `if repo is cached locally then use that else clone the remote one and cache it locally and then use that one, or don't cache it, or avoid doing networking connectivity and fail depending on what options the user passed in' | 15:39 |
| richard_maw | ssam2: sounds like it would be better as a separate project that provided something like git-remote-localcache so you could do `git clone localcache::git://git.baserock.org/delta/gcc.git`, which knows how to maintain a cache locally | 15:39 |
| richard_maw | the baserock build tool could then make use of it without needing to have its own cache logic | 15:40 |
| ssam2 | yes, that's the idea | 15:40 |
| * richard_maw was thinking of adding a wrapper around https://github.com/fishface60/python-gitcache that would do that | 15:41 | |
| richard_maw | so you wouldn't need to invoke a different command to clone it | 15:41 |
| richard_maw | after I've settled down again I was planning to go back to it and finish it up, but I lost my free time | 15:42 |
| ssam2 | yes, that sounds ideal | 15:56 |
| ssam2 | looking at python-gitcache seems you've come up with the concept of a 'coarse' git cache.. what would a less 'course' git cache be? one that shared all the objects between all the repos? | 16:00 |
| ssam2 | less 'coarse' I mean | 16:00 |
| richard_maw | coarse vs fine-grained was named about the implementation detail of how I wanted to do the locking | 16:02 |
| richard_maw | but yes, a fine-grained one could have all the objects in the same object store, and have the refs of all the repositories available as git namespaces | 16:02 |
| richard_maw | you'd want to be very careful about when you decided to repack a cache like that | 16:03 |
| locallycompact | Has there been any thought given as to what would be required to shoehorn^Btranslate maven dependencies into definitions? | 16:06 |
| ssam2 | you mean, writing an import tool that would create a .morph file listing all the dependencies from a Maven POM file? | 16:07 |
| ssam2 | I think the only tricky thing there is how to go from a Maven version string to a Git commit | 16:08 |
| ssam2 | for released versions I guess the version string should correspond to a tag | 16:08 |
| locallycompact | It's more that maven expect to be able to contact the internet | 16:08 |
| locallycompact | unless I'm misunderstanding maven | 16:08 |
| ssam2 | oh, the question is how to *run* `mvn` inside a Baserock sandbox? | 16:08 |
| ssam2 | yeah that's a bit tricky | 16:08 |
| ssam2 | I guess the only "clean" way to do this is include a Maven repo that contains everything you need, inside the sandbox | 16:09 |
| ssam2 | building all that from scratch from a C compiler with only Baserock tools sounds like a multi-year project | 16:10 |
| ssam2 | but personally I think it makes sense to embrace binary delivery for Java artifacts, that is the whole point of Java existing after all | 16:10 |
| ssam2 | so you'd just need a way of injecting the various .jars you need, plus a Java runtime, into the staging area, and setting up Maven to look in the right place for all the jars... I guess | 16:11 |
| ssam2 | some kind of 'maven-bootstrap' git repo that contained all those binary .jar files would be the quickest route to 'something' I guess | 16:11 |
| paulsherwood | i think i remain philosophically opposed to binaries in this context, but i may be alone :) | 16:11 |
| locallycompact | I have no strong opinion on binaries, but in that case I can skip maven entirely and just import riemann | 16:12 |
| paulsherwood | lol | 16:13 |
| locallycompact | building riemann requires clojure, building clojure requires maven and maven requires maven | 16:13 |
| ssam2 | makes sense | 16:15 |
| ssam2 | i think getting strong reproducibility in Java isn't something we can do as a side effect of other work, it'd have to be a major focus for us to have any kind of success | 16:17 |
| jmacs | I agree | 16:20 |
| *** ctbruce has quit IRC | 16:23 | |
| locallycompact | hmm, it doesn't seem like java made it into this java-build-system | 16:24 |
| paulsherwood | locallycompact: you could list /src/artifacts/*/*.unpacked for clues | 16:25 |
| locallycompact | nvm, it did my brain just rebelled | 16:25 |
| * jmacs notes that java.morph still points to his github account :S | 16:26 | |
| rjek | A definition that points at github is not reproducable | 16:27 |
| rjek | :) | 16:27 |
| jmacs | Why not? | 16:27 |
| rjek | In the general sense; I often find links to older projects on github are dead | 16:28 |
| ssam2 | yes, it's done that way as a cheat so we don't have to commit it to a repo on git.baserock.org | 16:29 |
| ssam2 | it's not at all a good foundation for building real systems that use Java | 16:29 |
| ssam2 | for the Gerrit system, I deployed it with Baserock but just injected a binary Java runtime environment at deploy time | 16:30 |
| ssam2 | I think that's a more sensible approach than trying to stick binary artifacts into source control | 16:30 |
| ssam2 | i mean, I built it with Baserock, I deployed it with Ansible :-) | 16:30 |
| locallycompact | ssam2, is that somewhere? | 16:31 |
| jmacs | But it breaks traceability somewhat. You'd need to keep track of which version of the JRE you injected and ideally store that version somewhere | 16:31 |
| ssam2 | http://git.baserock.org/cgit/baserock/baserock/infrastructure.git/tree/baserock_gerrit/instance-config.yml | 16:31 |
| ssam2 | the Ansible script is a bit of a mess, I'm not sure that's the best way to do this either, but I think the principle is better | 16:32 |
| locallycompact | is that ansible? | 16:32 |
| ssam2 | that link points to an ansible playbook | 16:33 |
| ssam2 | which basically installs the JRE and gerrit onto a running system | 16:33 |
| ssam2 | and does other setup | 16:33 |
| ssam2 | so, it does record exactly what version of the JRE is injected | 16:34 |
| locallycompact | how does that know where to download | 16:34 |
| ssam2 | it depends on Oracle hosting the JRE, but I think that if they mess that up then it's a problem for the whole JAva world | 16:34 |
| ssam2 | ah | 16:34 |
| ssam2 | i forgot something | 16:34 |
| ssam2 | it doesn't download the JRE because of the stupid Javascript license checks Oracle do | 16:34 |
| ssam2 | so you have to download it yourself and put it locally | 16:35 |
| ssam2 | I'm not sure if the JRE license allows publically hosting the binaries somewhere else (redistributing them) or not | 16:35 |
| locallycompact | put it locally when/where? | 16:36 |
| ssam2 | when you run the Ansible script | 16:36 |
| ssam2 | on whatever machine you run the Ansible script on (probably the same one you used for the rest of the build and deploy process) | 16:37 |
| locallycompact | oh I thought ansible ran on the target | 16:37 |
| locallycompact | ok | 16:37 |
| ssam2 | no, it just needs ssh access to the target | 16:37 |
| jmacs | It'd be good to check the sha1 of the jre tarball | 16:37 |
| ssam2 | yes, that'd be good | 16:37 |
| *** toscalix has quit IRC | 17:50 | |
| *** edcragg has quit IRC | 17:58 | |
| *** jonathanmaw has quit IRC | 18:00 | |
| *** bashrc has quit IRC | 18:01 | |
| *** ssam2 has quit IRC | 18:16 | |
| *** Lachlan1975 has quit IRC | 18:17 | |
| *** locallycompact has quit IRC | 18:18 | |
| *** rdale has quit IRC | 18:53 | |
| *** cosm has joined #baserock | 19:37 | |
| *** cosm has quit IRC | 22:32 | |
| *** cosm has joined #baserock | 22:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!