| juergbi | tristan: i don't think this was correct: https://gitlab.com/BuildStream/buildstream/commit/b0a196a8cb34964e7c0af2fa74c8ada8bc0408c8 | 07:13 |
|---|---|---|
| juergbi | runtime dependencies of build dependencies can influence the build and they aren't covered by the cache key of the build dependencies, afaict | 07:13 |
| juergbi | assuming we don't want to intentionally be fuzzy about this to speed up builds | 07:14 |
| *** tristan has quit IRC | 07:33 | |
| *** tristan has joined #buildstream | 07:58 | |
| *** tristan has quit IRC | 07:59 | |
| *** tristan has joined #buildstream | 07:59 | |
| *** jonathanmaw has joined #buildstream | 08:22 | |
| juergbi | tristan: did you get my messages or were they lost with your disconnect? | 08:27 |
| *** tlater has joined #buildstream | 08:29 | |
| gitlab-br-bot | push on buildstream@testing-colors (by Tristan Van Berkom): 3 commits (last: widget.py: print_heading() now takes a styling boolean) https://gitlab.com/BuildStream/buildstream/commit/4a4b2f77c272d26bcbbb221957eaac0c9687e45c | 08:51 |
| *** tiagogomes has quit IRC | 08:53 | |
| tristan | oh sorry, I missed it | 08:59 |
| tristan | juergbi, I think it is correct | 09:00 |
| tristan | but wait | 09:00 |
| tristan | no you are right | 09:00 |
| tristan | so then... what is correct, hmm | 09:00 |
| tristan | juergbi, build dependencies + runtime dependencies of each build dependency, non recursively ? | 09:00 |
| tristan | recursive everything seems a bit much | 09:01 |
| tristan | and while including runtime dependencies of 'self' solves it for the depending elements, it's incorrect I thikn | 09:01 |
| tristan | *think | 09:01 |
| juergbi | Scope.BUILD + recurse should be correct | 09:01 |
| tristan | + recurse completely I guess ? | 09:01 |
| juergbi | yes | 09:01 |
| tristan | cause runtime of runtime can change | 09:01 |
| juergbi | exactly | 09:01 |
| tristan | yeah, so the entire Element.dependencies(Scope.BUILD) | 09:02 |
| juergbi | yes | 09:02 |
| tristan | juergbi, care to just bypass the entire gitlab foolishness and push a fix directly to master for that ? | 09:02 |
| juergbi | sure, will do | 09:02 |
| tristan | Most of you have permission to do so | 09:03 |
| tristan | and if you want to create your own branches on the main repo, that might make your lives easier too :) | 09:03 |
| jonathanmaw | tristan: have you started on issue 44 or shall I pick it up? | 09:04 |
| jonathanmaw | (I'm blocked on all the other things I'd be doing) | 09:04 |
| * juergbi can hopefully unblock jonathanmaw very soon | 09:05 | |
| *** tiagogomes has joined #buildstream | 09:10 | |
| *** ssam2 has joined #buildstream | 09:12 | |
| ssam2 | morning all | 09:12 |
| ssam2 | tristan were you thinking of a buildstream BoF at GUADEC ? | 09:13 |
| gitlab-br-bot | push on buildstream@master (by Jürg Billeter): 1 commit (last: element.py: Fix dependency handling in cache key calculation) https://gitlab.com/BuildStream/buildstream/commit/40e9d16dce74bdf3abb8728fcd976fe80b3d1df3 | 09:13 |
| ssam2 | if so, signup is open now .. https://wiki.gnome.org/GUADEC/2017/Unconference | 09:13 |
| tristan | ssam2, good ! | 09:16 |
| tristan | thanks for the pointer :) | 09:16 |
| tristan | juergbi, so I'm quite worried about this artifact server situation... I let it run all night | 09:17 |
| tristan | every failure says "read: Connection reset by peer" | 09:18 |
| juergbi | not a single successful push? and this is with or without control master? | 09:18 |
| tristan | I wonder... juergbi is it possible that it took my old (incorrect, lacking 'artifacts@') configuration and cached it somewhere else and is using that instead ? | 09:18 |
| tristan | maybe as a remote or smth ? | 09:18 |
| juergbi | no, the ssh host/path is not cached anywhere | 09:19 |
| tristan | Hmm | 09:19 |
| tristan | Nah I would doubt anyway, I've seen it doing something and mount properly | 09:19 |
| tristan | but it eventually screws up | 09:19 |
| juergbi | can you try to manually copy a file to a sshfs mount of that server? see how fast it is / whether it works? | 09:20 |
| tristan | Not a single successful push at all no | 09:20 |
| tristan | I have tried to "touch" one so far, that worked | 09:20 |
| juergbi | that's with 4 max pushers? 1 max pusher might help with bad network connectivity | 09:20 |
| juergbi | i could setup a OSTree repo on my own server to test whether this is a server/connectivity-specific issue (would still be in Europe but a less complex network path) | 09:22 |
| tristan | Just copied a file with 2744 bytes | 09:22 |
| tristan | it took 1.571s | 09:22 |
| juergbi | ok, difficult to extrapolate performance for large files from that but it's already bad for small files | 09:23 |
| tristan | Ok but what I recall | 09:23 |
| tristan | Is that the connection stays at around 3/4 kbps for long periods of time (for my computer in general, so not necessarily all ostree push related traffic) | 09:24 |
| juergbi | 0.729s for 50 kB here | 09:24 |
| tristan | then every once and a while, it spikes up to 200kbps or even 900kbps | 09:24 |
| tristan | and it remains at that speed for about 0.1 seconds | 09:24 |
| juergbi | i've seen variations as well but it was between 100kbps and 10Mbps or so, iirc | 09:25 |
| juergbi | average of my session based on iftop was 500kbps | 09:25 |
| tristan | so what it feels like, is that it is making many many round trips for system calls, each at 350ms per trip | 09:25 |
| tristan | and one day... it decides, OK upload a file ! | 09:26 |
| tristan | and that happens immediately | 09:26 |
| tristan | and back to roundtripping | 09:26 |
| juergbi | could well be the case | 09:26 |
| juergbi | we briefly discussed the potential slowness of sshfs a while ago | 09:26 |
| juergbi | when i discovered that regular ostree-push only works with archive-z2 repos | 09:27 |
| tristan | hmmm | 09:27 |
| tristan | Ok but arent we using archive-z2 repo locally ? | 09:27 |
| juergbi | no, we use bare user, iirc | 09:27 |
| tristan | Oh right | 09:27 |
| tristan | That is needed for the optimized checkouts, we use archive-z2 for ostree sources | 09:28 |
| juergbi | yes. one option would be to have both locally or maybe we can get something like original ostree-push to work with bare-user as well | 09:28 |
| tristan | Not really | 09:29 |
| tristan | Well, first one, will make an artifact cache double in size | 09:29 |
| tristan | it would be anyway really huge... but... | 09:29 |
| tristan | We could do something temporary for a half way | 09:29 |
| juergbi | the archive-z2 one would at least be compressed | 09:29 |
| juergbi | however, i agree, it doesn't sound ideal | 09:30 |
| tristan | for push, we could checkout and create an archive-z2 repo for the purpose of pushing | 09:30 |
| tristan | that should not be too bad, all artifacts are usually pretty different, and most of them relatively small | 09:30 |
| juergbi | yes, two-stage push should work. first pull-local and then push via ssh | 09:31 |
| tristan | so create the archive-z2 repo, push from there, and delete, all in a temp dir | 09:31 |
| tristan | Before going down that road, we should see how performance is | 09:31 |
| juergbi | and also see whether we can't modify ostree-push to directly work with bare user repos | 09:32 |
| tristan | Maybe that | 09:34 |
| tristan | I dont know much the details of ostree push | 09:34 |
| tristan | jonathanmaw, ok so, to get this out of the way I'll just tell you... | 09:49 |
| tristan | jonathanmaw, utils.py:_relative_symlink_target() | 09:49 |
| tristan | does os.path.relpath() (find the relative path), between the absolute symlink, and a concatenation of the root and the target | 09:50 |
| tristan | i.e. the symlink's "realpath" | 09:50 |
| tristan | jonathanmaw, it needs to os.path.join(os.path.realpath(root), target) in this case, not os.path.join(root, target) | 09:51 |
| tristan | jonathanmaw, just please test that it still works with your symlinked build directory configuration after changing that line, and it will be enough | 09:51 |
| jonathanmaw | ta tristan | 09:52 |
| tlater | Looks like there are no callbacks for TarFile extractall() | 09:55 |
| tristan | tlater, ok so there is another approach which might be easier and, maybe better also | 10:00 |
| tristan | tlater, in the function you split up... Element._stage_sources_at()... | 10:02 |
| tristan | tlater, we just recursively force mtime on everything using utils._set_deterministic_mtime() | 10:02 |
| tristan | tlater, you can create a different utility to do the same, but force the uid/gid ownership of everything | 10:04 |
| tlater | Are you suggesting to set ui/dgid in a similar function? | 10:05 |
| tlater | Oh, yeah | 10:05 |
| tristan | to os.geteuid() and os.getegid() | 10:05 |
| tristan | tlater, that way if every this arises with a different source type, it's already covered | 10:05 |
| tristan | I guess it's plausible that it could happen with ostree sources | 10:06 |
| tlater | Yeah, that might be better | 10:06 |
| tristan | tlater, on a different note, I've been trying to fool around with your test cases, but having a hard time to install my branch of buildstream | 10:07 |
| tristan | seems I can `pip3 install --user -e .`... oh maybe I have to setup PATH | 10:07 |
| tristan | tlater, anyway for your branch... it should run the tests against whatever is arbitrarily buildstream master at the time the pipeline runs | 10:07 |
| tristan | not a fixed version of buildstream coded into a docker image | 10:07 |
| tlater | I think I removed that while debugging something, but the code for that is still in the main repo. | 10:08 |
| tristan | same with the buildstream pipeline itself, it should run the tests from whatever is buildstream-tests master | 10:08 |
| tlater | I'll copy it over in a minute | 10:08 |
| tristan | ok no worry, just saying because it should be what we merge | 10:08 |
| tristan | and we should merge soon :) | 10:08 |
| tlater | Yeah, I'll fix that | 10:09 |
| tristan | jonathanmaw, while you are kicking around, you might fix some random things, like say this one which you reported: https://gitlab.com/BuildStream/buildstream/issues/36 | 10:11 |
| jonathanmaw | tristan: good call | 10:12 |
| tristan | tlater, another heads up... it would be good to have a variable like ${bst_args} which can be shared across all scripts | 10:14 |
| tristan | So any script which calls `bst foo <target>`, instead does `bst ${bst_args} foo <target>` | 10:15 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch testing-colors | 10:16 |
| gitlab-br-bot | push on buildstream@testing-colors (by Tristan Van Berkom): 2 commits (last: widget.py: print_heading() now takes a styling boolean) https://gitlab.com/BuildStream/buildstream/commit/4a4b2f77c272d26bcbbb221957eaac0c9687e45c | 10:16 |
| gitlab-br-bot | buildstream: merge request (44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks->master: WIP: Resolve "Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") #47 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/47 | 10:16 |
| tlater | Ah, gotcha. | 10:16 |
| tlater | Though I think if we start adding functionality like that it might be worth switching to a third party tool for testing | 10:16 |
| tristan | looks like the tests still lack a common include script for testing outputs, which would have been a good place to define that ? | 10:17 |
| tristan | tlater, shell scripts should be perfectly fine, lets not complicate things | 10:17 |
| tristan | we just need a common include which everything sources for doing stuff | 10:17 |
| gitlab-br-bot | push on buildstream@44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks (by Jonathan Maw): 1 commit (last: Fix symlink generation being confused by symlinks to the sandbox root) https://gitlab.com/BuildStream/buildstream/commit/a1e24faafca082de3d82053f8a31e9228b321d15 | 10:18 |
| gitlab-br-bot | buildstream: merge request (44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks->master: WIP: Resolve "Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") #47 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/47 | 10:18 |
| tlater | Ok, I just think that argument parsing will get quite hard when I start adding options. | 10:18 |
| tristan | To the main script ? | 10:19 |
| tlater | Yeah | 10:19 |
| tristan | nah | 10:19 |
| tristan | tlater, take this random example: https://gitlab.com/BuildStream/jhbuild2bst/blob/master/autoconvert/autoconvert.sh | 10:20 |
| tristan | as template | 10:20 |
| tristan | just copy paste, arg parsing is just a loop in bash like that | 10:20 |
| tristan | juergbi, did something fancy with $(getopt ...) but meh | 10:20 |
| tristan | Admittedly his help() function is nicer to write though | 10:21 |
| tristan | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/data/ostree-pull-ssh#L26 | 10:22 |
| tristan | usage() | 10:22 |
| tlater | I tend to use getopt, too. I guess I just thought it was messy after getting to know argparse. | 10:22 |
| tristan | I dont know why getopt actually | 10:23 |
| tristan | what does it give you ? | 10:23 |
| tlater | iirc more gnu-compliant options | 10:23 |
| tristan | what is that ? | 10:23 |
| tlater | Things like being able to do ./test -hvv | 10:23 |
| tlater | Which would set a double verbose help menu or something | 10:24 |
| tristan | meh | 10:24 |
| tlater | (In this case total overkill) | 10:24 |
| tlater | x) | 10:24 |
| tristan | ok well, whatever, this is not the kind of thing worth spending time talking about :) | 10:24 |
| tlater | Yeah | 10:24 |
| tristan | tlater, https://gitlab.com/BuildStream/buildstream-tests/-/jobs/21024518 | 10:29 |
| tristan | check that out :) | 10:29 |
| tlater | Sweet, why didn't they work before? | 10:29 |
| tristan | tlater, because you are not passing --ansi-colors to bst | 10:29 |
| ssam2 | ah nice | 10:30 |
| tristan | tlater, and because the option does not exist in bst master (yet) :) | 10:30 |
| tlater | Fair enough | 10:30 |
| tristan | tlater, colors are only enabled if you are connected to a terminal, but as I suspected; gitlab runners support ANSI color code rendering despite not being a real terminal | 10:30 |
| tristan | tlater, thats what I asked about setting up an argument variable in an include too, but there are other reasons | 10:31 |
| tristan | for instance, we should probably also have --on-error continue | 10:31 |
| tlater | There should be environment variables that can be set through the CI scripts. | 10:31 |
| tristan | export ? | 10:31 |
| tristan | sourcing a common script will be cleaner I think | 10:32 |
| tlater | The CI scripts have a variables section that could be used for this. It might be neater for some settings - like colors for example. | 10:33 |
| tristan | oh this is weird | 10:34 |
| tristan | Sorry I did not look at your scripts more closely | 10:34 |
| tristan | So... you are committing the expected output for each thing individually, and comparing it with a bst checkout ? | 10:35 |
| tristan | Seems quite rigid | 10:35 |
| tristan | I guess, whatever... I was hoping for a toolbox with some of the functions you have in run-tests.py, and some sparse checks here and there using those included functions to check various things | 10:36 |
| tlater | Do you have suggestions to change that? I thought it would be neatest to keep it a bit rigid. | 10:36 |
| tristan | Is every test going to be a checkout ? | 10:36 |
| tristan | I dont know | 10:36 |
| tristan | Is every test expected to pass ? | 10:36 |
| tristan | I dont know | 10:36 |
| tristan | Maybe the success criteria of a test is that it fails in a specific way ? | 10:37 |
| tlater | Ok, I understand. Should not be too hard to change that, luckily. | 10:38 |
| tlater | I'm submitting a MR for the permission fix first... | 10:40 |
| gitlab-br-bot | buildstream: merge request (permission-fix->master: Ensure element permissions are set to euid and egid after staging) #48 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/48 | 10:45 |
| tristan | tlater, ok so... I'm going to recommend that you trash your gitlab repo | 10:47 |
| tristan | for any buildstream repo | 10:47 |
| tristan | tlater, because it's going to be easier; you are a 'Developer' in the group now... which means you can push branches to buildstream directly and submit MRs from it | 10:47 |
| tlater | Alright? Did this have strange merge issues again? | 10:47 |
| tristan | well, your MR is not fast forwardable, which means it is not rebased against todays master | 10:48 |
| tristan | which means you are forgetting to do that... and it's admittedly annoying to do when you work with a separate repo | 10:48 |
| tlater | Alright, yeah, doing that may be easier, I keep accidentally rebasing against my own master | 10:48 |
| tristan | Actually, I suggest that to everyone here who has permission; it's just easier | 10:48 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: widget.py: print_heading() now takes a styling boolean) https://gitlab.com/BuildStream/buildstream/commit/b9f3547ae19ecbb0566cfc08994610abe9e1c78a | 10:51 |
| gitlab-br-bot | buildstream: merge request (permission-fix->master: Ensure element permissions are set to euid and egid after staging) #48 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/48 | 10:53 |
| tristan | tlater, and you forgot to run ./setup.py test :) | 10:53 |
| tlater | Yup, just did that... | 10:54 |
| tristan | dont worry I'm merging that... | 10:54 |
| tristan | done | 10:54 |
| tlater | I need to write a submit-merge script. | 10:54 |
| gitlab-br-bot | buildstream: merge request (permission-fix->master: Ensure element permissions are set to euid and egid after staging) #48 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/48 | 10:55 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Ensure element permissions are set to euid and egid after staging) https://gitlab.com/BuildStream/buildstream/commit/956a003c018f850151d004f6c123e7e9703f8aaf | 10:55 |
| tlater | Alright, so next up I will change the scripts a little... | 10:55 |
| tristan | tlater, So now upstream master has bst --ansi-color / --no-ansi-colors | 10:58 |
| tristan | I mean --ansi-colors (plural) | 10:58 |
| tristan | I suppose it could have just been --colors / --no-colors | 10:58 |
| tlater | I guess the intention is for that to be the script default, with a flag --no-colors to disable that? | 10:59 |
| tristan | yeah I'll do that... --colors / --no-colors is simpler | 11:00 |
| tristan | tlater, no | 11:00 |
| tristan | tlater, the gitlab must specify --colors explicitly | 11:01 |
| tristan | the default is to enable it when connected to the terminal | 11:01 |
| tristan | when running bst ... > file ... or bst ... | cat ... it will be disabled by default | 11:01 |
| tristan | jonathanmaw, why is https://gitlab.com/BuildStream/buildstream/merge_requests/47 wip ? | 11:03 |
| tristan | jonathanmaw, just let me know, if it works for you... merge it | 11:03 |
| gitlab-br-bot | buildstream: merge request (44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks->master: WIP: Resolve "Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") #47 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/47 | 11:04 |
| gitlab-br-bot | push on buildstream@44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks (by Tristan Van Berkom): 5 commits (last: widget.py: print_heading() now takes a styling boolean) https://gitlab.com/BuildStream/buildstream/commit/b9f3547ae19ecbb0566cfc08994610abe9e1c78a | 11:04 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: main.py: Renamed option to --colors / --no-colors) https://gitlab.com/BuildStream/buildstream/commit/f6d3f961bec54a51311df407b27a5b0b39be8f02 | 11:05 |
| gitlab-br-bot | push on buildstream@sam/host-and-target-arch (by Sam Thursfield): 2 commits (last: Add --host-arch and --target-arch, and 'host-arches' conditional) https://gitlab.com/BuildStream/buildstream/commit/d4bc1491deec1813c2ca56a376dbab2bc4814b9c | 12:01 |
| gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/host-and-target-arch | 12:02 |
| gitlab-br-bot | buildstream: merge request (sam/host-and-target-arch->master: Add --host-arch and --target-arch, and 'host-arches' conditional) #34 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34 | 12:02 |
| ssam2 | hmm, so another thought on --host-arch and --target-arch before committing that | 12:04 |
| ssam2 | *merging that | 12:04 |
| ssam2 | the host-arch and target-arch vars should go in the element cache keys, not the context cache key | 12:04 |
| ssam2 | especially because some elements can ignore target-arch, e.g. import elements | 12:04 |
| ssam2 | i.e. it's pointless to import the x86_^4 Freedesktop SDK for times, once for each cross-bootstrap | 12:05 |
| ssam2 | *x86_64 | 12:05 |
| ssam2 | i'll fix that up now and push another version | 12:05 |
| ssam2 | *four times | 12:05 |
| ssam2 | my spelling going well today | 12:05 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: utils.py: Fixed recent _set_deterministic_user() addition to not follow symlinks.) https://gitlab.com/BuildStream/buildstream/commit/5ac9f5de3a42c527c319950aa2a6d5ec5483d3b8 | 12:06 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch testing-colors | 12:06 |
| tristan | ssam2, I dont know that I agree or understand | 12:07 |
| ssam2 | ok | 12:08 |
| tristan | ssam2, the "arch" (which is now the target arch as I understand it) has forever been in the Context, does that cease to be true ? | 12:08 |
| ssam2 | that was my idea | 12:08 |
| ssam2 | we'll have to do that at some point anyway | 12:08 |
| tristan | Hmmm | 12:08 |
| ssam2 | maybe now is not the time though ... | 12:08 |
| ssam2 | the cost of not doing this is just a bunch of extra time copying the Freedesktop SDK into the artifact cache under different IDs, even though it's the same thing | 12:09 |
| ssam2 | which I guess only affects me :-) | 12:09 |
| ssam2 | ok, keep it simple and just merge what I have for now | 12:09 |
| tristan | I would like to understand why, though | 12:09 |
| tristan | Ok so you are building for a target | 12:09 |
| tristan | and you need to stage a host-arch artifact | 12:09 |
| tristan | I think I see | 12:10 |
| ssam2 | yeah, it's to do with import elements | 12:10 |
| ssam2 | which are not affected by --target-arch | 12:10 |
| ssam2 | well, actually that's nonsense | 12:10 |
| ssam2 | they could choose to download something different based on the target arch | 12:10 |
| tristan | Right | 12:10 |
| tristan | but I see your meaning | 12:10 |
| ssam2 | ok, cool | 12:11 |
| ssam2 | this has to happen at some point once we start actually having elements with different arches in a single pipeline, but it can wait til cross sandboxes are a thing | 12:11 |
| tristan | I guess that is sensible | 12:12 |
| tristan | ssam2, for some reason I'm tempted to say... maybe we can yank it entirely; but... lets not do that ? | 12:13 |
| tristan | Well | 12:13 |
| tristan | is it bad ? | 12:13 |
| tristan | Well yeah it is | 12:13 |
| ssam2 | the effects are actually minor | 12:13 |
| tristan | because target arch will be important for your cross builds | 12:13 |
| ssam2 | oh, yeah the effects of removing target-arch would be bad | 12:13 |
| tristan | The thing I just noticed is, in a world without any cross building at all, we need not use the arch at all (because we'll select a different runtime at the base) | 12:14 |
| tristan | but anyway yeah, lets not touch it for now | 12:14 |
| tristan | tlater, note: https://gitlab.com/BuildStream/buildstream/commit/5ac9f5de3a42c527c319950aa2a6d5ec5483d3b8 | 12:17 |
| tristan | dont worry about it though, just pointing it out :) | 12:17 |
| gitlab-br-bot | buildstream: merge request (44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks->master: Resolve "Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") #47 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/47 | 12:18 |
| gitlab-br-bot | buildstream: merge request (44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks->master: Resolve "Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") #47 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/47 | 12:20 |
| gitlab-br-bot | buildstream: issue #44 ("Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/44 | 12:20 |
| tristan | So I'm evaluating ostree push stuff... right now using: ostree pull-local --repo TESTMOUNT/repo ~/.cache/buildstream/artifacts/ostree gnome/base-linker-priority/900fc0bb9dee8da986ba3f14e038a76e4ef461f6c2cc6e741a60735050700290 | 12:30 |
| tlater | tristan: Ah, sorry, I thought it did that by default. | 12:30 |
| tristan | that artifact is a couple of directories, with a one line ld.so.conf.d file | 12:30 |
| tristan | so it should be reaaaaaly fast to upload that artifact | 12:30 |
| tristan | I got: [long pause] ... Scanning metadata 1 | 12:31 |
| tristan | 2 | 12:31 |
| tristan | 3 | 12:31 |
| tristan | 4 | 12:31 |
| tristan | and now it's hanging there | 12:31 |
| tristan | after 4min, I tried to strace it | 12:32 |
| tristan | and it got interrupted system call | 12:32 |
| tristan | not sure if my observations falsified the evidence | 12:32 |
| tristan | lets try it under strace | 12:32 |
| tristan | Oh yeah I forgot, at the beginning it says 'receiving metadata objects' | 12:33 |
| juergbi | jonathanmaw: my updated artifact-storage branch (also see https://gitlab.com/BuildStream/buildstream/merge_requests/45 ) now supports dynamic public data | 12:38 |
| jonathanmaw | \o/ | 12:39 |
| juergbi | i did some minimal testing. it might make sense for you to directly test that and let me know if you encounter any issues, instead of me testing more extensively | 12:39 |
| juergbi | API-wise you can simply modify what you get back from get_public_data('bst') | 12:39 |
| gitlab-br-bot | buildstream: merge request (artifact-storage->master: WIP: Artifact storage enhancements) #45 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45 | 12:40 |
| juergbi | (inside your Element's assemble() method) | 12:40 |
| tristan | It's mostly doing a shit load of this: https://bpaste.net/show/b20d387abace | 12:41 |
| tristan | crawling the repo objects | 12:41 |
| tristan | and system calls, very slowly | 12:41 |
| tristan | ok so, I'm going to try the ostree-push method | 12:42 |
| tristan | with a local archive-z2 repo | 12:43 |
| tristan | in which I do a pull local to get the artifact I want to push | 12:43 |
| juergbi | sshfs would likely also not scale at all for large repositories | 12:43 |
| tristan | lets see | 12:43 |
| tristan | juergbi, I have a feeling that the target repo (which will probably be very large), has an impact on this yes | 12:43 |
| juergbi | as it seems to enumerate all object directories. although, that's at least not too bad latency wise | 12:43 |
| tristan | oh shit | 12:44 |
| juergbi | tristan: do you want me to help with / work on artifact sharing or shall i start with dual cache keys? | 12:44 |
| tristan | it's not gonna work either | 12:45 |
| juergbi | why not? | 12:45 |
| tristan | juergbi, ostree-push wants your user machine to be an http server, according to the second paragraph here: https://github.com/dbnicholson/ostree-push ?? | 12:45 |
| tristan | crap | 12:45 |
| tristan | cant we have a daemon which wakes up on an ssh trigger on the artifact server which we can push data to directly ? | 12:46 |
| tristan | or maybe I'm reading that README wrong | 12:46 |
| tristan | and it's part of a problem statement justifying the project | 12:46 |
| tristan | right | 12:46 |
| tristan | okay | 12:46 |
| tristan | so lets try this | 12:46 |
| tristan | would be nice if it had instructions to set it up | 12:47 |
| juergbi | iirc, you mainly need to install ostree-receive (symlink to ostree-push) in the path of the target server | 12:47 |
| juergbi | and then you can use ostree-push like ostree-push-ssh | 12:48 |
| juergbi | i mean, like the shell script | 12:48 |
| tristan | hmmm | 12:55 |
| tristan | https://github.com/dbnicholson/ostree-push/blob/master/ostree-push#L58 | 12:55 |
| tristan | __main__.PushException: Unrecognized message byteorder T | 12:55 |
| tristan | This does not bode well... | 12:55 |
| juergbi | huch? | 12:57 |
| tristan | ostree-push --debug -v --repo test-ostree-push-repo/ artifacts@gnome7.codethink.co.uk:repo gnome/base-linker-priority/900fc0bb9dee8da986ba3f14e038a76e4ef461f6c2cc6e741a60735050700290 | 12:57 |
| tristan | That was the command | 12:57 |
| tristan | can't find capital T in any string in ostree-push python file | 12:58 |
| tristan | any ideas ? | 12:58 |
| juergbi | i assume you're using the same ostree-push version on both systems | 12:58 |
| tristan | Just installed it on both yeah | 12:58 |
| tristan | lemme try uninstall and see if it gives me the same message | 12:59 |
| tristan | Yup | 12:59 |
| tristan | there's a good indication | 12:59 |
| tristan | Same exception when ostree-push is not installed on gnome7 | 12:59 |
| tristan | So I assume, maybe something has to be done to the repo | 13:00 |
| juergbi | maybe SSH connection fails somehow? | 13:00 |
| tristan | Or sshd_config needs some frobnication | 13:00 |
| tristan | to tell it to actually run the thing there | 13:00 |
| juergbi | and it's interpreting ssh error output? | 13:00 |
| juergbi | ah, of course, SFTP access is not sufficient that way | 13:00 |
| tristan | Ah also the port | 13:01 |
| tristan | No it's not ? | 13:01 |
| tristan | crap | 13:01 |
| juergbi | isn't port handled via your ~/.ssh/config? | 13:01 |
| tristan | oh yeah it is | 13:01 |
| juergbi | this needs regular SSH command execution privileges | 13:01 |
| juergbi | SSH can be restricted to just allow execution of ostree-receive, but that's definitely required | 13:02 |
| tristan | it should be I guess | 13:02 |
| tristan | Do you have any idea how to do it ? | 13:02 |
| tristan | hahaha | 13:02 |
| * tristan emacs /etc/sshd_config on gnome7 and assumes he will be thoroughly lost | 13:03 | |
| tristan | I see there is now: ForceCommand internal-sftp | 13:04 |
| tristan | juergbi, I guess I can change this to s/internal-sftp/ostree-receive/ and restart the daemon ? | 13:04 |
| juergbi | not sure whether this would still pass command-line args | 13:04 |
| juergbi | maybe it doesn't need any, don't know | 13:05 |
| *** tlater has left #buildstream | 13:05 | |
| juergbi | you could simply uncomment ForceCommand, removing the restriction for now | 13:05 |
| *** tlater has joined #buildstream | 13:05 | |
| juergbi | and lock it down later | 13:05 |
| ssam2 | you can do clever stuff with ForceCommand, because you can access the actual command the client sent with the SSH_ORIGINAL_COMMAND environment variable | 13:07 |
| tristan | Same | 13:08 |
| tristan | 'T' | 13:08 |
| tristan | If I knew this kind of thing, I would probably be perusing a log file that is hiding somewhere which would tell me what happened with the connection attempt, on gnome7 | 13:09 |
| juergbi | did you restart sshd / verify that you actually have shell access now? | 13:09 |
| juergbi | i still get | 13:09 |
| juergbi | This service allows sftp connections only. | 13:09 |
| tristan | This service allows sftp connections only. | 13:09 |
| tristan | Yup | 13:09 |
| tristan | Just found out where this infamous 'T' is coming from ! | 13:09 |
| tristan | I did restart the service though | 13:10 |
| tristan | Crap | 13:10 |
| tristan | Ok so, I have no idea what to do with this config | 13:10 |
| tristan | there is one mention of sftp | 13:10 |
| tristan | Subsystem sftp /usr/lib/openssh/sftp-server | 13:10 |
| ssam2 | it could be that DavePage set up the server that is doing port forwarding to also only allow SFTP | 13:11 |
| tristan | ssam2, not from what I can tell in the config | 13:13 |
| tristan | but I need to ssh into that machine anyway | 13:13 |
| tristan | from there I am allowed to shell in | 13:13 |
| gitlab-br-bot | buildstream: merge request (artifact-storage->master: WIP: Artifact storage enhancements) #45 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/45 | 13:25 |
| tristan | juergbi, does that 'pass' in SanitizedDict really work ? | 13:47 |
| tristan | I guess just by virtue of it being an OrderedDict in the first place ? | 13:48 |
| juergbi | SanitizedDict appears to work fine | 13:48 |
| juergbi | is this odd coding style? | 13:48 |
| tristan | incidentally, I think it will make nicer `bst show` output for the more exotic options | 13:48 |
| tristan | juergbi, what ? using 'pass' like that ? nah I think it's fine :) | 13:49 |
| juergbi | yes, ok :) | 13:49 |
| tristan | now I notice doing that 'integration' thing should never have been on 'element.py' in the first place (should have been in buildelement.py) | 13:49 |
| tristan | but anyway, nice to remove | 13:50 |
| juergbi | true, base Element shouldn't know about that | 13:50 |
| *** tristan has quit IRC | 13:58 | |
| *** tristan has joined #buildstream | 14:18 | |
| *** ChanServ sets mode: +o tristan | 14:18 | |
| jonathanmaw | juergbi: when I try to build using your "artifact-storage" branch, I end up with https://pastebin.com/vZfibxep | 14:34 |
| jonathanmaw | I'm currently checking the ruamel documentation for what it expects | 14:35 |
| juergbi | hm, that's odd, might this be due to a different ruamel.yaml version? | 14:35 |
| juergbi | i have 0.14.12 here on Python 3.6.1 | 14:35 |
| jonathanmaw | aha, my ruamel is 0.13.4 | 14:36 |
| juergbi | do you think it would be an issue to depend on the more recent version? | 14:37 |
| jonathanmaw | juergbi: the version of ruamel I'm currently using is the one provided by debian jessie | 14:38 |
| jonathanmaw | I'll have to figure out how hard it is to uninstall the old ruamel and install a newer one. | 14:38 |
| juergbi | ah ok, i think i installed mine with pip, don't remember | 14:38 |
| juergbi | maybe we could make it work with both versions, i haven't checked the differences | 14:39 |
| jonathanmaw | I suspect https://stackoverflow.com/questions/44703881/how-to-use-representer-in-older-version-version-0-11-of-ruamel-yaml would be useful | 14:39 |
| jonathanmaw | I was googling ruamel roundtriprepresenter and it popped up, which was my first clue that version differences might be at fault | 14:39 |
| juergbi | ah, if it's as simple as that, that should be fine | 14:40 |
| tristan | note, iiuc; you will need to wipe any prior artifact cache | 15:21 |
| juergbi | it probably makes sense to do so but the cache key generation has changed, so it will simply not use any of the previous items in case you don't wipe the cache, but it shouldn't cause any issues | 15:23 |
| tristan | ah indeed | 15:24 |
| tristan | juergbi, you probably already saw the couple comments I made, I think asides from that the branch seems pretty fine | 15:25 |
| tristan | pretty straight forward stuff | 15:25 |
| tlater | tristan: Could you tell me what the expected behaviour for this test case is? https://gitlab.com/tlater/buildstream-tests/blob/tristan/test-suite/compose-test/elements/compose-no-test.bst | 15:29 |
| tlater | Keeping in mind the project.conf in the parent dir | 15:30 |
| tlater | I'm trying to exclude a domain called "test", which I define in the project config | 15:30 |
| tlater | But I'm definitely doing something wrong. | 15:31 |
| tristan | it could be a bug | 15:38 |
| tristan | so what I think should happen, is the only files which should be included are A.) Captured by the 'runtime' domain B.) Orphans (not captured by any domain) | 15:38 |
| tristan | tlater, that's the intention anyway | 15:39 |
| tlater | As I understand it, I set it up so that the file in tests/ is neither in the runtime domain nor an orphan | 15:39 |
| tlater | Yet it's included | 15:39 |
| tlater | I'm just not sure my syntax is correct | 15:39 |
| tristan | well, your '- runtime' looks overindented, but I think it's correct | 15:41 |
| tlater | Hm, alright, then it's probably a bug; I'll ignore the test case for now and note it down in an issue. | 15:41 |
| tlater | MR should be up as soon as I get cmake to run... | 15:42 |
| tristan | tlater, this is not correct: https://gitlab.com/tlater/buildstream-tests/blob/tristan/test-suite/compose-test/project.conf | 15:42 |
| tlater | Globbing is wrong? | 15:43 |
| tristan | if you want all the files directly in /tests, it should be /tests/* | 15:43 |
| tlater | Whoops, yeah, I thought I changed that | 15:43 |
| tristan | if you want all files *recursively* in /tests, then /tests/** | 15:43 |
| tlater | Shouldn't be necessary for this test, perhaps in another one to check globbing though. | 15:44 |
| tristan | nod | 15:44 |
| gitlab-br-bot | push on buildstream@44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks (by Tristan Van Berkom): 1 commit (last: Fix symlink generation being confused by symlinks to the sandbox root) https://gitlab.com/BuildStream/buildstream/commit/7f365869aa377841b8111662bb48dd7b01bc614f | 15:47 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: Fix symlink generation being confused by symlinks to the sandbox root) https://gitlab.com/BuildStream/buildstream/commit/7f365869aa377841b8111662bb48dd7b01bc614f | 15:52 |
| gitlab-br-bot | push on buildstream@44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks (by Jonathan Maw): 4 commits (last: main.py: Renamed option to --colors / --no-colors) https://gitlab.com/BuildStream/buildstream/commit/f6d3f961bec54a51311df407b27a5b0b39be8f02 | 15:54 |
| tlater | Egh, that gitlab downtime came just at the wrong time | 16:10 |
| tlater | I'll get the MR up ASAP tomorrow morning, it's *actually* almost done this time. | 16:11 |
| *** tlater has quit IRC | 16:12 | |
| jonathanmaw | hrm, I'm now getting "execvp sh: No such file or directory" when trying to run commands from a ScriptElement | 16:13 |
| jonathanmaw | as far as I can tell, sh exists, (/bin/sh is a symlink to /bin/dash, which exists, in the sandbox debris). Also, I think PATH exists and includes /bin, since I hacked in a call to sandbox's _get_environment to print the environment variables before running the build | 16:15 |
| ssam2 | that error can also mean that the dynamic loader is missing -- /lib64/ld.so.something usually | 16:15 |
| jonathanmaw | ssam2: /lib64 doesn't have anything beginning in ld.so, but there's an /etc/ld.so.conf | 16:17 |
| ssam2 | if this is a normal x86_64 system, you almost certainly need /lib64/ld-linux-x86_64.so.2 or something like that | 16:19 |
| ssam2 | every dynamically linked binary is loaded through that, and if it's missing then all your dynamically linked binaries fail with "Not found" | 16:20 |
| ssam2 | which is really confusing because they don't tell you that it's ld.so that wasn't found | 16:20 |
| ssam2 | `ldd` will tell you what the `sh` binary is linked against | 16:21 |
| jonathanmaw | aha, lib64/ld-linux-x86-64.so.2 is a bad symlink | 16:25 |
| jonathanmaw | ta ssam2 | 16:29 |
| *** ssam2 has quit IRC | 16:43 | |
| *** ssam2 has joined #buildstream | 16:52 | |
| *** jonathanmaw has quit IRC | 17:02 | |
| *** ssam2 has quit IRC | 17:23 | |
| *** jude has quit IRC | 17:51 | |
| *** jude has joined #buildstream | 17:52 | |
| *** tristan has quit IRC | 19:35 | |
| *** jude has quit IRC | 21:02 | |
| *** violeta has quit IRC | 21:05 | |
| *** tiagogomes has quit IRC | 21:07 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!