| *** tristan has quit IRC | 03:26 | |
| *** jude has joined #buildstream | 08:00 | |
| *** tlater has joined #buildstream | 08:18 | |
| *** jonathanmaw has joined #buildstream | 08:39 | |
| *** ssam2 has joined #buildstream | 09:10 | |
| jonathanmaw | I'm getting errors when I try to build that seem to be caused by project.conf not having a host-arches field. (stack trace from https://pastebin.com/1VejnNxK) | 09:15 |
|---|---|---|
| jonathanmaw | looking at the code, my first guess is that project.conf's format has changed, and the sam/buildstream-bootstrap doesn't convert it properly any more, but buildstream-tests doesn't seem to have made the host-arches field mandatory, so I'm guessing not. | 09:16 |
| jonathanmaw | I went into the code and made sure it checked a field exists before it tries to delete it, and then it managed to finish loading the elements, but hangs when checking elements | 09:17 |
| jonathanmaw | so I suspect there's something going wrong when loading the elements | 09:18 |
| ssam2 | project.conf should have arches, not host-arches | 09:26 |
| ssam2 | must be a recent regression, anyway | 09:26 |
| ssam2 | 0a4f05f6a0486d6a58a83d50e9b607ddb747befe has worked fine for me | 09:27 |
| jonathanmaw | I'll try that commit out and cross my fingers :) | 09:29 |
| *** semanticdesign has joined #buildstream | 09:30 | |
| ssam2 | latest commit of buildstream works for me too | 09:43 |
| jonathanmaw | ssam2: yep, and the commit you linked has the same problem for me. | 09:50 |
| jonathanmaw | ssam2: do you know how buildstream figures out the arch to pick one of the items in host-arches, by the way? | 09:52 |
| ssam2 | the --host-arch argument | 09:52 |
| ssam2 | which defaults to the --arch argument | 09:52 |
| jonathanmaw | I'm not sure whether I should be adding a field named "aarch64", "armv8" or "armv8l64" | 09:52 |
| ssam2 | which defaults to $(uname -m) | 09:52 |
| ssam2 | buildstream itself has no notion of architecture name | 09:52 |
| ssam2 | baserock defines the moonshot's arch as "armv8l64" | 09:53 |
| ssam2 | but (maybe this is the issue?) if `uname -m` reports something different you'll need to run `bst --arch=armv8l64` | 09:53 |
| ssam2 | when building etc. | 09:53 |
| jonathanmaw | seems not , unfortunately. | 09:54 |
| ssam2 | ok | 09:54 |
| jonathanmaw | `bst --host-arch=armv8l64 build gnu-toolchain/base.bst` is still giving me KeyError: 'host-arches' | 09:54 |
| ssam2 | weird | 09:54 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Add fallback integration tests to CI) https://gitlab.com/BuildStream/buildstream/commit/7d57ea24a63374520d28be7c9c46244230369076 | 09:56 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 09:58 |
| adds68 | Could anyone point me in the direction of what i could build using bst as a first time project? | 10:21 |
| adds68 | Something preferably small so i can get a hang of the syntax and commands? | 10:21 |
| ssam2 | i can only point you to baserock. it's not small though :-) | 10:23 |
| tlater | Tristan has told us to build this for testing: https://gnome7.codethink.co.uk/gnome-modulesets.git/ | 10:23 |
| tlater | Also not small, but I end up just repeatedly building base/ninja.bst | 10:23 |
| tlater | You'll find the size doesn't matter much because you can specify individual elements :) | 10:24 |
| adds68 | Thanks guys, i shall take a look | 10:26 |
| adds68 | tlater, any idea on where to start with building the modules, any guides? | 10:28 |
| tlater | There's the buildstream documentation, and bst --help. | 10:28 |
| tlater | Ah, also this blog post: https://blogs.gnome.org/tvb/2017/06/01/continuous-bst-conversions-of-gnome-modulesets/ | 10:30 |
| adds68 | thanks :) | 10:32 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 1 commit (last: .gitlab-ci.yml: Add fallback integration tests to CI) https://gitlab.com/BuildStream/buildstream/commit/bd1edb840350fd21916e107cf0737f5d14929bc4 | 11:40 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 11:41 |
| *** tristan has joined #buildstream | 12:06 | |
| tlater | tristan: Still not sure what your time zone is, do you mind touching on a few of the comments on cross_platform? | 12:21 |
| gitlab-br-bot | push on buildstream@sam/embed-fusepy (by Tristan Van Berkom): 4 commits (last: bst push: Check connectivity to cache before trying to push) https://gitlab.com/BuildStream/buildstream/commit/0cf6c6bcd8e69554872b05efb267f5530a6b86d4 | 12:21 |
| gitlab-br-bot | buildstream: merge request (sam/embed-fusepy->master: Embed fusepy and add support for ppc64 platforms) #95 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/95 | 12:21 |
| tristan | tlater, shoot :) | 12:24 |
| tlater | On this comment: https://gitlab.com/BuildStream/buildstream/merge_requests/81#note_41459448 | 12:24 |
| tlater | I simply removed it because it was annoying me and forgot about it x) Supposedly this should check for the version at least in setup.py | 12:25 |
| tlater | But how exactly should it react if we don't find ostree - do we warn somewhere, perhaps even when running buildstream? | 12:25 |
| tristan | Yup, so - I asked where it ended up, suspecting you left it out but thinking perhaps it might have become a runtime check | 12:25 |
| tristan | it's a good question | 12:26 |
| tlater | Also, checking for the version when running buildstream is probably not a bad idea - just in case we bump version requirements or something | 12:26 |
| tristan | Now, I think the way you have it setup, at the time you look for a platform; you are doing some checks for what is installed instead of checking the os with python | 12:26 |
| tristan | Which is also a bit weird, and somewhat related to this | 12:27 |
| tristan | I.e., it's not really a great idea to say; oh well, you didnt install bwrap so I'll just require you be root and use chroot | 12:27 |
| tristan | it's better to say: You are on linux, get off your ass and install the requirements | 12:28 |
| tlater | Heh. Should they still be hard requirements though, given that buildstream can run without those requirements if need be? | 12:28 |
| tristan | tlater, what you say about runtime checks and upgrades is practically true; but I dont want to care about that honestly | 12:28 |
| tristan | I.e., people are currently running from gits | 12:29 |
| tristan | But they should be installing releases | 12:29 |
| tristan | We havent made any releases yet, of course | 12:30 |
| tristan | but conceptually speaking, I can't really vouch for a version in the field unless it's a release, nor should we have to care about developer git checkouts, installing should be an expected step | 12:31 |
| tristan | Next: Should it be a hard requirement | 12:31 |
| tristan | Not necessarily, but if it's optional (which it need not be), it should be a configuration before install | 12:31 |
| tristan | i.e. ./configure --with-platform=unix | 12:32 |
| tristan | something like that | 12:32 |
| tristan | It should certainly not start behaving differently | 12:32 |
| tlater | How would this work in python/setuptools land? I don't think we have anything similar yet | 12:33 |
| tristan | setuptools is horse crap | 12:33 |
| tristan | and since it's certainly not something we need, I am perfectly fine not allowing one to install without the linux requirements, when installing on linux | 12:34 |
| tristan | tlater, if someone cares that much about being able to run with tar caches and chroots instead of ostree/bwrap, on a perfectly linux host; they can put in that effort | 12:35 |
| tristan | but I dont suspect anyone would, and I wouldnt want to recommend that | 12:35 |
| tlater | Alright, that clears that up. So I suppose add back the setuptools checks, and additionally add a runtime check, as well as detecting the host OS? | 12:36 |
| tristan | tlater, on another point; please dont worry too much about configurability of network access in chroot sandboxes; if you can see an easy way to control that; it's nice to have: More importantly is reasonable assumption that at least non-malicious/runtimes build instructions cannot access the network | 12:37 |
| tristan | (where malicious would mean: Does some crazy mounts or writing to dev / proc to attempt to forcibly break out of a chroot jail) | 12:38 |
| tlater | tristan: Heh, that was going to be my next question - there is a reasonably sensible way to do this with ctypes, but exclusively for linux. | 12:38 |
| tlater | I'd be using unshare for that. | 12:38 |
| tristan | I see, so it doesnt apply here | 12:39 |
| tristan | because chroot is specifically for !linux | 12:39 |
| tlater | There doesn't really seem to be any cross platform way to deal with this, unfortunately. | 12:39 |
| tlater | Unless ssam2 has more insights. | 12:39 |
| tristan | However, I recall that ybd shells did not generally have network access | 12:39 |
| tristan | with sandboxlib | 12:40 |
| tristan | And looking at the code yesterday, which is not large... I cannot explain _why_ it does not have network access | 12:40 |
| tlater | tristan: atm buildstream shells don't *really* since the host configuration isn't copied. You could certainly force it though. | 12:40 |
| tristan | I want to know that | 12:40 |
| tristan | heh | 12:40 |
| tlater | I think sandboxlib might have the same 'fix' x) | 12:40 |
| benbrown | by passing --unshare-net to linux-user-chroot | 12:41 |
| tristan | benbrown, with the chroot backend specifically, though | 12:41 |
| benbrown | AFAIA, there is no network sandboxing with chroot | 12:42 |
| tristan | (to be honest, linux-user-chroot never really worked for me, I would constantly get too many mounts and such) | 12:42 |
| tristan | Ok so that's unfortunate; I might have thought that... excluding any network devices from being mounted into /dev would make any runtime running in there unable to use it | 12:43 |
| tristan | and that a malicious runtime/build instructions might try to mknod one or smth | 12:44 |
| tristan | (and would have to guess that it's the right device for your laptop) | 12:44 |
| tlater | tristan: If that *does* prevent network access, then buildstream already does that. | 12:44 |
| tristan | yeah | 12:44 |
| tlater | But I think that processes simply create new network devices. | 12:45 |
| tristan | This is what I dont know, and would really like to | 12:45 |
| * tlater probably knows less about the linux networking interface than you do :) I can have a read at least though. | 12:46 | |
| tristan | tlater, I wouldnt be so quick about that honestly | 12:46 |
| tristan | if there is one area of knowledge I am alergic to... its that | 12:46 |
| tristan | in one ear, out the other | 12:46 |
| tristan | what did you say ? you want to have dinner on the routing table ? | 12:47 |
| tristan | ok | 12:47 |
| tlater | If I ever get bored of programming I'm starting a restaurant whose gimmick is routing tables. | 12:47 |
| tristan | So... tlater I wont block on that; it's been known to work with YBD, code looks quite similar enough - just make sure there is a note in _sandboxchroot.py about this uncertainty | 12:48 |
| tristan | a FIXME comment or such, most important is we keep knowing about this | 12:48 |
| tristan | which is why, ideally I would have liked that comment to be an explanation about why this *happens* to work | 12:49 |
| tristan | or such, if we had certainty :) | 12:49 |
| tlater | Alright, that makes things a lot simpler. I'll see if I can find something on that quickly, and then I should be through your comments - once again. | 12:50 |
| *** xjuan has joined #buildstream | 12:51 | |
| tlater | tristan: Actually, adding network support would probably be possible simply by copying the host /etc/resolv.conf | 12:51 |
| gitlab-br-bot | buildstream: issue #95 ("please provide some real examples") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/95 | 12:51 |
| tlater | Should I add that functionality? | 12:52 |
| tristan | tlater, not for now no | 12:54 |
| tristan | tlater, lets keep that idea separate; and if we look into that; we'll have to check if it actually works with bwrap | 12:55 |
| tristan | and then if it does, we'll have to wonder *why* it works with bwrap, when the runtime does not have the hosts resolv.conf | 12:56 |
| tristan | so yeah, rabbit hole... too deep | 12:56 |
| gitlab-br-bot | buildstream: issue #83 ("FUSE SafeHardlinkOps filesystem doesn't work on ppc64") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/83 | 13:04 |
| gitlab-br-bot | buildstream: merge request (sam/embed-fusepy->master: Embed fusepy and add support for ppc64 platforms) #95 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/95 | 13:04 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: Fork and embed fusepy) https://gitlab.com/BuildStream/buildstream/commit/bf0f2863e8cea73e41d5d9efca3126e4075fc3ae | 13:04 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch sam/embed-fusepy | 13:04 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 4 commits (last: mount_simple.py: Add mount tests) https://gitlab.com/BuildStream/buildstream/commit/844d25525897efb0d59d2b6994fb3c2f15f81367 | 13:11 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 13:12 |
| gitlab-br-bot | push on buildstream@75-source-bundle-generated-script-fails-when-a-build-element-has-no-source (by Tristan Van Berkom): 3 commits (last: Fork and embed fusepy) https://gitlab.com/BuildStream/buildstream/commit/bf0f2863e8cea73e41d5d9efca3126e4075fc3ae | 13:15 |
| gitlab-br-bot | buildstream: merge request (75-source-bundle-generated-script-fails-when-a-build-element-has-no-source->master: build-module.sh.in: Don't attempt to copy empty sources) #97 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/97 | 13:15 |
| ssam2 | tristan, sandboxlib chroot backend didn't do anything to block network access, but it also didn't copy in /etc/resolv.conf which meant DNS would usually break | 13:25 |
| ssam2 | which effectively blocks most kinds of network access that builds would be doing | 13:25 |
| * tlater was right \o/ | 13:28 | |
| tristan | ssam2, I presume that `bst shell` doesnt really see the network for this same reason | 13:31 |
| ssam2 | probably yeah | 13:32 |
| tristan | seeing as we only omit the bwrap --unshare-net | 13:32 |
| ssam2 | `docker run` automatically links in /etc/resolv.conf IIRC | 13:32 |
| ssam2 | as does `flatpak run`, it seems | 13:32 |
| tristan | but this is starting to look pretty runtime specific, would be bad for the build tool to start looking at a whole category of bugs in bst shell that are not really relevant to building and debugging | 13:33 |
| tristan | Maybe we should just drop that tentative ability to allow networking from a shell sandbox from the start | 13:34 |
| gitlab-br-bot | buildstream: issue #75 ("source-bundle: generated script fails when a build element has no source") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/75 | 13:46 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: build-module.sh.in: Don't attempt to copy empty sources) https://gitlab.com/BuildStream/buildstream/commit/5d5d388e9894c575ecc982caf78e957dab9875d2 | 13:46 |
| gitlab-br-bot | buildstream: merge request (75-source-bundle-generated-script-fails-when-a-build-element-has-no-source->master: build-module.sh.in: Don't attempt to copy empty sources) #97 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/97 | 13:46 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 9 commits (last: Move sandboxes to separate directory) https://gitlab.com/BuildStream/buildstream/commit/1b029555dded3d2d6b75e988fdb78847a7836fe7 | 13:48 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 13:48 |
| tlater | tristan: Alright, that concludes the newest changes. With a bit of luck the coverage issue is finally fixed too. Should I open an issue about the networking thing as suggested or is a FIXME enough? | 13:50 |
| *** semanticdesign has quit IRC | 13:51 | |
| tristan | tlater, just a comment is fine; I have to think about this networking thing; it may be needed for accessing the local session bus (without needing resolv.conf hacks); which is important for running anything | 13:51 |
| *** semanticdesign has joined #buildstream | 13:51 | |
| tristan | anyway, just comment is fine | 13:51 |
| jonathanmaw | tristan: do you know if there have been any significant changes to the yaml loading logic? I'm getting errors whenever something tries to delete the "host-arch" field from a dict loaded from project.conf, or host-arch from an element, depends from an element, or variants from an element. | 13:58 |
| jonathanmaw | though it's also perplexing that I'm the only one encountering it with master, too :/ | 13:59 |
| tristan | jonathanmaw, your KeyError note I saw in the irc logs... looks like a bug that you managed to reproduce | 14:00 |
| tristan | "deleting a field from a dict" as far as I know, is not something that there is any way to do whatsoever | 14:00 |
| tristan | what did I miss ? | 14:00 |
| gitlab-br-bot | buildstream: Tristan Maat created branch 81-non-empty-read-only-directories-not-handled-during-bst-build-and-others | 14:00 |
| jonathanmaw | tristan: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L178 for example | 14:01 |
| tristan | jonathanmaw, honestly I dont immensely care about a bug with how 'arches' conditionals work, assuming that for right now; if you follow ssam2's instructions it should still work | 14:02 |
| tristan | that would be because, I'm in the process of nuking those conditionals from existence in favor of more generic project options | 14:02 |
| tristan | (so the fix will be obsolete, hopefully next week) | 14:02 |
| tristan | for now; using it the way that it works would be better :) | 14:02 |
| jonathanmaw | looking beyond arch conditionals, it also fails to load elements here https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L413 | 14:03 |
| jonathanmaw | because my "base" element doesn't have any depends | 14:03 |
| jonathanmaw | afaict | 14:03 |
| jonathanmaw | and it also fails at https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_loader.py#L256 | 14:04 |
| ssam2 | note that base.bst has a host-arches switch statement | 14:04 |
| ssam2 | which sets 'sources' for that element | 14:04 |
| ssam2 | if your --host-arch doesn't match anything in the switch, base.bst will end up with no sources | 14:04 |
| ssam2 | and thus buildstream will complain that it can't build it | 14:04 |
| ssam2 | it's a bit unintuitive, but i'm not sure how to give a better error in that situation | 14:05 |
| tristan | jonathanmaw, I find that very strange (*every* project starts with an import that has no dependencies), but making like 413 safer is not a bad thing | 14:05 |
| tristan | ok looking at 256 which is same thing | 14:06 |
| tristan | it would seem that you have a python that behaves differently on 'del' ? | 14:06 |
| tristan | very odd, though | 14:06 |
| jonathanmaw | python 3.5.2 | 14:06 |
| tristan | especially if this is on the moonshot | 14:06 |
| jonathanmaw | yep, moonshot | 14:07 |
| tristan | very odd | 14:07 |
| tristan | anyway yeah, I have 3.5.3 locally and havent seen this | 14:09 |
| tristan | that is *supposed* to raise KeyError, and never has | 14:11 |
| tristan | jonathanmaw, yeah just fix those sillinesses | 14:11 |
| tristan | cant really explain why it's triggering for you and not me, but fixing it is certainly harmless | 14:12 |
| jonathanmaw | so far, I've been working off the assumption that it's related to the cause of things going wrong down the line | 14:12 |
| jonathanmaw | i.e. I'm currently hanging on Checking elements | 14:12 |
| jonathanmaw | I think I've figured out why it was hanging. I was using a source of kind "local", which was a chroot I'd populated to build with. | 14:29 |
| jonathanmaw | it has devices set up in /dev, so it was trying to read all of /dev/random | 14:29 |
| ssam2 | wow, that's a good one | 14:30 |
| jonathanmaw | shall I submit a bug for that? | 14:30 |
| jonathanmaw | or is it suitably weird that it's practically forbidden? | 14:30 |
| ssam2 | i don't know how to really defend against that | 14:32 |
| tristan | nobody would ever do that :) | 14:32 |
| tristan | hehe | 14:32 |
| jonathanmaw | hrm, now the build's failing because ruamel.yaml has no attribute 'round_trip_dump' | 14:40 |
| ssam2 | your ruamel.yaml is broken, then | 14:42 |
| jonathanmaw | yep, going to try a ruamel.yaml installed from pip | 14:42 |
| jonathanmaw | \o/ it's building! | 14:56 |
| *** jonathanmaw has quit IRC | 15:01 | |
| *** jonathanmaw has joined #buildstream | 15:01 | |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 1 commit (last: setup.py: Add note about requirements) https://gitlab.com/BuildStream/buildstream/commit/7ca36931aff4bbf7ca5921889215c4790bbf96de | 15:27 |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 15:27 |
| *** jonathanmaw has quit IRC | 15:36 | |
| *** jonathanmaw has joined #buildstream | 15:36 | |
| jonathanmaw | aw, build failed at stage2-glibc because gawk was missing from the toolchain I provided | 15:48 |
| *** tlater has quit IRC | 16:11 | |
| *** ssam2 has quit IRC | 16:59 | |
| *** jonathanmaw has quit IRC | 17:03 | |
| *** semanticdesign has quit IRC | 19:42 | |
| *** xjuan has quit IRC | 20:15 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!