| *** tristan has joined #buildstream | 05:31 | |
| *** tristan has quit IRC | 05:43 | |
| *** tristan has joined #buildstream | 06:28 | |
| *** ChanServ sets mode: +o tristan | 06:28 | |
| *** jude has joined #buildstream | 07:17 | |
| *** franred has joined #buildstream | 08:08 | |
| *** tlater has joined #buildstream | 08:09 | |
| *** franred is now known as noisecell | 08:09 | |
| *** tlater has quit IRC | 08:31 | |
| *** tlater has joined #buildstream | 08:32 | |
| *** jonathanmaw has joined #buildstream | 08:45 | |
| *** ssam2 has joined #buildstream | 09:10 | |
| *** Chris has quit IRC | 09:15 | |
| *** tlater has quit IRC | 09:29 | |
| tristan | ssam2, ok regarding that comment, I am thinking that there should be an invariant: If a Source is in an Consistency.INCONSISTENT state (i.e. it has no ref), then ->fetch() will never be called on it | 09:47 |
|---|---|---|
| ssam2 | seems fair | 09:47 |
| tristan | I think it was my mistake to put that comment in _pipeline.py: Pipeline.build() | 09:48 |
| tristan | instead of just asserting that first | 09:48 |
| tristan | What would be the right human language for a source which has no ref ? | 09:49 |
| tristan | for an error message | 09:49 |
| ssam2 | it depends on the source | 09:50 |
| ssam2 | 'tar' is a weird case I think | 09:50 |
| tristan | Yeah thats what I mean | 09:50 |
| tristan | well, tar is indeed a weird case | 09:50 |
| ssam2 | probably OK to just say 'try running `bst track` to set the ref fields' | 09:50 |
| tristan | but lets ignore that | 09:50 |
| tristan | "Exact versions are missing for the the following elements, try running `bst track` on these elements first:" | 09:51 |
| tristan | That is the real meaning, ignoring the fact that tar is weird | 09:51 |
| ssam2 | yeah sounds good to me | 09:51 |
| tristan | ssam2, while I'm at it... I have something else... right now in `bst fetch` I just filter out inconsistent sources | 10:03 |
| tristan | I think it would be better to make the same assertion there | 10:03 |
| tristan | Thoughts ? | 10:03 |
| tristan | Also I'm going to add the --track option to `bst fetch` (like `bst build`, it can be done in one go) | 10:04 |
| tristan | Also... I was thinking of that --sever option (we need a better name), it will be interesting for other purposes (especially track) | 10:07 |
| tristan | Will be interesting to be able to express branches of the build graph to chop off... for track this means "I want to build the latest of my high level stuff, but I dont want to rebuild the whole damn runtime" | 10:07 |
| ssam2 | yes indeed | 10:12 |
| ssam2 | that will be good for people switching from jhbuild I think | 10:13 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 3 commits (last: setup.py: Added comment about OSTree.YEAR_VERSION & OSTree.RELEASE_VERSION) https://gitlab.com/BuildStream/buildstream/commit/0ea7c45f5fea46f1ecf606fdf4090069fb2662b8 | 10:21 |
| tristan | ssam2, alright assertions of consistency is there... | 10:21 |
| tristan | If you can check your MR and put only what is still needed, that would be great :) | 10:22 |
| ssam2 | will do | 10:22 |
| tristan | Also now `bst fetch --track` exists :) | 10:22 |
| gitlab-br-bot | buildstream: merge request (sam/traceback-fixes->master: Replace a few tracebacks with actual error messages) #24 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24 | 10:38 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: import.py: Raise an error when no files are found) https://gitlab.com/BuildStream/buildstream/commit/2620bfe2a15abcfdb15f4e70d8ddb1e943a2d97f | 10:44 |
| gitlab-br-bot | buildstream: merge request (sam/traceback-fixes->master: Replace a few tracebacks with actual error messages) #24 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/24 | 10:45 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: scriptelement.py: Fixing some fallout in scriptelement) https://gitlab.com/BuildStream/buildstream/commit/b076d2e88518a225767844e06f30e9dbaca83015 | 10:55 |
| *** ssam2 has quit IRC | 11:00 | |
| jonathanmaw | tristan: I think https://gitlab.com/BuildStream/buildstream/commit/b076d2e88518a225767844e06f30e9dbaca83015 might give you trouble | 11:06 |
| jonathanmaw | If you want to instantiate it with __layout = [], doing it in the class definition means that every element instantiated from it will start with that array | 11:07 |
| jonathanmaw | putting it in the init method would probably be better | 11:07 |
| jonathanmaw | I remember tripping up on something similar before | 11:07 |
| *** ssam2 has joined #buildstream | 11:16 | |
| tristan | jonathanmaw, I added a note to layout_add() about that | 11:23 |
| tristan | jonathanmaw, since you already do 'if not self.__layout: ... self.__layout = []' it shouldnt be an issue | 11:23 |
| tristan | (i.e. you are not appending to a mutable class data) | 11:23 |
| jonathanmaw | aha | 11:23 |
| tristan | (you create a new instance level list) | 11:23 |
| tristan | since empty list is falsy, the rest need not change | 11:24 |
| tristan | anyway, yeah... mutable things in python are tricky | 11:24 |
| tristan | for instance, watch out for mutable keyword argument default values | 11:25 |
| tristan | they are not instantiated on a per call basis but global data for program lifetime (ouch) | 11:25 |
| ssam2 | is important host tools from a tarball something that we want to support ? | 12:23 |
| ssam2 | i've hit a few issues, most seriously 'Error staging source: [Errno 1] Operation not permitted' | 12:23 |
| ssam2 | am trying to import Alpine Linux as a base to avoid dealing with multiple GBs of GNOME SDK every time I want to run a build | 12:23 |
| ssam2 | i could always put it in an OSTree repo outside of BuildStream, but since they provide binary tarballs I figured I'd try that first | 12:24 |
| ssam2 | hmm, I guess the issue will be that my user doesn't have permission to extract a tarfile with stuff owned by root | 12:25 |
| tristan | ssam2, the org.freedesktop.BaseRuntime and BaseSdk should not be that large really | 12:25 |
| ssam2 | ah yeah it's the FreeDesktop one | 12:25 |
| tristan | it's just the output of the base runtime yocto build | 12:25 |
| tristan | not the GNOME SDL | 12:25 |
| tristan | K | 12:25 |
| tristan | Hmm | 12:25 |
| ssam2 | i'm still having to wait around quite a bit though, and most of it is unneeded since i'm just building a toolchain | 12:25 |
| tristan | not sure how big that really is but it's supposed to only be the compiler, runtime and some tools | 12:26 |
| tristan | In any case, you should be able to use a tarball as a source for an import element, to answer more directly | 12:27 |
| tristan | ssam2, if you can untar the tarball, then buildstream should be able to | 12:27 |
| ssam2 | right, the issue is that I can't | 12:27 |
| ssam2 | which I guess isn't buildstream's problem | 12:27 |
| tristan | if you cant, then buildstream cant | 12:27 |
| tristan | right | 12:27 |
| ssam2 | in terms of freedesktop SDK ... i started a build there with my changes stashed, i'm on 1:38 of staging sources for base-platform.bst and base-sdk.bst and still waiting for anything to build | 12:28 |
| tristan | ssam2, honestly though, tarballs are supposed to be doable to some most extent as regular user, it will normally untar as active user if you are not root, etc | 12:28 |
| ssam2 | yeah... i think this might have device nodes or something weird | 12:28 |
| ssam2 | i'll write a script to put it in an ostree repo, or at least make a more sane tarball | 12:28 |
| tristan | yeah that we wont support, if it has xattrs then maybe we can strip them along the way | 12:29 |
| tristan | ssam2, of *staging* ? | 12:29 |
| ssam2 | yeah | 12:29 |
| tristan | not of fetching ? | 12:29 |
| ssam2 | no, staging | 12:29 |
| tristan | ssam2, odd really, you seem to be the winner of the slowest harddisk award | 12:29 |
| ssam2 | that wouldn't surprise me, this laptop is quite old | 12:30 |
| tristan | ssam2, so what I might also check... is `stat -c "%h"` on the files in ~/.cache/buildstream/artifacts/extract/... | 12:30 |
| tristan | ssam2, do you have bleeding edge ostree ? | 12:30 |
| ssam2 | not bleeding edge | 12:30 |
| tristan | I'm not *entirely* certain, I think you really should have hardlink checkouts with buildstream as-of-this-week and any version of ostree | 12:31 |
| tristan | because bare-user repository + user-mode checkout should give you that | 12:31 |
| tristan | ssam2, actually wait... | 12:32 |
| ssam2 | `stat -c "%h"` is showing 1 link for everything | 12:32 |
| tristan | you are staging the source not the artifact, so that you can cache an artifact for later use... right ? | 12:32 |
| ssam2 | which seems wrong.. | 12:32 |
| tristan | ssam2, yes that does seem wrong | 12:32 |
| tristan | ssam2, maybe you did not wipe your artifact cache since the changes | 12:32 |
| ssam2 | i didn't | 12:32 |
| ssam2 | it seemed to be in bare-user mode already | 12:33 |
| ssam2 | but will try that | 12:33 |
| tristan | ssam2, check the ostree/config file | 12:33 |
| tristan | it's like a git config ini | 12:33 |
| ssam2 | yes, mode=bare-user | 12:33 |
| tristan | will tell you that it is archive-z2 if it's from before my fixes | 12:33 |
| tristan | hmmm | 12:33 |
| tristan | strange | 12:33 |
| ssam2 | i can try wiping it though, I need to go eat anyway :-) | 12:33 |
| tristan | Ok do that | 12:34 |
| tristan | ssam2, I think that staging an ostree import will still take time, as it will uncompress things to stage initially | 12:34 |
| tristan | ssam2, and once you have it as an artifact, it should be hardlinks all the way | 12:34 |
| *** tlater has joined #buildstream | 12:35 | |
| ssam2 | ohh, what might have happened is that i've changed cache keys | 12:35 |
| ssam2 | quite likely with the host-arch stuff | 12:35 |
| ssam2 | so it'll be re-importing it from the sources/ repo | 12:35 |
| tristan | That part of staging the source ostree will take much more time yes | 12:36 |
| tristan | which is one reason it will be great to have artifact sharing... so nobody has to almost ever do that :) | 12:37 |
| tristan | It could be that we should be using bare-user repositories for the ostree source, too... but in that case it seems to make sense to use compression | 12:37 |
| ssam2 | still not entirely sure why you can't pull everything into the one repo | 12:38 |
| ssam2 | but we've discussed that before | 12:38 |
| tristan | It's not that it's not technically possible, it's that it's a horrible thing to do to the code, and artifact shares should mostly solve it anyway | 12:40 |
| tristan | ssam2, if you can make every Source implementation mirror directly into ostree instead of separate git,tar,bzr,etc mirrors, then yes, lets do it :) | 12:41 |
| tristan | actually even then, lets not do it | 12:41 |
| ssam2 | I suppose you're right that it's only needed when building from scratch | 12:42 |
| tristan | because then what happens when we port to environments which dont have ostree... ugh, what a mess it would be | 12:42 |
| tristan | sources are just not artifacts, they are different | 12:42 |
| ssam2 | hmm, that raises an interesting other point | 12:42 |
| ssam2 | of how to get host tools into systems that don't have ostree | 12:43 |
| ssam2 | I guess tarballs is the only way | 12:43 |
| tristan | Right: You cannot use an ostree source on a system that doesnt have ostree | 12:43 |
| tristan | Just like you cannot use tarball sources on systems that cant use tar | 12:43 |
| tristan | ssam2, you do it with something else, tarballs is one way | 12:43 |
| gitlab-br-bot | buildstream: merge request (sam/tar-base-dir-fix->master: tar.py: Consider the 'base-dir' as part of the unique key) #25 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25 | 12:54 |
| tristan | ssam2, ah good point | 12:55 |
| ssam2 | that stumped me for a minute or two :-) | 12:56 |
| tristan | not sure I agree with the reasoning in the comment/commit message, but setting base-dir will certainly effect what is staged, so it should be considered :) | 12:56 |
| ssam2 | yeah i should probably have not written anything | 12:57 |
| tristan | ssam2, anyway the default '*' should work for almost everything we care about; it's rare that you have dangerous tarballs which start with files directly in '.' | 12:57 |
| ssam2 | that's true for source tarballs | 12:57 |
| tristan | anyway I hit the merge button | 12:57 |
| ssam2 | but not for binary things | 12:57 |
| ssam2 | thanks | 12:57 |
| ssam2 | going to go eat lunch rather than try and arrange any more words into meaningful orders :-) | 12:58 |
| tristan | ssam2, not sure I agree: Its true for binary tarballs created by wreckless people | 12:58 |
| tristan | who think that sort of thing is just okay :) | 12:58 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 1 commit (last: tar.py: Consider the 'base-dir' as part of the unique key) https://gitlab.com/BuildStream/buildstream/commit/c7a75c038b5b03c899a709bf9d106f2e217e9c71 | 12:58 |
| gitlab-br-bot | buildstream: merge request (sam/tar-base-dir-fix->master: tar.py: Consider the 'base-dir' as part of the unique key) #25 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/25 | 12:58 |
| tlater | How do I determine what elements are build elements? element.get_kind() doesn't seem to be what I'm looking for, and I'd like to avoid introspection. | 13:39 |
| tristan | tlater, you dont | 13:45 |
| tristan | I mean, you *can*, but that wouldnt be good, instead you dont | 13:46 |
| tlater | How do I check which objects do implement my write_script interface then? Letting them throw ImplError feels like it might cause issues | 13:46 |
| tristan | tlater, yeah I'm not sure... I have a similar conundrum with `bst shell` I havent decided how to solve yet | 13:50 |
| tristan | I have two options, 1: construct a fake sandbox to pass to configure_sandbox() and/or stage() and then handle ImplError - sucks for the core but better/smaller API surface for plugins | 13:51 |
| tristan | 2: Add some kind of can_shell() method, cleaner for the core but means one more superfluous method a plugin must implement | 13:51 |
| tristan | I'm a bit torn about this | 13:51 |
| tlater | I prefer 2, personally | 13:51 |
| tristan | At least for shell, I dont need to error out early really; so I can do 1 and just fail gracefully | 13:52 |
| tristan | I honestly prefer 1, if at all feasible | 13:52 |
| tlater | Why is the fake sandbox necessary? | 13:53 |
| tristan | For an element to be shell-able, we need to know that it implements configure_sandbox, stage and assemble separately; launching a shell is the first two without the third | 13:53 |
| tristan | those methods take a sandbox and do stuff | 13:53 |
| tristan | If I constructed a fake sandbox and tried to catch ImplError, then I could determine whether it's possible without going further | 13:54 |
| tristan | But, for shell it's not all that important to handle it early | 13:54 |
| tlater | Gotcha, I didn't know some elements don't implement it | 13:54 |
| tristan | I can just catch it at that time | 13:54 |
| tristan | Yeah, well some elements just implement assemble(), originally there was only assemble() but this became problematic for shelling | 13:55 |
| tlater | This could work for the source_bundle too | 13:55 |
| tlater | I don't see why the error would need to be caught too early | 13:55 |
| tlater | Although it does waste some time... | 13:55 |
| tristan | Indeed, it's not a huge process if you generate the scripts before staging the sources | 13:56 |
| tristan | tlater, we make attempts to error early in regular operations which use the scheduler, cause it sucks to be building 500 modules for 3 hours and then find out that you screwed up, and could have been told at launch time :) | 13:56 |
| tlater | Yeah, that does seem like a pain | 13:57 |
| tristan | I suppose as long as you generate the scripts before you stage the sources to some output place (which could also take a long time), that's pretty fine | 13:57 |
| tlater | The ImplError should never actually show up for the source_bundle command though, should it? | 13:57 |
| tristan | even if it were 500 modules, you can generate all the scripts in probably less than a second | 13:58 |
| tristan | You would have to explicitly trap it and raise something more appropriate | 13:58 |
| tristan | tlater, to clarify; stack traces should never reach the user | 13:58 |
| tlater | No, but I mean, if I for example try: `bst source_bundle busybox.bst some_dir1 | 13:59 |
| tristan | So when it's code in _pipeline.py, it can be self.message(... MessageType.ERROR ...) | 13:59 |
| tristan | also, please dashes no underscores :) | 14:00 |
| tristan | heh | 14:00 |
| tlater | Oh, I'll have to fiddle with click then... But yeah, that does look neater | 14:00 |
| tlater | Either way, what's the expected behaviour of trying to bundle a stack target? Should it create a bundle containing all dependencies, with scripts to build those, or throw an error because a stack target is not a build element? | 14:01 |
| tristan | right, dont add huge code to main.py, let it call into the pipeline and do highlevel operations there | 14:01 |
| tristan | tlater, I dont know, it depends on how your surrounding stuff is going to work | 14:02 |
| tristan | tlater, I'm ok if it generates a noop | 14:02 |
| tristan | in any case it will be a serialized build script, so the order of pipeline.dependencies() should be correct | 14:03 |
| tlater | Could I then just ignore ImplError? | 14:04 |
| tlater | Wait, no, because that may be thrown elsewhere. Hm :/ | 14:04 |
| tristan | tlater, say you start with an Pipeline.source_bundle() internal API... and it has a try: with a loop over the dependencies to generate these scripts | 14:05 |
| tristan | tlater, the except ImplError: of that try is where you self.message(... MessageType.ERROR ...) and early return | 14:06 |
| tlater | The problem is that if I then try to bundle something like busybox, I would expect to get a bundle with all dependencies, and not an error. This implementation would always error then. | 14:07 |
| tristan | Sorry ? | 14:26 |
| tristan | tlater, I dont follow | 14:26 |
| tristan | tlater, AHHHH, ok that's what the --sever conversation was about | 14:26 |
| tristan | But I need a better name for that option | 14:26 |
| tristan | I want to have it also for `bst track` | 14:26 |
| tristan | tlater, i.e. someone could run `bst track --deps all --sever base-deps.bst gnome.bst`... and then they would be expressing a pipeline of "All of gnome.bst's dependencies, except for base-deps.bst and what that depends on, as long as those deps are not common with other things that gnome.bst depends on" | 14:28 |
| tristan | That would let me say, upgrade to new source references for my highlevel stuff, except keep the same baseline versions | 14:29 |
| tristan | tlater, for `bst source-bundle` you would probably need to use that to eliminate the base imports, actually you would want that to eliminate the entire gnu-toolchain.bst part | 14:29 |
| tristan | So anyway in real life, people will cross build gnu-toolchain.bst and produce an artifact for a foreign architecture, then they would run `bst source-bundle --sever gnu-toolchain.bst element-providing-buildstream-and-its-deps.bst` | 14:30 |
| tristan | To give you a build script that gets you from a base runtime on that arch -> to a system that can run buildstream | 14:31 |
| tristan | so you _need_ to be able to address a part of a pipeline | 14:31 |
| tristan | but I dont like the word --sever | 14:31 |
| tristan | maybe --chop | 14:31 |
| tlater | --except? | 14:32 |
| ssam2 | --start-from ... but that's misleading because you start *after* | 14:32 |
| tristan | --except sounds nice yeah | 14:32 |
| ssam2 | i like --except | 14:32 |
| tristan | --start-from is misleading because it implies its a serial build model, and that it can only be specified once | 14:32 |
| tristan | --except may be specified multiple times | 14:32 |
| tlater | I implemented --all from build because of consistency, too, should that affect --except? | 14:33 |
| tristan | Another approach for handling ImplError is to fire a MessageType.WARNING and just keep going, excluding that element entirely (and also marking that element for exclusion of it's sources being wrapped) | 14:33 |
| tristan | But seems not all that useful anyway | 14:34 |
| tristan | I think --except overrides, but --all from `bst build` I think is nonsensical in this context | 14:35 |
| tristan | because `bst build` will reason about the build plan depending on what is in the cache | 14:35 |
| tristan | and what was already built as a build-only dependency of something already cache will be discarded from the plan | 14:36 |
| tristan | (unless --all is specified) | 14:36 |
| tlater | Yeah, that doesn't make much sense. --all could be used to toggle warnings, but that should probably be named something else then. | 14:36 |
| tristan | I think you probably want the --deps options from `bst show`, excepting for the 'plan' option | 14:37 |
| tristan | and --except effects the pipeline after that selection | 14:37 |
| tristan | that gives most flexibility anyway | 14:38 |
| tristan | without offering the `--deps plan` which just makes no sense for source-bundle | 14:38 |
| tristan | damn ninja just doesnt work in the sandbox | 14:38 |
| tlater | Yup, that seems right. Hadn't seen --deps yet. | 14:39 |
| tristan | been trying to strace the crap out of this and looking at ugly C++ code for hours https://github.com/ninja-build/ninja/issues/1288 | 14:39 |
| ssam2 | nasty | 14:43 |
| tristan | I think I found the nasty | 14:51 |
| tristan | This baby is used to find if a file exists: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L72 | 14:54 |
| tristan | by checking the mtime here: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L61 | 14:54 |
| tristan | That mtime (which is 0 for /usr/bin/glib-mkenums), gets compared with -1 in https://github.com/ninja-build/ninja/blob/master/src/graph.cc#L30 | 14:55 |
| tristan | hmmm, have to hunt down exists() | 14:56 |
| * tristan gotta run | 14:56 | |
| tristan | AHA ! | 14:57 |
| tristan | mtime != 0 == exists !!!! | 14:57 |
| tristan | NOT TRUE ! | 14:57 |
| ssam2 | d'oh | 15:03 |
| ssam2 | yeah that's a silly assumption | 15:03 |
| ssam2 | does BuildStream set mtime to 0 then ? might make sense to set it to a fixed non-zero value instead, in case other things do that | 15:03 |
| *** tristan has quit IRC | 15:04 | |
| *** tristan has joined #buildstream | 15:14 | |
| *** tristan has quit IRC | 15:16 | |
| *** tristan has joined #buildstream | 15:16 | |
| *** ChanServ sets mode: +o tristan | 15:16 | |
| *** jonathanmaw has quit IRC | 16:02 | |
| *** noisecell has quit IRC | 16:42 | |
| *** tiagogomes has quit IRC | 16:44 | |
| *** jude has quit IRC | 17:24 | |
| *** tlater has quit IRC | 17:29 | |
| ssam2 | any thoughts on why GCC builds would die from out of memory inside BuildStream ? | 17:40 |
| ssam2 | it's partly a GCC bloat/design issue | 17:40 |
| ssam2 | but I can build it on this laptop outside BuildStream, so there must be something else making it worse | 17:40 |
| ssam2 | it says `MAKEFLAGS=-j4` which should be OK | 17:40 |
| ssam2 | ah, maybe Firefox using even more memory than usual | 17:41 |
| * ssam2 needs to set up a build VM to avoid this crap | 17:41 | |
| ssam2 | one last try | 17:41 |
| tristan | strange | 17:43 |
| tristan | ssam2, there shouldnt be any memory restrictions in the sandbox, maybe it's firefox indeed | 17:43 |
| tristan | let me know if you identify anything | 17:44 |
| tristan | fwiw the issue I hunted down was a dup of https://github.com/ninja-build/ninja/issues/1120 | 17:45 |
| *** ssam2 has quit IRC | 18:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!