| *** tristan has joined #buildstream | 05:04 | |
| *** ChanServ sets mode: +o tristan | 06:00 | |
| * tristan pushed the gcc build failure issue to build-essential in buildstream-tests fwiw | 10:05 | |
| ironfoot | ta | 10:12 |
|---|---|---|
| *** ssam2 has joined #buildstream | 10:16 | |
| tristan | cache keys fixed with this: https://gitlab.com/tristanvb/buildstream/commit/bf7643a131b3d964b64e970b4be0d1f362e4958f ... will rebuild world | 11:46 |
| tristan | now would be the right time to land a cache key version | 11:46 |
| tristan | (and test case) | 11:47 |
| ironfoot | great :) | 11:51 |
| * tristan is cooking up something nice actually right now :) | 11:54 | |
| ironfoot | heh, i dont know if you are talking about code, or food :) | 11:55 |
| * tristan hasnt cooked food in many years :-/ | 11:55 | |
| ironfoot | lol | 11:57 |
| tristan | does ramen noodles count ? | 11:57 |
| tristan | heh | 11:57 |
| ironfoot | :) | 11:58 |
| tristan | mkay, network unreachable pipeline failure https://gitlab.com/tristanvb/buildstream/builds/10599320 | 12:23 |
| tristan | meh | 12:23 |
| ironfoot | I'm testing I can build the converted core.bst | 12:35 |
| ironfoot | and.. I'm still waiting :/ | 12:35 |
| tristan | waiting how so ? | 12:36 |
| * tristan pushed his last but it doesnt provide the awesome that he wanted to include, it will have to wait for a bit of the frontend/UI refactor | 12:36 | |
| ironfoot | tristan: I've pushed https://gitlab.com/palvarez89/buildstream-tests/commits/pedro/defs2bst-tests if you want to give them a try | 12:39 |
| tristan | ironfoot, are you using the <target>.yml now ? | 12:42 |
| tristan | ironfoot, also... I'm curious about "I'm still waiting :/" | 12:43 |
| tristan | waiting... for a download to complete ? | 12:43 |
| tristan | ironfoot, also it may be good to do that in a fork of buildstream-tests, branched off of build-essential for now | 12:43 |
| ironfoot | waiting for any output | 12:44 |
| ironfoot | and yes, using the <target>.yml now | 12:45 |
| tristan | for any output | 12:45 |
| tristan | hmm :-/ | 12:45 |
| tristan | what does ps tell you ? | 12:45 |
| tristan | Oh sorry I didnt notice it was actually based on a fork of buildstream-tests | 12:47 |
| ironfoot | i don't know how to use `ps` to get useful information | 12:47 |
| ironfoot | :P | 12:47 |
| tristan | ps axf ? | 12:47 |
| tristan | and look for buildstream process | 12:47 |
| tristan | see if it's got any children | 12:47 |
| ironfoot | ah | 12:48 |
| ironfoot | nope | 12:48 |
| tristan | ironfoot, it could be that, maybe ostree cloning is stuck on something ? | 12:48 |
| tristan | maybe something choking on a stale lock from a previous CNTL-C ? | 12:48 |
| ironfoot | if I build a single element from core/ seems to work | 12:48 |
| ironfoot | I'll do some straces | 12:48 |
| tristan | ironfoot, ok so first of all; looking at your output... | 12:49 |
| tristan | you are setting dependencies all to type: build | 12:49 |
| tristan | See https://people.gnome.org/~tvb/buildstreamdocs/format.html#dependency-types | 12:49 |
| tristan | ironfoot, "build" dependencies are dependencies only required for building, as in they are not required to run; also they are not transitive as such | 12:50 |
| tristan | ironfoot, so if a "build" depends on b which "build" depends on c, c is not staged to build a | 12:50 |
| tristan | only implicitly brought in if c is also required for b to "run" | 12:51 |
| tristan | ironfoot, simple shorthand notation of the element path (with no filename: or type:) implies both build & run | 12:51 |
| ironfoot | yeah, I assumed I wasn't going to get all the dependencies right | 13:06 |
| ssam2 | that's a nice design, more expressive than Baserock without losing the simplicity | 13:06 |
| tristan | yeah I think so :) | 13:08 |
| tristan | we offload a lot of complexity by treating distribution separately, and this is one place it pays off | 13:08 |
| tristan | (no need to consider split "package" based dependencies in the model, which costs brain cells) | 13:09 |
| ironfoot | strace shows no activity | 13:12 |
| ironfoot | strace shows no activity after opening the files described in core.bst | 13:13 |
| tristan | mkay, odd | 13:13 |
| tristan | circular dependencies should be caught | 13:13 |
| tristan | ironfoot, no core is at 100% cpu ? | 13:13 |
| ironfoot | heheh yes | 13:15 |
| ironfoot | bst, 100% | 13:15 |
| ironfoot | I've removed all the deps in core.bst but one, and it started building | 13:15 |
| tristan | Ok, there is some context :) | 13:15 |
| tristan | So I presume our test cases are not finding all of the possible circular dependencies, or smth | 13:16 |
| * tristan doesnt see other ways we can spin, circular deps did make us spin | 13:16 | |
| tristan | ironfoot, see if you can find any circular dep in your conversion first, and I'll have to reproduce and fix | 13:20 |
| tristan | ironfoot, or you could reproduce and fix if you like of course :) ... in any case a bug report with the offending .bst situation would be great | 13:21 |
| *** tristan has quit IRC | 13:51 | |
| *** tristan has joined #buildstream | 14:13 | |
| *** ChanServ sets mode: +o tristan | 14:14 | |
| ironfoot | sure, I'll investigate a bit more :) | 14:23 |
| * tristan was on the verge of a nice interactive UX today... build fails, pause current tasks: "Sir, this build has failed, would you like to: a.) shell into the build sandbox and debug the failure ? b.) Abort current tasks ? c.) Continue building components which do not depend on this build ?" | 14:30 | |
| tristan | but the frontend UI is in need of a cleanup first | 14:30 |
| tristan | all the context is in place though | 14:31 |
| ironfoot | --interactive mode | 14:32 |
| tristan | I dont even think that's necessary, perhaps non-interactive optional switch | 14:32 |
| tristan | ironfoot, one of the nice things about the click library, is it "knows" if you are connected to a terminal | 14:33 |
| ironfoot | maybe the other way around, yes. | 14:33 |
| ironfoot | ah, good | 14:33 |
| tristan | so enable interactive mode by default only when connected to terminal | 14:33 |
| tristan | in fact, I think it has input prompts which automatically do that, even | 14:34 |
| ironfoot | I'm unsure this is a case of circular deps | 15:32 |
| ironfoot | but I do think this is related to how BuildStream calculates the dependencies | 15:33 |
| *** tristan has quit IRC | 15:39 | |
| ironfoot | for example. 4m to start building core/libtool.bst | 15:41 |
| *** tristan has joined #buildstream | 15:43 | |
| *** ChanServ sets mode: +o tristan | 15:43 | |
| ironfoot | heh, so disabling the circular deps check, is the bit that makes it go really really slow | 17:41 |
| ironfoot | erm.. words | 17:41 |
| ironfoot | but you know what I mean, I'll try to understand it and see if we can improve it | 17:42 |
| ironfoot | It's really expensive right now, making BuildStream unusable to build core.bst | 17:42 |
| ssam2 | it's quite hard to solve | 17:43 |
| ironfoot | I left the build running for one hour, when having lunch, and hadn't started when I came back | 17:45 |
| ssam2 | Morph just added each source it encountered to a list, and if it finds something with the same name already in the list, raises an error | 17:45 |
| ssam2 | http://git.baserock.org/cgit/baserock/baserock/morph.git/tree/morphlib/sourceresolver.py#n392 | 17:45 |
| ironfoot | I assume it can be improved :) | 17:45 |
| ssam2 | hah, yeah | 17:45 |
| ssam2 | YBD I think just starts building without resolving the build graph, then raises an error if it's building something it built before | 17:46 |
| ssam2 | but that approach limits what you can reuse the code to do | 17:46 |
| ironfoot | i think the first approach fits better with the current codebase | 17:49 |
| tristan | ironfoot, the circular deps check itself takes hours to complete ?! | 17:50 |
| ironfoot | you can test it yourself, and tell me that I'm wrong | 17:50 |
| * tristan does not see how that many build paths are possible | 17:50 | |
| tristan | I will | 17:50 |
| tristan | fwiw, BuildStream's dependency graph resolution is exponentially tricky because of variants (but wont be unless there *are* variants) | 17:51 |
| tristan | it's besides the point but explains the crazy loops which were needed in _loader.py | 17:51 |
| tristan | that could probably be optimized, but I'm not convinced that its required | 17:52 |
| tristan | heh, /me sees his own comment: # XXX FIXME: This is horribly expensive | 17:53 |
| tristan | ironfoot, question: Have you ever seen it complete ? | 17:56 |
| ironfoot | yep | 17:57 |
| tristan | wow | 17:57 |
| ironfoot | i mean | 17:57 |
| ironfoot | for the entire core.bst? nope | 17:57 |
| tristan | Oh | 17:57 |
| ironfoot | for parts of core.bst, yes (i tried removing some components to check) | 17:57 |
| tristan | And how long did that take ? | 17:58 |
| tristan | when you saw it complete | 17:58 |
| tristan | and how much of core.morph did you need to remove ? | 17:58 |
| ironfoot | it takes about 4 minutes to start building some of the components in core/*.bst | 17:58 |
| tristan | where some = ... 5 or 10 components ? | 17:59 |
| tristan | core is not *that* huge after all | 17:59 |
| ironfoot | some = 1 | 17:59 |
| tristan | wow so essentially, build-essential with all of its components is almost free | 18:00 |
| ironfoot | I mean, 4 minutes to start building only linux-pam | 18:00 |
| tristan | but adding one component which depends on it, costs 4 minutes ? | 18:00 |
| tristan | Ok how many components was in the dependency graph for it to take 4 minutes to check circular deps ? | 18:00 |
| ironfoot | I haven't noted down my findings sadly, I thought i was tracking down other problems | 18:00 |
| tristan | well ok... in the meantime, ssam2 I'm looking at that sourceresolver.py and I'm unsure how/what it's doing | 18:02 |
| tristan | ssam2, of course the same element shows up multiple times in a build graph, that is expected right ? | 18:02 |
| tristan | so you cant bail out | 18:02 |
| tristan | Or this is all thanks to the containing nature of "strata", which allow this to be true ? | 18:03 |
| tristan | Or rather, morph was never even able to build more than one component in parallel ? | 18:03 |
| ssam2 | yeah, maybe strata made that true | 18:04 |
| tristan | but even within one stratum, you have multiple chunks which depend on a common chunk, so its a bit confusing :-/ | 18:05 |
| ssam2 | yeah | 18:06 |
| tristan | well, I'm sure we can optimize that well, although... it really didnt strike me as something that would take more than say, 10 seconds when you have > 10,000 chunks to build | 18:06 |
| tristan | but eh | 18:06 |
| * tristan was considering some denormalization at load time, making sort-by-depth less of a contorted thing, perhaps that might help | 18:08 | |
| tristan | although, seems that would you try to calculate depth, that could be dangerous pre-circular-dep-check | 18:08 |
| * tristan thinks he has a plan... the list approach will work | 18:32 | |
| tristan | if, a.) its a recursive algorithm descending into dependency branches | 18:32 |
| tristan | and b.) the said list is copied and modified for each recursion | 18:33 |
| *** ssam2 has quit IRC | 19:38 | |
| *** tristan has quit IRC | 21:36 | |
| *** jjardon has quit IRC | 22:40 | |
| *** jjardon has joined #buildstream | 22:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!