| *** semanticdesign has quit IRC | 01:05 | |
| *** tristan has quit IRC | 04:58 | |
| *** tristan has joined #buildstream | 05:27 | |
| *** adds68 has joined #buildstream | 08:22 | |
| *** valentind has joined #buildstream | 08:31 | |
| *** tlater has joined #buildstream | 09:47 | |
| tristan | tlater, I dont feel super strongly about the fix for https://gitlab.com/BuildStream/buildstream/merge_requests/133/diffs fwiw | 09:54 |
|---|---|---|
| tlater | tristan: | 09:54 |
| tristan | tlater, just curious if there is a reason why we had to change the API at all | 09:54 |
| tlater | Gah, stupid enter key. Well, it turns out the platform needed arguments, and it seemed odd to have mandatory arguments for get_platform() yet not use them most of the time. | 09:55 |
| tristan | I mean, we could roll with that... but feels weird that a singleton needs to have a new "create_instance" explicitly called | 09:55 |
| tristan | tlater, that is a fair point | 09:56 |
| *** jude has joined #buildstream | 09:56 | |
| tlater | I was a little iffy about that too. | 09:56 |
| tristan | tlater, but... waait a sec | 09:56 |
| tristan | Why does it need a project ? | 09:56 |
| tlater | tristan: I... don't remember. I think it needed some of the API | 09:57 |
| tristan | ostree cache seems to need a project | 09:57 |
| tristan | Mkay | 09:58 |
| tristan | So, that is going to have to change | 09:58 |
| tristan | tlater, alright well lets roll with your patch as is then | 09:58 |
| * tristan expects that juergbi's branch will remove the 'project' parameter from Platform() constructors, and either will create a separate ArtifactCache for each project, or make a single ArtifactCache be able to work with multiple projects | 10:00 | |
| tristan | juergbi, that makes sense right ? | 10:00 |
| tristan | afiacs, the Project being a parameter of the singleton Project, is only there as a detail for the ArtifactCache to read per-project configuration about where to push/pull to/from | 10:01 |
| gitlab-br-bot | push on buildstream@platform_singleton (by Tristan Van Berkom): 9 commits (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/8c4d8ca2b7c886690fb933428c6768c745667db7 | 10:01 |
| gitlab-br-bot | buildstream: merge request (platform_singleton->master: Make the platform object a singleton) #133 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/133 | 10:01 |
| juergbi | tristan: at this point a single ArtifactCache is used with the configuration from the top-level project, but yes, we probably want to switch to one of these two options | 10:05 |
| juergbi | at least for the future import-style 'subprojects' | 10:05 |
| *** ssam2 has joined #buildstream | 10:07 | |
| tristan | juergbi, I would honestly expect that initially, in contrast with project options; shouldnt the artifact cache default to the project.conf declaring the elements ? | 10:09 |
| tristan | juergbi, rationale here (and original rationale for ensuring projects declare their own artifact caches), is that we mostly want to "by default" make sure that people dont push stuff to the wrong place | 10:10 |
| tristan | i.e. with inter-dependent projects, you can have proprietary code building with FOSS code, and you should not be able to easily and accidentally push proprietary binaries to a public share | 10:10 |
| juergbi | yes, it's probably the way to go | 10:11 |
| juergbi | it's not completely clear cut as for downstream kind of operation you might not want to push things with a tweaked build to upstream cache | 10:12 |
| juergbi | but it's probably still sensible as default | 10:12 |
| tristan | indeed | 10:12 |
| tristan | I think we're also looking into multiple artifact cache bla-bla-ness in the next cycle | 10:12 |
| tristan | so some flexibility in configuration will probably come along with that, too | 10:13 |
| juergbi | if the top-level project is proprietary, pushing FOSS subprojects to your top-level cache is not really an issue, though | 10:13 |
| juergbi | and i don't expect FOSS projects to have junctions to proprietary projects | 10:13 |
| tristan | True | 10:13 |
| juergbi | so i don't consider it essential | 10:14 |
| juergbi | from cache reuse perspective, it still makes sense to me, though | 10:14 |
| tristan | Hrrrm | 10:14 |
| tristan | maybe you're right - I wont get too deep into this right now | 10:15 |
| tristan | I do find it a bit weird that "A Platform singleton needs a project" but "A session can be building multiple projects" | 10:15 |
| tristan | juergbi, We do interrogate the element for it's project for most purposes in the artifact cache code | 10:15 |
| tristan | And untangling things is always a priority | 10:16 |
| tristan | in general | 10:16 |
| juergbi | i agree | 10:16 |
| juergbi | untangling definitely makes sense, no matter what policy we want in the end | 10:16 |
| tristan | yes :) | 10:16 |
| *** givascu has joined #buildstream | 10:26 | |
| tlater | tristan: You said you wanted to change something about the completion scripts - all my current failing test cases somehow revolve around that, could that perhaps be cleared up by your intended changes? | 10:27 |
| tristan | tlater, that part wont change | 10:27 |
| tristan | tlater, the completion tests unfortunately, need to be updated when some (not all) of the frontend command lines change | 10:28 |
| tristan | i.e. we use some of our own commands to test that completion features work | 10:28 |
| * tlater wonders if there's a way to do that automatically through click | 10:28 | |
| tristan | it's not like we check every single command, but we do some | 10:28 |
| tristan | tlater, our completions are a hack around click, so not really | 10:29 |
| tristan | because click's completions behave like crap, and the maintainer is overworked and doesnt have time to consider the patches I've sent | 10:30 |
| tristan | in fairness, they're not *that* bad | 10:30 |
| tristan | but ours are much better :) | 10:30 |
| *** tristan has quit IRC | 10:58 | |
| *** tristan has joined #buildstream | 11:02 | |
| *** tiagogomes has joined #buildstream | 11:09 | |
| gitlab-br-bot | buildstream: merge request (platform_singleton->master: Make the platform object a singleton) #133 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/133 | 11:33 |
| gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 2 commits (last: Make the platform object a singleton) https://gitlab.com/BuildStream/buildstream/commit/d64ea749379100f61ed2c3342bcda4d8e5a46d71 | 11:33 |
| gitlab-br-bot | buildstream: Tristan Van Berkom deleted branch platform_singleton | 11:33 |
| *** tiagogomes has quit IRC | 11:56 | |
| *** adds68 has quit IRC | 11:57 | |
| *** tiagogomes has joined #buildstream | 11:57 | |
| *** adds68 has joined #buildstream | 11:58 | |
| tlater | tristan: I'm really struggling to understand the completions code... If I try to complete 'bst build ', what does that trigger? options/arguments, their values, subcommands or chained commands? | 12:03 |
| tlater | Logically it would be argument values, but that only gives the default values click specifies, and so we get 'bst build ./*', instead of 'bst build elements/*' | 12:05 |
| tristan | tlater, I dont understand why you have to understand completions code | 12:13 |
| tlater | tristan: Well, we support unlimited numbers of arguments for various commands now, and that seems to break the completions code | 12:13 |
| tlater | I just have no clue why yet. | 12:13 |
| tristan | tlater, in tests/completions/completions.py, there are a few expected lists | 12:14 |
| tristan | and when a test fails, it shows you what was reported in the output | 12:14 |
| tristan | all that should be needed, is updating those lists with the new things found in the output, right ? | 12:14 |
| tristan | Ah, unlimited number breaks that ? | 12:14 |
| tristan | interesting | 12:14 |
| tlater | tristan: No, the actual completions are broken - it doesn't complete the correct paths anymore. Presumably because we're dealing with lists and not strings now. | 12:14 |
| tristan | ahhh, probably has to do with code I mean to refactor | 12:15 |
| tristan | tlater, so... right now there is some yucky stuff in _frontend/main.py which decides what to delegate to what completion | 12:17 |
| tristan | namely, the target option | 12:17 |
| tristan | tlater, if you've renamed target to targets for instance... you'll need to update that stuff | 12:18 |
| tlater | Ah | 12:18 |
| tlater | That makes sense | 12:18 |
| tristan | tlater, seems right now there is target or element | 12:18 |
| tristan | I mean to take care of refactoring that at the same time as supporting the except stuff | 12:18 |
| tristan | by subclassing the click.File() or click.Path() param types | 12:18 |
| tristan | and casing on that instead | 12:19 |
| tristan | for your purposes, dont get too deep, just update the hack is fine :) | 12:19 |
| tristan | main.py:override_completions() | 12:19 |
| tristan | is what you're looking for | 12:19 |
| tlater | Cool, I just couldn't track back to that since it doesn't show up when throwing exceptions in related functions. | 12:20 |
| tlater | ta tristan :) | 12:20 |
| tristan | yeah, the imported complete.py originating from click is not the easiest to follow | 12:20 |
| * tristan hopes maybe we can upstream this stuff next year | 12:21 | |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 14 commits (last: main.py: Adjust app to multiple targets) https://gitlab.com/BuildStream/buildstream/commit/1aaf031a0dd3f543aff07823b43384f0ff66e5bc | 12:41 |
| tlater | Hm, is anything that doesn't specify a dependency type just a runtime dependency? | 14:11 |
| * tlater finds that a bit counter-intuitive, especially when runtime dependencies are built *with* their parents. | 14:11 | |
| tristan | tlater, both | 14:22 |
| tristan | tlater, if it's not specified, it's assumed to be required for building and also required for running | 14:23 |
| tlater | tristan: Oh, so then for the purposes of planning it should be considered a build element since it is required to have been built before its parent? | 14:23 |
| tristan | tlater, a dependency that is only required for runtime, can be - something like a font, or a dbus service | 14:24 |
| tristan | suchlike | 14:24 |
| tristan | no | 14:24 |
| tristan | well, I dont know what you mean by that | 14:24 |
| tlater | tristan: If I understood juergbi 's code correctly RUN dependencies are built at the same time as their parents, whereas BUILD dependencies are built beforehand. | 14:24 |
| * paulsherwood notices that tristan seems to be enjoying the bd role :) | 14:25 | |
| tristan | http://buildstream.gitlab.io/buildstream/format.html#format-dependencies <-- of course that is worth a read | 14:25 |
| tlater | tristan: With my current implementation these elements are built twice... | 14:25 |
| juergbi | tlater: deps of type 'both' definitely need to be built beforehand, yes | 14:26 |
| tlater | Gah, that's going to look nasty, but I guess there's no way around it. | 14:27 |
| tristan | tlater, also take note of the Note: in the referred docs - if you only build depend on something, that thing is assumed to be required to run | 14:27 |
| tlater | tristan: Ah, right, ta :) | 14:29 |
| tristan | it's also not a huge requirement to like, rewrite juergbi's whole algorithm which has been working well enough, there must be a possible approach which just transforms that algo to work on a list of toplevel targets rather than a single one | 14:30 |
| juergbi | i think so as well | 14:31 |
| juergbi | isn't it more or less the same as an implicit stack with all the top-level targets? | 14:31 |
| juergbi | (haven't given it much thought yet) | 14:31 |
| tlater | juergbi: If you extend a long line of nodes from the graph and also use that as a target you end up with an order that misses some parallelization. | 14:33 |
| tlater | Kahn's algorithm builds this a little differently so all elements that can be built simultaneously end up next to each other. | 14:34 |
| juergbi | not sure what you mean with a long line of nodes. it would just be one extra level independent of the number of target nodes | 14:35 |
| juergbi | for optimal parallelization we'd anyway need more dynamic scheduling | 14:35 |
| juergbi | i don't know how significant the differences between the approaches would be in practice | 14:36 |
| juergbi | to me it seems you're doing two things at once | 14:36 |
| juergbi | adding support for multiple targets | 14:36 |
| juergbi | and switching to a different algorithm that may allow more parallelization | 14:37 |
| juergbi | i don't think the two really need to be done at the same time | 14:37 |
| tlater | juergbi: Fair, although I think currently with only one target this one creates the correct ord | 14:37 |
| tlater | er anyway | 14:38 |
| juergbi | how is it different compared to creating a stack and adding all targets as dep? | 14:38 |
| tlater | s/this one/the old one/ | 14:38 |
| tlater | juergbi: Kahn's algorithm is a breadth first search, so you end up visiting all nodes at their deepest depth. | 14:39 |
| tlater | That also means that sibling nodes are found together, as opposed to the depth search where you essentially build each target individually. | 14:40 |
| juergbi | doesn't the current depth sorting take care of this? | 14:41 |
| juergbi | we determine the maximum depth as well, just in a different manner | 14:41 |
| tlater | juergbi: Not with multiple elements, from some manual testing | 14:41 |
| juergbi | hm, siblings should definitely get the same depth value | 14:42 |
| * tlater tries this again | 14:42 | |
| juergbi | ah, is the issue that we return a flat list in the end? | 14:42 |
| juergbi | but we do have the depth values, so we could possible do something smarted with the result | 14:43 |
| juergbi | *smarter | 14:43 |
| tlater | juergbi: No, I don't think that's an issue, if the algorithm does assign the correct depth values with multiple targets | 14:44 |
| juergbi | an example may help if you think supporting multiple top-level targets with the current/old algorithm introduces an issue (functional or performance) | 14:47 |
| *** jude has quit IRC | 14:47 | |
| juergbi | if it's 'just' an optimization mostly independent of the multiple target support, i'd suggest to splitting the branch | 14:47 |
| juergbi | -to | 14:47 |
| *** paulsherwood has quit IRC | 14:48 | |
| *** waltervargas[m] has quit IRC | 14:48 | |
| *** benbrown has quit IRC | 14:48 | |
| *** laurenceurhegyi has quit IRC | 14:48 | |
| tlater | juergbi: Yeah, that's sensible. Also why I asked for review so early, didn't want to waste time on a fancy new algorithm if it's not necessary :) | 14:49 |
| *** ptomato[m] has quit IRC | 14:49 | |
| * tlater looks to have made a mistake while calculating this... | 14:52 | |
| tlater | Aaaand the old algorithm performs just fine. | 14:52 |
| tlater | Damnit, I knew I should have checked another time. | 14:53 |
| *** laurenceurhegyi has joined #buildstream | 14:59 | |
| *** paulsherwood has joined #buildstream | 15:03 | |
| *** benbrown has joined #buildstream | 15:03 | |
| *** llo has joined #buildstream | 15:15 | |
| llo | THOSE STUPID MUSLIM SAND NIGGERS HAVE KILLED INNOCENT AMERICANS AGAIN | 15:15 |
| llo | THE NAZI ORGANIZATION OF AMERICA IS PLANNING AN EMERGENCY MEETING | 15:15 |
| persia | Apologies for joining the discussion late, but there are a few cases where something is required to build, but not to run. Examples include a) optional bindings that would be dynamically loaded at runtime, if available (e.g. codec support), b) toolchain elements (e.g. compilers or image transcoders), c) source transformation utilities (e.g. yacc, sed, etc.) used by a buildsystem | 15:15 |
| llo | TODAY @ #/JOIN ON IRC.FREENODE.NET. DO NOT COMPLAIN IN #FREENODE | 15:15 |
| llo | THIS MEETING IS INTENDED TO BE FOR MORE LIKEMINDED INDIVIDUALS | 15:15 |
| llo | IF YOU HAVE QUESTIONS PLEASE DONT HESISTATE SENDING A MESSAGE TO | 15:15 |
| llo | VAP0R ON IRC.FREENODE.NET. | 15:15 |
| llo | benbrown paulsherwood laurenceurhegyi adds68 tiagogomes tristan givascu ssam2 tlater valentind juergbi kailueke[m] cs_shadow gitlab-br-bot brlogger anahuelamo mrmcq2u[m] cgmcintyre[m] mattiasb jjardon[m] ironfoot persia hergertme | 15:15 |
| *** llo has left #buildstream | 15:16 | |
| * persia repeats, to ease later log editing, if anyone is bothering | 15:16 | |
| persia | Apologies for joining the discussion late, but there are a few cases where something is required to build, but not to run. Examples include a) optional bindings that would be dynamically loaded at runtime, if available (e.g. codec support), b) toolchain elements (e.g. compilers or image transcoders), c) source transformation utilities (e.g. yacc, sed, etc.) used by a buildsystem | 15:16 |
| juergbi | persia: yes, build-only dependencies are supported | 15:18 |
| persia | Ah, good. I feared that might not be the case from "Note: in the referred docs - if you only build depend on something, that thing is assumed to be required to run " | 15:20 |
| tlater | persia: It's assumed to be required to run while building, i.e., it's runtime dependencies are required. | 15:21 |
| persia | Ah,. yes. That makes sense. | 15:21 |
| persia | Thank you both for resolving my confusion. | 15:22 |
| * tlater slaps himself for automatically adding ' to every "its" he types | 15:23 | |
| persia | There are a few things in Debian that build-depend on odd bits like XSL templates or RFC text, which don't actually "run" as such, but I presume the providers would have no runtime dependencies for those, so that assumption would have no ill side effects. | 15:23 |
| persia | tlater: It's one of the most common typos :) | 15:23 |
| juergbi | yes, the runtime terminology may not make sense for some elements but i don't think it's an issue in practice. it's also in line with other build systems, afaik | 15:25 |
| *** waltervargas[m] has joined #buildstream | 15:41 | |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 20 commits (last: _loader.py: Adjust the loader to support multiple targets) https://gitlab.com/BuildStream/buildstream/commit/1027a8e3c8f3df712b2e8682a0e92dda91417234 | 15:42 |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 1 commit (last: load.py: Add tests for loading multiple targets) https://gitlab.com/BuildStream/buildstream/commit/34fb37af3ed848d4a007fe8ac47ea280c23b8e7a | 15:51 |
| *** ptomato[m] has joined #buildstream | 15:54 | |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 1 commit (last: load.py: Add tests for loading multiple targets) https://gitlab.com/BuildStream/buildstream/commit/85b93c73d4d915ccc215455bc23e4c1c954bd32a | 15:59 |
| gitlab-br-bot | buildstream: Tristan Maat created branch tracking-changes | 16:00 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 16:00 |
| tlater | So right now all commands except `bst shell|source-bundle|checkout|workspace *` support multiple targets. I think it makes little sense with the first three, but `bst workspace close|reset` could have multiple targets. Should that also be a goal? | 16:08 |
| tlater | Actually, thinking about it, the other three might also want this, so that I can have a result that contains multiple elements that aren't related through dependencies... | 16:10 |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 20 commits (last: _pipeline.py: Adjust to new loader API) https://gitlab.com/BuildStream/buildstream/commit/601e4c3c5591f3abf9a0de147b38310c3ed5aeb8 | 17:19 |
| tristan | tlater, nah I dont think so | 17:38 |
| tristan | tlater, its gonna be weird I think for checkout, nonsensical for workspace or shell | 17:39 |
| tristan | source-bundle, maybe ? | 17:39 |
| tlater | tristan: Same use case as build, I suppose | 17:39 |
| tristan | yeah source-bundle is similar, if it's not problematic to do | 17:40 |
| tristan | if it causes even the most minor of side effect changes, I'd leave it alone and spend time on actually important things | 17:40 |
| tlater | tristan: It might be, because it turns out to be quite difficult to create a dummy element, which we need to stage a sandbox. | 17:40 |
| tristan | talking about different things are we ? | 17:41 |
| tlater | Ah, wait, we don't do that anymore | 17:41 |
| * tristan should stop ever using the word "it" in any conversation | 17:41 | |
| tlater | tristan: nvm, I misremembered how source-bundle worked. | 17:41 |
| tlater | tristan: Yeah, it looks possible for source-bundle | 17:42 |
| tristan | source-bundle might require refactoring of how the main script (is there a main script ?) works | 17:43 |
| tristan | in which case I'd just say screw it, seconds add up to minutes, those seconds would be wasted | 17:43 |
| tlater | tristan: That's fair. | 17:44 |
| tlater | tristan: In which case, I'll set up an MR for this soon, and move on to --except tomorrow. | 17:44 |
| tristan | it wouldnt be an API break to extend it to support multiple targets afaics, so not a blocker either | 17:44 |
| tristan | there I go again with the hand waving vague "it" word ;-) | 17:45 |
| tlater | tristan: Hard to go without 'it' ;P | 17:46 |
| * tlater blesses his linter for pointing out accidental pipeline.target references | 17:46 | |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 20 commits (last: _pipeline.py: Fix metaelement resolution) https://gitlab.com/BuildStream/buildstream/commit/44a91df2a2cc57b1f878edd5be416f092c2180e4 | 17:52 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 17:58 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:08 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:10 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:11 |
| gitlab-br-bot | buildstream: merge request (incremental-build->master: Add support for doing incremental builds) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:11 |
| *** tlater has quit IRC | 18:12 | |
| *** ssam2 has quit IRC | 18:13 | |
| *** tristan has quit IRC | 19:57 | |
| *** tristan has joined #buildstream | 19:58 | |
| jjardon[m] | hi! | 20:29 |
| jjardon[m] | does bst provides a tool to check if a project / bst file is "bst compliant" ? | 20:30 |
| tristan | jjardon[m], what is "bst compliant" ? | 20:31 |
| tristan | jjardon[m], if there is any problem with a project / element, running `bst show` on it will bail out with an error message indicating the filename, line, column and reason of the error | 20:32 |
| tristan | good enough ? | 20:32 |
| tristan | running `bst anything` for that matter, will do the same | 20:33 |
| jjardon[m] | tristan: a bst file that instead having "kind: tar" has kind: archive", when that plugin doesnt exist, for example. Yeah, that looks exactly what I need, cheers! | 20:36 |
| jjardon[m] | we simply want to check our conversion script does the correct thing, even if its not perfect | 20:38 |
| tristan | jjardon[m], yeah `bst show` is not perfect for validation but it almost is | 20:42 |
| tristan | it might not catch problems on untested sides of conditional statements | 20:43 |
| tristan | and it wont catch elements it doesnt load | 20:43 |
| tristan | jjardon[m], maybe a `bst validate` command could be a reasonable feature request, to validate every possible project option and bst file | 20:44 |
| tristan | but still, you are quite 'almost there' with just `bst show` | 20:44 |
| *** givascu has quit IRC | 21:10 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!