*** valentind has quit IRC | 00:22 | |
*** tristan has joined #buildstream | 05:37 | |
bochecha | fwiw, this will become unnecessary with the next pip release :) https://gitlab.com/BuildStream/buildstream/blob/master/integration-tests/pip-test/run-pip-test.sh#L58-63 | 06:44 |
---|---|---|
gitlab-br-bot | buildstream: issue #134 ("Only push the objects that are not in the remote artifact share") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/134 | 06:45 |
tristan | bochecha, interesting, I have a feeling though that we only care about this for the test case, I think | 06:49 |
bochecha | tristan: yes | 06:50 |
tristan | if we are somehow relying on that when using `pip install .` in a pip element, then I would better track that and think of complicated things :) | 06:50 |
bochecha | but I (building apps and runtimes with buildstream's pip plugin) care very much about having my builds reproducible :P | 06:50 |
tristan | Yeah, is that file itself installed ? | 06:51 |
bochecha | it is | 06:51 |
bochecha | it's how `pip uninstall` works | 06:51 |
tristan | I guess it would mean... it will just be more reproducible all by itself :) | 06:51 |
tristan | Ah right, that | 06:51 |
*** jonathanmaw has joined #buildstream | 07:53 | |
gitlab-br-bot | buildstream: issue #135 ("Expire artifacts in local cache") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/135 | 07:54 |
gitlab-br-bot | buildstream: issue #136 ("Expire artifacts on artifact share servers") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/136 | 08:01 |
ironfoot | tristan: I haven't done any changes to the bot or configuration :/ | 08:08 |
ironfoot | just re-installed what the same bot/config we had in the past | 08:08 |
*** valentind has joined #buildstream | 08:21 | |
*** jonathanmaw has quit IRC | 08:23 | |
*** jude has joined #buildstream | 08:25 | |
gitlab-br-bot | buildstream: issue #137 ("buildstream hangs in operations with --except") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/137 | 08:26 |
tristan | ironfoot, alright | 08:28 |
tristan | ironfoot, seems that in reality, when you do something once and intend to improve it later - later never happens | 08:29 |
* tristan already knows this lesson, but somehow overlooked it for this one ;-) | 08:29 | |
ironfoot | heh, it might be better to find another bot with more feature | 08:30 |
ironfoot | I think this one doesn't support it, | 08:30 |
tristan | ironfoot, thats more work than just editing the JS to case out branch names, I think | 08:37 |
tristan | Also, I looked at like 5 or 10 similar bots and found this one to actually have the most features I wanted | 08:38 |
* ironfoot wonders if upstream has moved | 08:44 | |
*** jude has quit IRC | 08:47 | |
ironfoot | it has, and it looks like this is supported | 08:48 |
ironfoot | maybe time to try to upstream my changes | 08:48 |
ironfoot | multiple networks support | 08:48 |
*** tlater has joined #buildstream | 08:50 | |
tristan | ironfoot, awesome sauce :) | 08:53 |
*** jonathanmaw has joined #buildstream | 08:54 | |
*** jonathanmaw has quit IRC | 09:00 | |
*** ssam2 has joined #buildstream | 09:00 | |
*** adds68_ has joined #buildstream | 09:23 | |
*** jonathanmaw has joined #buildstream | 09:25 | |
*** adds68_ has quit IRC | 09:28 | |
*** jonathanmaw has quit IRC | 09:58 | |
tristan | tlater, so fwiw, I dont know where you are with the ticker thing; to be frank; the side effect of not ticking in that very specific case does not bother me as much as the many more serious bugs we have; *although*, the fact that in that specific case things get weird seems an indication of a deeper problem | 10:04 |
* tlater is debugging mostly for that reason | 10:04 | |
tristan | tlater, I'm raising it because; it you feel you are far off from solving that, it would not be too bad to move on to other things and just document any findings you have in the bug | 10:04 |
tristan | right, the ticker breaking is really not bothersome | 10:05 |
tlater | I'm more concerned about the SIGINT failing, but I suppose it's not top priority for now | 10:05 |
tristan | yeah it seems to get blocked without getting unblocked, in very specific circumstances | 10:06 |
* tlater would like to finish up the tracking changes instead then | 10:07 | |
tristan | well, that sounds like another can of worms, but a design one instead :) | 10:08 |
gitlab-br-bot | buildstream: Jonathan Maw created branch 97-apply-pep-3102-to-all-public-api-surfaces | 10:08 |
gitlab-br-bot | buildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/127 | 10:08 |
tlater | tristan: Well, I think the current design covers all cases I can think of | 10:09 |
tristan | tlater, please keep https://gitlab.com/BuildStream/buildstream/issues/129 and https://gitlab.com/BuildStream/buildstream/issues/131 in mind | 10:09 |
tristan | and note that --except is also already totally broken in other ways too | 10:09 |
tristan | e.g. found this today: https://gitlab.com/BuildStream/buildstream/issues/137 | 10:09 |
* tlater wonders why that happens, it works with `bst build --track-except ` | 10:10 | |
tlater | Just to confirm, the hang happens after tracking finishes? | 10:10 |
tristan | seems like scheduler waiting on something that's never gonna happen | 10:10 |
tristan | yeah | 10:10 |
tristan | everything completes, status area stops updating with anything (left empty), and hang indefinitely | 10:11 |
tlater | Another tracking-related scheduler bug? Sounds fun! | 10:11 |
tlater | x) | 10:11 |
tristan | I feel like it's an --except related scheduler bug | 10:11 |
tristan | but *maybe* to do with tracking, I dunno for sure | 10:11 |
* tlater will have a look at #129 and #131 first | 10:12 | |
tristan | seems like we removed some of the things the scheduler intends to do, without properly informing the scheduler that it wont have to do that - but, that's just how it *feels* | 10:12 |
tlater | I think my current implementation would make #129 simple anyway | 10:12 |
tristan | tlater, solving both of those first I feel like, would be very very good yes | 10:13 |
tristan | you can let me take care of https://gitlab.com/BuildStream/buildstream/issues/133 | 10:13 |
tristan | also --except related | 10:14 |
gitlab-br-bot | push on buildstream@97-apply-pep-3102-to-all-public-api-surfaces (by Jonathan Maw): 1 commit (last: Apply pep 3102 to Element.dependencies) https://gitlab.com/BuildStream/buildstream/commit/76da54f53c3bfcdd11887d53b6307efe581c600d | 10:14 |
tristan | it's totally orthogonal | 10:14 |
* tristan has an idea how to improve on that completion code anyway | 10:14 | |
tlater | Alright, that seems sensible. Ta :) | 10:14 |
*** valentind has quit IRC | 10:15 | |
tristan | cool, just dont forget to dump anything you know about what you were working on, in the bug report - before context switching that brain of yours :D | 10:15 |
tristan | tlater, ^^ | 10:15 |
* tlater did that while finishing up yesterday, unfortunately I ended up just chasing a red herring | 10:16 | |
tristan | if you catch them, and pickle them, I hear they are delicious :) | 10:16 |
tlater | I should try that next time ;P | 10:17 |
*** adds68_ has joined #buildstream | 10:22 | |
gitlab-br-bot | buildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/127 | 10:23 |
*** adds68_ has quit IRC | 10:26 | |
*** adds68 has joined #buildstream | 10:27 | |
tristan | tlater, so I guess you have to change the _loader.py code to parse a list of toplevel targets and return a list of toplevels in the same order, right ? | 10:27 |
tristan | tlater, ensuring that you always get one-instance-per-element, but still have your toplevels in place for traversing and creating build pipelines with ? | 10:28 |
*** tiagogomes_ has joined #buildstream | 10:30 | |
tlater | tristan: I'm not sure what you mean - can this not be solved in the pipeline? I'm taking notes on #128 first, realized I didn't mention everything I found yet. | 10:30 |
*** tiagogomes has quit IRC | 10:32 | |
tristan | tlater, not only no | 10:40 |
tristan | tlater, if you need to load multiple targets, they will have intersecting elements | 10:41 |
tristan | dependencies | 10:41 |
* tlater is almost up to speed on this now, that seems like the way to go | 10:41 | |
tristan | tlater, same with loading the --except elements to find the intersections | 10:41 |
tlater | I used a set() in my code perviously to avoid loading multiple of the same element | 10:41 |
tristan | tlater, so two problems arise, you get separate MetaElements to sort through messily in _pipeline.py, YUCK | 10:41 |
tlater | But doing this in the loader seems better | 10:41 |
tristan | tlater, and also, you parse most of the pipeline various times - SLOOOOW | 10:42 |
tlater | tristan: I suppose a workaround would be to create a dummy toplevel element that 'depends' on all toplevel elements? | 10:50 |
tlater | Although I'm not sure that would end up very neat for the frontend. | 10:51 |
tristan | tlater, nah | 10:53 |
tristan | tlater, also it doesnt solve that you need TWO anyway | 10:54 |
tristan | because you must differentiate between excepts and targets | 10:54 |
* tlater feels like there is no solution besides a fairly complex tree traversal, and that only solves things being slow | 11:00 | |
tristan | tlater, fairly complex tree traversals is what we do best :) | 11:02 |
tlater | Wel, I suppose this is where that CS degree pays off | 11:04 |
tristan | tlater, so I expect the first thing to do is to support multiple targets before the except thing | 11:04 |
tristan | That part, you should only need to modify the Planner part of _pipeline.py | 11:04 |
tristan | so that it does it's thing on multiple toplevels instead of just the one | 11:04 |
tristan | Well, that and making the _loader.py load multiple filenames and return multiple targets in the same order, in one go | 11:05 |
tristan | although the _loader.py part is superduper trivial | 11:05 |
tristan | or maybe not completely trivial, you might need some internal virtual target in order to compute the circular deps efficiently | 11:06 |
tlater | It's fairly trivial relative to the other issues | 11:06 |
tristan | baby steps, that one helps unblock the others | 11:08 |
tristan | the except thing is finding an intersection and applying a rule | 11:08 |
tlater | After those two steps I can worry about --except then :) | 11:08 |
tristan | knowing what that rule is, is half of the work | 11:08 |
tristan | finding the intersections should not be hard, and might be worth adding a feature for in Element API along side the .search() and .dependencies() APIs | 11:09 |
tristan | then again, doing it efficiently, might be tricky - but not variants tricky, just tricky :D | 11:09 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 11:11 |
tristan | Hmmm, there may be some tricky stuff there; lets not get into duplicating elements though | 11:11 |
tristan | in the sense that, all we want from this is a list, which we can later sort through the Planner() thing | 11:12 |
tristan | So subtracting one target from another target, does not produce a new tree of elements; just a result set | 11:12 |
tristan | anyway, seems to me that this is the fun stuff | 11:13 |
tlater | tristan: Isn't the dependency ordering important? | 11:13 |
tristan | Yes | 11:13 |
tristan | tlater, well actually it's not in real life, I think | 11:13 |
tristan | tlater, because of the nature of the scheduler | 11:14 |
tristan | But, you basically still need the Planner to do it's thing on the *result set* | 11:14 |
tlater | tristan: Ah, right, it sorts out dependencies independently | 11:14 |
tristan | tlater, alternatively, you can pass in the result set as additional data to the planner, and let it traverse the tree but ignore parts which are not in the real set | 11:14 |
tristan | right, dependency ordering is important for two things | 11:15 |
tristan | the Planner does depth sort for build optimization | 11:15 |
tristan | The Element sorts it's immediate dependencies independently, in order to produce a correct and stable staging order (and iteration order all around) | 11:15 |
tristan | I once asked juergbi if the way we sort dependencies, and the code which iterates on Element.dependencies(), could in some way contribute to, or replace the Planner, efficiently | 11:16 |
tristan | But I think the answer was no | 11:16 |
tristan | The planner may also by now be considering other things, like non-strict build mode, and whether an artifact that is a build-only dependencies is downloadable | 11:16 |
tristan | so I suppose it's a beast of it's own by now anyway | 11:17 |
tlater | Yeah, probably little point in trying to get a tree from subtracting two elements | 11:17 |
tlater | Anyway, that's probably very future detail - I'll be busy with this one for a while, I'm sure :) | 11:18 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 11:21 |
tristan | I think the user configuration for 'strict' should be removed from the toplevel, and added as a project specific option | 11:25 |
tristan | juergbi, ^^^^ thoughts ? | 11:25 |
tristan | I'm considering recommending for GNOME developers to set that to False in their user config, but specifically for projects: -> gnome: -> strict: False | 11:26 |
ssam2 | hmmm that's an interesting idea | 11:28 |
ssam2 | i like it | 11:28 |
* tristan tries to quickly nail that | 11:29 | |
gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/artifactcache-preflight-check | 11:31 |
gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/ci-update-docker-image | 11:31 |
gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/fix-docs | 11:31 |
gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/no-install-python-deps | 11:31 |
gitlab-br-bot | buildstream: Sam Thursfield deleted branch sam/use-via-docker | 11:31 |
* ssam2 seems to have found a bug in conditionals: https://pastebin.com/V1SUvAL3 | 11:35 | |
ssam2 | going to go for lunch, but will try and fix it this afternoon | 11:35 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 11:40 |
tristan | ssam2, yeah, gotta catch that exception | 11:40 |
tristan | ssam2, fwiw, that is not supported; conditionals only work on dictionaries, not lists | 11:40 |
tristan | ssam2, please include a test case, maybe tests/yaml is the right place, otherwise tests/format | 11:41 |
*** jonathanmaw has joined #buildstream | 11:48 | |
*** jonathanmaw has quit IRC | 11:55 | |
*** adds68 has quit IRC | 11:56 | |
*** jonathanmaw has joined #buildstream | 12:07 | |
*** jonathanmaw has quit IRC | 12:17 | |
*** jonathanmaw has joined #buildstream | 12:18 | |
*** jonathanmaw has quit IRC | 12:19 | |
*** jonathanmaw has joined #buildstream | 12:19 | |
gitlab-br-bot | push on buildstream@master (by Tristan Van Berkom): 4 commits (last: context.py: Moving strict mode configuration to be project specific) https://gitlab.com/BuildStream/buildstream/commit/cf1a2d326af47e0cdf49cd99d1e4447f494de231 | 12:27 |
*** tristan has quit IRC | 13:03 | |
*** adds68 has joined #buildstream | 13:30 | |
*** xjuan has joined #buildstream | 13:54 | |
*** adds68 has quit IRC | 14:29 | |
*** semanticdesign has joined #buildstream | 14:33 | |
*** semanticdesign_ has joined #buildstream | 14:33 | |
*** adds68 has joined #buildstream | 14:35 | |
*** valentind has joined #buildstream | 14:47 | |
*** bochecha has quit IRC | 15:00 | |
*** bochecha has joined #buildstream | 15:04 | |
*** jonathanmaw has joined #buildstream | 15:07 | |
*** bochecha has quit IRC | 15:10 | |
*** adds68 has quit IRC | 15:14 | |
*** ssam2 has quit IRC | 15:15 | |
*** ssam2 has joined #buildstream | 15:16 | |
*** bochecha has joined #buildstream | 15:17 | |
*** jonathanmaw has quit IRC | 15:18 | |
*** bochecha has quit IRC | 15:21 | |
*** adds68 has joined #buildstream | 15:25 | |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 2 commits (last: tests/yaml: Rename 'invalid' file to make way for more) https://gitlab.com/BuildStream/buildstream/commit/6a22ee9b2de8b044c8c7902be1ceee1fcb33a9d5 | 15:49 |
*** adds68 has quit IRC | 15:56 | |
tlater | Hm, I think I need a count of incoming dependencies to solve the dependency resolution for multiple targets properly. | 15:57 |
tlater | I could traverse the graph while computing that, but that is silly and inefficient - I also don't want to add an 'incoming' property to metaelements though. | 15:58 |
*** adds68 has joined #buildstream | 15:59 | |
*** tiagogomes_ has quit IRC | 16:01 | |
tlater | Aww, stackoverflow has an answer, so boring :( | 16:02 |
tlater | It actually doesn't, they choose the inefficient "walk through the entire graph first" solution | 16:05 |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to composite a list) https://gitlab.com/BuildStream/buildstream/commit/f22998f418dd268444f3f1fc13dbe323ea69917c | 16:19 |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to composite a list) https://gitlab.com/BuildStream/buildstream/commit/a27626b247f0629622a4bebefa31778a522145d5 | 16:22 |
*** adds68 has quit IRC | 16:25 | |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/634bc0ea72ab5ee1905d4dc292d5bc709ad4d9c8 | 16:27 |
*** semanticdesign_ has quit IRC | 16:29 | |
*** semanticdesign has quit IRC | 16:29 | |
gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 16:36 |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/318c5ce36a9cdc451d685f3c1a3f1f5f62deedf0 | 16:40 |
gitlab-br-bot | buildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/129 | 16:41 |
gitlab-br-bot | push on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/0dac630fe6a852016d4b79e8bee6af4b70e7e0f7 | 16:50 |
gitlab-br-bot | buildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/129 | 16:50 |
gitlab-br-bot | buildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/129 | 16:53 |
*** ssam2 has quit IRC | 16:56 | |
*** tristan has joined #buildstream | 17:00 | |
gitlab-br-bot | buildstream: issue #138 ("aborting bst push command causes stack trace") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/138 | 17:15 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 17:57 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:04 |
gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 15 commits (last: _pipeline.py: Improve error message when --except fails.) https://gitlab.com/BuildStream/buildstream/commit/640e7e57842ca4c2ec3a8641258f54ff7c42d3c0 | 18:11 |
gitlab-br-bot | buildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 18:12 |
*** tlater has quit IRC | 18:12 | |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:21 |
cs_shadow | tristan: I still need to add tests to https://gitlab.com/BuildStream/buildstream/merge_requests/126 but in case you have some time, it'll be nice to get some initial feedback to see if I'm headed in the right direction | 18:23 |
tristan | cs_shadow, ok ! I'll keep a tab open on it and check it tomorrow | 18:25 |
tristan | seeing as it's 3:30am :) | 18:25 |
* tristan wonders what he's doing home so early on a friday night | 18:26 | |
cs_shadow | thanks! | 18:26 |
tristan | cs_shadow, I couldnt help myself | 18:27 |
tristan | cs_shadow, first impression... why would we want that to be optional ? | 18:27 |
tristan | I think, just make the default better, and dont double our potential bugs attack surface | 18:28 |
cs_shadow | I think it's just a question of how the workspaces should behave like. I thought some users might want to keep their workspaces "clean" | 18:29 |
cs_shadow | but if you think that this default is good, i'd be happy to change that | 18:29 |
tristan | yeah better just make that default | 18:29 |
tristan | when you build from a checkout, you dirty your tree, without buildstream at all | 18:29 |
tristan | seems expected to me, we're just fixing it to work properly | 18:30 |
cs_shadow | sure thing. by default, do you mean making the option True by default or removing the option altogether? | 18:30 |
tristan | (plus, if you give the user a choice, then you're stuck supporting both options... forever) | 18:30 |
tristan | cs_shadow, I mean removing the choice | 18:30 |
cs_shadow | okay, makes sense | 18:30 |
cs_shadow | that also makes my life easier as i don't have to think about what to call the option. (it was changed from dirty -> incremental -> mount-workspaces) | 18:32 |
tristan | plus, if the objective is cleanliness (purpose of the option), then the correct thing is to add support for builddir != sourcedir across the board (at least for build systems which support that) | 18:33 |
tristan | but always as a subdir of the builddir | 18:33 |
tristan | I think that's desirable anyway; for the type of modules who just want to run CI and ensure that their autotools-foo is working correctly with builddir != srcdir | 18:34 |
cs_shadow | fair point | 18:35 |
gitlab-br-bot | buildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/126 | 18:48 |
cs_shadow | updated | 18:49 |
cs_shadow | doh! I broke the tests, let me fix those | 18:52 |
gitlab-br-bot | buildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/128 | 20:15 |
*** xjuan has quit IRC | 20:35 | |
*** valentind has quit IRC | 23:25 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!