IRC logs for #buildstream for Friday, 2017-10-27

*** valentind has quit IRC00:22
*** tristan has joined #buildstream05:37
bochechafwiw, this will become unnecessary with the next pip release :) https://gitlab.com/BuildStream/buildstream/blob/master/integration-tests/pip-test/run-pip-test.sh#L58-6306:44
gitlab-br-botbuildstream: issue #134 ("Only push the objects that are not in the remote artifact share") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/13406:45
tristanbochecha, interesting, I have a feeling though that we only care about this for the test case, I think06:49
bochechatristan: yes06:50
tristanif we are somehow relying on that when using `pip install .` in a pip element, then I would better track that and think of complicated things :)06:50
bochechabut I (building apps and runtimes with buildstream's pip plugin) care very much about having my builds reproducible :P06:50
tristanYeah, is that file itself installed ?06:51
bochechait is06:51
bochechait's how `pip uninstall` works06:51
tristanI guess it would mean... it will just be more reproducible all by itself :)06:51
tristanAh right, that06:51
*** jonathanmaw has joined #buildstream07:53
gitlab-br-botbuildstream: issue #135 ("Expire artifacts in local cache") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/13507:54
gitlab-br-botbuildstream: issue #136 ("Expire artifacts on artifact share servers") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/13608:01
ironfoottristan: I haven't done any changes to the bot or configuration :/08:08
ironfootjust re-installed what the same bot/config we had in the past08:08
*** valentind has joined #buildstream08:21
*** jonathanmaw has quit IRC08:23
*** jude has joined #buildstream08:25
gitlab-br-botbuildstream: issue #137 ("buildstream hangs in operations with --except") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/13708:26
tristanironfoot, alright08:28
tristanironfoot, seems that in reality, when you do something once and intend to improve it later - later never happens08:29
* tristan already knows this lesson, but somehow overlooked it for this one ;-)08:29
ironfootheh, it might be better to find another bot with more feature08:30
ironfootI think this one doesn't support it,08:30
tristanironfoot, thats more work than just editing the JS to case out branch names, I think08:37
tristanAlso, I looked at like 5 or 10 similar bots and found this one to actually have the most features I wanted08:38
* ironfoot wonders if upstream has moved08:44
*** jude has quit IRC08:47
ironfootit has, and it looks like this is supported08:48
ironfootmaybe time to try to upstream my changes08:48
ironfootmultiple networks support08:48
*** tlater has joined #buildstream08:50
tristanironfoot, awesome sauce :)08:53
*** jonathanmaw has joined #buildstream08:54
*** jonathanmaw has quit IRC09:00
*** ssam2 has joined #buildstream09:00
*** adds68_ has joined #buildstream09:23
*** jonathanmaw has joined #buildstream09:25
*** adds68_ has quit IRC09:28
*** jonathanmaw has quit IRC09:58
tristantlater, so fwiw, I dont know where you are with the ticker thing; to be frank; the side effect of not ticking in that very specific case does not bother me as much as the many more serious bugs we have; *although*, the fact that in that specific case things get weird seems an indication of a deeper problem10:04
* tlater is debugging mostly for that reason10:04
tristantlater, I'm raising it because; it you feel you are far off from solving that, it would not be too bad to move on to other things and just document any findings you have in the bug10:04
tristanright, the ticker breaking is really not bothersome10:05
tlaterI'm more concerned about the SIGINT failing, but I suppose it's not top priority for now10:05
tristanyeah it seems to get blocked without getting unblocked, in very specific circumstances10:06
* tlater would like to finish up the tracking changes instead then10:07
tristanwell, that sounds like another can of worms, but a design one instead :)10:08
gitlab-br-botbuildstream: Jonathan Maw created branch 97-apply-pep-3102-to-all-public-api-surfaces10:08
gitlab-br-botbuildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12710:08
tlatertristan: Well, I think the current design covers all cases I can think of10:09
tristantlater, please keep https://gitlab.com/BuildStream/buildstream/issues/129 and https://gitlab.com/BuildStream/buildstream/issues/131 in mind10:09
tristanand note that --except is also already totally broken in other ways too10:09
tristane.g. found this today: https://gitlab.com/BuildStream/buildstream/issues/13710:09
* tlater wonders why that happens, it works with `bst build --track-except `10:10
tlaterJust to confirm, the hang happens after tracking finishes?10:10
tristanseems like scheduler waiting on something that's never gonna happen10:10
tristanyeah10:10
tristaneverything completes, status area stops updating with anything (left empty), and hang indefinitely10:11
tlaterAnother tracking-related scheduler bug? Sounds fun!10:11
tlaterx)10:11
tristanI feel like it's an --except related scheduler bug10:11
tristanbut *maybe* to do with tracking, I dunno for sure10:11
* tlater will have a look at #129 and #131 first 10:12
tristanseems like we removed some of the things the scheduler intends to do, without properly informing the scheduler that it wont have to do that - but, that's just how it *feels*10:12
tlaterI think my current implementation would make #129 simple anyway10:12
tristantlater, solving both of those first I feel like, would be very very good yes10:13
tristanyou can let me take care of https://gitlab.com/BuildStream/buildstream/issues/13310:13
tristanalso --except related10:14
gitlab-br-botpush on buildstream@97-apply-pep-3102-to-all-public-api-surfaces (by Jonathan Maw): 1 commit (last: Apply pep 3102 to Element.dependencies) https://gitlab.com/BuildStream/buildstream/commit/76da54f53c3bfcdd11887d53b6307efe581c600d10:14
tristanit's totally orthogonal10:14
* tristan has an idea how to improve on that completion code anyway10:14
tlaterAlright, that seems sensible. Ta :)10:14
*** valentind has quit IRC10:15
tristancool, just dont forget to dump anything you know about what you were working on, in the bug report - before context switching that brain of yours :D10:15
tristantlater, ^^10:15
* tlater did that while finishing up yesterday, unfortunately I ended up just chasing a red herring10:16
tristanif you catch them, and pickle them, I hear they are delicious :)10:16
tlaterI should try that next time ;P10:17
*** adds68_ has joined #buildstream10:22
gitlab-br-botbuildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12710:23
*** adds68_ has quit IRC10:26
*** adds68 has joined #buildstream10:27
tristantlater, so I guess you have to change the _loader.py code to parse a list of toplevel targets and return a list of toplevels in the same order, right ?10:27
tristantlater, ensuring that you always get one-instance-per-element, but still have your toplevels in place for traversing and creating build pipelines with ?10:28
*** tiagogomes_ has joined #buildstream10:30
tlatertristan: I'm not sure what you mean - can this not be solved in the pipeline? I'm taking notes on #128 first, realized I didn't mention everything I found yet.10:30
*** tiagogomes has quit IRC10:32
tristantlater, not only no10:40
tristantlater, if you need to load multiple targets, they will have intersecting elements10:41
tristandependencies10:41
* tlater is almost up to speed on this now, that seems like the way to go10:41
tristantlater, same with loading the --except elements to find the intersections10:41
tlaterI used a set() in my code perviously to avoid loading multiple of the same element10:41
tristantlater, so two problems arise, you get separate MetaElements to sort through messily in _pipeline.py, YUCK10:41
tlaterBut doing this in the loader seems better10:41
tristantlater, and also, you parse most of the pipeline various times - SLOOOOW10:42
tlatertristan: I suppose a workaround would be to create a dummy toplevel element that 'depends' on all toplevel elements?10:50
tlaterAlthough I'm not sure that would end up very neat for the frontend.10:51
tristantlater, nah10:53
tristantlater, also it doesnt solve that you need TWO anyway10:54
tristanbecause you must differentiate between excepts and targets10:54
* tlater feels like there is no solution besides a fairly complex tree traversal, and that only solves things being slow11:00
tristantlater, fairly complex tree traversals is what we do best :)11:02
tlaterWel, I suppose this is where that CS degree pays off11:04
tristantlater, so I expect the first thing to do is to support multiple targets before the except thing11:04
tristanThat part, you should only need to modify the Planner part of _pipeline.py11:04
tristanso that it does it's thing on multiple toplevels instead of just the one11:04
tristanWell, that and making the _loader.py load multiple filenames and return multiple targets in the same order, in one go11:05
tristanalthough the _loader.py part is superduper trivial11:05
tristanor maybe not completely trivial, you might need some internal virtual target in order to compute the circular deps efficiently11:06
tlaterIt's fairly trivial relative to the other issues11:06
tristanbaby steps, that one helps unblock the others11:08
tristanthe except thing is finding an intersection and applying a rule11:08
tlaterAfter those two steps I can worry about --except then :)11:08
tristanknowing what that rule is, is half of the work11:08
tristanfinding the intersections should not be hard, and might be worth adding a feature for in Element API along side the .search() and .dependencies() APIs11:09
tristanthen again, doing it efficiently, might be tricky - but not variants tricky, just tricky :D11:09
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12611:11
tristanHmmm, there may be some tricky stuff there; lets not get into duplicating elements though11:11
tristanin the sense that, all we want from this is a list, which we can later sort through the Planner() thing11:12
tristanSo subtracting one target from another target, does not produce a new tree of elements; just a result set11:12
tristananyway, seems to me that this is the fun stuff11:13
tlatertristan: Isn't the dependency ordering important?11:13
tristanYes11:13
tristantlater, well actually it's not in real life, I think11:13
tristantlater, because of the nature of the scheduler11:14
tristanBut, you basically still need the Planner to do it's thing on the *result set*11:14
tlatertristan: Ah, right, it sorts out dependencies independently11:14
tristantlater, alternatively, you can pass in the result set as additional data to the planner, and let it traverse the tree but ignore parts which are not in the real set11:14
tristanright, dependency ordering is important for two things11:15
tristanthe Planner does depth sort for build optimization11:15
tristanThe Element sorts it's immediate dependencies independently, in order to produce a correct and stable staging order (and iteration order all around)11:15
tristanI once asked juergbi if the way we sort dependencies, and the code which iterates on Element.dependencies(), could in some way contribute to, or replace the Planner, efficiently11:16
tristanBut I think the answer was no11:16
tristanThe planner may also by now be considering other things, like non-strict build mode, and whether an artifact that is a build-only dependencies is downloadable11:16
tristanso I suppose it's a beast of it's own by now anyway11:17
tlaterYeah, probably little point in trying to get a tree from subtracting two elements11:17
tlaterAnyway, that's probably very future detail - I'll be busy with this one for a while, I'm sure :)11:18
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12611:21
tristanI think the user configuration for 'strict' should be removed from the toplevel, and added as a project specific option11:25
tristanjuergbi, ^^^^ thoughts ?11:25
tristanI'm considering recommending for GNOME developers to set that to False in their user config, but specifically for projects: -> gnome: -> strict: False11:26
ssam2hmmm that's an interesting idea11:28
ssam2i like it11:28
* tristan tries to quickly nail that11:29
gitlab-br-botbuildstream: Sam Thursfield deleted branch sam/artifactcache-preflight-check11:31
gitlab-br-botbuildstream: Sam Thursfield deleted branch sam/ci-update-docker-image11:31
gitlab-br-botbuildstream: Sam Thursfield deleted branch sam/fix-docs11:31
gitlab-br-botbuildstream: Sam Thursfield deleted branch sam/no-install-python-deps11:31
gitlab-br-botbuildstream: Sam Thursfield deleted branch sam/use-via-docker11:31
* ssam2 seems to have found a bug in conditionals: https://pastebin.com/V1SUvAL311:35
ssam2going to go for lunch, but will try and fix it this afternoon11:35
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12611:40
tristanssam2, yeah, gotta catch that exception11:40
tristanssam2, fwiw, that is not supported; conditionals only work on dictionaries, not lists11:40
tristanssam2, please include a test case, maybe tests/yaml is the right place, otherwise tests/format11:41
*** jonathanmaw has joined #buildstream11:48
*** jonathanmaw has quit IRC11:55
*** adds68 has quit IRC11:56
*** jonathanmaw has joined #buildstream12:07
*** jonathanmaw has quit IRC12:17
*** jonathanmaw has joined #buildstream12:18
*** jonathanmaw has quit IRC12:19
*** jonathanmaw has joined #buildstream12:19
gitlab-br-botpush on buildstream@master (by Tristan Van Berkom): 4 commits (last: context.py: Moving strict mode configuration to be project specific) https://gitlab.com/BuildStream/buildstream/commit/cf1a2d326af47e0cdf49cd99d1e4447f494de23112:27
*** tristan has quit IRC13:03
*** adds68 has joined #buildstream13:30
*** xjuan has joined #buildstream13:54
*** adds68 has quit IRC14:29
*** semanticdesign has joined #buildstream14:33
*** semanticdesign_ has joined #buildstream14:33
*** adds68 has joined #buildstream14:35
*** valentind has joined #buildstream14:47
*** bochecha has quit IRC15:00
*** bochecha has joined #buildstream15:04
*** jonathanmaw has joined #buildstream15:07
*** bochecha has quit IRC15:10
*** adds68 has quit IRC15:14
*** ssam2 has quit IRC15:15
*** ssam2 has joined #buildstream15:16
*** bochecha has joined #buildstream15:17
*** jonathanmaw has quit IRC15:18
*** bochecha has quit IRC15:21
*** adds68 has joined #buildstream15:25
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 2 commits (last: tests/yaml: Rename 'invalid' file to make way for more) https://gitlab.com/BuildStream/buildstream/commit/6a22ee9b2de8b044c8c7902be1ceee1fcb33a9d515:49
*** adds68 has quit IRC15:56
tlaterHm, I think I need a count of incoming dependencies to solve the dependency resolution for multiple targets properly.15:57
tlaterI could traverse the graph while computing that, but that is silly and inefficient - I also don't want to add an 'incoming' property to metaelements though.15:58
*** adds68 has joined #buildstream15:59
*** tiagogomes_ has quit IRC16:01
tlaterAww, stackoverflow has an answer, so boring :(16:02
tlaterIt actually doesn't, they choose the inefficient "walk through the entire graph first" solution16:05
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to composite a list) https://gitlab.com/BuildStream/buildstream/commit/f22998f418dd268444f3f1fc13dbe323ea69917c16:19
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to composite a list) https://gitlab.com/BuildStream/buildstream/commit/a27626b247f0629622a4bebefa31778a522145d516:22
*** adds68 has quit IRC16:25
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/634bc0ea72ab5ee1905d4dc292d5bc709ad4d9c816:27
*** semanticdesign_ has quit IRC16:29
*** semanticdesign has quit IRC16:29
gitlab-br-botbuildstream: merge request (list_relative_paths_opt->master: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12816:36
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/318c5ce36a9cdc451d685f3c1a3f1f5f62deedf016:40
gitlab-br-botbuildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12916:41
gitlab-br-botpush on buildstream@sam/compose-list-exception (by Sam Thursfield): 1 commit (last: Catch attempts to compose a list) https://gitlab.com/BuildStream/buildstream/commit/0dac630fe6a852016d4b79e8bee6af4b70e7e0f716:50
gitlab-br-botbuildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12916:50
gitlab-br-botbuildstream: merge request (sam/compose-list-exception->master: Catch attempts to compose a list) #129 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12916:53
*** ssam2 has quit IRC16:56
*** tristan has joined #buildstream17:00
gitlab-br-botbuildstream: issue #138 ("aborting bst push command causes stack trace") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/13817:15
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12617:57
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12618:04
gitlab-br-botpush on buildstream@tracking-changes (by Tristan Maat): 15 commits (last: _pipeline.py: Improve error message when --except fails.) https://gitlab.com/BuildStream/buildstream/commit/640e7e57842ca4c2ec3a8641258f54ff7c42d3c018:11
gitlab-br-botbuildstream: merge request (tracking-changes->master: WIP: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/11918:12
*** tlater has quit IRC18:12
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12618:21
cs_shadowtristan: I still need to add tests to https://gitlab.com/BuildStream/buildstream/merge_requests/126 but in case you have some time, it'll be nice to get some initial feedback to see if I'm headed in the right direction18:23
tristancs_shadow, ok ! I'll keep a tab open on it and check it tomorrow18:25
tristanseeing as it's 3:30am :)18:25
* tristan wonders what he's doing home so early on a friday night18:26
cs_shadowthanks!18:26
tristancs_shadow, I couldnt help myself18:27
tristancs_shadow, first impression... why would we want that to be optional ?18:27
tristanI think, just make the default better, and dont double our potential bugs attack surface18:28
cs_shadowI think it's just a question of how the workspaces should behave like. I thought some users might want to keep their workspaces "clean"18:29
cs_shadowbut if you think that this default is good, i'd be happy to change that18:29
tristanyeah better just make that default18:29
tristanwhen you build from a checkout, you dirty your tree, without buildstream at all18:29
tristanseems expected to me, we're just fixing it to work properly18:30
cs_shadowsure thing. by default, do you mean making the option True by default or removing the option altogether?18:30
tristan(plus, if you give the user a choice, then you're stuck supporting both options... forever)18:30
tristancs_shadow, I mean removing the choice18:30
cs_shadowokay, makes sense18:30
cs_shadowthat also makes my life easier as i don't have to think about what to call the option. (it was changed from dirty -> incremental -> mount-workspaces)18:32
tristanplus, if the objective is cleanliness (purpose of the option), then the correct thing is to add support for builddir != sourcedir across the board (at least for build systems which support that)18:33
tristanbut always as a subdir of the builddir18:33
tristanI think that's desirable anyway; for the type of modules who just want to run CI and ensure that their autotools-foo is working correctly with builddir != srcdir18:34
cs_shadowfair point18:35
gitlab-br-botbuildstream: merge request (incremental-build->master: WIP: Incremental build) #126 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12618:48
cs_shadowupdated18:49
cs_shadowdoh! I broke the tests, let me fix those18:52
gitlab-br-botbuildstream: merge request (list_relative_paths_opt->master: WIP: utils.list_relative_paths: avoid os.path.relpath) #128 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/12820:15
*** xjuan has quit IRC20:35
*** valentind has quit IRC23:25

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!