| gitlab-br-bot | jjardon merged MR !1218 (jjardon/bst-external-table->master: README.rst: Add table with package status of bst-external) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1218 | 00:14 |
|---|---|---|
| *** nimish has joined #buildstream | 01:31 | |
| *** nimish has quit IRC | 03:24 | |
| *** tpollard has joined #buildstream | 05:32 | |
| *** hergertme has quit IRC | 05:47 | |
| *** hergertme has joined #buildstream | 05:49 | |
| *** hergertme has joined #buildstream | 05:50 | |
| *** hergertme has joined #buildstream | 05:50 | |
| *** hergertme has joined #buildstream | 05:51 | |
| *** tpollard has quit IRC | 06:49 | |
| *** tpollard has joined #buildstream | 07:19 | |
| *** tpollard has quit IRC | 07:33 | |
| *** alatiera has joined #buildstream | 08:18 | |
| *** toscalix has joined #buildstream | 08:45 | |
| *** toscalix has quit IRC | 08:46 | |
| *** toscalix has joined #buildstream | 08:46 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1226 (aevri/tmpdir_for_cache_size->master: cascache: atomically save size via tmpdir instead) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1226 | 08:54 |
| *** tpollard has joined #buildstream | 09:14 | |
| *** rdale has joined #buildstream | 09:23 | |
| *** raoul has joined #buildstream | 09:40 | |
| gitlab-br-bot | juergbi approved MR !1226 (aevri/tmpdir_for_cache_size->master: cascache: atomically save size via tmpdir instead) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1226 | 09:46 |
| *** tpollard has quit IRC | 09:50 | |
| *** jonathanmaw has joined #buildstream | 10:08 | |
| Kinnison | For those who care... the state of the yaml rework is: | 10:10 |
| Kinnison | 208 failed, 981 passed, 126 skipped in 788.54 seconds | 10:10 |
| Kinnison | almost all of those failures are track related because that's the one remaining big algorithm to be rewritten | 10:10 |
| Kinnison | juergbi: unless it turns out to be a pig causing delays, please could you not cancel the latest pipeline on shared/yaml-rework since I'd like to see if the test results are comparable elsewhere. I expect it to be failing all over :D | 10:20 |
| juergbi | ok | 10:23 |
| *** lachlan has joined #buildstream | 10:38 | |
| *** toscalix has quit IRC | 10:48 | |
| *** toscalix has joined #buildstream | 10:50 | |
| *** toscalix has quit IRC | 10:55 | |
| *** toscalix has joined #buildstream | 10:55 | |
| juergbi | aevri: !1231 still has WIP status. remove it quickly, before Marge notices :-) | 11:05 |
| gitlab-br-bot | MR !1231: WIP: project: don't do _find_project_dir if we're a junction https://gitlab.com/BuildStream/buildstream/merge_requests/1231 | 11:05 |
| gitlab-br-bot | aevri opened (was WIP) MR !1231 (aevri/direct_load_junction_projects->master: project: don't do _find_project_dir if we're a junction) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1231 | 11:05 |
| aevri | Done, thanks! :) | 11:05 |
| gitlab-br-bot | marge-bot123 merged MR !1226 (aevri/tmpdir_for_cache_size->master: cascache: atomically save size via tmpdir instead) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1226 | 11:08 |
| *** alatiera has quit IRC | 11:15 | |
| *** swick has quit IRC | 11:24 | |
| laurence | how are people finding marge? as i commented the other day i have seen quite a few notifications of CI taking too lon | 11:25 |
| juergbi | I think the timeout is 90min right now? should probably be increased to at least 120min | 11:26 |
| * Kinnison likes that he can assign to marge and stop worrying | 11:26 | |
| juergbi | as WSL can occasionally take a long time | 11:26 |
| juergbi | however, in general I definitely like the setup | 11:26 |
| Kinnison | WSL is horrendously slow | 11:27 |
| Kinnison | 120m is probably a good idea | 11:27 |
| juergbi | it seems to vary a lot | 11:27 |
| laurence | ok, will change to 120 mins | 11:28 |
| phildawson | It's definitely nice to be able to assign and forget. I agree it would be good to up her timeout though. It would probably make sense to set it to whatever our CI is set to. | 11:28 |
| juergbi | our CI might currently have a very long timeout due to the overnight tests | 11:28 |
| juergbi | not sure whether we can have separate timeouts for the regular tests | 11:29 |
| juergbi | in that case the same timeout would definitely make sense | 11:29 |
| phildawson | Fair point. | 11:30 |
| laurence | valentind, pls could you approve/merge https://gitlab.com/BuildStream/website/merge_requests/118#note_150868055 ? | 11:36 |
| laurence | or anyone else who has permissions on the website repo | 11:37 |
| valentind | laurence, done. | 11:41 |
| *** mohan43u_ has joined #buildstream | 11:41 | |
| *** mohan43u has quit IRC | 11:41 | |
| *** lachlan has quit IRC | 11:42 | |
| laurence | valentind, tvm. johnward could you pls press the merge button? | 11:43 |
| valentind | laurence, sorry, I thought you had the right to merge. | 11:43 |
| valentind | I merged | 11:44 |
| laurence | valentind, ah, ok (wasn't sure if you have merge rights or just approver rights) - cheers | 11:44 |
| gitlab-br-bot | LaurenceUrhegyi closed issue #924 (Update BuildStream's documentation and website with an update on the Release Policy change) on buildstream https://gitlab.com/BuildStream/buildstream/issues/924 | 11:50 |
| *** lachlan has joined #buildstream | 11:58 | |
| benschubert | juergbi: does https://gitlab.com/BuildStream/buildstream/commit/c9de0cb803d7dbf57316730c9e7e4b64e6a47ef1 makes sense? | 12:15 |
| juergbi | benschubert: yes, I think so | 12:26 |
| benschubert | juergbi: ok thanks! I'll continue my work then | 12:29 |
| *** raoul has quit IRC | 12:33 | |
| gitlab-br-bot | marge-bot123 merged MR !1231 (aevri/direct_load_junction_projects->master: project: don't do _find_project_dir if we're a junction) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1231 | 12:33 |
| *** mohan43u_ has quit IRC | 12:54 | |
| *** mohan43u has joined #buildstream | 12:56 | |
| *** lachlan has quit IRC | 12:58 | |
| *** nimish2711 has joined #buildstream | 13:18 | |
| *** raoul has joined #buildstream | 13:39 | |
| *** raoul has quit IRC | 13:52 | |
| *** raoul has joined #buildstream | 13:57 | |
| benschubert | juergbi: I'm still confused with something. Can we build an element that has no __cache_key, but only a weak one? | 13:57 |
| *** lachlan has joined #buildstream | 13:59 | |
| juergbi | benschubert: no, I don't think this can happen | 14:00 |
| juergbi | workspaces are build with __cache_key set to None but weak is set to None as well in that case | 14:01 |
| benschubert | ok yeah | 14:01 |
| juergbi | in all non-workspace cases we should have both __cache_key and __weak_cache_key when starting a build | 14:01 |
| benschubert | I'm really confusing myself with my code right now. I'm not triggering a build that should be -_-' | 14:01 |
| juergbi | the overall logic is more complex than it'd seem at first, indeed easy to get confused | 14:02 |
| juergbi | it would be nice if we could somehow move the complexity of non-strict builds and workspaces ou tof the core cache key logic | 14:03 |
| benschubert | that's in the back of my head | 14:05 |
| gitlab-br-bot | aevri opened (was WIP) MR !1227 (aevri/you_only_write_once->master: cleanupjob, cascache: don't write cache size twice) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1227 | 14:23 |
| *** lachlan has quit IRC | 14:27 | |
| *** nimish2711 has quit IRC | 14:40 | |
| *** lachlan has joined #buildstream | 14:52 | |
| *** raoul has quit IRC | 15:05 | |
| *** nimish2711 has joined #buildstream | 15:06 | |
| gitlab-br-bot | mbodmer closed issue #962 (Question: Incremental builds for multiple build configurations side-by-side) on buildstream https://gitlab.com/BuildStream/buildstream/issues/962 | 15:07 |
| *** swick has joined #buildstream | 15:09 | |
| *** raoul has joined #buildstream | 16:06 | |
| *** lachlan has quit IRC | 16:19 | |
| raoul | For people that wanted to know about ux of the new pipeline with source pushing, here it is for testing the autotools example along with some remote execution: https://asciinema.org/a/LneYtAas5RvvczdrGVHXzItty | 16:40 |
| raoul | looks like it doesn't say in the pipeline summary so I should add that, and some other shorter name for the current status may be better than "Source pushed" | 16:41 |
| *** toscalix has quit IRC | 16:49 | |
| *** lachlan has joined #buildstream | 16:50 | |
| *** raoul has quit IRC | 16:52 | |
| *** raoul has joined #buildstream | 16:57 | |
| benschubert | juergbi: are you around? could you look at https://gitlab.com/BuildStream/buildstream/merge_requests/new/diffs?merge_request%5Bsource_branch%5D=bschubert%2Fpipeline-next&merge_request%5Btarget_branch%5D=master and tell me if the changes seem sensible to you? If so, I'll cleanup and push to !1070 | 16:59 |
| gitlab-br-bot | MR !1070: WIP: Refactor _update_state() to be called only when needed https://gitlab.com/BuildStream/buildstream/merge_requests/1070 | 16:59 |
| raoul | he's just headed off to the airport so he may not be online for a bit | 17:01 |
| doras[m] | What would be the correct element to use if I only want it to run integration commands? | 17:02 |
| benschubert | raoul: oh shoot, thanks! | 17:02 |
| tlater[m] | doras: Probably a compose element: https://docs.buildstream.build/elements/compose.html | 17:04 |
| tlater[m] | Though, that's assuming you have an element to depend on | 17:04 |
| tlater[m] | Otherwise you might want a manual element | 17:04 |
| doras[m] | Wouldn't compose include the artifcats of those elements in itself? I don't want it to have any files of its own, I just want it tto run some commands upon integration. | 17:06 |
| tlater[m] | doras: Hm, what do you want those commands to do? | 17:07 |
| tlater[m] | You can suppress output by setting include-orphans to false and simply not including any domains | 17:07 |
| tlater[m] | But I don't really see the point of such an element | 17:07 |
| doras[m] | It would have elements that it depends upon to run the integration commands, but it should not "become" its dependencies. | 17:08 |
| doras[m] | The point is that I have a compose element that runs the integration commands of its various elements. However, some commands aren't inherently part of an element, but rather optional. In order to run them only in the context of my compose element, I want to make them part of an element dedictaed purely to running them. | 17:10 |
| doras[m] | Strange use case, but makes some sense. | 17:11 |
| doras[m] | It also makes no sense to run them before integration, because they operate on files created by the integration commands of other elements. | 17:12 |
| doras[m] | This is where I hope I can control the order by which integration commands are run. | 17:12 |
| * tlater[m] isn't sure he quite understands the use case - can't really see what the input (in terms of filesystem) is, and what the expected output would be | 17:13 | |
| tlater[m] | Sorry |: | 17:13 |
| doras[m] | I could just as well make a script element that stages the compose element and then runs those commands. | 17:14 |
| doras[m] | But I somehow feel that I would be missing the point of integration commands. | 17:14 |
| tlater[m] | Yep, I see where you're coming from | 17:14 |
| tlater[m] | I suspect a clever use of split rules can do this | 17:14 |
| tlater[m] | But I can't really help much with that without a clearer picture | 17:15 |
| tlater[m] | doras: Reading through that again, do you mean: element -> integration-command-element -> compose? | 17:17 |
| doras[m] | Yes. | 17:18 |
| doras[m] | More like "elements -> integration-command-element -> compose", but yeah. | 17:18 |
| tlater[m] | Err, there's an important distinction here | 17:18 |
| tlater[m] | Is the mapping N:N:1 | 17:19 |
| tlater[m] | Or N:1:1 | 17:19 |
| doras[m] | N:N:1, if this works. | 17:20 |
| doras[m] | I have more use cases for "optional" integration commands. | 17:20 |
| *** alatiera has joined #buildstream | 17:21 | |
| tlater[m] | doras: I think best practice for "integration-command-element" would still be a compose element in this case | 17:22 |
| tlater[m] | You object to it "becoming" the dependency though, right? | 17:22 |
| tlater[m] | Why is that? | 17:22 |
| doras[m] | It is dependent on tools that I don't want to include in the final compose. They are just integration tools. | 17:22 |
| tlater[m] | So "integration-command-element" would depend on both the thing you want to integrate and a handful of dev tools? | 17:23 |
| doras[m] | Only dev tools, I think. The thing I want to integrate is the ouput of the integration command. | 17:25 |
| tlater[m] | That sounds like a script element... Where do the integration commands come from? | 17:26 |
| doras[m] | Dev tools. | 17:26 |
| doras[m] | Dev tools run integration on elements that are part of the compose, produce output. I'd like that output to become part of the compose as well. | 17:27 |
| tlater[m] | In either case, I think the options are 1) compose element with very fancy split rules or 2) script element | 17:27 |
| doras[m] | Kind of like a compose element running its own compose commands. | 17:27 |
| doras[m] | err, integration commands* | 17:27 |
| tlater[m] | This certainly seems like a case for a script element to me | 17:28 |
| tlater[m] | Because you're not really using buildstream's concept of integration commands in the first place | 17:28 |
| tlater[m] | Unless the integration tools have integration commands, but that seems wrong because it's not the commands used to integrate the integration tools, but rather the elements they are integrating | 17:29 |
| doras[m] | What is BuildStream's concept of integration commands? | 17:30 |
| tlater[m] | The way I perceive it is that we associate a set of commands that need to be run for an element if it's to be integrated with other tools | 17:30 |
| tlater[m] | s/tools/elements/ | 17:30 |
| tlater[m] | Say, something to update font caches | 17:30 |
| doras[m] | What if I optionally want to update font caches? | 17:31 |
| tlater[m] | That depends on what "optionally" means :) | 17:32 |
| doras[m] | Some compose elements with integration enabled would depend on the element that can update font caches, but not actually update the font caches during integration. Some compose elements would. | 17:33 |
| doras[m] | The real issue here is that this element comes with many dev tools, some I want to run during integration and some I do not. | 17:34 |
| tlater[m] | doras: Could you get away with scripting this in the integration commands? | 17:34 |
| tlater[m] | I.e., have some check in the integration commands that checks if you need to run the dev tool for the current state? | 17:35 |
| doras[m] | I could. It would be unfortunate if I have to, though. Detection could be complicated. | 17:36 |
| doras[m] | Right now integration commands are implicit. My use case is explicit integration commands. | 17:37 |
| doras[m] | I guess it's not really "supported". | 17:37 |
| tlater[m] | Yep, not unexpected. There's a chance I simply don't know how it is supported. | 17:37 |
| tlater[m] | I'd suggest seeing if tristan pops up later and pitching this to him | 17:37 |
| tlater[m] | But generally elements are pretty static, little should change between invocations of the same element. | 17:38 |
| *** jonathanmaw has quit IRC | 17:40 | |
| doras[m] | I could use script elements, but they basically don't have the concept of integration commands since they can't have runtime dependencies. | 17:43 |
| doras[m] | Which is unfortunate, I don't understand why. | 17:43 |
| doras[m] | A script can create an output that can be run. Say, another script. Why not allow running it? :\ | 17:46 |
| tlater[m] | ^ If anything, that should be documented | 17:46 |
| tlater[m] | Agreed. I'm trying to remember why, but it's been a while since I played with script elements |: | 17:47 |
| doras[m] | Well, it does say that "Script elements may only specify build dependencies." | 17:47 |
| doras[m] | In here: https://docs.buildstream.build/elements/script.html | 17:47 |
| tlater[m] | It doesn't say why though :) | 17:48 |
| benschubert | tlater[m]: if you have time can you have a look at https://gitlab.com/BuildStream/buildstream/commits/bschubert/pipeline-next and tell me if you spot some obvious problems? I'm still in the process of cleaning up but should be pretty much ready :) only one test not passing. For more context, !1070 contains the previous iteration | 17:52 |
| gitlab-br-bot | MR !1070: WIP: Refactor _update_state() to be called only when needed https://gitlab.com/BuildStream/buildstream/merge_requests/1070 | 17:52 |
| tlater[m] | "blah" is a pretty good commit message, benschubert ;) | 17:52 |
| benschubert | Will be merged with the PQ commit ;) | 17:53 |
| benschubert | (and I apparently broken something just before pushing, 1 test in all is failing o/) | 17:57 |
| benschubert | uh? no it's not broken | 17:59 |
| tlater[m] | benschubert: From a quick read through it seems fine | 18:05 |
| tlater[m] | Much prefer the new UniquePQ | 18:05 |
| benschubert | tlater[m]: thanks! I'll cleanup, extract the PQ in a _collections.py and will update !1070 with it. We might get it merged on monday :D | 18:06 |
| benschubert | once I've got juergbi and tristan's reviews | 18:06 |
| tlater[m] | Yeah, you'll definitely want more rigorous review | 18:06 |
| * tlater[m] has been pretty far removed from this particular issue | 18:06 | |
| benschubert | just missing the benchmarks on it :) | 18:06 |
| benschubert | agreed :) I'm just trying to have as much people as possible to have a look at this one, since it's pretty core :) | 18:07 |
| tlater[m] | Let's hope those turn out good :) | 18:07 |
| benschubert | tlater[m]: they will at least for cases with >8 builders | 18:08 |
| benschubert | :D | 18:08 |
| tlater[m] | If you want to use some fresh-off-the-press beta code, you can use the new local benchmarking to get some initial estimates. | 18:08 |
| benschubert | you mean jennis' stuff? | 18:08 |
| tlater[m] | ... but I suspect lachlan wouldn't like me advertising that | 18:08 |
| benschubert | ah? | 18:08 |
| benschubert | too late :) | 18:08 |
| tlater[m] | Nah, we're revamping the original benchmarking repo so people can use it on their dev machines | 18:09 |
| benschubert | tlater[m]: I like that! | 18:09 |
| benschubert | but I went ahead and scripted my own https://gitlab.com/BenjaminSchubert/bst-benchmarks for local stuff | 18:09 |
| benschubert | but I would be very happy to use something else | 18:09 |
| tlater[m] | Oh, i've actually seen that, yeah | 18:09 |
| tlater[m] | benschubert: Keep an eye on this branch: https://gitlab.com/BuildStream/benchmarks/tree/lachlanmackenzie/LocalRepoBenchmarkingTesting | 18:09 |
| tlater[m] | That said, it's still quite the end from ready | 18:09 |
| benschubert | tlater[m]: do we have tests with different number of runners and/or remote exec? | 18:10 |
| tlater[m] | Not yet. Ideally once this gets a little more exposure those tests will magically appear. | 18:11 |
| benschubert | tlater[m]: please keep me updated on this, I've been looking a lot at performance :) | 18:11 |
| tlater[m] | And by "magically" I mean lachlan and I will write them as people complain our metrics are lacking ;) | 18:11 |
| benschubert | Then add number of builders to your list ;) | 18:11 |
| tlater[m] | benschubert: will do! | 18:11 |
| benschubert | because https://gitlab.com/snippets/1832618 because it's not glorious :/ | 18:12 |
| tlater[m] | Oh, interesting | 18:13 |
| * tlater[m] finishes up for the day, in any case, some uni work to finish off before tomorrow... o/ | 18:14 | |
| benschubert | good luck with that! | 18:21 |
| juergbi | benschubert: decrement at the end of update_state: shouldn't this be done only if self.__remaining_runtime_deps_cache_keys_to_discover is 0? | 18:24 |
| juergbi | have to go to the gate now | 18:24 |
| *** raoul has quit IRC | 18:25 | |
| juergbi | and if it's not 0, we need to do this decrement when that happens | 18:25 |
| *** nimish2711 has quit IRC | 18:26 | |
| benschubert | juergbi: gah that's true, we need to do it recursively also | 18:30 |
| benschubert | I'll amend that thanks! | 18:30 |
| benschubert | and benchmark tonight | 18:30 |
| *** rdale has quit IRC | 19:43 | |
| *** alatiera has quit IRC | 20:17 | |
| *** TheScaia has joined #buildstream | 20:29 | |
| *** TheScaia has quit IRC | 20:30 | |
| *** lachlan has quit IRC | 21:03 | |
| *** alatiera has joined #buildstream | 22:26 | |
| *** nimish2711 has joined #buildstream | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!