IRC logs for #buildstream for Thursday, 2019-03-14

*** nimish has joined #buildstream01:38
*** alatiera has quit IRC02:33
*** tristan has joined #buildstream03:32
*** nimish has quit IRC04:15
*** nimish has joined #buildstream04:16
*** mohan43u has quit IRC04:26
*** mohan43u has joined #buildstream04:27
*** tristan has quit IRC04:51
*** nimish has quit IRC04:53
*** nimish has joined #buildstream04:53
gitlab-br-bottpollard opened MR !1228 (tpollard/fix-complete-nonetype->master: _frontend/cli.py: Don't attempt to path NoneType in complete_target()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/122805:12
*** nimish has quit IRC05:26
*** tpollard has joined #buildstream06:06
*** mohan43u has quit IRC06:09
gitlab-br-bottristanvb approved MR !1228 (tpollard/fix-complete-nonetype->master: _frontend/cli.py: Don't attempt to path NoneType in complete_target()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/122806:10
*** nimish has joined #buildstream06:11
*** tristan has joined #buildstream06:13
*** mohan43u has joined #buildstream06:14
*** nimish has quit IRC06:17
*** mohan43u has quit IRC06:33
gitlab-br-botmarge-bot123 closed issue #958 (Autocompletion outside of a project produces stack trace) on buildstream https://gitlab.com/BuildStream/buildstream/issues/95806:36
gitlab-br-botmarge-bot123 merged MR !1228 (tpollard/fix-complete-nonetype->master: _frontend/cli.py: Don't attempt to path NoneType in complete_target()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/122806:36
*** tristan has quit IRC07:31
*** mohan43u has joined #buildstream07:32
*** tpollard has quit IRC07:37
*** tpollard has joined #buildstream07:37
*** tpollard has quit IRC07:44
gitlab-br-botmarge-bot123 merged MR !1124 (raoul/440-source-cache->master: Source cache) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/112407:53
*** tpollard has joined #buildstream08:36
*** benschubert has quit IRC09:01
*** benschubert has joined #buildstream09:05
*** toscalix has joined #buildstream09:07
*** toscalix has quit IRC09:14
*** toscalix has joined #buildstream09:16
*** tpollard has quit IRC09:25
*** slaf has quit IRC09:29
*** slaf has joined #buildstream09:32
*** slaf has joined #buildstream09:32
*** slaf has joined #buildstream09:33
*** raoul has joined #buildstream09:33
*** mohan43u has quit IRC09:48
*** mohan43u has joined #buildstream09:48
*** jonathanmaw has joined #buildstream10:04
*** toscalix has quit IRC10:32
*** toscalix has joined #buildstream10:32
*** lachlan has joined #buildstream10:38
phildawsonraoul, I've fixed the commit messages on !1215. Are you happy for me to marge it?10:45
raoulyep go for it :)10:45
phildawsonCheers10:46
*** alatiera has joined #buildstream10:47
gitlab-br-botmarge-bot123 merged MR !1215 (phil/consolidate-repo-tests->master: Consolidate templated source tests) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/121510:52
benschubertphildawson: :'( soo many conflicts with that10:55
phildawsonI know. Rebasing it was a massive pita :/10:56
Kinnisonphildawson: Feh10:57
Kinnisonphildawson: You should try rebasing my branch10:57
* phildawson is glad to avoid that particular ball of fun10:57
KinnisonIt only touches 90% of anything which ever thinks about yaml10:57
benschubertphildawson: I've got a 26 commits, 55 changes on the tests that were coming, now I need to rebase on yours :'D10:57
phildawsonI'm just glad I got in first :P10:58
benschubert20 commits :D10:58
*** lachlan has quit IRC11:16
raouljuergbi, the more I look into getting the logic in push to work with sources sensibly, the more I think it makes sense to add another queue, any thoughts? Unless I've missed something it doesn't look adding a new queue would be too hard11:17
juergbiraoul: as discussed, separate queue probably makes sense anyway long term11:24
juergbiif it can indeed be done without too much effort, would be great to already fold it into this MR11:24
raoulCool, shall attempt to do this11:24
juergbiraoul: I suspect the frontend/UI aspect might not be trivial, though11:25
juergbitwo queues called 'push' would obviously be confusing11:25
raoulYeah not sure what to call it, could just be source_push but that might be too long for the UI11:26
gitlab-br-botjuergbi opened MR !1230 (juerg/cas->master: _casbaseddirectory.py: Use variable-length argument list) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/123011:31
*** lachlan has joined #buildstream11:37
Kinnisonbenschubert: I've got a 102 commit branch to rebase soon :/11:49
benschubert102 commits? Oo11:49
*** alatiera has quit IRC12:05
*** alatiera has joined #buildstream12:06
*** alatiera has quit IRC12:25
gitlab-br-botmarge-bot123 merged MR !1230 (juerg/cas->master: _casbaseddirectory.py: Use variable-length argument list) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/123012:27
*** lachlan has quit IRC12:31
*** raoul has quit IRC12:36
*** alatiera has joined #buildstream12:41
toscalixThis limitation we experience with the website repo and subgroups  has been fixed: https://gitlab.com/gitlab-org/gitlab-ce/issues/3054813:08
*** nimish has joined #buildstream13:16
*** raoul has joined #buildstream13:24
*** nimish has quit IRC14:04
KinnisonIs there a way to know what Marge is doing for an MR?14:05
KinnisonI assigned her an MR which needs rebasing but so far she seems to have been sitting on it14:05
*** nimish has joined #buildstream14:07
*** tpollard has joined #buildstream14:08
juergbimaybe adds68 knows?14:10
*** lachlan has joined #buildstream14:11
benschubertKinnison: she might have other MRs to rebase first14:17
Kinnisonaah14:17
KinnisonOK14:17
benschubertit will go in a fifo and rebase only one at a time :D14:17
benschuberthence not overflowing our builders with useless runs14:17
*** lachlan has quit IRC14:17
KinnisonLooking at https://gitlab.com/BuildStream/buildstream/merge_requests?scope=all&utf8=%E2%9C%93&state=opened&assignee_username=marge-bot123 - she only has the one MR14:17
benschubertthen I don't know14:17
*** lachlan has joined #buildstream14:27
*** tpollard has quit IRC14:29
*** lachlan has quit IRC14:32
adds68Kinnison, as benschubert said :)14:32
Kinnisonadds68: so, since she only has the one MR assigned, and it's the one I assigned, what's up?14:33
adds68Hm, yes i just saw that, i am not sure14:33
adds68usually if there is a problem she will comment on the MR asking to be fixed14:33
Kinnisonthere's lots of pipelines running right now I s'pose14:34
KinnisonNone mine though :(14:34
gitlab-br-botmbodmer opened issue #962 (Question: How to have parallel builds for different architectures) on buildstream https://gitlab.com/BuildStream/buildstream/issues/96214:40
KinnisonShe finally rebased it \o/14:41
juergbishe was just out for lunch?14:41
Kinnisonperhaps :D14:42
*** lachlan has joined #buildstream14:49
*** lachlan has quit IRC14:53
raouljuergbi, for !1214 I've now put sources being pushed into a separate pipeline. They also give the status push, but as in the info and finished status it's source push(ed) I think it's clear enough. Still need to add some tests but let me know if you think it's alright (or anyone else that wants to have a look)15:03
gitlab-br-botMR !1214: Remote source cache https://gitlab.com/BuildStream/buildstream/merge_requests/121415:03
*** tristan has joined #buildstream15:04
*** tristan has quit IRC15:05
*** lachlan has joined #buildstream15:05
juergbiraoul: ok, will take a look15:16
juergbiwe should probably also add a `bst source push` command, right?15:16
juergbinot a blocker, though, imo15:16
raoulyeah we probably should, can add it to follow up work15:26
KinnisonIf I have a bare number in a field (e.g. strip-level: 1) should I expect that track might quote that ('1') ?15:29
*** lachlan has quit IRC15:31
*** lachlan has joined #buildstream15:37
juergbijonathanmaw: WSL seems to be very slow now16:18
juergbimarge is upset ;) I'm doing a quick manual merge16:18
* jonathanmaw pokes WSL to see if something's amiss16:19
juergbiI haven't seen any hangs but it takes >90 min to complete16:20
gitlab-br-botjuergbi merged MR !1209 (jennis/remove_node_chain_stuff->master: Rip out ChainMap(), node_chain_copy(), node_list_copy()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/120916:21
jonathanmawhmm, seeing a bunch of "Failed to process runner" errors16:22
jonathanmawand failing to communicate with the job coordinator16:23
jonathanmawcode 403. forbidden? how odd.16:23
Kinnisonphildawson: Is there a way to limit tox to only running the internal test suite?  I'm not sure much else will run yet on my branch but since I rebased recently I'm getting odd things from it trying to pip install -e16:38
juergbiKinnison: the fix was merged yesterday evening16:46
Kinnisonthanks16:47
KinnisonAnd now begins conflict'o'rama with my rebase16:48
Kinnison:D16:48
KinnisonI fear this will take some time16:51
Kinnisonartifact abstraction, source cache, cas changes, all joyful conflicts16:51
*** lachlan has quit IRC16:51
* raoul does not envy Kinnison16:53
KinnisonFrankly, 'tis basically hometime, I might pretend I've not started this rebase, and go to sleep instead16:54
*** toscalix has quit IRC16:54
KinnisonIn fact, I shall16:54
benschubertjuergbi: are you around? do you have a minute? I've got question about !1070 and the runtime_cache_key we were speaking about17:02
gitlab-br-botMR !1070: WIP: Refactor _update_state() to be called only when needed https://gitlab.com/BuildStream/buildstream/merge_requests/107017:02
juergbibenschubert: sure17:02
benschubertI was wondering how was the best way of keeping this key up to date. having a "update_runtime_key()" method and split rdeps on whether they are required for (build|all) or only runtime and when only runtime, call the update_runtime_key whenever we do the update_state_recursively?17:03
benschubertjuergbi: does this seem sensible? Do you see a better way?17:06
juergbibenschubert: I was originally not thinking about incremental updates17:08
juergbiin most situations, keys don't change later on, they mainly just go from None to the final key17:09
juergbiand thus, if the key is calculated only if all dependencies also have a key, I wouldn't expect the overhead to be very big17:09
benschubertjuergbi: so only using the strong_cache_key in case of runtime-only deps? Since this is the only one we need17:11
benschubertso whenever our strong_cache_key gets updated, we go through our runtime_only_rdeps and update the runtime_key, and the normal key of all their rdeps?17:12
juergbiuse the strong cache key of the self combined with the runtime key of the runtime dependencies17:12
benschubertjuergbi: not sure why we need to combine there?17:12
juergbithat way we'd have a single trigger. that's how I was thinking about it17:13
juergbinot sure whether splitting up would also work without issues17:13
juergbiwe definitely need to cover indirect runtime dependencies17:13
juergbiso if we use the combined key, that's definitely covered17:13
juergbiif the two aspects are considered separately, we don't get that for free, afaict17:14
benschubertjuergbi: for indirect combined deps, we only care when they are ALL discovered right? So we can set our runtime_deps_resolved to True once all our runtime_deps have it set to true and done no?17:14
benschubertor am I missing something17:14
juergbiyes but then we would still have to recursively check all runtime deps17:15
juergbiI think it would be cheaper to create a combined hash and then only check direct deps17:15
benschubertjuergbi: only the direct ones17:15
benschubertand if they don't have this flag to true, it means one of their rdep is not ready17:15
juergbiright, for the check that should suffice17:15
juergbihowever, to actually calculate the runtime key, you'd need to recurse, wouldn't you?17:16
juergbiwhile with a combined key, you can skip the recursion also for the runtime key calculation17:16
benschubertso the algo would be:17:17
benschubertwhen a strong cache key is discovered, go to all runtime_only reverse dependencies (others are already covered) and for each one do:17:17
benschubert   - if all direct runtime deps are ready: update all reverse dependencies keys17:17
benschubert  - otherwise pass17:17
benschubertOr am I missing something?17:17
juergbithe runtime key is a hash that includes the runtime dependencies, not the rdeps17:18
juergbithat key is used as a trigger to update rdeps17:18
juergbibut the key itself is of regular deps17:18
gitlab-br-botmarge-bot123 merged MR !1210 (aevri/nodefaultsset->master: element.__init_default: treat `None`  plugin_conf as if missing file + refactor) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/121017:19
benschubertjuergbi: so in what case would the above algorithm break?17:19
juergbiwhen a runtime dependency changes, we may need to update state/keys of reverse dependencies17:20
benschubertyes17:20
benschubertwhen a runtime dependency has its strong_cache_key change only though correct?17:20
juergbiyes (including indirect runtime dependencies)17:21
juergbiso if a.bst has a key change17:22
juergbiand b.bst is a reverse runtime-only dependency17:22
juergbib.bst's strong key will not change no matter what happens in a.bst (because it's runtime-only)17:23
juergbiso the algo wouldn't trigger an update in c.bst, which may build depend on b.bst17:23
juergbii.e., c.bst build-depends on b.bst. b.bst runtime-depends on a.bst17:24
juergbic.bst's strong key will change when a.bst's strong key changes17:24
juergbihowever, b.bst's strong key will not change when a.bst's strong key changes17:25
juergbior am I misunderstanding your algo?17:25
juergbi(I have my head wrapped around the other algo which works quite differently)17:25
benschubertSo:17:26
benschubert1) store BUILD|ALL direct reverse dependencies in a "reverse_dep_for_build_or_all", and all RUNTIME only direct dependencies in a "reverse_dep_for_run"17:26
benschubert2) when a strong cache key is discovered for an element, go through all elements in "reverse_dep_for_run".17:26
benschubert    - if all direct runtime dependency of this element have a strong_cache_key discovered (not None):17:26
benschubert      - update the state of all "reverse_dep_for_build_or_all"17:26
benschubert     - if the element has a strong cache key, goto 2 for this element.17:26
benschubert  - otherwise, pass17:26
benschubertIs that clearer?17:26
juergbihm, maybe that would work with the single indirect runtime dependency17:27
juergbibut if we add one additional indirection (c.bst build-depends on b1.bst which runtime-depends on b2.bst which runtime-depends on a.bst), does this still work?17:27
juergbiin case of key change17:28
benschubertSo a.bst would get a new strong_cache_key. We would go through the reverse_dep_for_run, that means "b2.bst" only.17:30
benschubert1) b2.bst doesn't has anything in "reverse_dep_for_build_or_all", so it doesn't do anything there17:30
benschubert2) two cases:17:30
benschubert     2.1) b2.bst has a strong cache key, we repeat for b1.bst17:30
benschubert     2.2) b2.bst doesn't have a strong cache key, there is no point in updating b1.bst, since b2.bst is not ready17:30
benschubertDoes that seem correct to you?17:31
juergbihm, but then we're essentially going over all reverse dependencies, recursively, even  if the key doesn't change in intermediate elements17:32
juergbias we also have to recurse further for build rdeps17:32
benschubertbut that's the case with the hash anyways17:33
juergbithat should work but that's the slower approach17:33
juergbino, with the hash you only have to go further if the hash changes17:33
benschubertsame here, but I only have a boolean17:33
juergbiyou have to recurse even if the strong cache key of intermediate elements doesn't change, though17:33
juergbimy scenario was when a.bst's key changes, not when it's initially set17:34
benschubertjuergbi: the strong_cache_key never changes once discovered17:34
juergbiah, we don't set it for workspaces until after assembly?17:34
* juergbi forgot the details17:35
benschubertjuergbi: if I understand well https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1227-1238 prevents it right?17:36
juergbiyes, that should be correct17:36
benschubertin that case my version would be simpler and quicker correct?17:37
benschubertand correct too17:37
benschubertjuergbi: or do you see something wrong with it?17:43
juergbi"- if all direct runtime dependency of this element have a strong_cache_key discovered (not None):"17:43
juergbiwhat if an indirect runtime dependency doesn't have a strong cache key yet?17:43
juergbithis might mean that the strong cache key of a reverse build dependency can't be calculated yet17:44
juergbimight this be an issue?17:44
benschubertmmh we would go through everything and we shouldn't true.17:44
benschubertSo we need a boolean "ready_for_runtime", that is all(runtime_deps.ready_for_runtime) and self.__strong_cache_key is not None17:45
benschubertcorrect?17:45
juergbiyes17:46
juergbiin a way it's a 1 bit variant of my combined hash17:46
benschubertagreed17:46
benschubertbut reduce the need for hashing :D17:46
juergbiyes, as we don't need to care about hash changes, but only about hash being available17:46
benschubertSo you would agree with a variant like that (assuming it does work)?17:46
juergbiwe can skip the actual hashing, but conceptually the same algo still makes sense17:47
juergbiprobably the most efficient approach17:47
juergbiand hopefully correct ;)17:47
benschubertgreat! If you think of hard test cases we don't cover in the tests, please let me know so we can add them. I've already got two from Tristan17:48
benschubertI'll implement this today/tomorrow and we should be able to review that soon17:48
juergbigreat17:48
juergbiwould we still calculate weak cache key early on or also only when deps are ready?17:48
benschubertjuergbi: that is a change that jonathanmaw is looking into if I remember well, correct jonathanmaw?17:49
jonathanmawbenschubert: I'm looking into all the stuff _update_state does17:50
* jonathanmaw reads backscroll for context17:50
juergbibenschubert: btw: 'ready_for_runtime' might be a bit misleading as it won't mean that the elements are actually built17:51
benschubertjuergbi: it would though, if we incorporate the "__strong_cache_key is not None" in the bool right?17:51
juergbino, cache key available doesn't mean that the element was built17:52
benschubertjuergbi: or we can have "all_runtime_deps_ready", which might be better right?17:52
juergbifor simple cases (strict build plan, no workspace, etc), we can calculate all strong cache keys early on17:52
juergbiwithout having built anything17:52
benschubertah gotcha17:53
benschubertso "__runtime_deps_strong_cache_keys_resolved" ?17:53
juergbiI think that would be accurate17:54
benschubertand for the elements needing the current element only at runtime: "__elements_with_dependencies_on_us_at_runtime_only"? It's starting to look like a paragraph but this part of the code is complex17:55
* jonathanmaw is not sure whether you only need to worry about the strong cache key when updating reverse-dependencies. Elements with BST_STRICT_REBUILD calculate cache keys based on their dependencies' weak cache keys instead of their names17:56
benschubertjonathanmaw: we are only looking in the case where this is a dependency for runtime only, which is only taken into account in the strong_cache_key17:57
benschubertafaik17:57
juergbibenschubert: I'd prefer reverse_dependencies to dependencies_on_us. __runtime_only_reverse_dependencies?17:57
juergbihowever, should we even separate this and the build reverse dependencies?17:58
benschubertjuergbi: we have two types of reverse dependencies, the ones needing us at runtime only, and the ones needing us at build time too. reverse_dependencies would not be enough17:58
benschubertI think that yes17:58
benschubertbecause for build, there are other moments where we need the weak cache keys no?17:58
juergbiwe might not actually need them earlier than when the strong cache key becomes available, in which case trigger on strong cache key / deps setting would suffice18:01
juergbibut I might be missing an aspect here18:01
juergbiit would be a nice simplification, though18:02
benschubertmmh I'm sure something is missing there, but I can try18:02
juergbiit might delay pull attempts of strict artifacts in non-strict mode18:06
juergbinot sure whether that would actually be an issue, though18:06
benschubertso the tests would fail18:06
juergbiI'm not sure18:07
juergbijust different scheduling18:07
juergbiso I wouldn't expect tests to fail18:07
juergbibut I might be missing something18:07
*** lachlan has joined #buildstream18:07
benschubertsure18:08
benschubertI'll try with that, see if tests work18:08
juergbiok, thanks18:09
benschubertotherwise I'll go with the other solution (keep build deps as we have in !1070 and do what we said for runtime_deps18:09
gitlab-br-botMR !1070: WIP: Refactor _update_state() to be called only when needed https://gitlab.com/BuildStream/buildstream/merge_requests/107018:09
benschubertjuergbi: why does the strict_cache_key takes precedence over the strong_cache_key?18:15
juergbibenschubert: in what aspect/situation?18:15
*** lachlan has quit IRC18:16
benschubertwe have if context.getstrict(): cache_key = strict; elif _cached(): cache_key = strong_key18:16
juergbiin strict mode, there is no difference anyway18:17
juergbiin non-strict mode, we need to load the strong cache key from the cached artifact, it's impossible to purely calculate it18:17
juergbi(if we're reusing a cached artifact, not building the element)18:18
benschubertoh ok18:18
juergbii.e., in strict mode, there is exactly one possible strong key and that's the strict key18:18
juergbiin non-strict mode there are arbitrary number of possible strong keys18:18
benschubertbut a single strict18:18
benschubertI see18:18
juergbiyes, there is always only a single strict one18:19
juergbihowever, the strict one is not relevant in non-strict mode, except when attempting to pull the 'perfect' strict artifact from the artifact server18:19
juergbi(and the same for the check in the local cache first, of course)18:19
benschubertjuergbi: so having only the cache_key as a test, test_push_pull_track_non_strict doesn't pass18:19
benschubertso my guess is, it's not possible like that, we need to use the previous iteration18:20
*** raoul has quit IRC18:20
juergbimaybe, I'd have to take a closer look. about to head out now18:20
benschubertok let's sync tomorrow18:20
benschubertif that's good with you :)18:20
juergbisure18:23
*** nimish has quit IRC18:31
*** nimish has joined #buildstream18:31
*** jonathanmaw has quit IRC18:38
*** alatiera has quit IRC19:17
*** nimish has quit IRC19:47
*** rdale has quit IRC20:31
gitlab-br-botjuergbi merged MR !1223 (aevri/doc_artifact_log->master: 'artifact log': document the 'artifacts' argument) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/122320:50
*** alatiera has joined #buildstream23:16
*** alatiera has quit IRC23:19

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!