IRC logs for #buildstream for Monday, 2019-09-09

*** ahmed89 has joined #buildstream05:32
*** ahmed89 has quit IRC05:45
*** ahmed89 has joined #buildstream05:46
*** mohan43u has joined #buildstream05:50
*** mohan43u has quit IRC05:53
*** mohan43u has joined #buildstream06:04
*** ahmed89 has quit IRC06:21
*** ahmed89 has joined #buildstream06:22
*** ahmed89 has quit IRC06:26
*** ahmed89 has joined #buildstream06:27
*** ahmed89 has quit IRC06:47
*** ahmed89 has joined #buildstream06:47
*** ahmed89 has quit IRC07:07
*** ahmed89 has joined #buildstream07:08
*** ahmed89 has quit IRC07:17
*** ahmed89 has joined #buildstream07:18
*** rdale has joined #buildstream07:53
gitlab-br-botBenjaminSchubert closed issue #1127 (Allow sources to access their element's variables) on buildstream https://gitlab.com/BuildStream/buildstream/issues/112707:54
*** bochecha has joined #buildstream08:17
*** ahmed89_ has joined #buildstream09:04
*** ahmed89 has quit IRC09:06
*** ahmed89_ is now known as ahmed8909:06
*** jonathanmaw has joined #buildstream09:10
*** tpollard has joined #buildstream09:10
*** tiagogomes has quit IRC09:28
*** traveltissues has joined #buildstream09:39
gitlab-br-botBenjaminSchubert opened issue #1128 (Exception thrown when retrying a task) on buildstream https://gitlab.com/BuildStream/buildstream/issues/112809:39
benschubertjuergbi: are you around? Any idea why I might be seeing:09:40
benschubert    grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:09:40
benschubert        status = StatusCode.UNAVAILABLE09:40
benschubert        details = "Socket closed"09:40
benschubert        debug_error_string = "{"created":"@1568021528.559529300","description":"Error received from peer unix:/tmp/buildstream2k0p4_l4/casd.sock","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Socket closed","grpc_status":14}"09:40
tpollardbenschubert: yes, I've made an issue for it09:41
tpollardhttps://gitlab.com/BuildStream/buildstream/issues/1120 we should probably collate them09:41
juergbinot the same 'details' but it's definitely close09:41
benschubertah agreed09:42
benschubertthis completely blew up my buildstream09:42
juergbiwould be great if manage to trigger this in a test case09:42
benschubertctrl+c doesn't work anymore09:42
benschubertjuergbi: trying to fetch a remote that doesn't exist and hitting 'retry' seems to have triggered for me quite a bit09:43
benschubertbut requires interactive mode09:43
juergbiok09:43
tpollardyep, we can't recover from interrupt, manual or otherwise09:43
benschubertjuergbi: also thanks for your comment on !1126, that makes much more sense :)09:47
gitlab-br-botMR !1126: WIP: yamlcache: Lookup files in the cache without reading the file https://gitlab.com/BuildStream/buildstream/merge_requests/112609:47
juergbiyw, I assume you mean #112609:48
gitlab-br-botIssue #1126: Cache resolution is slow https://gitlab.com/BuildStream/buildstream/issues/112609:48
benschubertcorrect09:48
benschubertjuergbi: oh, another interesting emssage with BuildBox-casd: Resource exhausted. An artifact is too big for the cache... Any fix for that on the top of your head?09:51
juergbibenschubert: that should only happen if the artifact is larger than the configured quota - or you're running out of disk space09:52
benschubertjuergbi: I haven't set a quota, let me double check for disk space09:53
*** cs-shadow has joined #buildstream09:54
benschubertI also have at least 200GB of disk space in my docker machine09:55
benschubert    grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:09:55
benschubert        status = StatusCode.RESOURCE_EXHAUSTED09:55
benschubert        details = "Received message larger than max (4660631 vs. 4194304)"09:55
benschubert        debug_error_string = "{"created":"@1568022618.298376400","description":"Error received from peer unix:/tmp/buildstreamlzh5pivm/casd.sock","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Received message larger than max (4660631 vs. 4194304)","grpc_status":8}"09:55
juergbioh, I need to look into that09:57
benschubertDo you want me to open an issue?09:57
juergbiplease do09:57
benschubertthat was triggered trying to source push part of freedesktop-sdk to my local cache09:57
juergbithat error message is from gRPC core09:57
*** tiagogomes has joined #buildstream09:59
juergbiprobably failing to split a gRPC request with tons of blobs09:59
gitlab-br-botBenjaminSchubert opened issue #1129 (Error when src-push to a remote: Message larger than max) on buildstream https://gitlab.com/BuildStream/buildstream/issues/112910:01
benschubertjuergbi: ^ here you go!10:01
juergbita10:01
benschubertlet me know if you need more logs/etc10:02
juergbiwill do but it's probably enough10:03
*** ahmed89_ has joined #buildstream10:13
*** ahmed89 has quit IRC10:14
*** ahmed89_ is now known as ahmed8910:14
* tlater[m] wonders why our master documentation is still tagged as 1.3.0+something10:14
juergbiwe should make a 1.90.0 tag to fix this10:15
benschubertCould someone give me merge rights on bst-plugins-experimental by any chance? :)10:26
jjardontlater[m]: last tag available in master branch10:27
jjardonjuergbi: yes please :)10:27
gitlab-br-botBenjaminSchubert approved MR !1587 (jennis/update_python_deps->master: Update all Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158710:27
tlater[m]jjardon: I know, just wanted to know why we hadn't made another tag yet ;)10:28
juergbibenschubert: done10:29
benschubertthanks juergbi !10:29
benschubertcoldtom: I've set https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/28 to be merged. Let me know once you rebased, I'll review your PR for tox so we can finally get it in :)10:30
benschubertadditionally, that might fix our nightly tests10:30
qinusty:D Good news tlater[m], good job on the merge10:44
tlater[m]:D10:45
traveltissuescan i please get a review for !158810:52
gitlab-br-botMR !1588: use show_default https://gitlab.com/BuildStream/buildstream/merge_requests/158810:52
gitlab-br-botBenjaminSchubert approved MR !1588 (traveltissues/1110->master: use show_default) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158810:54
traveltissuesty benschubert10:54
tlater[m]jennis: Mind taking another quick look at this? https://gitlab.com/BuildStream/docs-website/merge_requests/3/diffs?diff_id=54243600&start_sha=5fd92cf3ef6263a9cb4a59ef5bb95ffa18c3f33810:55
jennistlater[m], sure will be in 30 mins or so though10:55
tlater[m]np, just scared of the big blue button ;)10:56
tlater[m]Ok, I ended up cleaning that up a *tiny* bit after all: https://gitlab.com/BuildStream/docs-website/merge_requests/3/diffs?diff_id=54244555&start_sha=5fd92cf3ef6263a9cb4a59ef5bb95ffa18c3f33811:01
tlater[m]-> replaced a hardcoded path11:02
gitlab-br-botcoldtom opened MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158911:02
gitlab-br-botBenjaminSchubert approved MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158911:03
*** ahmed89 has quit IRC11:13
tlater[m]Oh, this is cute, our 1.0 docs still use the old theme: https://docs.buildstream.build/1.0.1/index.html12:23
coldtomty benschubert, https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/27 is rebased and ready for another round of review12:26
benschubertcoldtom: I'll atake a look now, thanks!12:31
gitlab-br-botmarge-bot123 closed issue #1110 (use `show_default` for Click.options) on buildstream https://gitlab.com/BuildStream/buildstream/issues/111012:46
gitlab-br-botmarge-bot123 merged MR !1588 (traveltissues/1110->master: use show_default) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158812:46
gitlab-br-bottpollard approved MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158913:13
* tlater[m] is looking at https://gitlab.com/BuildStream/buildstream/issues/1068 and feels like it isn't all that useful in the end13:15
tlater[m]I think it might be better to individually review each and every subcommand13:15
tlater[m]And instead track the per-subcommand progress in that issue13:16
tlater[m]This would probably make feeding back into the design principles discussion easier too13:16
tlater[m]Would anyone object to my creating an issue for each subcommand?13:17
* tlater[m] was about to mailing list that question, but it seems like a relatively trivial thing13:17
gitlab-br-bottraveltissues closed issue #1123 (test failures on wsl) on buildstream https://gitlab.com/BuildStream/buildstream/issues/112313:21
traveltissuesty for the help with #1123 juergbi13:22
gitlab-br-botjennis opened MR !1590 (jennis/update_source_checkout->master: Ensure `source checkout` is symmetric to `artifact checkout`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/159013:24
jennistlater[m], I agree with you13:29
jennisThat's a good idea13:29
tlater[m]I also think we should start this off with a big ML thread for each subcommand13:30
jennisyeah13:30
tlater[m]So that we can gather use cases and come up with a design from scratch13:30
gitlab-br-botmarge-bot123 merged MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158913:49
Kinnisonjennis: 👍 to !1587 but it needs a rebase14:02
gitlab-br-botMR !1587: Update all Python dependencies https://gitlab.com/BuildStream/buildstream/merge_requests/158714:02
gitlab-br-botjennis opened issue #1130 (Discuss and document policy for updating BuildStream's Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/issues/113014:04
jennisKinnison, yep. Also opened #1130 as a followup14:07
*** phoenix has joined #buildstream14:07
Kinnisonnice14:07
Kinnisonjennis: probably worth posting to the ML about that14:08
Kinnisonin case people don't notice it14:08
benschubertIs it me or the CI runners is flaky again?14:08
benschubert(the bastions-runners-overnight)14:09
*** phoenix has quit IRC14:13
gitlab-br-bottraveltissues opened issue #1131 (Error loading elements for freedesktop-sdk) on buildstream https://gitlab.com/BuildStream/buildstream/issues/113114:33
gitlab-br-botmarge-bot123 merged MR !1587 (jennis/update_python_deps->master: Update all Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/158714:49
valentindbenschubert, Do you care about that CI of your branch on Freedesktop SDK? Maybe you can just disable it. We have bst master docker images, though. But if you run your own CI for it, maybe just disable the one on Freedesktop SDK.14:52
benschubertvalentind: oh I don't care no, sorry about that14:55
benschuberthow do I disable it?14:55
benschubertit was just in order to update jjardon's bst2 branch14:55
*** phildawson has quit IRC14:59
*** phildawson has joined #buildstream14:59
*** phildawson has quit IRC14:59
Kinnisonjennis: 👍 on !1590 -- looks like a good step to closer UX14:59
gitlab-br-botMR !1590: Ensure `source checkout` is symmetric to `artifact checkout` https://gitlab.com/BuildStream/buildstream/merge_requests/159015:00
benschubert^ now only missing bst source push :D15:00
Kinnisonheh15:00
jennisthink tlater[m] has a branch for that?15:02
tlater[m]Umm, yes15:02
tlater[m]Though it's heavily WIP because I basically wrote the code to get something to use for manual smoke testing15:02
*** phildawson has joined #buildstream15:04
benschuberttlater[m]: for bst source push? :)15:06
*** phildawson_ has joined #buildstream15:49
*** phildawson has quit IRC15:50
jennisbenschubert: https://gitlab.com/BuildStream/buildstream/merge_requests/156116:02
benschubertis it ready for review?16:07
benschubert        status = StatusCode.UNAVAILABLE16:10
benschubert        details = "failed to connect to all addresses"16:10
benschubertWhat could be the cause of this in the push queue? Roughly half of my elements fail to be pushed like that16:10
jennisNope, "heavily WIP"16:11
juergbibenschubert: I assume the CAS server is not overloaded. does it retry and fail 3 times?16:17
juergbialso, it's not clear whether this is an issue connecting to casd or to the remote CAS16:17
benschubertjuergbi: it doesn't retry as it's an unhandled exception. The cas server has a single client, so I hope it's not overloaded16:18
benschubertit's in 'get_local_cas'16:19
juergbiLinux or WSL?16:19
benschubertjuergbi: https://gitlab.com/snippets/189343216:19
benschubertinside a ubuntu 19.04 container16:19
benschubertand my whole build completely dead locked with that16:20
juergbiok :-/16:20
juergbiI'm wondering whether it could be an issue on casd side or why else a local socket connection would fail16:21
valentindbenschubert, you can just remove .gitlab-ci.yml.16:22
benschubertjuergbi: oh, the exception seems to happen after the first try failed, when doing the second try16:32
juergbibenschubert: interesting, so it might be related to retry failing in interactive mode16:33
benschubertpossible yes16:33
benschuberteven though this time I had '--on-error=continue'16:34
juergbialthough, I really wouldn't expect any connection failures with UNIX sockets - unless the service crashed or similar16:34
benschubertwhich seems possible. Any ways I can check that?16:34
juergbibenschubert: besides checking with `ps` you could add a self._casd_process.is_alive() to _get_local_cas()16:36
juergbiif it's a real segfault, it might also show up in system log and you might get a coredump - depending on system configuration16:36
benschubertI do have a 'buildbox-casd' subprocess16:36
juergbiand it's not a zombie?16:36
benschubertbuildst+ 69156 27.5  0.0      0     0 pts/0    Z+   16:26   3:31 [buildbox-casd] <defunct>16:39
benschubertIt... is not in a good shape -_-16:39
tlater[m]benschubert: I don't really have the time to get `bst source pushll` over the line atm :(16:39
traveltissuesis there any instance where it can be expected that a parent process will receive NoneType child process data?16:39
benschuberttlater[m]: no worries, I was just curious :)16:39
traveltissuesin terms of the scheduler16:40
benschubertIs it expected that we would call 'get_local_cas' roughly 50-100 times between 'SUCCESS: caching' and 'SUCCESS build'?16:49
traveltissuesidk but that seems wrong16:50
tlater[m]I'm looking at issue #1093 - does anyone know whether our plugins can already send "normal" status reports without text?16:52
gitlab-br-botIssue #1093: Add progress reporting as a generic string https://gitlab.com/BuildStream/buildstream/issues/109316:52
juergbibenschubert: right now, yes. we're currently adding files one by one. batching those calls (or rather, capture the whole directory tree in one call) is a planned optimization16:52
juergbibenschubert: it should only attempt to connect once per job child process, though, and then stay connected16:52
juergbi(i.e., the other times it should hit the fast path)16:52
tlater[m]I feel like #1092 says they can't, but #1093 doesn't refer to it16:53
gitlab-br-botIssue #1092: Add progress reporting across jobs in subprocesses https://gitlab.com/BuildStream/buildstream/issues/109216:53
benschubertyep, I was just suprised, I added a print in get_local_cas, and it ended up printing almost only that16:53
tpollardtlater[m]: could we just use the our own Message()?16:54
tlater[m]tpollard: The idea is that we have a continuous ticker16:54
tlater[m]For when multiple things happen at the same time16:55
tlater[m]#1093 wants to always display the last line from build output16:55
tlater[m]As part of a task16:55
tlater[m]The little menu at the bottom that shows all the fetches/builds/tracks currently happening :)16:55
tpollardyep16:56
tlater[m]We want builds to be able to display the last line of their output there.16:56
tlater[m]tpollard: So an arbitrary plugin *can* currently send things to the frontend already?16:56
tpollardall plugins have access to info()16:57
tpollardnot exactly what we'd want here, but it's a case where we enable it16:58
tpollardand status() of course16:58
tpollardand in terms of subprocessing, I think the Message() should be picklable now16:59
tlater[m]Well, I'm wondering how much API exists for accessing our tasks shenanigans16:59
tlater[m]But message isn't nearly enough for that, is it?16:59
benschubertjuergbi: ok, so the casd server doesn't crash17:02
benschubertjuergbi: however, GetCapabilitiesRequest() fails to connect to all addresses...17:02
benschubertDo we have to close requests ourselves? are we leaking something?17:03
benschubertThat seems like a leak and it fails once we opened too many connections17:03
*** phoenix has joined #buildstream17:05
juergbibenschubert: on the bst client side I think it's unlikely to be a leak because it's handled in job subprocess and an potential leak would be cleaned up on subprocess termination17:09
tpollardtlater[m]: I'm not 100% upto date on the State() side of things sadly17:10
juergbithere could be a leak on the casd side17:10
benschubertjuergbi: mmh, not sure how to start debugging this17:12
tpollardtlater[m]: I've tried to ensure that Messenger() will still work when process separation happens, but I do worry there's extra's in State() that also need adapting17:14
tlater[m]Yeah, I suspect State() is going to be a bit of a nightmare with this stuff17:14
* tlater[m] will need to take a deep dive to figure out how hard these things are...17:15
benschubertjjardon: Are those kind of errors expected, or would that be a bug in the plugins: https://gitlab.com/BuildStream/buildstream/-/jobs/291029431 ? I'm trying to fix the nightly tests17:15
juergbibenschubert: #1120 sounds like it might be easy to reproduce. I should probably start debugging that and maybe it has the same root cause as what you're seeing17:16
gitlab-br-botIssue #1120: Interrupt handling during a task that involves cascache grpc processes is very flaky https://gitlab.com/BuildStream/buildstream/issues/112017:16
juergbifirst step would probably be to get it into the test suite17:16
jjardonbenschubert: I have never seen that error before17:16
benschubertjuergbi: let me know if you need more info17:17
juergbiok, will do17:17
juergbibenschubert: do you use a very large number of job subprocesses?17:17
juergbidoes it happen also with few builders/pushers17:17
benschubertthe default17:17
juergbiok, so not that many, odd17:17
jjardonMigth be because some changes on master, not sure sorry17:17
benschubertand trying to build freedesktop/base-platform.bst is enough to trigger it17:18
benschubertjjardon: no problem, thanks! If I update the freedesktop pinned version we are using in the tests, would that be fine with you?17:18
juergbibenschubert: only if you have a push artifact server configured or even purely local?17:18
benschubertjuergbi: haven't tried in purely local17:18
benschubertlet me check17:18
jjardonbenschubert: sure17:18
juergbibenschubert: btw: don't you see is_single_threaded() assertion failures due to ostree?17:19
benschubertjuergbi: none17:19
juergbibst-plugins-experimental master?17:19
benschubertyep17:20
juergbiwondering whether this might depend on ostree/gobject version17:20
*** phoenix has quit IRC17:28
*** jonathanmaw has quit IRC17:29
benschubertjuergbi: the problem seems to not be happening when no cache is available17:34
benschubertI'm still running the build just in case, but when pushing I was never able to build that many elements17:34
juergbiok, as you've seen the issue for build jobs, not push jobs, I still suspect the issue is on the casd side17:37
juergbibut don't have an idea what exactly is going wrong there17:37
juergbimight make sense to run casd with valgrind17:37
benschubertgood point I'll try this setup17:37
benschubertDoes casd writes logs?17:38
juergbistderr17:38
juergbiyou can increase verbosity17:38
juergbi--verbose17:38
juergbior for the maximum --log-level=trace17:38
benschubertbut we are not storing them anywhere. Would it make sense to save them somewhere and add configuration for it in buildstream.conf ?17:39
juergbinot sure, do we support this for general buildstream logs right now?17:40
juergbiiirc, we only store element-specific logs in the artifact17:40
benschubertwell, we store the messages17:40
benschubertcoldtom: fine if I merge https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/27 ? :)18:01
coldtombenschubert: absolutely fine, glad it's finally in a fit state to merge :)18:05
*** traveltissues has quit IRC18:17
*** phildawson_ has quit IRC18:21
benschubertcoldtom: I can imagine, thanks a lot for the hard work :)18:28
benschubertjuergbi: looking at logs, the only thing I can see (once) is : E0909 20:34:53.104015208   18690 client_context.cc:85]       assertion failed: call_ == nullptr20:35
*** rdale has quit IRC20:36
*** cs-shadow has quit IRC21:24
*** phoenix has joined #buildstream21:34
*** phoenix has quit IRC21:59
*** swick has quit IRC22:16
*** swick has joined #buildstream22:22

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!