*** ahmed89 has joined #buildstream | 05:32 | |
*** ahmed89 has quit IRC | 05:45 | |
*** ahmed89 has joined #buildstream | 05:46 | |
*** mohan43u has joined #buildstream | 05:50 | |
*** mohan43u has quit IRC | 05:53 | |
*** mohan43u has joined #buildstream | 06:04 | |
*** ahmed89 has quit IRC | 06:21 | |
*** ahmed89 has joined #buildstream | 06:22 | |
*** ahmed89 has quit IRC | 06:26 | |
*** ahmed89 has joined #buildstream | 06:27 | |
*** ahmed89 has quit IRC | 06:47 | |
*** ahmed89 has joined #buildstream | 06:47 | |
*** ahmed89 has quit IRC | 07:07 | |
*** ahmed89 has joined #buildstream | 07:08 | |
*** ahmed89 has quit IRC | 07:17 | |
*** ahmed89 has joined #buildstream | 07:18 | |
*** rdale has joined #buildstream | 07:53 | |
gitlab-br-bot | BenjaminSchubert closed issue #1127 (Allow sources to access their element's variables) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1127 | 07:54 |
---|---|---|
*** bochecha has joined #buildstream | 08:17 | |
*** ahmed89_ has joined #buildstream | 09:04 | |
*** ahmed89 has quit IRC | 09:06 | |
*** ahmed89_ is now known as ahmed89 | 09:06 | |
*** jonathanmaw has joined #buildstream | 09:10 | |
*** tpollard has joined #buildstream | 09:10 | |
*** tiagogomes has quit IRC | 09:28 | |
*** traveltissues has joined #buildstream | 09:39 | |
gitlab-br-bot | BenjaminSchubert opened issue #1128 (Exception thrown when retrying a task) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1128 | 09:39 |
benschubert | juergbi: are you around? Any idea why I might be seeing: | 09:40 |
benschubert | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: | 09:40 |
benschubert | status = StatusCode.UNAVAILABLE | 09:40 |
benschubert | details = "Socket closed" | 09:40 |
benschubert | debug_error_string = "{"created":"@1568021528.559529300","description":"Error received from peer unix:/tmp/buildstream2k0p4_l4/casd.sock","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Socket closed","grpc_status":14}" | 09:40 |
tpollard | benschubert: yes, I've made an issue for it | 09:41 |
tpollard | https://gitlab.com/BuildStream/buildstream/issues/1120 we should probably collate them | 09:41 |
juergbi | not the same 'details' but it's definitely close | 09:41 |
benschubert | ah agreed | 09:42 |
benschubert | this completely blew up my buildstream | 09:42 |
juergbi | would be great if manage to trigger this in a test case | 09:42 |
benschubert | ctrl+c doesn't work anymore | 09:42 |
benschubert | juergbi: trying to fetch a remote that doesn't exist and hitting 'retry' seems to have triggered for me quite a bit | 09:43 |
benschubert | but requires interactive mode | 09:43 |
juergbi | ok | 09:43 |
tpollard | yep, we can't recover from interrupt, manual or otherwise | 09:43 |
benschubert | juergbi: also thanks for your comment on !1126, that makes much more sense :) | 09:47 |
gitlab-br-bot | MR !1126: WIP: yamlcache: Lookup files in the cache without reading the file https://gitlab.com/BuildStream/buildstream/merge_requests/1126 | 09:47 |
juergbi | yw, I assume you mean #1126 | 09:48 |
gitlab-br-bot | Issue #1126: Cache resolution is slow https://gitlab.com/BuildStream/buildstream/issues/1126 | 09:48 |
benschubert | correct | 09:48 |
benschubert | juergbi: oh, another interesting emssage with BuildBox-casd: Resource exhausted. An artifact is too big for the cache... Any fix for that on the top of your head? | 09:51 |
juergbi | benschubert: that should only happen if the artifact is larger than the configured quota - or you're running out of disk space | 09:52 |
benschubert | juergbi: I haven't set a quota, let me double check for disk space | 09:53 |
*** cs-shadow has joined #buildstream | 09:54 | |
benschubert | I also have at least 200GB of disk space in my docker machine | 09:55 |
benschubert | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: | 09:55 |
benschubert | status = StatusCode.RESOURCE_EXHAUSTED | 09:55 |
benschubert | details = "Received message larger than max (4660631 vs. 4194304)" | 09:55 |
benschubert | debug_error_string = "{"created":"@1568022618.298376400","description":"Error received from peer unix:/tmp/buildstreamlzh5pivm/casd.sock","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Received message larger than max (4660631 vs. 4194304)","grpc_status":8}" | 09:55 |
juergbi | oh, I need to look into that | 09:57 |
benschubert | Do you want me to open an issue? | 09:57 |
juergbi | please do | 09:57 |
benschubert | that was triggered trying to source push part of freedesktop-sdk to my local cache | 09:57 |
juergbi | that error message is from gRPC core | 09:57 |
*** tiagogomes has joined #buildstream | 09:59 | |
juergbi | probably failing to split a gRPC request with tons of blobs | 09:59 |
gitlab-br-bot | BenjaminSchubert opened issue #1129 (Error when src-push to a remote: Message larger than max) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1129 | 10:01 |
benschubert | juergbi: ^ here you go! | 10:01 |
juergbi | ta | 10:01 |
benschubert | let me know if you need more logs/etc | 10:02 |
juergbi | will do but it's probably enough | 10:03 |
*** ahmed89_ has joined #buildstream | 10:13 | |
*** ahmed89 has quit IRC | 10:14 | |
*** ahmed89_ is now known as ahmed89 | 10:14 | |
* tlater[m] wonders why our master documentation is still tagged as 1.3.0+something | 10:14 | |
juergbi | we should make a 1.90.0 tag to fix this | 10:15 |
benschubert | Could someone give me merge rights on bst-plugins-experimental by any chance? :) | 10:26 |
jjardon | tlater[m]: last tag available in master branch | 10:27 |
jjardon | juergbi: yes please :) | 10:27 |
gitlab-br-bot | BenjaminSchubert approved MR !1587 (jennis/update_python_deps->master: Update all Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1587 | 10:27 |
tlater[m] | jjardon: I know, just wanted to know why we hadn't made another tag yet ;) | 10:28 |
juergbi | benschubert: done | 10:29 |
benschubert | thanks juergbi ! | 10:29 |
benschubert | coldtom: I've set https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/28 to be merged. Let me know once you rebased, I'll review your PR for tox so we can finally get it in :) | 10:30 |
benschubert | additionally, that might fix our nightly tests | 10:30 |
qinusty | :D Good news tlater[m], good job on the merge | 10:44 |
tlater[m] | :D | 10:45 |
traveltissues | can i please get a review for !1588 | 10:52 |
gitlab-br-bot | MR !1588: use show_default https://gitlab.com/BuildStream/buildstream/merge_requests/1588 | 10:52 |
gitlab-br-bot | BenjaminSchubert approved MR !1588 (traveltissues/1110->master: use show_default) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1588 | 10:54 |
traveltissues | ty benschubert | 10:54 |
tlater[m] | jennis: Mind taking another quick look at this? https://gitlab.com/BuildStream/docs-website/merge_requests/3/diffs?diff_id=54243600&start_sha=5fd92cf3ef6263a9cb4a59ef5bb95ffa18c3f338 | 10:55 |
jennis | tlater[m], sure will be in 30 mins or so though | 10:55 |
tlater[m] | np, just scared of the big blue button ;) | 10:56 |
tlater[m] | Ok, I ended up cleaning that up a *tiny* bit after all: https://gitlab.com/BuildStream/docs-website/merge_requests/3/diffs?diff_id=54244555&start_sha=5fd92cf3ef6263a9cb4a59ef5bb95ffa18c3f338 | 11:01 |
tlater[m] | -> replaced a hardcoded path | 11:02 |
gitlab-br-bot | coldtom opened MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1589 | 11:02 |
gitlab-br-bot | BenjaminSchubert approved MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1589 | 11:03 |
*** ahmed89 has quit IRC | 11:13 | |
tlater[m] | Oh, this is cute, our 1.0 docs still use the old theme: https://docs.buildstream.build/1.0.1/index.html | 12:23 |
coldtom | ty benschubert, https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/27 is rebased and ready for another round of review | 12:26 |
benschubert | coldtom: I'll atake a look now, thanks! | 12:31 |
gitlab-br-bot | marge-bot123 closed issue #1110 (use `show_default` for Click.options) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1110 | 12:46 |
gitlab-br-bot | marge-bot123 merged MR !1588 (traveltissues/1110->master: use show_default) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1588 | 12:46 |
gitlab-br-bot | tpollard approved MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1589 | 13:13 |
* tlater[m] is looking at https://gitlab.com/BuildStream/buildstream/issues/1068 and feels like it isn't all that useful in the end | 13:15 | |
tlater[m] | I think it might be better to individually review each and every subcommand | 13:15 |
tlater[m] | And instead track the per-subcommand progress in that issue | 13:16 |
tlater[m] | This would probably make feeding back into the design principles discussion easier too | 13:16 |
tlater[m] | Would anyone object to my creating an issue for each subcommand? | 13:17 |
* tlater[m] was about to mailing list that question, but it seems like a relatively trivial thing | 13:17 | |
gitlab-br-bot | traveltissues closed issue #1123 (test failures on wsl) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1123 | 13:21 |
traveltissues | ty for the help with #1123 juergbi | 13:22 |
gitlab-br-bot | jennis opened MR !1590 (jennis/update_source_checkout->master: Ensure `source checkout` is symmetric to `artifact checkout`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1590 | 13:24 |
jennis | tlater[m], I agree with you | 13:29 |
jennis | That's a good idea | 13:29 |
tlater[m] | I also think we should start this off with a big ML thread for each subcommand | 13:30 |
jennis | yeah | 13:30 |
tlater[m] | So that we can gather use cases and come up with a design from scratch | 13:30 |
gitlab-br-bot | marge-bot123 merged MR !1589 (coldtom/fix-status-job-render->master: _frontend/status.py: Fix non-formatted terminator) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1589 | 13:49 |
Kinnison | jennis: 👍 to !1587 but it needs a rebase | 14:02 |
gitlab-br-bot | MR !1587: Update all Python dependencies https://gitlab.com/BuildStream/buildstream/merge_requests/1587 | 14:02 |
gitlab-br-bot | jennis opened issue #1130 (Discuss and document policy for updating BuildStream's Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1130 | 14:04 |
jennis | Kinnison, yep. Also opened #1130 as a followup | 14:07 |
*** phoenix has joined #buildstream | 14:07 | |
Kinnison | nice | 14:07 |
Kinnison | jennis: probably worth posting to the ML about that | 14:08 |
Kinnison | in case people don't notice it | 14:08 |
benschubert | Is it me or the CI runners is flaky again? | 14:08 |
benschubert | (the bastions-runners-overnight) | 14:09 |
*** phoenix has quit IRC | 14:13 | |
gitlab-br-bot | traveltissues opened issue #1131 (Error loading elements for freedesktop-sdk) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1131 | 14:33 |
gitlab-br-bot | marge-bot123 merged MR !1587 (jennis/update_python_deps->master: Update all Python dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1587 | 14:49 |
valentind | benschubert, Do you care about that CI of your branch on Freedesktop SDK? Maybe you can just disable it. We have bst master docker images, though. But if you run your own CI for it, maybe just disable the one on Freedesktop SDK. | 14:52 |
benschubert | valentind: oh I don't care no, sorry about that | 14:55 |
benschubert | how do I disable it? | 14:55 |
benschubert | it was just in order to update jjardon's bst2 branch | 14:55 |
*** phildawson has quit IRC | 14:59 | |
*** phildawson has joined #buildstream | 14:59 | |
*** phildawson has quit IRC | 14:59 | |
Kinnison | jennis: 👍 on !1590 -- looks like a good step to closer UX | 14:59 |
gitlab-br-bot | MR !1590: Ensure `source checkout` is symmetric to `artifact checkout` https://gitlab.com/BuildStream/buildstream/merge_requests/1590 | 15:00 |
benschubert | ^ now only missing bst source push :D | 15:00 |
Kinnison | heh | 15:00 |
jennis | think tlater[m] has a branch for that? | 15:02 |
tlater[m] | Umm, yes | 15:02 |
tlater[m] | Though it's heavily WIP because I basically wrote the code to get something to use for manual smoke testing | 15:02 |
*** phildawson has joined #buildstream | 15:04 | |
benschubert | tlater[m]: for bst source push? :) | 15:06 |
*** phildawson_ has joined #buildstream | 15:49 | |
*** phildawson has quit IRC | 15:50 | |
jennis | benschubert: https://gitlab.com/BuildStream/buildstream/merge_requests/1561 | 16:02 |
benschubert | is it ready for review? | 16:07 |
benschubert | status = StatusCode.UNAVAILABLE | 16:10 |
benschubert | details = "failed to connect to all addresses" | 16:10 |
benschubert | What could be the cause of this in the push queue? Roughly half of my elements fail to be pushed like that | 16:10 |
jennis | Nope, "heavily WIP" | 16:11 |
juergbi | benschubert: I assume the CAS server is not overloaded. does it retry and fail 3 times? | 16:17 |
juergbi | also, it's not clear whether this is an issue connecting to casd or to the remote CAS | 16:17 |
benschubert | juergbi: it doesn't retry as it's an unhandled exception. The cas server has a single client, so I hope it's not overloaded | 16:18 |
benschubert | it's in 'get_local_cas' | 16:19 |
juergbi | Linux or WSL? | 16:19 |
benschubert | juergbi: https://gitlab.com/snippets/1893432 | 16:19 |
benschubert | inside a ubuntu 19.04 container | 16:19 |
benschubert | and my whole build completely dead locked with that | 16:20 |
juergbi | ok :-/ | 16:20 |
juergbi | I'm wondering whether it could be an issue on casd side or why else a local socket connection would fail | 16:21 |
valentind | benschubert, you can just remove .gitlab-ci.yml. | 16:22 |
benschubert | juergbi: oh, the exception seems to happen after the first try failed, when doing the second try | 16:32 |
juergbi | benschubert: interesting, so it might be related to retry failing in interactive mode | 16:33 |
benschubert | possible yes | 16:33 |
benschubert | even though this time I had '--on-error=continue' | 16:34 |
juergbi | although, I really wouldn't expect any connection failures with UNIX sockets - unless the service crashed or similar | 16:34 |
benschubert | which seems possible. Any ways I can check that? | 16:34 |
juergbi | benschubert: besides checking with `ps` you could add a self._casd_process.is_alive() to _get_local_cas() | 16:36 |
juergbi | if it's a real segfault, it might also show up in system log and you might get a coredump - depending on system configuration | 16:36 |
benschubert | I do have a 'buildbox-casd' subprocess | 16:36 |
juergbi | and it's not a zombie? | 16:36 |
benschubert | buildst+ 69156 27.5 0.0 0 0 pts/0 Z+ 16:26 3:31 [buildbox-casd] <defunct> | 16:39 |
benschubert | It... is not in a good shape -_- | 16:39 |
tlater[m] | benschubert: I don't really have the time to get `bst source pushll` over the line atm :( | 16:39 |
traveltissues | is there any instance where it can be expected that a parent process will receive NoneType child process data? | 16:39 |
benschubert | tlater[m]: no worries, I was just curious :) | 16:39 |
traveltissues | in terms of the scheduler | 16:40 |
benschubert | Is it expected that we would call 'get_local_cas' roughly 50-100 times between 'SUCCESS: caching' and 'SUCCESS build'? | 16:49 |
traveltissues | idk but that seems wrong | 16:50 |
tlater[m] | I'm looking at issue #1093 - does anyone know whether our plugins can already send "normal" status reports without text? | 16:52 |
gitlab-br-bot | Issue #1093: Add progress reporting as a generic string https://gitlab.com/BuildStream/buildstream/issues/1093 | 16:52 |
juergbi | benschubert: right now, yes. we're currently adding files one by one. batching those calls (or rather, capture the whole directory tree in one call) is a planned optimization | 16:52 |
juergbi | benschubert: it should only attempt to connect once per job child process, though, and then stay connected | 16:52 |
juergbi | (i.e., the other times it should hit the fast path) | 16:52 |
tlater[m] | I feel like #1092 says they can't, but #1093 doesn't refer to it | 16:53 |
gitlab-br-bot | Issue #1092: Add progress reporting across jobs in subprocesses https://gitlab.com/BuildStream/buildstream/issues/1092 | 16:53 |
benschubert | yep, I was just suprised, I added a print in get_local_cas, and it ended up printing almost only that | 16:53 |
tpollard | tlater[m]: could we just use the our own Message()? | 16:54 |
tlater[m] | tpollard: The idea is that we have a continuous ticker | 16:54 |
tlater[m] | For when multiple things happen at the same time | 16:55 |
tlater[m] | #1093 wants to always display the last line from build output | 16:55 |
tlater[m] | As part of a task | 16:55 |
tlater[m] | The little menu at the bottom that shows all the fetches/builds/tracks currently happening :) | 16:55 |
tpollard | yep | 16:56 |
tlater[m] | We want builds to be able to display the last line of their output there. | 16:56 |
tlater[m] | tpollard: So an arbitrary plugin *can* currently send things to the frontend already? | 16:56 |
tpollard | all plugins have access to info() | 16:57 |
tpollard | not exactly what we'd want here, but it's a case where we enable it | 16:58 |
tpollard | and status() of course | 16:58 |
tpollard | and in terms of subprocessing, I think the Message() should be picklable now | 16:59 |
tlater[m] | Well, I'm wondering how much API exists for accessing our tasks shenanigans | 16:59 |
tlater[m] | But message isn't nearly enough for that, is it? | 16:59 |
benschubert | juergbi: ok, so the casd server doesn't crash | 17:02 |
benschubert | juergbi: however, GetCapabilitiesRequest() fails to connect to all addresses... | 17:02 |
benschubert | Do we have to close requests ourselves? are we leaking something? | 17:03 |
benschubert | That seems like a leak and it fails once we opened too many connections | 17:03 |
*** phoenix has joined #buildstream | 17:05 | |
juergbi | benschubert: on the bst client side I think it's unlikely to be a leak because it's handled in job subprocess and an potential leak would be cleaned up on subprocess termination | 17:09 |
tpollard | tlater[m]: I'm not 100% upto date on the State() side of things sadly | 17:10 |
juergbi | there could be a leak on the casd side | 17:10 |
benschubert | juergbi: mmh, not sure how to start debugging this | 17:12 |
tpollard | tlater[m]: I've tried to ensure that Messenger() will still work when process separation happens, but I do worry there's extra's in State() that also need adapting | 17:14 |
tlater[m] | Yeah, I suspect State() is going to be a bit of a nightmare with this stuff | 17:14 |
* tlater[m] will need to take a deep dive to figure out how hard these things are... | 17:15 | |
benschubert | jjardon: Are those kind of errors expected, or would that be a bug in the plugins: https://gitlab.com/BuildStream/buildstream/-/jobs/291029431 ? I'm trying to fix the nightly tests | 17:15 |
juergbi | benschubert: #1120 sounds like it might be easy to reproduce. I should probably start debugging that and maybe it has the same root cause as what you're seeing | 17:16 |
gitlab-br-bot | Issue #1120: Interrupt handling during a task that involves cascache grpc processes is very flaky https://gitlab.com/BuildStream/buildstream/issues/1120 | 17:16 |
juergbi | first step would probably be to get it into the test suite | 17:16 |
jjardon | benschubert: I have never seen that error before | 17:16 |
benschubert | juergbi: let me know if you need more info | 17:17 |
juergbi | ok, will do | 17:17 |
juergbi | benschubert: do you use a very large number of job subprocesses? | 17:17 |
juergbi | does it happen also with few builders/pushers | 17:17 |
benschubert | the default | 17:17 |
juergbi | ok, so not that many, odd | 17:17 |
jjardon | Migth be because some changes on master, not sure sorry | 17:17 |
benschubert | and trying to build freedesktop/base-platform.bst is enough to trigger it | 17:18 |
benschubert | jjardon: no problem, thanks! If I update the freedesktop pinned version we are using in the tests, would that be fine with you? | 17:18 |
juergbi | benschubert: only if you have a push artifact server configured or even purely local? | 17:18 |
benschubert | juergbi: haven't tried in purely local | 17:18 |
benschubert | let me check | 17:18 |
jjardon | benschubert: sure | 17:18 |
juergbi | benschubert: btw: don't you see is_single_threaded() assertion failures due to ostree? | 17:19 |
benschubert | juergbi: none | 17:19 |
juergbi | bst-plugins-experimental master? | 17:19 |
benschubert | yep | 17:20 |
juergbi | wondering whether this might depend on ostree/gobject version | 17:20 |
*** phoenix has quit IRC | 17:28 | |
*** jonathanmaw has quit IRC | 17:29 | |
benschubert | juergbi: the problem seems to not be happening when no cache is available | 17:34 |
benschubert | I'm still running the build just in case, but when pushing I was never able to build that many elements | 17:34 |
juergbi | ok, as you've seen the issue for build jobs, not push jobs, I still suspect the issue is on the casd side | 17:37 |
juergbi | but don't have an idea what exactly is going wrong there | 17:37 |
juergbi | might make sense to run casd with valgrind | 17:37 |
benschubert | good point I'll try this setup | 17:37 |
benschubert | Does casd writes logs? | 17:38 |
juergbi | stderr | 17:38 |
juergbi | you can increase verbosity | 17:38 |
juergbi | --verbose | 17:38 |
juergbi | or for the maximum --log-level=trace | 17:38 |
benschubert | but we are not storing them anywhere. Would it make sense to save them somewhere and add configuration for it in buildstream.conf ? | 17:39 |
juergbi | not sure, do we support this for general buildstream logs right now? | 17:40 |
juergbi | iirc, we only store element-specific logs in the artifact | 17:40 |
benschubert | well, we store the messages | 17:40 |
benschubert | coldtom: fine if I merge https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/27 ? :) | 18:01 |
coldtom | benschubert: absolutely fine, glad it's finally in a fit state to merge :) | 18:05 |
*** traveltissues has quit IRC | 18:17 | |
*** phildawson_ has quit IRC | 18:21 | |
benschubert | coldtom: I can imagine, thanks a lot for the hard work :) | 18:28 |
benschubert | juergbi: looking at logs, the only thing I can see (once) is : E0909 20:34:53.104015208 18690 client_context.cc:85] assertion failed: call_ == nullptr | 20:35 |
*** rdale has quit IRC | 20:36 | |
*** cs-shadow has quit IRC | 21:24 | |
*** phoenix has joined #buildstream | 21:34 | |
*** phoenix has quit IRC | 21:59 | |
*** swick has quit IRC | 22:16 | |
*** swick has joined #buildstream | 22:22 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!