| *** slaf has joined #buildstream | 00:04 | |
| *** slaf has joined #buildstream | 00:06 | |
| *** slaf has joined #buildstream | 00:08 | |
| *** dylan-m_ has quit IRC | 00:59 | |
| *** dylan-m_ has joined #buildstream | 02:32 | |
| *** dylan-m_ has quit IRC | 02:37 | |
| *** misterwhatever has quit IRC | 02:45 | |
| *** kapip has joined #buildstream | 02:53 | |
| *** dylan-m_ has joined #buildstream | 04:53 | |
| *** dylan-m_ has quit IRC | 05:08 | |
| *** traveltissues has joined #buildstream | 08:50 | |
| *** rdale has joined #buildstream | 09:01 | |
| coldtom | anyone seen push errors like this before? the remote is a bst-artifact-server https://gitlab.com/celduin/burn-sdk/-/jobs/360715846 | 09:05 |
|---|---|---|
| coldtom | it's consistently those artifacts that have trouble too | 09:08 |
| Kinnison | Do you have anything visible in logs on the server? | 09:13 |
| coldtom | afaict there aren't any logs for that job.... | 09:17 |
| *** bochecha has joined #buildstream | 09:18 | |
| Kinnison | A brief googling points the finger for "Unexpected error in RPC handling" at potentially bad error handling server-side | 09:19 |
| benschubert | are the server and client on the same version? | 09:24 |
| coldtom | both are using master, but i doubt it's the same commit /o\ | 09:25 |
| coldtom | that could be it, i'll try and run with the same bst version | 09:26 |
| Kinnison | Also check that buildbox-casd is consistent | 09:28 |
| benschubert | and do you have a proxy between both? (Like nginx on the server machine?) | 09:28 |
| coldtom | i think that the client is talking directly to bst-artifact-server running inside of docker | 09:30 |
| Kinnison | So your build system has a server and the client in the same docker instance? | 09:31 |
| coldtom | the server has a bst-artifact-server running inside docker, the client is on an entirely different machine, sorry, i should have been clearer there | 09:33 |
| *** SotK has quit IRC | 09:35 | |
| benschubert | Ok, then it's probably not somet4hing due to the size of requests, I had this error for that some times ago | 09:35 |
| gitlab-br-bot | juergbi opened issue #1218 (Source files are read-only hard links with buildbox-run) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1218 | 09:36 |
| *** SotK has joined #buildstream | 09:46 | |
| Kinnison | Is there a page/repo/wiki/whatever which carries the project's official images and rights of re-use etc? | 10:00 |
| benschubert | Is marge having problems again: https://gitlab.com/BuildStream/buildstream/merge_requests/1733#note_250442070 ? | 10:04 |
| Kinnison | gitlab itself is being super-slow for me today | 10:05 |
| Kinnison | could it be related? | 10:05 |
| benschubert | that could yes | 10:05 |
| gitlab-br-bot | traveltissues opened issue #1219 (SAST not detecting correct analyzer) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1219 | 10:11 |
| *** jonathanmaw has joined #buildstream | 10:15 | |
| benschubert | juergbi: tlater[m] : is there a reason the sourcecache is on the element and not the source? | 10:20 |
| coldtom | hmm so using the same commit of bst doesn't help me, and i'm *fairly* sure that i'm using the same version of buildbox-casd | 10:20 |
| benschubert | coldtom: can you reproduce locally? like spin up a docker locally and try it? | 10:21 |
| juergbi | benschubert: for elements with multiple sources, the combined source is (also) cached | 10:22 |
| benschubert | I see, thanks, that will complexify refactoring all of this, oh well :) | 10:22 |
| *** narispo has quit IRC | 10:24 | |
| *** narispo has joined #buildstream | 10:24 | |
| benschubert | juergbi: also, '_source_cached' on element is specifically about sources being in the source cache right? | 10:24 |
| juergbi | yes | 10:28 |
| *** lachlan has joined #buildstream | 10:32 | |
| gitlab-br-bot | traveltissues opened (was WIP) MR !1734 (traveltissues/tmp->master: .gitlab-ci.yml: define repo language as python) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1734 | 10:33 |
| gitlab-br-bot | BenjaminSchubert approved MR !1734 (traveltissues/tmp->master: .gitlab-ci.yml: define repo language as python) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1734 | 10:34 |
| benschubert | Should we start merging manually since nothing is working? | 10:48 |
| traveltissues | what's wrong with marge? | 10:50 |
| *** akvilebirgelyte__ has quit IRC | 10:51 | |
| benschubert | gitlab is timing out on the API | 10:51 |
| benschubert | so marge is just giving us back an error | 10:51 |
| traveltissues | right | 10:53 |
| traveltissues | merging is fine by me in this case | 10:53 |
| *** lachlan has quit IRC | 10:55 | |
| benschubert | Ok, I'll merge mine and then feel free to do yours :) | 10:55 |
| gitlab-br-bot | BenjaminSchubert merged MR !1733 (bschubert/track-cleanup->master: _stream.py: Remove dead code and comments about tracking) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1733 | 10:55 |
| *** lachlan has joined #buildstream | 10:59 | |
| *** cs-shadow has joined #buildstream | 11:01 | |
| traveltissues | the windows runner is down | 11:02 |
| *** lachlan has quit IRC | 11:14 | |
| traveltissues | should be unstuck now | 11:14 |
| *** lachlan has joined #buildstream | 11:23 | |
| *** phildawson_ has joined #buildstream | 11:30 | |
| *** phildawson has quit IRC | 11:32 | |
| *** lachlan has quit IRC | 11:35 | |
| coldtom | aha, reproduced locally, but only using an old docker image for the artifact server | 11:38 |
| coldtom | i think an update should do it | 11:38 |
| gitlab-br-bot | traveltissues closed issue #1219 (SAST not detecting correct analyzer) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1219 | 11:39 |
| gitlab-br-bot | traveltissues merged MR !1734 (traveltissues/tmp->master: .gitlab-ci.yml: define repo language as python) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1734 | 11:39 |
| *** lachlan has joined #buildstream | 11:45 | |
| *** lachlan has quit IRC | 11:51 | |
| *** lachlan has joined #buildstream | 11:53 | |
| *** phildawson_ has quit IRC | 11:54 | |
| *** lachlan has quit IRC | 12:00 | |
| *** phildawson_ has joined #buildstream | 12:02 | |
| *** phildawson_ has quit IRC | 12:04 | |
| *** phildawson_ has joined #buildstream | 12:04 | |
| tlater[m] | > FAILURE hello-junction.bst:hello.bst: tar source at hello-junction.bst:hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.1.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)> | 12:08 |
| * tlater[m] wonders what's going on with ftpmirror.gnu.org | 12:08 | |
| tlater[m] | I can happily download locally | 12:10 |
| juergbi | maybe at least one mirror is still flaky and handshake failure is sporadic? | 12:10 |
| *** lachlan has joined #buildstream | 12:14 | |
| coldtom | ...well updating the server got me a different error message, so that's progress | 12:34 |
| juergbi | coldtom: a useful error message? | 12:36 |
| coldtom | it's just the number 13 | 12:37 |
| Kinnison | somniloquy% errno 13 | 12:37 |
| Kinnison | EACCES 13 Permission denied | 12:37 |
| Kinnison | Maybe something wrong with where the cas is vs. what user it's running as? | 12:37 |
| *** lachlan has quit IRC | 12:52 | |
| coldtom | https://paste.gnome.org/pxotjzssw server logs suggest there's a digest mismatch? | 12:53 |
| Kinnison | 611 megs vs 268 megs | 12:56 |
| Kinnison | those are some big files | 12:56 |
| Kinnison | I wonder if the stream dies midway | 12:56 |
| coldtom | i'm pushing an artifact of openjdk which would explain the large files | 12:56 |
| * coldtom wonders if it is related to artifact size | 12:57 | |
| coldtom | i think the problem in the failed pipeline was gcc, which i suspect also has some big files | 12:58 |
| juergbi | it seems individual blobs are fairly large: 611.7 MB | 12:59 |
| juergbi | not just the whole artifact | 12:59 |
| juergbi | and only 268.4 MB are processed | 13:00 |
| Kinnison | coldtom: OOI does that container have enough disk space? | 13:00 |
| Kinnison | coldtom: in /data/tmp | 13:00 |
| coldtom | /data should be big enough (300G, so i'd hope so), and i think /data/tmp is just a directory | 13:04 |
| *** phildawson_ has quit IRC | 13:06 | |
| *** phildawson_ has joined #buildstream | 13:06 | |
| juergbi | coldtom: have you also checked the currently available disk space? expiry/cleanup might be triggered too late with large files | 13:10 |
| coldtom | there's 298G free | 13:11 |
| juergbi | as this is the server side, https://gitlab.com/BuildStream/buildstream/merge_requests/1645 probably affects the behavior | 13:11 |
| coldtom | the failure seems to happen on multiples of megabytes if that helps | 13:11 |
| *** lachlan has joined #buildstream | 13:12 | |
| juergbi | hm, the failing size is exactly 0x10000000 | 13:12 |
| juergbi | suspicious | 13:12 |
| *** phildawson_ has quit IRC | 13:13 | |
| coldtom | there are other failures at 64M and 128M | 13:13 |
| juergbi | tlater[m]: as you've been working on the server side, maybe you could look into this issue? | 13:14 |
| juergbi | it might even already be fixed by your branch, but trying to reproduce this with master and a large file would be good | 13:15 |
| tlater[m] | juergbi/coldtom: I'll take a look | 13:16 |
| juergbi | ta | 13:16 |
| *** lachlan has quit IRC | 13:19 | |
| coldtom | i can't seem to reproduce on my host machine (although i'm using http not https) | 13:27 |
| * coldtom will try locally with https after lunch | 13:28 | |
| benschubert | coldtom: how much ram? | 13:31 |
| benschubert | I used to be able to push files >40Go/file | 13:31 |
| *** phildawson_ has joined #buildstream | 14:05 | |
| coldtom | ahh, server has 1G of ram | 14:07 |
| coldtom | that could be the issue | 14:07 |
| juergbi | coldtom: it might indeed. we fixed a few issues with excessive memory consumption but it's possible that we still require more RAM than we should for large files | 14:09 |
| tlater[m] | Would explain the failing size being exactly 1G :D | 14:10 |
| juergbi | might be memory allocation failures that we don't handle well at all | 14:11 |
| juergbi | might be more an issue on the casd side than buildstream | 14:12 |
| tlater[m] | coldtom: Can you reproduce it if you set a memory limit on the artifact server process? | 14:13 |
| tlater[m] | Locally, that is | 14:13 |
| tlater[m] | (since you apparently already have everything in place to test :)) | 14:14 |
| *** lachlan has joined #buildstream | 14:35 | |
| *** lachlan has quit IRC | 14:41 | |
| tlater[m] | juergbi: Hm, it looks like the request we get from old clients contains only a key, which we can't forward as a digest, because we don't have the size_bytes | 14:49 |
| tlater[m] | Any idea how to convert between a key and a digest when we potentially don't have the proto locally, in the case of GetReference? | 14:50 |
| juergbi | tlater[m]: hm, didn't we store the whole digest in the old format? | 14:50 |
| juergbi | in what method/line do you see an issue? | 14:51 |
| tlater[m] | We probably stored it in the old format, but the old client doesn't use the full digest when talking to GetReference | 14:51 |
| tlater[m] | One sec | 14:51 |
| tlater[m] | juergbi: In this instance request.key is just a key, without the size_bytes: https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_cas/casserver.py#L405 | 14:52 |
| tlater[m] | Ah, I see, we used to resolve_ref there | 14:52 |
| tlater[m] | I suppose we still need to do that, can't get around it with FetchTree | 14:52 |
| juergbi | that's the buildstream cache key, that never has a size | 14:53 |
| juergbi | right, resolve_ref is still needed | 14:53 |
| tlater[m] | Oh o\ | 14:53 |
| * tlater[m] thought that would be an artifact digest, but of course it isn't | 14:54 | |
| tlater[m] | Yeah, shame | 14:54 |
| valentind | I suppose the monthly IRC team meeting for buildstream is not a thing anymore. | 15:01 |
| coldtom | would you like me to raise an issue for the server-side issues i've been having? if so, in buildstream or buildbox-casd? | 15:04 |
| *** lachlan has joined #buildstream | 15:05 | |
| *** akvilebirgelyte__ has joined #buildstream | 15:08 | |
| tlater[m] | coldtom: On buildstream is probably more appropriate, given that we don't actually *know* whether it's a memory issue. | 15:09 |
| tlater[m] | (Nor where it really occurs) | 15:09 |
| gitlab-br-bot | cs-shadow opened (was WIP) MR !1700 (chandan/refactor-default-target->master: tests/frontend: Refactor tests for default targets) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1700 | 15:13 |
| Kinnison | valentind: I imagine that if people had things to discuss and were prepared to organise a meeting, then others would attend | 15:14 |
| traveltissues | anymore on !1682? | 15:15 |
| gitlab-br-bot | MR !1682: Support RE workspaces (non-incremental) https://gitlab.com/BuildStream/buildstream/merge_requests/1682 | 15:15 |
| gitlab-br-bot | traveltissues merged MR !1721 (traveltissues/incrementaltest->master: Enable the incremental workspace test in integration) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1721 | 15:15 |
| *** lachlan has quit IRC | 15:16 | |
| coldtom | giving the server more ram fixed the issue | 15:19 |
| gitlab-br-bot | traveltissues opened issue #1220 (Review results of bandit on master) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1220 | 15:19 |
| Kinnison | Oh dear | 15:19 |
| Kinnison | streaming is, in part, meant to solve that kind of thing | 15:19 |
| * coldtom gets that pipeline going again, and hopes he doesn't need to make the server even bigger | 15:20 | |
| tlater[m] | benschubert: Any final comments on !1645? | 15:22 |
| gitlab-br-bot | MR !1645: Refactor casserver.py: Stop relying on the buildstream-internal `CASCache` implementation https://gitlab.com/BuildStream/buildstream/merge_requests/1645 | 15:22 |
| gitlab-br-bot | traveltissues approved MR !1700 (chandan/refactor-default-target->master: tests/frontend: Refactor tests for default targets) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1700 | 15:30 |
| benschubert | traveltissues: let me have a look :) | 15:35 |
| benschubert | oups I mean tlater[m] sorry | 15:36 |
| benschubert | Code quality improved on 13 points and degraded on 3 points that's pretty sweet! | 15:37 |
| benschubert | tlater[m]: any benchmarks run? | 15:39 |
| tlater[m] | benschubert: No, but I doubt we have any benchmarks that actually cover the artifact server | 15:40 |
| benschubert | fair :) | 15:40 |
| * tlater[m] doesn't expect performance to change much anyway, this just removes a layer of indirection :) | 15:41 | |
| gitlab-br-bot | coldtom opened issue #1221 (bst-artifact-server fails on blobs that are "too large") on buildstream https://gitlab.com/BuildStream/buildstream/issues/1221 | 15:49 |
| gitlab-br-bot | coldtom opened issue #1222 (Frequent failures with"timeout waiting for buildbox-casd") on buildstream https://gitlab.com/BuildStream/buildstream/issues/1222 | 15:49 |
| *** lachlan has joined #buildstream | 15:50 | |
| benschubert | tlater[m]: first round, let me know once you have addressed/answered and i'll give a second round :) | 16:02 |
| *** lachlan has quit IRC | 16:04 | |
| tlater[m] | benschubert: Will do - a couple of those are on moved code, but I suppose I might as well patch that up. | 16:06 |
| *** lachlan has joined #buildstream | 16:13 | |
| benschubert | ah sorry, didn't realize. Well as you wish :) | 16:14 |
| * tlater[m] wishes magit forge was usable on large repositories | 16:26 | |
| tlater[m] | It'd be so convenient to see people's comments in my editor... | 16:26 |
| tlater[m] | Or well, maybe my wish is that gitlab's API for these things was nearly as fast as github's | 16:27 |
| *** bochecha has quit IRC | 16:30 | |
| gitlab-br-bot | traveltissues closed issue #1213 (Compose elements cannot run integration commands) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1213 | 16:35 |
| gitlab-br-bot | traveltissues merged MR !1722 (valentindavid/compose-integration-commands->master: compose: Allow compose elements to run integration commands) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1722 | 16:35 |
| *** narispo has quit IRC | 16:44 | |
| *** phildawson has joined #buildstream | 17:31 | |
| *** phildawson_ has quit IRC | 17:32 | |
| *** traveltissues has quit IRC | 17:40 | |
| *** jonathanmaw has quit IRC | 18:04 | |
| *** lachlan has quit IRC | 18:06 | |
| gitlab-br-bot | cs-shadow opened (was WIP) MR !1716 (chandan/glossary->master: doc: Add glossary of common terms) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1716 | 18:54 |
| *** rdale has quit IRC | 19:02 | |
| gitlab-br-bot | cs-shadow opened MR !1735 (chandan/hacking/drop-line-length->master: doc/coding_guidelines: Drop section about line lengths) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1735 | 19:15 |
| *** phildawson has quit IRC | 19:23 | |
| *** bochecha has joined #buildstream | 21:45 | |
| *** bochecha has joined #buildstream | 21:56 | |
| *** kapip has quit IRC | 22:02 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!