| *** phoenix has quit IRC | 00:01 | |
| jjardon | benschubert: buildstream should be compatible with buildbarn to use it as a remote cache. It will not work if you use it for remote execution (yet) | 01:10 |
|---|---|---|
| *** tristan has joined #buildstream | 04:01 | |
| *** ChanServ sets mode: +o tristan | 04:01 | |
| *** tristan has quit IRC | 05:17 | |
| *** tristan has joined #buildstream | 06:57 | |
| *** mohan43u has quit IRC | 07:04 | |
| *** phil has joined #buildstream | 07:04 | |
| *** mohan43u has joined #buildstream | 07:10 | |
| *** mohan43u has quit IRC | 07:21 | |
| *** mohan43u has joined #buildstream | 07:21 | |
| *** ahmed89 has joined #buildstream | 08:08 | |
| gitlab-br-bot | marge-bot123 merged MR !1609 (coldtom/mark-filter-test-integration->master: tests: Add some missing integration marks) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1609 | 08:16 |
| benschubert | jjardon: oh great, thanks! | 08:19 |
| *** bochecha has joined #buildstream | 08:38 | |
| *** tiagogomes has joined #buildstream | 08:44 | |
| *** tristan has quit IRC | 08:59 | |
| *** traveltissues has joined #buildstream | 09:04 | |
| *** tristan has joined #buildstream | 09:07 | |
| *** tpollard has joined #buildstream | 09:08 | |
| benschubert | juergbi: question about the artifactserver, the way we store our artifact proto is not cas-based correct? Where would be the best place to look to understand how it works? | 09:15 |
| juergbi | benschubert: correct, the protos are stored as simple files outside CAS, the relative path matches the artifact cache key | 09:16 |
| benschubert | awesome, thanks! | 09:17 |
| benschubert | and the artifact proto is stored as the content of the file, correct? | 09:18 |
| juergbi | I don't think there is comprehensive documentation about this. relevant code is in _artifactcache.py and casserver.py | 09:18 |
| juergbi | yes | 09:18 |
| juergbi | and fields in that proto file point to CAS digests | 09:18 |
| benschubert | thanks! | 09:18 |
| *** raoul has joined #buildstream | 09:27 | |
| *** lachlan has joined #buildstream | 09:28 | |
| *** jonathanmaw has joined #buildstream | 09:46 | |
| benschubert | juergbi: (sorry for the amount of questions, I'm working on a talk for next week and want to make sure I'm not saying wrong stuff), For RE, does BuildStream need to download all sources and dependencies locally before launching the build? Or does it try first without uploading anything, and only download then reuploads the missing parts? | 09:53 |
| juergbi | benschubert: regarding dependencies, it initially downloads only the Directory protos, not the actual blobs. missing blobs are downloaded from the artifact server and then uploaded to RE CAS as needed right before sending the Action | 09:57 |
| benschubert | juergbi: right before sending the action or whenever we get a PRECONDITION_FAILED? | 09:58 |
| juergbi | the same should apply to sources, however, I forgot whether we actually got around implementing it | 09:59 |
| benschubert | also, what's the directory protos? I'm not sure I heard this one yet :/ | 09:59 |
| juergbi | we always call FindMissingBlobs before sending the action, so we do it in response of that | 09:59 |
| juergbi | given that we construct the staged tree for each action, it's unlikely that e.g. the toplevel directory is already on the server | 10:00 |
| juergbi | i.e., we very likely anyway have to upload a few blobs | 10:00 |
| juergbi | Directory is the CAS proto that describes directory entries (subdirectories and files) | 10:01 |
| benschubert | So the protocol is the following: fetch source proto if not available -> fetch tree proto for sources -> fetch artifact proto for build dependencies -> fetch tree proto for each dependency artifact -> merge both source and artifact input iinto a new input tree -> findMissingBlob() -> download then upload all missing -> create REAPI Action -> submit to RE server | 10:01 |
| benschubert | is that accurate? | 10:02 |
| juergbi | that's pretty close | 10:03 |
| benschubert | what's missing? :) | 10:03 |
| juergbi | we create the REAPI Action a bit earlier and check the action cache if one is configured. although that's probably not helpful typically | 10:03 |
| juergbi | and we only work with Directory protos in BuildStream, not complete Trees | 10:04 |
| benschubert | So REAPI Action is called before the findMissingBlob() correct? | 10:04 |
| benschubert | ok, so we fetch the directory protos instead of the tree protos? | 10:04 |
| juergbi | if the optional action cache endpoint is configured, we query that before findMissingBlobs | 10:05 |
| juergbi | however, this should probably normally not even be configured, assuming the RE server checks it internally | 10:05 |
| juergbi | (one less round trip) | 10:06 |
| benschubert | I see, thanks a lot | 10:07 |
| benschubert | I think I'll omit this part from the slides, but the rest is much clearer now, thanks | 10:07 |
| juergbi | we use Tree only for action output but otherwise plain Directory protos | 10:08 |
| juergbi | (Tree is anyway just a combination of Directory protos) | 10:09 |
| benschubert | Ok, so we combine the directory protos of each source and put them in a single one | 10:11 |
| *** tristan has quit IRC | 10:17 | |
| *** tristan has joined #buildstream | 10:24 | |
| *** lachlan has quit IRC | 11:10 | |
| *** lachlan has joined #buildstream | 11:17 | |
| *** lachlan has quit IRC | 11:31 | |
| *** narispo has joined #buildstream | 12:45 | |
| *** lachlan has joined #buildstream | 13:10 | |
| *** lachlan has quit IRC | 13:18 | |
| *** lachlan has joined #buildstream | 13:19 | |
| *** lachlan has quit IRC | 13:31 | |
| *** lachlan has joined #buildstream | 13:40 | |
| benschubert | File "/usr/local/lib/python3.7/dist-packages/buildstream/source.py", line 637, in mark_download_url | 13:40 |
| benschubert | "Primary URL marked twice with different URLs" | 13:40 |
| benschubert | AssertionError: Primary URL marked twice with different URLs | 13:40 |
| benschubert | Uh? How would I go debugging this? | 13:40 |
| *** lachlan has quit IRC | 13:48 | |
| *** lachlan has joined #buildstream | 13:50 | |
| juergbi | benschubert: is this with an upstream plugin? do you have a backtrace? | 13:51 |
| coldtom | i've seen that a few times with the git plugin i think, but i can't recall what the issue was | 13:55 |
| coldtom | maybe due to submodules iirc | 13:55 |
| benschubert | Yep, git plugin when trying to open a workspace | 13:55 |
| benschubert | And yep it has submodules | 13:55 |
| cs-shadow | do the submodules (including transitive ones) point to the same repo by any chance? | 13:56 |
| benschubert | That is possible, I have 15 submodules at the first level | 13:56 |
| benschubert | https://gitlab.com/snippets/1897698 for the stack | 13:57 |
| *** lachlan has quit IRC | 14:02 | |
| *** ahmed89 has quit IRC | 14:14 | |
| *** lachlan has joined #buildstream | 14:35 | |
| benschubert | ah yeah, two different submodule of the same repo | 14:56 |
| benschubert | Ok, listing explicitely all the submodules removed the problem | 15:09 |
| cs-shadow | sorry i was afk for a while, but bst could surely do a better job in this case if it's going to error out | 15:10 |
| benschubert | Yep, an assertion is definitely not what I would expect :'D | 15:10 |
| juergbi | yes, the question is whether we should catch this in core or in the git plugin | 15:13 |
| coldtom | https://gitlab.com/BuildStream/buildstream/issues/757 is a related issue to this | 15:14 |
| coldtom | is there a way i can tell BuildStream not to cache a certain element? | 15:16 |
| *** lachlan has quit IRC | 15:18 | |
| cs-shadow | juergbi: I think the source class at least should raise an error instead of asserting this. But that will likely be generic error about getting two URLs for the same primary. | 15:18 |
| cs-shadow | On top of that, the git source can perhaps raise a nicer error just before it marks all the submodule urls | 15:18 |
| tpollard | coldtom: not explicitly afaik | 15:18 |
| gitlab-br-bot | BenjaminSchubert opened issue #1142 (Autocompletion of artifacts is throwing stack traces) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1142 | 15:19 |
| juergbi | cs-shadow: if we consider it a plugin bug, isn't assertion failure correct? maybe we should be clearer in the message we present to the user that this is a bug in a particular plugin | 15:19 |
| cs-shadow | I'm not sure. For example, assertions may be disabled in production, which would make it more difficult to catch such errors if one has buggy plugins | 15:21 |
| cs-shadow | for the messaging, I agree that we should mention that it's plugin's fault | 15:21 |
| coldtom | tpollard: ack, essentially I want to build something against a proprietary binary blob, but I'm pretty sure caching said binary blob so that I could build against it would be redistribution | 15:24 |
| * coldtom will open an issue | 15:24 | |
| juergbi | cs-shadow: if we want to move away from assertions for bugs, we'll have to discuss this as a contribution policy change and provide a new mechanism that buildstream core handles properly. and possibly switch over the whole code base at once | 15:25 |
| tpollard | coldtom: hmm, can the `exclude:` provide something like that for you? | 15:28 |
| *** lachlan has joined #buildstream | 15:31 | |
| coldtom | tpollard: perhaps not `exclude` but I think craftily downloading them as sources and staging them separately could work? I'll probably have to investigate more | 15:32 |
| cs-shadow | juergbi: fair enough, maybe asserts to exceptions is an orthogonal issue | 15:32 |
| cs-shadow | in which case, I'd say this is a plugin issue and gitsourcebase should fix this | 15:33 |
| tpollard | coldtom: potentially yeh, might have to be aware of buildtrees though (not cached by default in master though) | 15:33 |
| gitlab-br-bot | traveltissues opened (was WIP) MR !1563 (traveltissues/985->master: workspaces via sourcecache 1) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1563 | 15:54 |
| *** lachlan has quit IRC | 15:55 | |
| *** lachlan has joined #buildstream | 16:00 | |
| *** bochecha has quit IRC | 16:29 | |
| *** lachlan has quit IRC | 16:35 | |
| *** lachlan has joined #buildstream | 16:52 | |
| laurence | juergbi, just doing some old ticket tidy up - do you think we can close off https://gitlab.com/BuildStream/buildstream/issues/866 ? | 16:54 |
| juergbi | laurence: no, that's still valid | 16:56 |
| juergbi | well, partially | 16:56 |
| juergbi | you no longer get a fatal error message like that (and with the pending casd MR the quota will automatically be adjusted) | 16:57 |
| juergbi | however, the potential confusion because the quota size only applies to cas (and not the separate sources directory) still holds | 16:58 |
| gitlab-br-bot | marge-bot123 closed issue #1088 (Do not force reset workspace cache data) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1088 | 17:00 |
| gitlab-br-bot | marge-bot123 merged MR !1563 (traveltissues/985->master: workspaces via sourcecache 1) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1563 | 17:00 |
| *** lachlan has quit IRC | 17:02 | |
| juergbi | laurence: I added a comment | 17:04 |
| laurence | alright, cheers juergbi | 17:06 |
| *** traveltissues has quit IRC | 17:08 | |
| *** jonathanmaw has quit IRC | 17:10 | |
| benschubert | uh? With the latest master, my resolve time for cached state jumped from 17s to 1:40m ? On a tree of 195 elements. Is that expected? | 17:10 |
| *** tristan has left #buildstream | 17:11 | |
| *** tristan has joined #buildstream | 17:11 | |
| *** tiagogomes has quit IRC | 17:13 | |
| tpollard | does that included !1563 ? | 17:21 |
| benschubert | tpollard: yep, that's my only change, including this MR :/ | 17:21 |
| benschubert | but the benchmarks don't seem to agree there | 17:22 |
| tpollard | in the MR it was reported that show took a hit | 17:22 |
| tpollard | although not to that severity.... | 17:22 |
| benschubert | and I don't have a workspace :/ | 17:22 |
| tpollard | yep indeed | 17:23 |
| tpollard | :S that's not quite ~20%..... | 17:23 |
| benschubert | and it's without anything cached either | 17:23 |
| benschubert | which is .35% in the benchmarks | 17:24 |
| benschubert | Oh well, I'll have a look tomorrow | 17:24 |
| *** lachlan has joined #buildstream | 17:35 | |
| *** lachlan has quit IRC | 17:38 | |
| *** traveltissues has joined #buildstream | 19:45 | |
| *** traveltissues has quit IRC | 20:20 | |
| *** tristan has quit IRC | 22:26 | |
| *** narispo has quit IRC | 22:59 | |
| *** narispo has joined #buildstream | 22:59 | |
| *** narispo has quit IRC | 23:08 | |
| *** narispo has joined #buildstream | 23:08 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!