| *** toscalix has joined #buildstream | 05:12 | |
| *** toscalix has quit IRC | 05:21 | |
| juergbi | benschubert: RW access to casd? do you mean for destructive import/capture? I don't think we do this anywhere | 06:41 |
|---|---|---|
| juergbi | we force RW when needed in utils.copy_files/link_files but that's not directly related to CAS | 06:42 |
| juergbi | there are also sources that ensure files are readable to allow import in CAS but that's not RW | 06:43 |
| juergbi | I could be missing something, of course | 06:43 |
| juergbi | and it's possible we did something along these lines before casd | 06:44 |
| *** mohan43u has quit IRC | 08:05 | |
| *** mohan43u has joined #buildstream | 08:11 | |
| benschubert | juergbi: > "there are also sources that ensure files are readable to allow import in CAS but that's not RW" actually I think that's the part I remember. Is it per source or do we have a global place for this? | 08:18 |
| juergbi | benschubert: it's the responsibility of each source | 08:22 |
| benschubert | juergbi: ok thanks! | 08:23 |
| juergbi | benschubert: e.g., tar overrides the tar info attributes to ensure all extracted items are readable | 08:23 |
| juergbi | avoiding a separate chmod pass | 08:23 |
| benschubert | Gah, that will be tedious with buildbox-run-userchroot :'D | 08:23 |
| juergbi | benschubert: how so? | 08:24 |
| juergbi | that should not be involved for source->cas operations | 08:24 |
| benschubert | we would need two users, one running buildstream and the build, and one running buildbox-casd, which means we need to rely on group permissions fo rmost things, and currently most of our sources are staging as 0600 | 08:25 |
| benschubert | and same, buildbox-casd will need to store its data as 0660 not 0600, since now the buildstream user will want to have access to it | 08:25 |
| juergbi | ah, right, issue due to running casd as another user | 08:25 |
| benschubert | yup | 08:25 |
| juergbi | buildbox-casd already does that, iirc | 08:25 |
| juergbi | explicitly for this use case | 08:25 |
| benschubert | is that recent? | 08:26 |
| juergbi | no, not that recent | 08:26 |
| benschubert | weird, I need to double check then | 08:26 |
| * juergbi hope it wasn't broken | 08:26 | |
| juergbi | *hopes | 08:26 |
| benschubert | I will try to have a double look for this afternoon | 08:27 |
| juergbi | // Creating a temp file and writing the blob data into it... | 08:28 |
| juergbi | buildboxcommon::TemporaryFile temp_file(d_temp_directory.c_str(), | 08:28 |
| juergbi | "cas-tmp", 0644); | 08:28 |
| juergbi | not 0660, read-only for others but that's what we want | 08:29 |
| benschubert | but then we can't update the timestamp of it | 08:29 |
| benschubert | for the modified timestamp | 08:30 |
| juergbi | true, we probably have to switch to gRPC call for that | 08:30 |
| juergbi | we can't make it read-write, otherwise the whole point of having separate users is moot | 08:30 |
| benschubert | for the storage that's fine | 08:30 |
| benschubert | the place we don't want it rw is in the sandbox | 08:31 |
| juergbi | with hardlinking there is no difference | 08:31 |
| benschubert | ah shoot | 08:32 |
| benschubert | ok, then using grpc for this would be an option | 08:32 |
| juergbi | hopefully we can always batch | 08:32 |
| juergbi | the contains_directory() utime part is already replaced by gRPC call in my latest branch | 08:33 |
| juergbi | but there are other places | 08:33 |
| juergbi | the other calls might only be relevant to bst-artifact-server, though | 08:34 |
| benschubert | oh then it's not that bad :) | 08:34 |
| juergbi | so my latest branch might already fix that for bst client itself | 08:34 |
| benschubert | are you working on the userchroot sandbox right now or what's this branch? :) | 08:35 |
| *** mohan43u has quit IRC | 08:35 | |
| juergbi | not yet, it's for #1126 | 08:35 |
| gitlab-br-bot | Issue #1126: Cache resolution is slow https://gitlab.com/BuildStream/buildstream/issues/1126 | 08:35 |
| juergbi | will work on buildbox-run soon, though | 08:35 |
| benschubert | ok, perfect | 08:36 |
| *** santix has joined #buildstream | 08:38 | |
| *** mohan43u has joined #buildstream | 08:38 | |
| gitlab-br-bot | juergbi opened MR !1641 (juerg/prepare-fork->master: Replace is_fork_allowed() with prepare_fork()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1641 | 08:41 |
| *** tiagogomes has joined #buildstream | 08:45 | |
| juergbi | benschubert: bug error detail something like "Background threads are active"? | 08:49 |
| benschubert | juergbi: yep, that would be better :) | 08:50 |
| juergbi | ok, will add | 08:50 |
| benschubert | thanks | 08:50 |
| *** jonathanmaw has joined #buildstream | 09:03 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1641 (juerg/prepare-fork->master: Replace is_fork_allowed() with prepare_fork()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1641 | 09:03 |
| *** lachlan has joined #buildstream | 09:29 | |
| *** traveltissues has joined #buildstream | 09:29 | |
| *** traveltissues has quit IRC | 09:36 | |
| benschubert | coldtom: are you still planning in pushing !1607 to the end or would you rather have somebody take it over? :) | 09:49 |
| gitlab-br-bot | MR !1607: WIP: Run standardised source tests from external plugins https://gitlab.com/BuildStream/buildstream/merge_requests/1607 | 09:49 |
| coldtom | benschubert: i'd like to get it done if i can find the time | 09:49 |
| benschubert | ok :) let me know how I can help, and it would also need some rebasing currently (I think it's mostly the only thing left to do here, but I'll re-review once you had time to rebase) | 09:50 |
| gitlab-br-bot | marge-bot123 merged MR !1641 (juerg/prepare-fork->master: Replace is_fork_allowed() with prepare_fork()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1641 | 09:54 |
| coldtom | thanks, i'll try and at least rebase tonight | 09:57 |
| benschubert | awesome! | 09:57 |
| *** traveltissues has joined #buildstream | 09:58 | |
| *** mohan43u has quit IRC | 10:00 | |
| *** mohan43u has joined #buildstream | 10:03 | |
| *** traveltissues has quit IRC | 10:05 | |
| tpollard | Would anyone be against me sending an MR to change references to self._artifacts to self._artifactcaches ? | 10:11 |
| *** mohan43u has quit IRC | 10:11 | |
| tpollard | specifically when we're inferring about remotes, such as in stream doing 'if self._artifacts.has_push_remotes' for example | 10:13 |
| *** mohan43u has joined #buildstream | 10:14 | |
| *** mohan43u has quit IRC | 10:18 | |
| *** mohan43u has joined #buildstream | 10:27 | |
| benschubert | tpollard: I'm for it, only question is will it conflict with tlater[m] rework of the caches? :) | 10:32 |
| * tlater[m] thinks he missed part of this conversation | 10:33 | |
| benschubert | tlater[m]: the rename of 'self_artifacts' to self._artifatcaches' would that conflict with your work? | 10:33 |
| tlater[m] | Nothing I'm doing at the moment | 10:34 |
| * tlater[m] wonders when this conversation started, but wants to express his enthusiasm for it anyway | 10:35 | |
| tlater[m] | benschubert: For reference, I'm working on getting the server end to talk directly to casd, rather than going through BuildStream internals | 10:36 |
| benschubert | it just started :) | 10:36 |
| tlater[m] | This is mostly a clean-up of the server end, but some of the client end can probably be refactored afterwards. | 10:36 |
| benschubert | ah ok, then I slightly misunderstood. tpollard seems good then :) | 10:36 |
| tpollard | I'm not really touching cache parts of the code, but I just think it makes the member a bit clearer in Stream | 10:39 |
| *** narispo has joined #buildstream | 10:45 | |
| juergbi | tpollard: if we do it, we should do it also in Element and Pipeline | 10:47 |
| tpollard | I agree about Pipeline | 10:48 |
| tpollard | and yeh Element | 10:49 |
| *** narispo has quit IRC | 10:54 | |
| *** lachlan has quit IRC | 11:01 | |
| *** narispo has joined #buildstream | 11:38 | |
| gitlab-br-bot | aevri opened (was WIP) MR !1638 (aevri/casdprocessmanager->master: cascache: refactor, extract CASDProcessManager) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1638 | 11:56 |
| *** phoenix has joined #buildstream | 12:01 | |
| *** traveltissues has joined #buildstream | 12:04 | |
| *** lachlan has joined #buildstream | 12:06 | |
| *** lachlan has quit IRC | 12:17 | |
| *** narispo has quit IRC | 12:27 | |
| *** narispo has joined #buildstream | 12:28 | |
| *** phoenix has quit IRC | 12:32 | |
| *** lachlan has joined #buildstream | 13:09 | |
| *** santix has quit IRC | 13:11 | |
| *** santix has joined #buildstream | 13:35 | |
| *** lachlan has quit IRC | 13:53 | |
| *** lachlan has joined #buildstream | 14:00 | |
| *** santix has quit IRC | 14:01 | |
| *** santix has joined #buildstream | 14:01 | |
| *** lachlan has quit IRC | 14:09 | |
| *** lachlan has joined #buildstream | 14:15 | |
| tpollard | https://www.python.org/downloads/release/python-380/ | 14:19 |
| *** rdale has quit IRC | 14:35 | |
| benschubert | time to add new runners :D | 14:39 |
| tpollard | 'on macOS, the spawn start method is now used by default in multiprocessing' | 14:40 |
| tpollard | this is interesting...... | 14:40 |
| benschubert | They have typed dict? Oo | 14:42 |
| benschubert | > multiprocessing can now use shared memory segments to avoid pickling costs between processes | 14:43 |
| benschubert | That seems awesome | 14:43 |
| tpollard | yes definitely | 14:43 |
| tpollard | might have to double check that none of our unix/macos checks allow process start method to be default, else we might get some bangs | 14:45 |
| tpollard | until we fully support all of the multiprocessing under spawn | 14:46 |
| *** rdale has joined #buildstream | 15:10 | |
| tlater[m] | Hah, is BuildStream 2.0 going to release for python3.8+ only? :D | 15:14 |
| benschubert | if only ;) | 15:16 |
| tlater[m] | Well, considering our target platforms we could probably get away with something like windows/mac only on 3.8? | 15:17 |
| tlater[m] | And afaiu we don't really have subprocess performance issues on other unixes anyway. | 15:18 |
| gitlab-br-bot | aevri opened (was WIP) MR !1643 (aevri/oldgit->master: testing/.../site: windows-friendly HAVE_OLD_GIT) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1643 | 15:19 |
| * tlater[m] is probably still thinking to wishfully | 15:21 | |
| tlater[m] | s/to/too/ | 15:21 |
| tlater[m] | juergbi: Is there any reason why we're using utils.save_file_atomic for the casserver over a simple write? | 15:43 |
| tlater[m] | Can we get concurrent access? | 15:44 |
| juergbi | normally not but we want it to be atomic | 15:44 |
| juergbi | e.g., write could fail in the middle | 15:44 |
| juergbi | ah, actually, we even do have concurrent access due to grpc | 15:45 |
| juergbi | so we want to make sure that it's not read before it's completely written | 15:45 |
| juergbi | atomic file write should be considered best practice even if there was no concurrency, imo | 15:46 |
| tlater[m] | Ok, makes sense. | 15:46 |
| tlater[m] | Probably. I'm just on a mission to remove all calls to BuildStream API and I was wondering if I could save a call there :D | 15:47 |
| * tlater[m] also notices that there's issues for adding this to python | 15:47 | |
| tlater[m] | So maybe this won't be necessary in the future... | 15:47 |
| tlater[m] | Or anyway if we ever get artifact servers in buildbox-casd | 15:48 |
| juergbi | tlater[m]: btw: you might also be interested in !1638 | 15:52 |
| gitlab-br-bot | MR !1638: cascache: refactor, extract CASDProcessManager https://gitlab.com/BuildStream/buildstream/merge_requests/1638 | 15:52 |
| tlater[m] | Ah, I was writing one of these | 15:53 |
| tlater[m] | juergbi: Do you think the imports to `Message` in there would be a problem for importing it from casserver? | 15:58 |
| tlater[m] | I suppose there's really no reason they should be. | 15:58 |
| juergbi | I don't expect an issue with the import itself | 15:59 |
| juergbi | if it fails to report an error when it should, that's another thing, of course | 15:59 |
| juergbi | as I mentioned, we don't have to make it completely standalone right now | 15:59 |
| tlater[m] | In any case, I don't think those thoughts should block the MR | 16:01 |
| * tlater[m] might extend the original or modify it to be a bit more generic | 16:01 | |
| tlater[m] | But yeah, I'd rather use this than roll my own, since it's very nicely decoupled from the rest | 16:01 |
| *** lachlan has quit IRC | 16:07 | |
| gitlab-br-bot | BenjaminSchubert opened MR !1644 (bschubert/yaml-tidying->master: Some tidying up of our yaml API) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1644 | 16:20 |
| *** lachlan has joined #buildstream | 16:31 | |
| *** bochecha has quit IRC | 16:33 | |
| *** jonathanmaw has quit IRC | 17:05 | |
| *** santix has quit IRC | 17:15 | |
| benschubert | Kinnison: would you mind having a lookt at !1644, it touches the yaml :) | 17:29 |
| Kinnison | Sure, it'll be tomorrow morning if that's okay? | 18:07 |
| *** tiagogomes has quit IRC | 18:22 | |
| *** narispo has quit IRC | 19:06 | |
| *** phoenix has joined #buildstream | 19:12 | |
| benschubert | sure :) | 19:14 |
| *** phoenix has quit IRC | 22:22 | |
| *** narispo has joined #buildstream | 22:30 | |
| *** phoenix has joined #buildstream | 22:54 | |
| *** phoenix has quit IRC | 23:02 | |
| *** narispo has quit IRC | 23:08 | |
| *** flatmush has quit IRC | 23:12 | |
| *** flatmush has joined #buildstream | 23:20 | |
| *** phoenix has joined #buildstream | 23:34 | |
| *** phoenix has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!