| *** nimish2711 has joined #buildstream | 03:21 | |
| *** alatiera has joined #buildstream | 04:28 | |
| *** tristan has quit IRC | 05:41 | |
| *** tristan has joined #buildstream | 06:05 | |
| *** mohan43u_ has quit IRC | 06:42 | |
| jennis | juergbi, tpollard, thanks for the review of !1299 | 07:33 |
|---|---|---|
| gitlab-br-bot | MR !1299: WIP: Replace _update_state with various methods in various CacheKey implementations https://gitlab.com/BuildStream/buildstream/merge_requests/1299 | 07:33 |
| *** mohan43u has joined #buildstream | 07:37 | |
| *** bochecha has joined #buildstream | 07:44 | |
| *** bochecha has quit IRC | 07:47 | |
| *** bochecha has joined #buildstream | 07:49 | |
| *** toscalix has joined #buildstream | 07:52 | |
| *** toscalix has quit IRC | 07:53 | |
| *** toscalix has joined #buildstream | 07:54 | |
| *** benschubert has joined #buildstream | 08:11 | |
| *** tristan has quit IRC | 08:13 | |
| *** rdale has joined #buildstream | 08:15 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1310 (jennis/revert_gc_management->master: Revert !1164 - Manage GC during pipeline load) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1310 | 08:28 |
| *** alatiera has quit IRC | 08:47 | |
| benschubert | phildawson: you around? | 08:57 |
| adds68 | Has anyone ever used bst on a public gitlab runner? | 08:57 |
| benschubert | adds68: no, however I guess you have troubles with it? What's the problem? :) | 08:58 |
| adds68 | benschubert, hey, yes i am haha, good guess | 08:59 |
| adds68 | I am seeing bwrap issues | 08:59 |
| adds68 | bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'. | 08:59 |
| benschubert | adds68: if I remember well, the public runners are docker containers and they are run without the privileged flag correct? | 08:59 |
| *** tristan has joined #buildstream | 08:59 | |
| adds68 | benschubert, yea i think that is the case :( | 09:00 |
| benschubert | If that's the case, then you can't run bubblewrap in them, since bubblewrap will not be able to set its namespaces correctly | 09:00 |
| adds68 | Ok i will have to try and set up a runner, is this going to change in bst2 or the new debian stable, would you know? | 09:00 |
| adds68 | or is it purely because docker is not running in privileged mode? | 09:01 |
| benschubert | it's purely because we are missing some privileges yep | 09:01 |
| *** ChanServ sets mode: +o tristan | 09:01 | |
| phildawson | benschubert, I am | 09:01 |
| tristan | right, exactly :) | 09:01 |
| benschubert | phildawson: i'm trying to understand how "{project_dir}" gets transformed to a correct path in testing/_sourcetests/project/project.conf, could you enlighten me? I'm probably missing something very simple | 09:03 |
| tristan | adds68, you might ask jjardon about setting up runners, but I don't know exactly what guarantees gitlab runners provide for it's hosts (as I recall, some hosts might not have user namespaces enabled) | 09:03 |
| adds68 | tristan, i think as benschubert said, the lack of privileges makes this a non starter =/ | 09:04 |
| benschubert | tristan: you can add tags to runners to force stuff, at home I have "privileged" runners, that have the permissiosn for such stuff for example :) | 09:04 |
| tristan | I guess you are not allowed to have privs on the public runners | 09:04 |
| adds68 | It's a shame that bst can't be run for "free" because of Gitlab though =/ | 09:04 |
| tristan | yeah that could be cool | 09:04 |
| tristan | maybe once unprivileged namespacing becomes mainstream that will be possible | 09:05 |
| benschubert | adds68: travis and other free CI tools suffer the same problem | 09:05 |
| *** jonathanmaw has joined #buildstream | 09:05 | |
| tristan | adds68, i.e. there is a host configuration you can ask the kernel to allow regular users to create namespaces, but it's not widely considered secure yet | 09:05 |
| adds68 | I mean it makes sense for security, just a shame | 09:06 |
| tristan | Well I think it is a goal to make that secure | 09:06 |
| tristan | adds68, i.e. it should theoretically be secure for a regular user to create a user namespace, it shouldnt be able to escalate privileges | 09:07 |
| tristan | just not enabled on most distros | 09:07 |
| adds68 | I wonder if gitlab have an issue open for this | 09:08 |
| * tristan doubts it is reasonable to ask for, I was just raising that as I suspect the situation might naturally get better over the years to come | 09:08 | |
| tristan | while user namespaces mature | 09:08 |
| tpollard | cs-shadow: yeh, I'll roll back and have a look | 09:12 |
| phildawson | benschubert, looks like it's filled in by the cli_integration fixture. Though I wasn't the original author of those tests, so there's always the possibility I'm missing something. | 09:15 |
| *** nimish2711 has quit IRC | 09:16 | |
| benschubert | phildawson: ok I'll look into that part, thanks! | 09:16 |
| benschubert | I need it in a non-integration test so we'll see x) | 09:16 |
| phildawson | :) | 09:18 |
| *** nimish2711 has joined #buildstream | 09:39 | |
| gitlab-br-bot | phildawson approved MR !1310 (jennis/revert_gc_management->master: Revert !1164 - Manage GC during pipeline load) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1310 | 09:40 |
| *** lachlan has joined #buildstream | 09:40 | |
| *** lachlan has quit IRC | 09:44 | |
| phildawson | cs-shadow, would you object if I were to make the build of the segfaulting fedora 30 image in the CI for buildstream-docker-images optional until a new none-broken fedora 30 image is released? | 09:49 |
| cs-shadow | phildawson: that would be good, thanks | 09:55 |
| phildawson | :) | 09:55 |
| phildawson | cs-shadow, https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/116 | 10:10 |
| *** lachlan has joined #buildstream | 10:12 | |
| *** lachlan has quit IRC | 10:16 | |
| gitlab-br-bot | jennis opened MR !1311 (jennis/cache_quota->master: Do not calculate cache quota when we don't need to) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1311 | 10:18 |
| jennis | tristan, if you have some time today, I'd appreciate a review of !1311, I've removed the calculation of the cache quota upon CASQuota instantiation, AFAICS it's not actually required, we seem to calculate the cache quota when we need it during the session | 10:21 |
| gitlab-br-bot | BenjaminSchubert opened issue #1010 (Bst track can't track when multiple sources are on the same element without ref.) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1010 | 10:27 |
| benschubert | ^ a funky error which might require a chunky rework of the tracking :/ | 10:28 |
| *** nimish2711 has quit IRC | 10:30 | |
| *** nimish2711 has joined #buildstream | 10:30 | |
| jennis | oh dear | 10:34 |
| jennis | I hope this isn't a YAML NWO regression :/ | 10:34 |
| *** lachlan has joined #buildstream | 10:41 | |
| tristan | ummmm, ok who was it... jonathanmaw was asking me for a review ? | 10:43 |
| tristan | right, lets see | 10:43 |
| tristan | ohhhh, big stuff ;-) | 10:44 |
| tristan | jonathanmaw, very very shallow preliminary review | 10:51 |
| tristan | jonathanmaw, honestly I think it's far from ready :-/ | 10:51 |
| *** lachlan has quit IRC | 10:55 | |
| jonathanmaw | tristan: thanks for the review | 10:55 |
| jonathanmaw | very shallow is fine, we were looking for sanity checks | 10:56 |
| tristan | Ok yeah, it does look like it has a lot of todo comments still | 10:56 |
| tristan | jonathanmaw, I worry a bit that it's might be a lot for one merge request | 10:56 |
| tristan | jonathanmaw, I wonder if classing of cache key calculation could be an orthogonal activity from untangling cache key calculations from orthogonal state bits | 10:57 |
| *** alatiera has joined #buildstream | 10:58 | |
| *** lachlan has joined #buildstream | 11:00 | |
| tristan | like, if we started with something like (A) Keep _update_state() as the focal point for updating all state (for now)... (B) have _update_state() call _update_state_cache_key() and then call _update_state_bits(), the latter of which would take care of _schedule_assemble() etc depending on what the conditions are... and then (C) be able to start replacing _update_state_bits() with more granular calls... and finally (D) Have _update_state_cache_key() | 11:00 |
| tristan | do the classed thing | 11:00 |
| tristan | jonathanmaw, I just think that tackling the whole thing at once might be very difficult, and clash with other ongoing work in difficult ways | 11:01 |
| jonathanmaw | yeah, it is quite big | 11:01 |
| tristan | Seems once update state internally has separated all of it's bits and pieces, we can start figuring out how to call it's bits and pieces at the right times | 11:01 |
| *** lachlan has quit IRC | 11:05 | |
| *** lachlan has joined #buildstream | 11:09 | |
| jonathanmaw | tristan: ok, most of that branch isn't going to be usable for that, though I have a couple of bits and pieces that reduce the amount that we call _update_state without making any radical changes. I'm going to cherry-pick those into a new MR so they can be reviewed separately | 11:09 |
| tristan | jonathanmaw, good idea to get those bits and pieces out of the way yeah | 11:18 |
| gitlab-br-bot | jonathanmaw opened MR !1312 (jonathan/reduce-update-state-calls->master: Reduce the amount of times we call Element._update_state()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1312 | 11:22 |
| jonathanmaw | tristan: as part of (C) in the outline above, do you mean replacing _update_state_bits() with more granular calls within _update_state(), or in other call sites elsewhere? | 11:25 |
| *** lachlan has quit IRC | 11:27 | |
| tristan | jonathanmaw, I mean A and B ends up with: _update_state() { _update_state_keys(); _update_state_bits(); }, where _update_state_bits() takes care of side effects not related to the key itself... then we have separation of keys and bits | 11:28 |
| *** lachlan has joined #buildstream | 11:29 | |
| tristan | jonathanmaw, Once we have that, we have this function _update_state_bits() and we can analyze where exactly those individual bits need to be changed, often it is in result of a cache key being resolved, plus some other conditions | 11:29 |
| tristan | so we can start splitting up those side effect updates once they are safely untangled from the cache key resolution algo | 11:30 |
| tristan | jonathanmaw, does that make sense ? | 11:30 |
| tristan | jonathanmaw, really, take this as a suggestion, it's probably how I would tackle the problem | 11:31 |
| jonathanmaw | tristan: just calling _update_state_keys and _update_state_bits is a bit messy because of what we do for workspaces that we need to assemble, plus __update_source_state needs to be called before we do either (and is more of a 'bits' thing than a 'keys' thing) | 11:39 |
| jonathanmaw | the broad approach makes sense to me, though | 11:40 |
| tristan | I mean the large problem is that we have a huge function which has many side effects, we need to untangle them into separate phases I think | 11:41 |
| tristan | only then we can start to selectively compute side effects | 11:41 |
| tristan | but we can hopefully do that untangling first, without changing callers to _update_state(), then half the work is done and the rest becomes much easier | 11:42 |
| *** lachlan has quit IRC | 12:02 | |
| benschubert | Is there a way with the artifactserver in buildstream to remove everything related to a given project? | 12:56 |
| tristan | I dont think there is even a way to explicitly delete anything from an artifact server | 13:00 |
| tristan | benschubert, however the (unfinished) plan for manipulating the local artifact cache was intended to accept wildcards for artifact names/refs, this would at least support `bst artifact delete %{project-name}/*` | 13:01 |
| tristan | benschubert, that said, if we optimize the cleanups so that they are object based and not ref based, I expect it would be less problematic that there is no way to explicitly delete things on remotes ? | 13:03 |
| *** lachlan has joined #buildstream | 13:08 | |
| *** lachlan has quit IRC | 13:10 | |
| *** lachlan has joined #buildstream | 13:10 | |
| *** tristan has quit IRC | 13:17 | |
| benschubert | tristan it's not a problem, I was just asking if I could do it to reclaim storage, but I can just wait for it to be evicted :) | 13:24 |
| *** tristan has joined #buildstream | 13:30 | |
| *** lachlan has quit IRC | 13:30 | |
| gitlab-br-bot | marge-bot123 merged MR !1310 (jennis/revert_gc_management->master: Revert !1164 - Manage GC during pipeline load) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1310 | 13:50 |
| *** lachlan has joined #buildstream | 14:26 | |
| phildawson | cs-shadow (or anyone else), any idea what's happening here https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/201693800 ? I've rerun it several times and it seems to be a consistent failure. | 14:31 |
| *** lachlan has quit IRC | 14:31 | |
| *** lachlan has joined #buildstream | 14:34 | |
| *** tristan has quit IRC | 14:35 | |
| tpollard | some of the others pipelines are failing with not being able to find the registry image | 14:35 |
| *** lachlan has quit IRC | 14:39 | |
| cs-shadow | phildawson: I have restarted the corresponding build job for that image, will hit rebuild on the test job next. Just in case the build was somehow buggy | 14:40 |
| jennis | juergbi, when we start an artifact server I'm seeing the sub-directories artifacts/refs, now to me, this looks pretty redundant, i.e. I can remove this and still use the server as intended | 14:51 |
| jennis | Looking at casserver.py, it looks like we might be using this to store the protos? | 14:52 |
| jennis | I'm not too sure though / not that au fait with this area of the code | 14:53 |
| juergbi | jennis: with master or the artifact as proto branch? | 14:53 |
| jennis | master | 14:53 |
| juergbi | maybe the directory is created already but only used in the WIP branch | 14:53 |
| juergbi | with artifact as a proto cas/refs will no longer be used for artifact refs | 14:53 |
| jennis | oh, so it's definitely going to be used in the future? | 14:53 |
| tpollard | yes | 14:54 |
| jennis | Ok, gotcha, thanks | 14:54 |
| tpollard | was added in preparation with https://gitlab.com/BuildStream/buildstream/merge_requests/1259/diffs#ccbdde14166de2d0d1229bad0ac2748f8f76b209 | 14:55 |
| phildawson | cs-shadow, cool hopefully that will solve it :) | 14:55 |
| jennis | tpollard thanks | 14:56 |
| jennis | Was about to revert it... | 14:56 |
| cs-shadow | yay! at least that step passed. The rest should hopefully be fine | 14:56 |
| phildawson | \o/ fresh docker images are so close now | 14:58 |
| tpollard | nice | 16:03 |
| tpollard | jonathanmaw: I'm happy with !1312 but I think it's probably worth getting approval from Tristan before a marge | 16:06 |
| gitlab-br-bot | MR !1312: Reduce the amount of times we call Element._update_state() https://gitlab.com/BuildStream/buildstream/merge_requests/1312 | 16:06 |
| *** jonathanmaw has quit IRC | 16:09 | |
| *** bochecha has quit IRC | 16:32 | |
| *** lachlan has joined #buildstream | 16:33 | |
| *** phildawson has quit IRC | 16:34 | |
| *** phildawson has joined #buildstream | 16:35 | |
| *** toscalix has quit IRC | 16:41 | |
| *** lachlan has quit IRC | 16:43 | |
| *** alatiera has quit IRC | 16:44 | |
| *** tpollard has quit IRC | 16:46 | |
| *** phildawson has quit IRC | 17:03 | |
| *** lachlan has joined #buildstream | 17:29 | |
| *** alatiera has joined #buildstream | 17:39 | |
| *** lachlan has quit IRC | 17:40 | |
| *** alatiera has quit IRC | 17:41 | |
| *** nimish2711 has quit IRC | 17:49 | |
| *** nimish2711 has joined #buildstream | 17:53 | |
| *** nimish2711 has joined #buildstream | 17:53 | |
| *** lachlan has joined #buildstream | 17:59 | |
| *** nimish2711 has quit IRC | 18:08 | |
| *** lachlan has quit IRC | 18:20 | |
| *** lachlan has joined #buildstream | 18:38 | |
| *** nimish2711 has joined #buildstream | 18:40 | |
| *** nimish2711 has quit IRC | 18:45 | |
| *** nimish2711 has joined #buildstream | 18:55 | |
| *** lachlan has quit IRC | 18:57 | |
| *** nimish2711 has quit IRC | 19:00 | |
| *** nimish2711 has joined #buildstream | 19:01 | |
| *** nimish2711 has quit IRC | 19:06 | |
| *** nimish2711 has joined #buildstream | 19:06 | |
| *** lachlan has joined #buildstream | 19:15 | |
| *** lachlan has quit IRC | 19:22 | |
| *** nimish2711 has quit IRC | 19:26 | |
| *** nimish2711 has joined #buildstream | 19:26 | |
| *** nimish2711 has quit IRC | 19:51 | |
| *** nimish2711 has joined #buildstream | 19:52 | |
| *** nimish2711 has quit IRC | 19:57 | |
| *** nimish2711 has joined #buildstream | 20:22 | |
| *** nimish2711 has quit IRC | 20:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!