IRC logs for #buildstream for Monday, 2019-11-04

Kinnisonjuergbi: I apologise for how slow I was getting back to !1667 (I've had a hectic time last week) but I think your fix in casd looks good and I've resolved that comment as a result.  I've also hit 'rebase' to ensure that your change is still good against master.  Do you need to push a change to .gitlab-ci.yml on that MR or is it OK to merge as-is and bring the race-condition fix from casd in later?08:28
gitlab-br-botMR !1667: cascache.py: Defer attempt to connect to casd until socket file exists https://gitlab.com/BuildStream/buildstream/merge_requests/166708:28
Kinnisonjuergbi: either way you have my 👍08:28
juergbiKinnison: thanks. I'm about to update buildbox-casd anyway as we'll need a new snapshot for other MRs08:29
* Kinnison nods08:29
juergbiso maybe it's best to defer the merge of this until the CI images are updates08:29
juergbi*updated08:29
KinnisonSure.08:29
KinnisonI won't assign to marge anyway08:29
KinnisonI'll leave that to you08:29
*** traveltissues has joined #buildstream08:35
*** tpollard has joined #buildstream09:13
benschubertjuergbi: !1642 is good to go I think, https://gitlab.com/BuildStream/buildstream/issues/1126#note_239345156 wqas my benchmark on freedesktop09:20
gitlab-br-botMR !1642: WIP: Reimplement contains_directory() with FetchTree() https://gitlab.com/BuildStream/buildstream/merge_requests/164209:20
juergbibenschubert: yes, I saw that. thanks for testing. currently updating buildbox-casd and then we can merge it09:20
benschubertgreat09:21
juergbibenschubert: I have to say I'm surprised by the uncached show speedup, though09:23
benschubertMy cache was entirely empty, maybe that was it?09:24
juergbithat could make a difference due to casd startup time09:27
*** santi has joined #buildstream09:30
*** tiagogomes has joined #buildstream09:39
*** rdale has joined #buildstream09:42
juergbitlater[m]: fyi, I'm updating buildbox-casd in CI etc., should help unblock !164509:45
gitlab-br-botMR !1645: WIP: Refactor casserver.py: Stop relying on the buildstream-internal `CASCache` implementation https://gitlab.com/BuildStream/buildstream/merge_requests/164509:45
*** lachlan has joined #buildstream09:58
*** jonathanmaw has joined #buildstream10:05
benschubertany MRs that need attention?10:34
tpollardbenschubert: would be beneficial if you could benchmark my subprocessing branch on your WSL setup10:35
tpollardwell, just a timed end to end of the debian target10:35
benschubert!1613? A whole build?10:36
gitlab-br-botMR !1613: WIP: Run bst build under a multiprocessing subprocess https://gitlab.com/BuildStream/buildstream/merge_requests/161310:36
tpollardI don't think it has to be a full build, and obviously when you've got the resources to do so :)10:38
benschubertSure I'll benchmark that and let you know :)10:38
tpollardI've got one set of results, which is good enough for now, there's no rush10:38
tpollardjust something that exercises click.echo heavily10:39
*** cs-shadow has joined #buildstream10:46
*** narispo has quit IRC10:47
*** phoenix has joined #buildstream10:47
*** narispo has joined #buildstream10:51
*** narispo has quit IRC10:52
tlater[m]juergbi: ta!11:00
gitlab-br-botjuergbi opened MR !1683 (juerg/casd->master: .gitlab-ci.yml: Update docker images for buildbox-casd 0.0.4) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168311:02
gitlab-br-botBenjaminSchubert approved MR !1683 (juerg/casd->master: .gitlab-ci.yml: Update docker images for buildbox-casd 0.0.4) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168311:03
*** phoenix has quit IRC11:03
gitlab-br-bottraveltissues approved MR !1673 (aevri/testutils_artifactshare->master: tests/artifactshare: safer cleanup_on_sigterm use) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/167311:34
* tpollard would very much appreciate some eyes on !1613 with any interactive testing (especially on wsl/macos) or review :) 11:48
tpollardhttps://gitlab.com/BuildStream/buildstream/merge_requests/161311:49
*** lachlan has quit IRC11:53
*** lachlan has joined #buildstream11:56
gitlab-br-botmarge-bot123 merged MR !1671 (chandan/remove-build-track->master: frontend: Remove tracking options from `bst build`) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/167112:09
*** akvilebirgelyte has quit IRC12:10
*** akvilebirgelyte has joined #buildstream12:11
*** lachlan has quit IRC12:21
*** akvilebirgelyte has quit IRC12:28
*** lachlan has joined #buildstream12:35
*** akvilebirgelyte has joined #buildstream12:41
gitlab-br-botmarge-bot123 merged MR !1683 (juerg/casd->master: .gitlab-ci.yml: Update docker images for buildbox-casd 0.0.4) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168312:50
tpollardhas anyone found themselves waiting around for a gitlab runner in the CI since friday ooi?13:24
tlater[m]tpollard: I've been waiting a fair amount, but that's on the docker images repo13:35
tlater[m]Not sure they share runners13:35
*** santi has quit IRC13:35
gitlab-br-botcs-shadow approved MR !1681 (traveltissues/changeflag->master: Replace flag name) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168113:35
gitlab-br-bottlater opened MR !1684 (tlater/casd-socket-permissions->master: cascache.py: Make the casd socket tempdir group-rwx) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168413:37
gitlab-br-botBenjaminSchubert approved MR !1684 (tlater/casd-socket-permissions->master: cascache.py: Make the casd socket tempdir group-rwx) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168413:41
tlater[m]Ta benschubert :)13:45
tlater[m]Though I'd like juergbi to approve, too, I think it'd be better to create that directory in buildbox-casd.13:45
*** santi has joined #buildstream13:49
tpollardtlater[m]: hmmm I'm not sure either, ta13:51
benschuberttlater[m]: if buildbox-casd does it, it will also need to set it 770, there is little change there. I wonder whether buildbox would need that. If so, that's where we should do it, otherwise BuildStream :)13:55
benschuberttlater[m]: also anythin needed on the removal of update state part?13:55
tlater[m]benschubert: Mostly review13:56
tlater[m]:)13:56
tlater[m]I'd like to give it another benchmark run, but I'm waiting for a new laptop13:56
tlater[m]Since this one's way too slow13:56
benschubertok I'll add it to my list :)13:56
tlater[m]\o/13:57
tlater[m]ta benschubert13:57
juergbitlater[m]: will think about it14:00
juergbiaccess is required by both processes14:01
juergbithe ideal approach, imo, would be to use socket activation, which would mean that casd wouldn't need access to it. however, gRPC doesn't support that (yet)14:01
traveltissuestlater[m], how long does it take to benchmark on that laptop14:02
juergbithe main issue is that buildstream generally has no idea who else belongs to the same group, i.e., whether that chmod would be a safe thing14:03
tlater[m]traveltissues: 4 hours14:03
traveltissuesthat's about normal for mine14:03
tlater[m]juergbi: Yes, that's why I'm hestitant.14:03
tlater[m]traveltissues: I'll probably have to bite the bullet eventually, but it's not that much of a priority right this instant.14:04
tlater[m]Especially not if review comes up with more changes.14:04
traveltissuesi often leave mine to run overnight14:05
tpollardI just restarted the bastions gitlab-runner service, might take a few moments to spin back up14:09
tpollardmight require any on-going pipelines to be restarted too14:11
tpollard(I've increased the amount of concurrent runners that docker-machine can spawn)14:13
benschuberttlater[m]: I can probably run the benchmark at home overnight if needed also14:24
tlater[m]benschubert: That's ok, I'll re-run them once I've had another round of review.14:25
benschuberttpollard: is !1613 rebased on master for the tests? :)14:28
gitlab-br-botMR !1613: WIP: Run bst build under a multiprocessing subprocess https://gitlab.com/BuildStream/buildstream/merge_requests/161314:28
juergbimarge is broken inside :-/14:29
tpollardbenschubert: yep, should be on master :)14:30
benschuberttpollard: awesome14:30
tpollardthe commit history is still in a WIP state, but the actual total diff is about ready for review14:30
*** lachlan has quit IRC14:31
*** lachlan has joined #buildstream14:31
benschubertyep no worries, I'm pulling, will be done in 1h or so, then I'll try14:32
tpollardbenschubert: also as we briefly discussed handling exceptions from the subprocess (i.e tracebacks being non-pickle) my current approach uses tblib, I'm not sure if you've come across it before14:34
benschubertnever, but I had a brief look at what it was, haven't looked at the code though14:34
benschubertI'd just wonder how much of it do we need, as I'd rather not add yet another dependency if it can be fixed in 50-100 lines :)14:35
* tpollard nods 14:36
*** lachlan has quit IRC14:38
*** lachlan has joined #buildstream14:41
gitlab-br-botmostynb opened MR !1685 (link_fix->master: fix broken user guide link in README.rst) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168514:44
gitlab-br-bottraveltissues approved MR !1685 (link_fix->master: fix broken user guide link in README.rst) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168514:45
gitlab-br-botmostynb closed MR !1665 (link_fix->master: fix broken user guide link in README.rst) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/166514:45
gitlab-br-botBenjaminSchubert approved MR !1685 (link_fix->master: fix broken user guide link in README.rst) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168514:45
gitlab-br-botmarge-bot123 merged MR !1681 (traveltissues/changeflag->master: Replace flag name) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168114:47
tlater[m]Grmbl, there's more temporary directories14:48
*** lachlan has quit IRC14:49
tlater[m]Any idea why local cas would be writing to these directories? https://hastebin.com/gicacahoni.sql14:49
tlater[m]For reference, the actual cas tmp directory is in cache/cas/tmp, not in cache/tmp14:50
tpollard:S14:58
tlater[m]Seems to be the directory elements/sources are staged into14:59
tlater[m]Why would buildbox-casd need to access that?14:59
tlater[m]It's going to be the virtual directory on top of that directory...15:02
tlater[m]grmbl, grmbl, grmbl15:03
tlater[m]juergbi: I'm starting to wonder if these setuid games are a good idea15:03
juergbitlater[m]: I'm not generally fond of setuid, but do you have a suitable alternative in mind?15:06
juergbiwe can't get around the need for two uids, afaict15:06
juergbiwe could require the user to start casd as separate uid manually / via something like systemd15:06
tlater[m]Is there no way to keep it at one uid?15:07
tlater[m]It could be done by listening to a socket with systemd15:07
tlater[m]But I suppose not all platforms have a systemd15:08
juergbialmost none do15:08
juergbiand on Linux the standard mechanism is planned to use FUSE, so we won't need two UIDs on Linux15:08
tlater[m]Yeah15:08
juergbihowever, for other platforms, if we want to use hardlinks and safeguard the cache from corruption, we need two UIDs, afaict15:08
juergbiif the platform doesn't support FUSE15:09
juergbiuserchroot also has an additional requirement for two UIDs, but we already need it for hardlink protection, so userchroot is not the main issue here15:09
juergbitlater[m]: are you concerned about the use of two uids in general or just about using setuid to achieve that?15:10
tlater[m]In general. It's bleeding into all our invocations of mkdtemp15:11
tlater[m]It feels wrong to essentially require two users to run buildstream15:12
tlater[m]And everything is resisting that, too.15:12
benschuberttpollard: which branch of debian-stretch-bst where you trying?15:12
tlater[m]I see why it's required, but I dislike it.15:12
tlater[m]juergbi: Obviously setuid is evil, too, but I think userchroot is a reasonable way of doing that.15:13
juergbitlater[m]: two uids: yes, I don't exactly like it either but unfortunately POSIX doesn't have proper immutable files (and not even Linux has that)15:14
juergbitlater[m]: not sure I understand your last comment. userchroot doesn't eliminate the need for casd setuid15:14
juergbiuserchroot itself needs to be setuid root but that's orthogonal15:14
tpollardbenschubert: locally I use jennis/add_all_files, I'm not 100% sure which branch was used for the wsl results. Do you remember traveltissues? ^15:14
juergbitlater[m]: btw: I've added a comment to your MR15:15
benschubertOk I'll use this one, thanks :)15:15
tlater[m]No, that's true15:15
benschubertgah, those checkout times x')15:15
tlater[m]juergbi: Ah, ta15:15
traveltissuestpollard, that's the branch i use15:15
tpollardcool, ta :)15:15
benschuberttraveltissues: ooi is checking out this branch also taking 10 minutes+ for you?15:16
traveltissuesyes15:16
tlater[m]juergbi: So create the socket in a buildstream-owned directory, and symlink to it instead to avoid having its path name be long?15:17
juergbitlater[m]: it's always casd that creates the socket, not buildstream (as we can't use socket activation)15:18
juergbiright now we pass something like /tmp/buildstream-foo/casd.sock to casd as socket path to use15:19
tlater[m]Ah, right15:19
juergbimy suggestion is to pass something like /tmp/buildstream-foo/cas/casd.sock15:19
juergbiwhere 'cas' is a symlink15:19
juergbi(maybe socket name also should be something unique, not fixed casd.sock to retain current behavior with multiple instances)15:20
tlater[m]That symlinks to somewhere deep inside $HOME?15:20
juergbiand the 'cas' symlink would point to the cas directory, i.e., the toplevel directory casd works with15:20
tlater[m]Or whatever other path buildstream's current cache directory is15:20
juergbiso casd should anyway have write access to that directory already15:20
juergbi(and buildstream read access)15:20
* tlater[m] doesn't dislike this, but we encounter a similar problem with FileBasedDirectory, which requires us to make the buildstream-created localcas directories rwx.15:21
tlater[m]That can't be avoided the same way, so we sort of need to leak build details to the user's group?15:22
tlater[m]At which point the socket directory permissions aren't that important anymore15:22
juergbitlater[m]: can you clarify what you mean with buildstream-created localcas directories?15:23
tlater[m]I suppose tightening that isn't too bad either way.15:23
tlater[m]juergbi: These paths: https://hastebin.com/uwezuhonox.sql15:24
tlater[m]BuildStream's FileBasedDirectory is backed by CAS, but the directories are created by BuildStream15:24
juergbiFileBasedDirectory is not backed by CAS15:25
juergbithat would be CasBasedDirectory15:25
tlater[m]If BuildBox-casd is setuid, that becomes a problem15:25
juergbithe goal should be that everything in the cas directory should be created by casd15:26
tlater[m]Well, FileBasedDirectory calls through to CasBasedDirectory15:26
juergbiI'd say buildstream should never have to write anything in there15:26
juergbibut neither of them should directly write into the cas directory15:26
juergbithat would be a bug15:26
* tlater[m] tries to figure out why BuildStream creates that15:29
juergbifor tests things might be more complicated than for non-test environments, as tests have to dynamically create these directory hierarchies whereas we can require some of those directories to be created manually for non-test environments as part of setup for two setuid platforms15:31
juergbion the other hand, adding test-only chmods would typically not be a security issue, so we should be able to work around that15:32
tlater[m]juergbi: The problem is that those directories only come into existence when we invoke BuildStream, and their names are random15:34
tlater[m]We could make a util that does tempfile.mkdtemp but respects umask, but there's a security consideration with that.15:35
juergbiyes, I don't think we should do that15:35
juergbiI'd rather add tets-specific code. e.g., to runcli15:35
juergbior even to the core codebase (conditional on running in test suite) if we have to, but better to keep it out of the core code15:36
tlater[m]Maybe a tempfile.mkdtemp that respects umask only when run with BST_TEST_SUITE?15:36
tlater[m]Yeah, but I don't see how we're going to predict temporary directory names.15:36
juergbiwould theoretically be an option but only if it's not possible with with additions in test code alone15:36
juergbiwe obviously won't15:36
juergbihowever, it should only be needed for the toplevel CAS directory structure for each casd instance15:37
juergbiand that we might be able to handle15:37
juergbibut I might be missing something15:37
tlater[m]Assuming that BuildStream doesn't create temporary directories inside CAS' directory15:37
juergbitlater[m]: I don't think we do but the issue might be that we create a temporary directory outside that casd needs to be able to read15:39
tlater[m]Why would it need to read that?15:40
juergbiimport files15:40
tlater[m]Is that to work around expensive copies?15:41
juergbino, it needs to read those files to create copies15:41
juergbiin a way it is an optimization in that we don't want to send everything over the socket, as that would be the only alternative to get files into CAS15:42
tlater[m]Ah right15:42
juergbitlater[m]: my comment above about not generally weakening mkdtemp security with umask was mainly about global locations such as /tmp, though. respecting umask for mkdtemp inside a buildstream-owned directory directory should be much less of an issue15:42
tlater[m]Yeah15:43
juergbithere we use mkdtemp only as random name generator, not as security mechanism15:43
juergbiand with the exception of the socket, I think we use mkdtemp always in those buildstream hierarchies15:43
juergbiso we can potentially change that15:43
tlater[m]I suppose those directories would be harder to get to15:44
juergbiyes, access can be controlled by parent directories15:44
tlater[m]So maybe a mkdtemp util that asserts parent directories' permissions aren't open enough to cause trouble?15:45
*** lachlan has joined #buildstream15:45
*** traveltissues has quit IRC15:49
*** traveltissues has joined #buildstream15:49
juergbitlater[m]: I don't think we should assert those permissions15:50
juergbithat should be up to the admin/user, imo15:51
gitlab-br-botmarge-bot123 merged MR !1667 (juerg/casd-connect->master: cascache.py: Defer attempt to connect to casd until socket file exists) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/166715:51
*** lachlan has quit IRC15:51
juergbias long as we still respect umask I don't see an issue15:51
juergbiI don't think we need to open up permissions beyond umask, right?15:51
*** lachlan has joined #buildstream15:51
juergbi(assuming systems that require the dual uid approach set a suitable umask)15:51
tlater[m]No, that seems right15:52
gitlab-br-botjuergbi opened (was WIP) MR !1642 (juerg/fetch-tree->master: Reimplement contains_directory() with FetchTree()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/164215:53
tlater[m]Well, let's see - ta juergbi, I would have been stuck for a couple more days without that explanation15:54
* tlater[m] gets on with that patch15:54
juergbireview/approval required for simple website MR: https://gitlab.com/BuildStream/website/merge_requests/12915:56
tpollardwill check it15:57
tpollarddone :)15:59
juergbita16:00
*** santi has quit IRC16:17
coldtomis there a way i can get bst to create a tmpfs at `/dev/shm` rather than mounting the host `/dev/shm` in? iiuc bst is using the host one16:20
*** lachlan has quit IRC16:22
tlater[m]coldtom: Assuming you're on Linux, that's handled by bwrap16:23
* tlater[m] doesn't see any options for this in bwrap16:24
coldtomtlater[m]: don't directories marked using `Sandbox.mark_directory()` get --dev-bind-ed in? which surely shares the host one?16:24
tlater[m]coldtom: In any case, that requires calling mknod, so you'd need to run BuildStream as root.16:25
tlater[m]coldtom: Oh, I suppose /dev is marked16:25
coldtomwouldn't the --tmpfs bwrap option work? that looks to be how /tmp gets created16:26
tlater[m]But yeah, I don't think we'd have permissions to create a tmpfs there.16:26
tlater[m]I'd imagine that conflicts with --dev-bind-ing /dev16:27
tlater[m]coldtom: Oh, it's actually done right16:28
tlater[m]coldtom: In non-interactive mode /dev/shm is not created at all16:28
tlater[m]You won't get a tmpfs, but at least you won't get host debris16:29
tlater[m]But to answer your actual question, there's no option to create a tmpfs for /dev/shm with BuildStream atm, you'd need to write a patch for that.16:30
tlater[m]Probably not too difficult.16:30
coldtomta, it's just that if something in the sandbox needs to use /dev/shm, i'd rather it not use the host one16:31
tlater[m]coldtom: You won't be16:34
tlater[m]Unless you're running a `bst shell`16:34
tlater[m]But `bst shell` doesn't produce anything anyway, so that should be ok.16:35
tlater[m]We only --dev-bind /dev/null|zero|full|...16:35
tlater[m]Except in interactive shells16:35
tlater[m]Where we bind everything so you can access your console16:35
*** traveltissues has quit IRC16:37
*** lachlan has joined #buildstream16:59
gitlab-br-botmarge-bot123 merged MR !1685 (link_fix->master: fix broken user guide link in README.rst) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/168517:01
*** santi has joined #buildstream17:03
*** lachlan has quit IRC17:06
*** lachlan has joined #buildstream17:06
*** lachlan has quit IRC17:11
*** lachlan has joined #buildstream17:16
*** lachlan has quit IRC17:24
*** tiagogomes has quit IRC17:43
*** santi has quit IRC18:02
*** phoenix has joined #buildstream18:03
*** lachlan has joined #buildstream18:17
benschuberttpollard: https://gitlab.com/BuildStream/buildstream/merge_requests/1613#note_239793750 :(18:23
*** lachlan has quit IRC18:24
*** jonathanmaw has quit IRC21:06
*** rdale has quit IRC21:06
*** phoenix has quit IRC21:20

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!