| *** tristan <tristan!tristan@223.62.169.92> has joined #buildstream | 02:52 | |
| *** ChanServ sets mode: +o tristan | 02:52 | |
| *** tristan <tristan!tristan@223.62.169.92> has quit IRC | 03:16 | |
| *** tristan <tristan!tristan@223.62.169.218> has joined #buildstream | 08:42 | |
| *** ChanServ sets mode: +o tristan | 08:42 | |
| *** tristan <tristan!tristan@223.62.169.218> has quit IRC | 08:45 | |
| juergbi | nanonyme: juerg/overlayfs branches of buildbox-common and buildbox-run-bubblewrap are mostly working | 14:42 |
|---|---|---|
| nanonyme | Cool | 14:43 |
| juergbi | seeing two buildstream integration test failures, one is the read-only test (with overlayfs, buildbox-fuse can't deny writes) and the other is an incremental workspace test. not sure why that's failing | 14:43 |
| juergbi | it might be in good enough state to build freedesktop-sdk, if you want to give this a try | 14:43 |
| juergbi | requires linux 5.11 (and unprivileged user namespaces) | 14:44 |
| nanonyme | Ungh. Looks like Debian stable has 5.10 https://packages.debian.org/search?keywords=linux-image-amd&searchon=names&suite=stable§ion=all | 14:45 |
| juergbi | is that what the runners are using or your local system? | 14:45 |
| nanonyme | It's what the runners are using, yes. x86_64 runners are running on top of Debian 11 AMI | 14:46 |
| nanonyme | + customizations | 14:46 |
| nanonyme | juergbi: might take a bit of extra sorting out | 14:48 |
| nanonyme | juergbi: wouldn't it be easier to just block the opens for write access than blocking the writes? | 14:49 |
| juergbi | nanonyme: we use the no-open optimization, so open requests won't reach buildbox-fuse | 14:50 |
| juergbi | however, that's not the issue here. the issue is that buildbox-fuse won't see any write-related activity with the overlay active | 14:51 |
| juergbi | an overlay's 'lower' filesystem is always treated as read-only | 14:51 |
| nanonyme | Yeah, but kernel will assumably block the write if file is opened in read-only mode | 14:51 |
| juergbi | yes but the overlay will trigger a copy-up, it won't allow the lower filesystem to block anything | 14:52 |
| nanonyme | Hm | 14:52 |
| juergbi | read-only bind mounts on top of the overlay might do the trick | 14:53 |
| juergbi | another option could be to mark the releveant files and directories read-only (via file mode). wouldn't prevent chmod and then writing but the whole thing is anyway about detection of incorrect/accidental writes, not security | 15:05 |
| nanonyme | Yeah | 18:13 |
| nanonyme | juergbi: do you think we could have some synthetic benchmarks between native, plain buildbox-fuse and buildbox-fuse+overlayfs? | 18:14 |
| nanonyme | I'm not sure we can immediately integrate that stuff due to kernel requirements | 18:14 |
| nanonyme | Just to see if this is worth further work | 18:16 |
| juergbi | nanonyme: I've tested flatpak build-export for platform+sdk+pre-bootstrap | 19:05 |
| juergbi | with plain buildbox-fuse it takes 42s, with buildbox-fuse + overlayfs it takes 32s | 19:05 |
| nanonyme | It's suspiciously fast even with plain buildbox-fuse | 19:07 |
| juergbi | it's just these three parts, not the whole stack, and on a reasonably fast desktop system | 19:08 |
| nanonyme | juergbi: did you use the debug extension? | 19:08 |
| nanonyme | There's several gigabytes of debuginfo | 19:08 |
| juergbi | no, that's without -debug.bst elements | 19:08 |
| juergbi | timing with the sdk-debug.bst would be useful? | 19:09 |
| nanonyme | Well, it's by far the biggest component there and also probably has biggest individual files | 19:10 |
| juergbi | let's see whether I'm not running out of disk space | 19:11 |
| juergbi | btw: there is still some optimization potential in the overlayfs variant, it currently has to hash new/modified files twice | 19:11 |
| juergbi | but with regards to writes it should already be efficient (could further be optimized with reflinks for capture, independent of overlayfs being used or not) | 19:12 |
| juergbi | nanonyme: also including sdk-debug it's 1m41s vs. 1m10s | 19:23 |
| nanonyme | juergbi: the thing is, it takes us *hours* to run full flatpak repo | 19:24 |
| nanonyme | So something is now clearly off | 19:24 |
| nanonyme | I am really getting the feeling we need to test with newer kernel | 19:25 |
| juergbi | this is with fast local I/O. ext4 (+LUKS) on NVMe. no VM | 19:25 |
| juergbi | 5.10 is not that old, though. I don't remember huge FUSE optimizations in the meantime | 19:26 |
| nanonyme | Well, what led us to suspect FUSE in the first place is bst1 on exact same runner was magnitude faster | 19:30 |
| nanonyme | Since as we know bst1 doesn't actually use FUSE | 19:31 |
| juergbi | (it does for some cases but presumably not in this context) | 19:32 |
| nanonyme | Well, we tested the build sandbox, it did not | 19:33 |
| nanonyme | And this is normal case for majority of builds. | 19:34 |
| juergbi | yes. it's mainly integration commands where FUSE is used, iirc | 19:36 |
| nanonyme | juergbi: I guess this overlayfs would also help heaps with integration commands? | 19:36 |
| nanonyme | Right | 19:37 |
| juergbi | if they are write-bottlenecked. not sure they are | 19:37 |
| nanonyme | I am reading through buildbox-fuse cmake definitions. Can you please open up in simple English whether https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/CMakeLists.txt#L69-74 ; does it have any impact other than for testing? | 19:53 |
| nanonyme | We use very limited build sandboxes and I'm pretty sure those things are not in it | 19:54 |
| juergbi | that's only for testing | 19:56 |
| nanonyme | Right. It looks almost impossible to build buildbox-fuse wrong then | 19:56 |
| juergbi | nanonyme: fdo-sdk docker-images seems to still be on buildbox-fuse 0.0.61, not 0.0.63, though | 19:58 |
| juergbi | it includes an opendir optimization. however, I don't expect it to make that much of a difference | 19:59 |
| nanonyme | I can fix that. We don't currently have automatic updates for docker repo, I've been asking every now and then if we should | 20:00 |
| juergbi | could we easily e.g. run top on the CI VM while it's building flatpak-release? | 20:04 |
| juergbi | just to figure out where CPU time is going or whether there is a lot of I/O wait? | 20:04 |
| nanonyme | Well, I can't. jjardon and benjaminb (https://matrix.to/#/#freedesktop-sdk:matrix.org) probable can at least get the to VM. | 20:13 |
| nanonyme | I am anyway updating all builbox components now https://gitlab.com/freedesktop-sdk/infrastructure/freedesktop-sdk-docker-images/-/merge_requests/356 | 20:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!