| *** tristan <tristan!tristan@223.62.216.84> has joined #buildstream | 03:49 | |
| *** ChanServ sets mode: +o tristan | 03:49 | |
| *** tristan <tristan!tristan@223.62.216.84> has quit IRC | 04:17 | |
| *** tristan <tristan!tristan@223.62.188.148> has joined #buildstream | 05:46 | |
| *** ChanServ sets mode: +o tristan | 05:46 | |
| *** tristan <tristan!tristan@223.62.188.148> has quit IRC | 07:23 | |
| *** tristan <tristan!tristan@223.62.188.148> has joined #buildstream | 07:23 | |
| *** ChanServ sets mode: +o tristan | 07:23 | |
| *** tristan <tristan!tristan@223.62.188.148> has quit IRC | 08:04 | |
| *** tristan <tristan!tristan@223.62.188.148> has joined #buildstream | 08:05 | |
| *** ChanServ sets mode: +o tristan | 08:05 | |
| *** tristan <tristan!tristan@223.62.188.148> has quit IRC | 08:38 | |
| *** tristan <tristan!tristan@223.33.164.138> has joined #buildstream | 09:57 | |
| *** ChanServ sets mode: +o tristan | 09:57 | |
| *** tristan <tristan!tristan@223.33.164.138> has quit IRC | 11:09 | |
| nanonyme | juergbi: | 19:18 |
|---|---|---|
| nanonyme | 2023-01-13T21:16:57.174+0200 [7187:140304246190016] [buildboxrun_bubblewrap.cpp:530] [DEBUG] Capturing command outputs... | 19:18 |
| nanonyme | 2023-01-13T21:16:57.175+0200 [7187:140304246190016] [buildboxcommon_localstageddirectory.cpp:122] [DEBUG] Uploading directory buildstream-build/. | 19:18 |
| nanonyme | 2023-01-13T21:16:57.176+0200 [7187:140304246190016] [buildboxcommon_casclient.cpp:396] [DEBUG] Uploading 102b51b9765a56a3e899f7cf0ee38e5251f9c503b357b330a49183eb7b155604 from string | 19:18 |
| nanonyme | 2023-01-13T21:16:57.176+0200 [7187:140304246190016] [buildboxcommon_casclient.cpp:451] [DEBUG] uploads/23b22f29-9754-4bde-a1a9-aef00e7421dc/blobs/102b51b9765a56a3e899f7cf0ee38e5251f9c503b357b330a49183eb7b155604/2: 2 bytes uploaded | 19:18 |
| nanonyme | 2023-01-13T21:16:57.176+0200 [7187:140304246190016] [buildboxrun_bubblewrap.cpp:532] [DEBUG] Finished capturing command outputs | 19:18 |
| nanonyme | 2023-01-13T21:16:57.177+0200 [7187:140304246190016] [buildboxcommon_localstageddirectory.cpp:65] [DEBUG] Unstaging /var/home/nanonyme/.cache/buildstream/cas/staging/overlayLYBkjS/overlay | 19:18 |
| nanonyme | That's with verbose | 19:18 |
| nanonyme | Ah, that's the end. So what's the beginning... | 19:20 |
| juergbi | nanonyme: isn't the full runner log part of the build log of the element? | 19:22 |
| nanonyme | Yeah. I was just trying to see it | 19:22 |
| nanonyme | I00000000 00:00:00.000000 7187 vlog_is_on.cc:213] RAW: Set VLOG level for "*" to 1 | 19:23 |
| nanonyme | 2023-01-13T21:16:57.042+0200 [7187:140304246190016] [buildboxcommon_runner.cpp:263] [DEBUG] [actionDigest=2286245010d61361d0d6cf56f257d8a0dd62ede77013e36a009a6807b0358af2/178] Initializing CAS client to connect to: "unix:/tmp/buildstreame | 19:23 |
| nanonyme | b3qhywm/cas/casserver-1x4p1b58.sock" | 19:23 |
| nanonyme | 2023-01-13T21:16:57.081+0200 [7187:140304246190016] [buildboxcommon_connectionoptions.cpp:262] [DEBUG] Creating grpc channel to [unix:/tmp/buildstreameb3qhywm/cas/casserver-1x4p1b58.sock] | 19:23 |
| nanonyme | 2023-01-13T21:16:57.114+0200 [7187:140304246190016] [buildboxcommon_casclient.cpp:95] [INFO] Setting d_maxBatchTotalSizeBytes = 4128768 bytes by default | 19:23 |
| nanonyme | 2023-01-13T21:16:57.116+0200 [7187:140304246190016] [buildboxcommon_runner.cpp:412] [DEBUG] [actionDigest=2286245010d61361d0d6cf56f257d8a0dd62ede77013e36a009a6807b0358af2/178] Fetching Command 21eabe3fbf69b69b6a9ebcbb751578cccb969fb9a8829051329e7933784d4c60/3304 | 19:23 |
| nanonyme | 2023-01-13T21:16:57.116+0200 [7187:140304246190016] [buildboxcommon_runner.cpp:433] [DEBUG] [actionDigest=2286245010d61361d0d6cf56f257d8a0dd62ede77013e36a009a6807b0358af2/178] Executing command | 19:23 |
| nanonyme | 2023-01-13T21:16:57.150+0200 [7187:140304246190016] [buildboxrun_bubblewrap.cpp:517] [DEBUG] Running in /var/home/nanonyme/.cache/buildstream/cas/staging/overlayLYBkjS/overlay/buildstream-build | 19:23 |
| nanonyme | 2023-01-13T21:16:57.150+0200 [7187:140304246190016] [buildboxcommon_runner.cpp:509] [DEBUG] [actionDigest=2286245010d61361d0d6cf56f257d8a0dd62ede77013e36a009a6807b0358af2/178] Created parent output directory: /var/home/nanonyme/.cache/buildstream/cas/staging/overlayLYBkjS/overlay/buildstream-build/.. | 19:23 |
| nanonyme | 2023-01-13T21:16:57.150+0200 [7187:140304246190016] [buildboxrun_bubblewrap.cpp:524] [DEBUG] Executing /usr/sbin/bwrap --unshare-pid --die-with-parent --bind /var/home/nanonyme/.cache/buildstream/cas/staging/overlayLYBkjS/overlay / --unshare-net --unshare-uts --hostname buildbox --unshare-ipc --dir buildstream-build --chdir buildstream-build --unshare-user --uid 0 --gid 0 --unsetenv SHELL --unsetenv COLORTERM --unsetenv XDG_MENU_PREFIX | 19:23 |
| nanonyme | --unsetenv HOSTNAME --unsetenv SSH_AUTH_SOCK --unsetenv DESKTOP_SESSION --unsetenv PWD --unsetenv XDG_SESSION_DESKTOP --unsetenv XDG_SESSION_TYPE --unsetenv TOOLBOX_PATH --unsetenv XAUTHORITY --unsetenv container --unsetenv HOME --unsetenv LANG --unsetenv XDG_CURRENT_DESKTOP --unsetenv VTE_VERSION --unsetenv WAYLAND_DISPLAY --unsetenv TERM --unsetenv USER --unsetenv DISPLAY --unsetenv SHLVL --unsetenv XDG_RUNTIME_DIR --unsetenv PS1 | 19:23 |
| nanonyme | --unsetenv XDG_DATA_DIRS --unsetenv PATH --unsetenv DBUS_SESSION_BUS_ADDRESS --unsetenv _ --unsetenv XDG_CACHE_HOME --unsetenv XDG_CONFIG_HOME --unsetenv XDG_DATA_HOME --setenv PATH /usr/bin:/bin:/usr/sbin:/sbin --setenv SHELL /bin/sh --setenv TERM dumb --setenv USER tomjon --setenv USERNAME tomjon --setenv LOGNAME tomjon --setenv LC_ALL en_US.UTF-8 --setenv HOME /tmp --setenv TZ UTC --setenv SOURCE_DATE_EPOCH 1320937200 --setenv CFLAGS | 19:24 |
| nanonyme | -O2 -pipe -g -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer --setenv CXXFLAGS -O2 -pipe -g -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection | 19:24 |
| nanonyme | -fno-omit-frame-pointer --setenv LDFLAGS -Wl,-z,relro,-z,now -Wl,--as-needed --setenv G_SLICE always-malloc --setenv PYTHON /usr/bin/python3 --setenv PYTHONHASHSEED 0 --setenv GZIP --no-name --setenv BSTARCH x86_64 --setenv FORCE_REBUILD 1 --setenv PWD /buildstream-build --proc /proc --tmpfs /tmp --tmpfs /dev/shm --dev-bind /dev/full /dev/full --dev-bind /dev/null /dev/null --dev-bind /dev/urandom /dev/urandom --dev-bind /dev/random | 19:24 |
| nanonyme | /dev/random --dev-bind /dev/zero /dev/zero sh -c -e (set -ex; sh -c -e 'sh Configure -des \ | 19:24 |
| nanonyme | I'm so going to be killing the IRC bridge | 19:24 |
| nanonyme | juergbi: did that truncate? | 19:24 |
| juergbi | yes, you really shouldn't paste that much in line | 19:24 |
| nanonyme | Myeah, with matrix it would work fine | 19:25 |
| nanonyme | juergbi: I can put that somewhere but not much useful stuff I can see. No failures or anything. | 19:26 |
| juergbi | and where do you see the `sh` error? | 19:28 |
| nanonyme | It's the previous line to 2023-01-13T21:16:57.174+0200 [7187:140304246190016] [buildboxrun_bubblewrap.cpp:530] [DEBUG] Capturing command outputs... | 19:29 |
| nanonyme | bwrap: execvp sh: No such file or directory | 19:29 |
| nanonyme | juergbi: so it sounds like this is potentially an empty sandbox but hard to say further than that since it was unmounted | 19:29 |
| nanonyme | juergbi: I tried opening just shell after the error and got https://nopaste.net/u84es8ULK0 | 19:34 |
| nanonyme | So it clearly looks as if what I'm getting is an empty sandbox | 19:35 |
| nanonyme | juergbi: I don't know though if it's fully empty sadly because this unmounting happens through atomic operation | 19:48 |
| juergbi | based on the log, the overlayfs mount doesn't seem to have failed, no idea why the sandbox would be empty or incomplete | 19:51 |
| juergbi | nanonyme: I did another quick test locally. it built a modified fdo-sdk element just fine | 19:52 |
| juergbi | what kernel version are you on? | 19:52 |
| juergbi | and can you verify that your `bwrap` is NOT installed setuid root? | 19:53 |
| nanonyme | Linux toolbox 6.0.18-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Jan 7 17:10:00 UTC 2023 x86_64 GNU/Linux | 19:53 |
| nanonyme | juergbi: it is not setuid root | 19:54 |
| juergbi | exactly the same (upstream) kernel version as here | 19:54 |
| nanonyme | Also absolutely no difference if I make it setuid root | 19:55 |
| juergbi | setuid root might rather break things. just wanted a sanity check | 19:55 |
| juergbi | wondering whether toolbox is the issue somehow but not sure how it could behave that way without the setup failing | 19:56 |
| nanonyme | Can we btw merge the checkout optimization or do you want further review? | 19:59 |
| juergbi | I saw a non-ostree test failure, will take another look | 20:00 |
| juergbi | nanonyme: maybe try running buildbox-run with strace -f and then copy the trace to pastebin? | 20:01 |
| juergbi | buildbox_command = [ | 20:01 |
| juergbi | "strace", | 20:01 |
| juergbi | "-f", | 20:01 |
| juergbi | self.__buildbox_run(), | 20:01 |
| juergbi | FAILED tests/artifactcache/expiry.py::test_never_delete_required - AssertionError: assert 'buildable' == 'cached' | 20:02 |
| juergbi | is this a generally flaky test? don't remember | 20:02 |
| nanonyme | Eh. I don't have strace in this container | 20:03 |
| nanonyme | I will need to create a new container that has it, I guess | 20:04 |
| juergbi | nanonyme: you might be able to use a copied strace binary. not sure if you have the library dependencies already | 20:05 |
| nanonyme | I can try that, kicked container rebuild in the meantime | 20:06 |
| nanonyme | juergbi: this is the strace https://nopaste.net/yitjVKiIdN | 20:14 |
| juergbi | hm, the setup looks all fine. don't see anything odd up to the /bin/sh ENOENT | 20:20 |
| nanonyme | I have manually tested overlay mount in toolbox using mount command line and it worked fine except mount binary wanted me to run it with sudo. | 20:23 |
| juergbi | what do you mean with 'wanted me to'? it failed without sudo? | 20:23 |
| nanonyme | mount: /var/home/nanonyme/foo: must be superuser to use mount. | 20:24 |
| nanonyme | But this is probably just built-in check in mount | 20:24 |
| juergbi | as unprivileged user, you do first have to enter an unprivileged user+mount namespace | 20:24 |
| juergbi | you can't mount anything (except for fuse) in the top-level mount namespace | 20:25 |
| nanonyme | Makes sense | 20:25 |
| juergbi | howerver, buildbox-run-bubblewrap handles that on its own. and overlay mount succeeds according to strace | 20:25 |
| juergbi | nanonyme: a hack for debugging could be to rename bwrap to bwrap.real and create a bwrap shell script that sleeps for a long time before executing bwrap.real | 20:26 |
| juergbi | at that point you could check what the overlay directory contains | 20:26 |
| juergbi | it is possible that the overlay directory is correct at that point but somehow bwrap can't bind it into its sandbox | 20:27 |
| juergbi | I'm not on the latest bubblewrap, still on 0.6.2. I wouldn't expect a significant behavior change in 0.7.0, though | 20:28 |
| nanonyme | juergbi: my bwrap was not executed by build o.O | 20:29 |
| nanonyme | Only when I try to enter shell | 20:29 |
| juergbi | cached build failure? | 20:30 |
| nanonyme | Ah, maybe | 20:30 |
| nanonyme | Anyway. /var/home/nanonyme/.cache/buildstream/cas/staging/overlayPvqnnZ/overlay/buildstream-build/ -> overlay upper work : three directories that are otherwise empty except work has a directory work in it that is restricted and when I try to ls it as superuser it says incompat | 20:32 |
| nanonyme | juergbi: where's the lower? | 20:33 |
| juergbi | nanonyme: a cas-tmpdir* in /var/home/nanonyme/.cache/buildstream/cas/staging | 20:33 |
| nanonyme | It has data | 20:34 |
| juergbi | oh, the issue is that the mount only applies to that mount namespace, of course. from the outside you can't actually access the contents :-/ | 20:34 |
| juergbi | including bin/sh ? | 20:35 |
| nanonyme | Yes | 20:36 |
| nanonyme | /var/home/nanonyme/.cache/buildstream/cas/staging/cas-tmpdirYQKfOA/bin/sh is a relative symlink to /var/home/nanonyme/.cache/buildstream/cas/staging/cas-tmpdirYQKfOA/bin/bash | 20:36 |
| juergbi | ok and I assume the latter exists as well | 20:38 |
| nanonyme | Yes | 20:38 |
| juergbi | this is the regular fuse mount, so that part seems to be working as expected | 20:38 |
| juergbi | wondering whether either the kernel or toolbox somehow have additional restrictions enabled | 20:39 |
| nanonyme | I still wonder if the problem is that we're already inside a user namespace when we run bwrap. Various tools like flatpak don't work properly inside toolbox | 20:39 |
| juergbi | really depends on how toolbox creates namespaces. that said, I would have expected some syscall to fail, not simply the overlay being empty | 20:41 |
| juergbi | do you know / have a pointer why flatpak doesn't work inside toolbox? that definitely seems like a bad sign | 20:42 |
| nanonyme | I can't recall. There was some discussion about it at some point. But now I just tried and it worked fine | 20:46 |
| nanonyme | juergbi: I guess I could try it in our regular CI which is not running inside toolbox. But I would really like it to gracefully fallback to old functionality even in toolbox if that cannot be supported | 20:50 |
| nanonyme | Or, well, I could but our kernel is too old :( | 21:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!