| *** narispo has quit IRC | 00:55 | |
| *** narispo has joined #buildstream | 00:55 | |
| *** narispo has quit IRC | 01:03 | |
| *** narispo has joined #buildstream | 01:03 | |
| *** narispo has quit IRC | 06:18 | |
| *** narispo has joined #buildstream | 06:18 | |
| *** slaf_ has joined #buildstream | 06:59 | |
| *** slaf_ has joined #buildstream | 06:59 | |
| *** slaf_ has joined #buildstream | 06:59 | |
| *** slaf_ has joined #buildstream | 06:59 | |
| *** slaf_ has joined #buildstream | 07:00 | |
| *** slaf_ has joined #buildstream | 07:00 | |
| *** slaf_ has joined #buildstream | 07:01 | |
| *** slaf_ has joined #buildstream | 07:01 | |
| *** slaf has quit IRC | 07:01 | |
| *** slaf has joined #buildstream | 07:01 | |
| *** slaf has joined #buildstream | 07:01 | |
| *** slaf has joined #buildstream | 07:02 | |
| *** bochecha has joined #buildstream | 07:12 | |
| gitlab-br-bot | juergbi opened MR !1583 (juerg/await-threads->master: utils.py: Wait for threads to exit in _is_single_threaded()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1583 | 07:43 |
|---|---|---|
| *** jjardon changes topic to "BuildStream 1.4.1 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://docs.buildstream.build/ | IRC logs: https://irclogs.baserock.org/buildstream | Mailing List: https://mail.gnome.org/mailman/listinfo/buildstream-list | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmaps" | 07:51 | |
| jjardon | Hi, would it be possible to have a 1.9.x snapshot of current master? last one was 1.3.1 and it can be a bit confused now that 1.4.0 is out | 07:55 |
| *** bochecha has quit IRC | 08:08 | |
| juergbi | jjardon: I think we only created a tag for 1.3.0 (for proper git version), not an actual release. we could do the same with 1.90.0 | 08:09 |
| jjardon | I think that would help yes | 08:14 |
| *** rdale has joined #buildstream | 08:23 | |
| juergbi | Kinnison: hopefully the last MR with regards to the thread issue: https://gitlab.com/BuildStream/buildstream/merge_requests/1583 | 08:23 |
| * Kinnison looks | 08:23 | |
| juergbi | I also added a _reset_global_state method to https://gitlab.com/BuildStream/buildstream/merge_requests/1576 | 08:24 |
| juergbi | (resulting in existing tests working as regression tests) | 08:25 |
| benschubert_ | juergbi: thanks a lot for this one :) | 08:26 |
| benschubert_ | juergbi: for the threads, couldn't we join them ourselves? | 08:27 |
| juergbi | benschubert_: not sure. gRPC can use both Python and low level threads. the low level ones we presumably can't portably join in Python | 08:32 |
| juergbi | at least some of the low level ones are actually joined by gRPC core on close/shutdown. however, apparently sometimes there is still a thread left | 08:32 |
| juergbi | it's not frequent that we have to actually wait, so I don't think it's a performance issue (the method is anyway not called often outside the test suite) | 08:34 |
| juergbi | as I can't reproduce it locally, it's a bit cumbersome to try out other approaches | 08:34 |
| juergbi | btw: gRPC python's (incomplete) prefork support actually also has a wait loop for its threads (although that uses condition variables, which we don't have access to, instead of sleep) | 08:35 |
| benschubert_ | juergbi: ok fair enough, if it's only the test suite I'm fine with it :) | 08:36 |
| juergbi | it's not just in the test suite but outside the test suite it's used max. twice per 'bst' process | 08:37 |
| gitlab-br-bot | BenjaminSchubert approved MR !1583 (juerg/await-threads->master: utils.py: Wait for threads to exit in _is_single_threaded()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1583 | 08:37 |
| juergbi | and it should never actually trigger waiting outside the test suite | 08:37 |
| Kinnison | juergbi: 👍 on both those changes | 08:38 |
| juergbi | ta | 08:38 |
| * Kinnison sees that the node test suite failed because of a thread :D | 08:38 | |
| gitlab-br-bot | BenjaminSchubert approved MR !1576 (juerg/synthetic-file-index->master: node.pyx: _SYNTHETIC_FILE_INDEX must not be module-private) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1576 | 08:40 |
| qinusty | Hey, I'm currently looking at the remote sandbox builds and noticing that for the autotools example, the working directory is set to `/buildstream/autotools/hello.bst/doc/amhello` before the autotools plugin script is ran. I'm not certain how this works exactly with buildgrid but the amhello directory of autotools doesn't appear to contain ./config | 08:41 |
| qinusty | ure, ./autogen or any of the other scripts in the configure step. | 08:41 |
| qinusty | Am I missing something? D: | 08:41 |
| juergbi | qinusty: doc/amhello contains configure.ac | 08:44 |
| qinusty | this is true, is this ran by autoreconf? As the if statement within the autotools.yaml plugin element file, doesn't appear to mention this | 08:46 |
| juergbi | qinusty: autoreconf -ivf creates configure from configure.ac | 08:47 |
| juergbi | autoreconf is essentially identical to a minimal autogen.sh | 08:48 |
| qinusty | Ah okay, I don't appear to have autoreconf installed on my sandboxed system which is likely causing me issues | 08:48 |
| juergbi | it should be part of autoconf | 08:50 |
| qinusty | I would expect this to be included in buildstream/buildstream:nightly-extra? | 08:53 |
| juergbi | qinusty: it might not be. buildstream itself doesn't need it | 08:54 |
| benschubert_ | qinusty: that autoreconf if from inside the sandbox, You'll need a build-dep on an autoconf package :) | 08:54 |
| qinusty | Using doc/examples/autotools though surely this works currently? I know it works with buildgrid currentlyu | 08:57 |
| qinusty | currenty* | 08:57 |
| benschubert_ | mmh that's correct | 08:58 |
| juergbi | yes, I assume it's part of the alpine base image | 08:59 |
| juergbi | that's independent of the buildstream nightly docker image | 08:59 |
| juergbi | when run with buildstream the alpine base image should automatically be staged in the sandbox | 08:59 |
| qinusty | autoreconf doesn't exist in alpine:latest | 08:59 |
| juergbi | or via remote execution that would be in the input root | 08:59 |
| juergbi | oh, hm | 08:59 |
| qinusty | I'm confused where it's coming from D: | 09:00 |
| juergbi | $ tar -tf integration-tests-base.v1.x86_64.tar.xz | grep autoreconf | 09:00 |
| juergbi | ./usr/bin/autoreconf | 09:00 |
| juergbi | it's in the alpine tarball I see here | 09:00 |
| qinusty | Ah, I pulled the docker container | 09:01 |
| juergbi | if you're referring to a docker image with 'alpine:latest', that's independent | 09:01 |
| juergbi | buildstream itself will pull the alpine tarball for the test and it's in there | 09:01 |
| qinusty | Aha, this might get me somewhere | 09:02 |
| juergbi | https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz | 09:03 |
| qinusty | Buildbarn is receiving /usr/bin/autoconf in the input_files but it appears that it isn't part of $PATH, This is likely because I have placed the input files as a subdir of / | 09:03 |
| qinusty | Therefore /usr/bin is no longer path | 09:03 |
| juergbi | yes, buildstream actions generally won't work if the input root is not the filesystem root | 09:05 |
| qinusty | Hm | 09:05 |
| qinusty | It is desirable to the point of avoiding awkwardness, and as suggested by Ed @ https://github.com/buildbarn/bb-remote-execution/issues/18#issuecomment-525313758 that the input_root be made a subdir of / | 09:06 |
| Kinnison | That comment is about buildbarn not actions | 09:07 |
| Kinnison | i.e. that buildbarn should stop putting extra droppings *inside* what the action provided | 09:07 |
| Kinnison | `input_root` is still where the action's input should go | 09:07 |
| qinusty | ├── .stdout.txt | 09:08 |
| qinusty | ├── .stderr.txt | 09:08 |
| qinusty | └── input_root | 09:08 |
| qinusty | ├── bin | 09:08 |
| qinusty | └── usr | 09:08 |
| qinusty | This is kinda how it looks right now, with other folders | 09:09 |
| qinusty | So all of buildstream provided input_files are thrown into input_root | 09:09 |
| Kinnison | Right, and the chroot should be `input_root` | 09:09 |
| * qinusty isn't sure if there is a chroot somewhere. If there is, can you chroot inside a chroot? 🤔 | 09:10 | |
| Kinnison | Yes you can; but buildstream actions won't work unless the `input_root` is the root of the filesystem tree as far as the build processes are concerned | 09:10 |
| Kinnison | this is done by means of buildbarn's worker chrooting into `input_root` before exec()ing the action | 09:11 |
| Kinnison | At least, it should be | 09:11 |
| Kinnison | :D | 09:11 |
| qinusty | This is likely already done but for build_directory | 09:11 |
| juergbi | bazel-oriented workers/platforms never do that, so that's not a surprise | 09:11 |
| Kinnison | Right, which is a bug | 09:11 |
| qinusty | Well, the idea of build_directory being separate to input_directory is neq | 09:12 |
| juergbi | bazel always expects tools present outside the input root | 09:12 |
| qinusty | new* | 09:12 |
| juergbi | to support buildstream clients, the server/worker either has to directly support this extra chroot, e.g., triggered via platform property | 09:13 |
| juergbi | or buildstream has to use chroot as action command | 09:13 |
| Kinnison | juergbi: the latter requiring that the action run as root presumably | 09:14 |
| juergbi | yes, although it can be root in a user namespace, or we could use bwrap instead of chroot | 09:14 |
| Kinnison | true | 09:14 |
| juergbi | it shouldn't be a big effort to try this out | 09:14 |
| * qinusty wonders if anyone has any proposals for a preferred solution | 09:15 | |
| juergbi | however, it's not so easy to do it right as this is a platform-specific thing, of course | 09:15 |
| juergbi | my preferred solution would be a platform property in standard REAPI that will trigger this on the server/worker side | 09:15 |
| juergbi | right now REAPI is underspecified in this regard. if you just use what's in the spec, the client has no idea what kind of environment the action will run in | 09:16 |
| juergbi | what path, what tools will be available etc. | 09:16 |
| juergbi | this generally can't work, also not for bazel | 09:16 |
| juergbi | Google RBE has a (non-standard) platform property to allow specifying a docker image for the action to run in | 09:16 |
| juergbi | that is theoretically good enough (but it's still a bit inconvenient for buildstream as it's platform specific, and doesn't work at all on non-Linux) | 09:17 |
| juergbi | we should probably make a proposal for such a platform property | 09:18 |
| juergbi | and ideally provide patches for buildbarn and buildfarm | 09:18 |
| juergbi | or at least bring it up on mailing list or in the monthly meeting | 09:19 |
| qinusty | And this would be a modification to the REAPI proto? | 09:20 |
| juergbi | addition | 09:22 |
| juergbi | could be part of the platform lexicon, i.e., considered optional to implement | 09:22 |
| *** jonathanmaw has joined #buildstream | 09:25 | |
| qinusty | I guess we should detail this up in an issue somewhere. I'm happy to summarise this somewhere, which issue tracker would be preferred for this? Buildstream/bazel etc? | 09:27 |
| juergbi | with the goal of a platform property, filing an issue in https://github.com/bazelbuild/remote-apis would be the right place | 09:30 |
| juergbi | I think it would be good to file this directly with a full proposal | 09:31 |
| juergbi | especially as I don't expect other clients to be interested at this point | 09:31 |
| gitlab-br-bot | marge-bot123 merged MR !1583 (juerg/await-threads->master: utils.py: Wait for threads to exit in _is_single_threaded()) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1583 | 09:41 |
| *** traveltissues has joined #buildstream | 09:44 | |
| gitlab-br-bot | marge-bot123 merged MR !1576 (juerg/synthetic-file-index->master: node.pyx: _SYNTHETIC_FILE_INDEX must not be module-private) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1576 | 10:12 |
| *** narispo has quit IRC | 10:47 | |
| *** narispo has joined #buildstream | 10:47 | |
| gitlab-br-bot | marge-bot123 closed issue #1118 (ProvenanceInformation has no attribute `toplevel`) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1118 | 11:12 |
| gitlab-br-bot | marge-bot123 merged MR !1582 (traveltissues/1118->master: Fix typo in node attribute) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1582 | 11:12 |
| *** rdale has quit IRC | 11:19 | |
| *** bochecha has joined #buildstream | 11:30 | |
| juergbi | trivial docker image MR to update buildbox for disk usage support | 11:50 |
| juergbi | https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/134 | 11:50 |
| juergbi | any objections? I've sanity checked the images with https://gitlab.com/BuildStream/buildstream/merge_requests/1564 | 11:50 |
| valentind | Is there an issue about being able to run commands with a different sysroot within the sandbox? | 11:56 |
| valentind | I see this as a recurring problem when wanting to run integration commands on some subdirectory. | 11:57 |
| valentind | Alternatively being able to have multiple sandboxes for one build of an element would be nice. | 11:58 |
| juergbi | valentind: I'm not aware of an issue for this. the closest is your !894 | 11:58 |
| gitlab-br-bot | MR !894: Add support for importing build dependencies in sysroots. https://gitlab.com/BuildStream/buildstream/merge_requests/894 | 11:59 |
| valentind | Right, we mention that in there. | 11:59 |
| juergbi | I generally agree that there are use cases where the current structure is not a very good fit | 12:00 |
| valentind | But there are plugins where we want to run integration commands, but we do not stage files in the root of the sandbox. | 12:00 |
| juergbi | however, I'm not sure yet how exactly we should improve it | 12:00 |
| juergbi | workaround with a helper element for integration is possible, right? but could definitely be cumbersome | 12:01 |
| valentind | I think it would be nice to have a new parameter to sandbox.run() | 12:01 |
| valentind | The problem is that you lose chmods | 12:01 |
| valentind | Currently. | 12:01 |
| juergbi | hm, ok, but we might be able to fix that soon | 12:02 |
| juergbi | based on earlier discussions I think tristan would prefer keeping elements simple (i.e., single sandbox) but providing more convenience for defining multiple elements at once | 12:03 |
| juergbi | for reference: support for additional file properties in CAS: https://github.com/bazelbuild/remote-apis/issues/90 | 12:03 |
| juergbi | I was considering 'generator elements' a while ago. i.e., plugins that can generate multiple elements to cover multi-step tasks | 12:05 |
| juergbi | this could be, e.g., tests or for different packaging mechanism | 12:05 |
| juergbi | I don't have a fleshed out proposal, though | 12:06 |
| jennis | Hi all, would appreciate reviews on !1565 (update checkout to handle artifact refs) and !1568 (update push/pull to handle artifact refs) before I merge | 12:10 |
| gitlab-br-bot | MR !1565: Update artifact checkout to handle artifact refs https://gitlab.com/BuildStream/buildstream/merge_requests/1565 | 12:10 |
| gitlab-br-bot | MR !1568: Update artifact push and pull so that they can handle artifact refs https://gitlab.com/BuildStream/buildstream/merge_requests/1568 | 12:10 |
| jennis | Hi guys, I've made a start on trying to solidify our design principles for the frontend. I've actually found this pretty hard to write down, so I'd appreciate any ideas/feedback on !1584: https://gitlab.com/BuildStream/buildstream/merge_requests/1584 | 12:19 |
| gitlab-br-bot | MR !1584: WIP: Document forntend design principles https://gitlab.com/BuildStream/buildstream/merge_requests/1584 | 12:19 |
| Kinnison | jennis: !1568 👍 | 12:24 |
| Kinnison | jennis: !1565 one nit | 12:26 |
| *** jward has quit IRC | 12:41 | |
| *** jward has joined #buildstream | 12:41 | |
| jjardon | Hi, is there a minimal BuildBox version to use BuildStream master? | 12:49 |
| tpollard | jjardon: https://buildstream.build/buildbox_install.html | 12:51 |
| jjardon | ah nevermind, already seen the comments at https://buildstream.build/buildbox_install.html#building-buildbox-components | 12:51 |
| tpollard | :) | 12:51 |
| jjardon | So I guess buildbox-casd replaces the local cache buildstream uses in 1.x ? | 12:54 |
| gitlab-br-bot | juergbi opened (was WIP) MR !1564 (juerg/cache-usage->master: Display cache usage in status bar) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1564 | 12:58 |
| juergbi | jjardon: yes. read access still uses direct filesystem access but all writes go via buildbox-casd, and expiry is also handled by buildbox-casd | 13:00 |
| juergbi | pulling from and pushing to remote CAS servers is also handled by buildbox-casd | 13:00 |
| jjardon | cool, thanks for the info | 13:01 |
| jjardon | what is buildbox-fuse used for? | 13:01 |
| juergbi | that's a replacement for hardlinking + pyfuse + bubblewrap sandboxing | 13:02 |
| juergbi | it's implemented as a sandboxing backend in buildstream. that's still considered experimental, though, and is missing a few features | 13:02 |
| juergbi | jjardon: fyi: buildbox-casd expiry is currently enabled only if a quota is configured, i.e., it doesn't monitor free disk space yet. will hopefully be fixed soon: https://gitlab.com/BuildGrid/buildbox/buildbox-casd/issues/28 | 13:04 |
| jjardon | I see, so it is meant to replace buildbox-run-* projects? Or is it a wrapper around them? | 13:04 |
| juergbi | the other way round. the bubblewrap sandboxing part will be moved from buildbox-fuse to buildbox-run-bubblewrap | 13:05 |
| juergbi | buildstream will then invoke buildbox-run and buildbox-run will handle the rest | 13:05 |
| jjardon | ah ok! | 13:06 |
| *** rdale has joined #buildstream | 13:13 | |
| *** rdale has quit IRC | 13:44 | |
| juergbi | does anyone want to review !1564 and hasn't done so yet? | 13:45 |
| juergbi | as website and docker images have been updated, it's no longer marked WIP | 13:45 |
| *** rdale has joined #buildstream | 13:45 | |
| juergbi | "Display cache usage in status bar" https://gitlab.com/BuildStream/buildstream/merge_requests/1564 | 13:45 |
| gitlab-br-bot | jjardon opened MR !1585 (jjardon/doc_install_update->bst-1: doc/source/install_linux_distro.rst: Update info) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1585 | 13:47 |
| tpollard | if it's adding multiprocessing to cascache aevri might be interested? | 13:47 |
| juergbi | good point. this might need a tweak for non-fork platforms, but I don't expect it requiring a big change | 13:49 |
| tpollard | yep nothing has set off any alarms for me | 13:51 |
| *** bochecha has quit IRC | 13:59 | |
| ironfoot | I've noticed in a couple of cases now that some aliases, that I didn't have definied in project.conf, worked, maybe because I had the repos cached? | 14:05 |
| ironfoot | is this a known issue? | 14:05 |
| juergbi | hm, doesn't sound familiar | 14:06 |
| ironfoot | then, my theory is that bst looks into the sources cache, using the alias, before trying to resolve the alias at all | 14:07 |
| ironfoot | I feel like the cache should be stored using the resolved alias | 14:08 |
| ironfoot | oh, maybe I've seen this issue only because the built artifact was cached, will check | 14:10 |
| juergbi | using the alias for caching is intentional, iirc, as changing the mirror shouldn't invalidate the cache | 14:14 |
| juergbi | we should in any case catch undefined aliases, though | 14:14 |
| aevri | !1564 seems to keep tests_spawn_multiprocess_start_method happy, so I'm happy too :) Thanks for thinking about spawn | 14:17 |
| gitlab-br-bot | MR !1564: Display cache usage in status bar https://gitlab.com/BuildStream/buildstream/merge_requests/1564 | 14:17 |
| ironfoot | true, using the alias may make sense after all :) | 14:18 |
| gitlab-br-bot | jjardon merged MR !1585 (jjardon/doc_install_update->bst-1: doc/source/install_linux_distro.rst: Update info) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1585 | 14:28 |
| ironfoot | confirmed, if the source is cached, then it will not fail to resolve the alias | 14:40 |
| tpollard | interrupt handling when we're doing source cache is really buggy, almost always grpc can't recover | 15:00 |
| tpollard | which means we error out because it's unhandled | 15:03 |
| juergbi | tpollard: interrupt handling as in connection interruption to a remote source cache? | 15:17 |
| juergbi | don't we handle this the same way as with artifact caches, with automatic retries? | 15:18 |
| gitlab-br-bot | palvarez89 opened issue #1119 (BuildStream fails to detect missing source alias if already cached) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1119 | 15:19 |
| ironfoot | ended up opening that for the issue I mentioned before ^ | 15:20 |
| tpollard | juergbi: a local process in this case, I'm just writing up a ticket for it | 15:22 |
| gitlab-br-bot | tpollard opened issue #1120 (Interrupt handling during a task that involves cascache grpc processes is very flaky) on buildstream https://gitlab.com/BuildStream/buildstream/issues/1120 | 15:29 |
| tpollard | juergbi ^ | 15:29 |
| juergbi | hm, can't connect to casd anymore. wondering whether it's an issue on the casd side | 15:32 |
| gitlab-br-bot | marge-bot123 merged MR !1573 (jennis/tasks->master: Improve long-running task reporting) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1573 | 15:38 |
| gitlab-br-bot | tpollard approved MR !1568 (jennis/update_push_pull->master: Update artifact push and pull so that they can handle artifact refs) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/1568 | 15:42 |
| gitlab-br-bot | jjardon closed issue #535 (BuildStream doesn't fix the version of its dependencies) on buildstream https://gitlab.com/BuildStream/buildstream/issues/535 | 16:18 |
| *** traveltissues has quit IRC | 16:54 | |
| *** phildawson has quit IRC | 17:04 | |
| *** tiagogomes has quit IRC | 17:04 | |
| *** jonathanmaw has quit IRC | 17:04 | |
| *** tpollard has quit IRC | 17:04 | |
| *** raoul has quit IRC | 17:04 | |
| *** tpollard has joined #buildstream | 17:04 | |
| *** tiagogomes has joined #buildstream | 17:05 | |
| *** jonathanmaw has joined #buildstream | 17:05 | |
| *** raoul has joined #buildstream | 17:05 | |
| *** phildawson has joined #buildstream | 17:05 | |
| *** tiagogomes_ has joined #buildstream | 17:12 | |
| *** tiagogomes has quit IRC | 17:14 | |
| *** jonathanmaw has quit IRC | 18:12 | |
| *** narispo has quit IRC | 20:48 | |
| *** narispo has joined #buildstream | 20:48 | |
| *** narispo has quit IRC | 21:16 | |
| *** narispo has joined #buildstream | 21:17 | |
| *** narispo has quit IRC | 21:47 | |
| *** narispo has joined #buildstream | 21:47 | |
| *** narispo has quit IRC | 22:18 | |
| *** narispo has joined #buildstream | 22:18 | |
| *** narispo has quit IRC | 22:48 | |
| *** narispo has joined #buildstream | 22:48 | |
| *** rdale has quit IRC | 22:53 | |
| *** narispo has quit IRC | 23:18 | |
| *** narispo has joined #buildstream | 23:19 | |
| *** narispo has quit IRC | 23:49 | |
| *** narispo has joined #buildstream | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!