| *** mcatanzaro has quit IRC | 00:42 | |
| *** mrmcq2u[m] has left #buildstream | 03:24 | |
| *** tristan has joined #buildstream | 08:15 | |
| *** tm has joined #buildstream | 08:18 | |
| *** tm has quit IRC | 08:39 | |
| *** tm has joined #buildstream | 08:40 | |
| *** valentind has joined #buildstream | 08:49 | |
| *** toscalix has joined #buildstream | 08:51 | |
| *** jonathanmaw has joined #buildstream | 09:00 | |
| tlater | juergbi: Regarding #205 (Workspace builds might not rebuild correctly when dependecies are updated), is there anything left to discuss regarding it? | 09:31 |
|---|---|---|
| tlater | I don't see any issues with what you proposed on the gitlab issue, but I didn't actively follow the conversation | 09:31 |
| *** aday has joined #buildstream | 09:34 | |
| *** dominic has joined #buildstream | 09:59 | |
| *** adds68 has joined #buildstream | 10:04 | |
| juergbi | tlater: i assume you mean #215 | 10:04 |
| *** adds68 has quit IRC | 10:04 | |
| tlater | Ack, sorry, yes | 10:04 |
| juergbi | i'm not aware of any issues with that per se | 10:05 |
| juergbi | however, the part with the failed builds feels a bit odd | 10:06 |
| juergbi | adding those files to the workspace metadata | 10:06 |
| tlater | Why do we make a distinction between failed builds and successful builds for workspaces anyway? | 10:07 |
| tlater | Regarding incremental builds, anyway | 10:08 |
| tlater | If I understand this correctly, we'll update the timestamps for files from both successful and failed builds | 10:08 |
| juergbi | we essentially update the timestamps of files since the last successful build | 10:09 |
| juergbi | however, intermediate failed builds could have involved various changes in dependencies | 10:09 |
| juergbi | and we need those file lists to figure out what files were changed since the last successful build | 10:10 |
| tlater | Right, I guess there's no way around that then | 10:10 |
| juergbi | we could handle it closer to regular builds but that would be a big change | 10:11 |
| juergbi | and would depend on incremental build support for non-workspace builds | 10:12 |
| juergbi | i.e., just copy files from the workspace into the build tree, don't actually mount the workspace | 10:12 |
| juergbi | (and then cache the build tree same as for incremental non-workspace builds) | 10:12 |
| *** aday has quit IRC | 10:13 | |
| juergbi | one issue is the last sentence in my comment. reopening a workspace | 10:13 |
| *** ssam2 has joined #buildstream | 10:14 | |
| juergbi | we don't currently have a way to force a clean rebuild, afaik | 10:14 |
| jmac | No | 10:14 |
| jmac | Other than deleting all the refs in ostree | 10:14 |
| juergbi | i meant with regards to workspaces | 10:14 |
| juergbi | reopening a workspace with a dirty build tree would cause issues | 10:15 |
| tlater | Hm, I suspect that would be difficult to do in general as well | 10:15 |
| tlater | Because we can't rely on `make clean` | 10:15 |
| juergbi | i was always favoring separating source and build tree for workspaces (handled internally) | 10:15 |
| juergbi | this would also simplify the cache key situation with workspaces | 10:16 |
| juergbi | but without overlay/union fs available, we'd need to physically copy things around | 10:16 |
| juergbi | maybe we should still switch to that approach once incremental non-workspace builds are available | 10:16 |
| tlater | How would we handle code-modifying builds? | 10:17 |
| tlater | I.e., systems that replace @something@ in a shell script or somesuch | 10:17 |
| * tlater realizes this is awful practice, but spent his weekend working around that in an ebuild file | 10:17 | |
| juergbi | the file from the workspace should take precedence | 10:18 |
| juergbi | i.e., with such conflicts, the workspace file should win and be staged with updated timestamp | 10:18 |
| juergbi | it might make incremental build slow in such cases but projects should just be fixed | 10:18 |
| juergbi | modifying vcs-controlled files is a big no-go | 10:18 |
| tlater | Yup, that makes sense | 10:19 |
| jmac | BuildStream modifies vcs-controlled files! | 10:19 |
| juergbi | with explicit track, ok | 10:19 |
| juergbi | ;) | 10:19 |
| juergbi | but not as part of bst build without explicit track-save | 10:19 |
| tlater | Maybe we should rephrase as "modifying vcs-controlled files without user intervention is a big no-go" | 10:20 |
| tlater | Otherwise simply modifying code counts as a big no-go ;P | 10:21 |
| * tlater gathers from this that the solution to #215 is more-or-less temporary until we get to implementing a nicer way to separate workspace files from built files | 10:22 | |
| juergbi | there hasn't been a decision on this but i'd go into that direction | 10:23 |
| juergbi | having to stage the sources may be a performance disadvantage for big projects, not sure whether relevant | 10:24 |
| jmac | Is our current release 1.1.0? | 10:26 |
| juergbi | development release, yes | 10:27 |
| juergbi | stable is 1.0.1 | 10:27 |
| * tlater wonders how one packages something with this versioning scheme | 10:28 | |
| ssam2 | same as any of the other 1000s of projects that use it | 10:29 |
| ssam2 | basically 1.1.x should only ever go in "unstable" distros | 10:29 |
| ssam2 | stable distros should have either 1.0.x or 1.2.x (when it arrives) | 10:29 |
| tlater | I guess bleeding edge distros then just ignore the fact that uneven is considered unstable | 10:30 |
| *** valentind has quit IRC | 10:36 | |
| ironfoot | should we change the topic to point to 1.0.1 then? | 10:39 |
| gitlab-br-bot | buildstream: merge request (jmac/configurable-logging->master: WIP: Configurable log line formatting) #282 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/282 | 10:39 |
| tlater | jmac: I believe you need to enable gitlab's CI runners | 10:40 |
| tlater | On your private branch | 10:40 |
| * tlater wonders if we should make a point about this in Hacking.rst or something | 10:40 | |
| juergbi | i'd recommend creating branches in the main branch | 10:41 |
| juergbi | *repo | 10:41 |
| juergbi | any contributor will be given access | 10:41 |
| *** ssam3 has joined #buildstream | 10:41 | |
| jmac | I don't think that's appropriate for all contributors | 10:42 |
| jmac | I mean I'm happy for anyone to get access but I don't want to ask permission before I fork and start hacking | 10:42 |
| *** aday has joined #buildstream | 10:42 | |
| juergbi | you don't have to ask permission to fork it, of course | 10:42 |
| tlater | Just having a note about this somewhere might help confused first-time contributors | 10:43 |
| juergbi | my recommendation is purely based on being able to use the buildstream CI with cache and allowing rebases in MRs | 10:43 |
| *** ssam2 has quit IRC | 10:43 | |
| juergbi | if someone wants to work in their own repo, that's fine, but it can cause some inconveniences for MRs | 10:43 |
| jmac | Does CI get me much more than running the tests locally? I've never seen a CI job pass so I'm not sure what it does. | 10:43 |
| juergbi | it also tests the fallback backend | 10:44 |
| juergbi | and it also runs the integration tests. those can be run locally as well but are not run by default | 10:44 |
| juergbi | and it generates docs, which can also fail | 10:44 |
| tlater | jmac: The integration tests are the most important reason IMO, as the time to run them locally is atrocious | 10:44 |
| juergbi | with sam's Alpine branch merged it's much better, though | 10:45 |
| tlater | Oh, that happened? | 10:45 |
| tlater | \o/ | 10:45 |
| * tlater wanted to make that part of his refactoring, but tristan vetoed | 10:45 | |
| juergbi | yes, shouldn't have taken so long but test was failing first | 10:45 |
| juergbi | he commented that he has no issues with using Alpine, iirc | 10:46 |
| juergbi | jmac: regarding #38. not sure whether that's already clear to you but given the constraints, it may not make sense to change anything in buildstream itself at this point | 10:46 |
| juergbi | i.e., it's not clear whether it makes sense to handle those parts that we could handle even though we can't handle everything | 10:47 |
| juergbi | so we might just document best practice how to live with this limitation for now | 10:48 |
| juergbi | (or we could implement subuid support but with this projects that use this could only be built on hosts that have subuids configured) | 10:50 |
| jmac | I have no intention of changing anything at the moment; I expect it'll take me today and tomorrow just to figure out what the problem is | 10:53 |
| juergbi | makes sense, just wanted to make the above clear | 10:53 |
| jmac | Indeed, thanks | 10:54 |
| *** aday has quit IRC | 11:31 | |
| tlater | BB standup notes: | 11:44 |
| tlater | - The project-generation script is going to be open sourced, should hit our repo soon (1-3 days, as far as I can tell) | 11:44 |
| tlater | - 1-2 people from the London team will join the catch up meeting | 11:44 |
| tlater | laurence ^^ | 11:45 |
| jonathanmaw | did you mean that for a different window? | 11:45 |
| tlater | Erm, yep | 11:45 |
| *** aday has joined #buildstream | 11:45 | |
| *** tm has quit IRC | 11:52 | |
| gitlab-br-bot | buildstream: merge request (239-use-pylint-for-linting->master: WIP: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/283 | 11:58 |
| *** tm has joined #buildstream | 11:59 | |
| tristan | juergbi, after discussing a lot with Michael in the dead of night, I'm pivoting to focus on better support for 'running stuff in bst shell', and right now looking back at https://gitlab.com/BuildStream/buildstream/merge_requests/265 | 12:06 |
| tristan | juergbi, I think I'll throw away my concerns about sandbox flags API, we started with that pattern already, maybe we need to keep it and maybe it makes sense to have this granularity | 12:07 |
| tristan | asides from this, I'm wondering about optionality and how to control it | 12:07 |
| gitlab-br-bot | buildstream: merge request (sam/disable-ostree-free-space-check->master: _ostree.py: Disable OSTree's minimum disk space check in our repos) #285 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/285 | 12:08 |
| tristan | I think for a start; a single user facing switch to control "Is this a weak sandbox or a strong one" is better than exposing SandboxFlags granularity on the cli | 12:09 |
| juergbi | i agree on the CLI switch | 12:11 |
| juergbi | tristan: as the sandbox API is public, we have to be careful about what new flags we expose, though | 12:11 |
| tristan | Yeah | 12:11 |
| juergbi | but yes, it does fit with the existing pattern which is why i've implemented it that way | 12:11 |
| tristan | That's annoying | 12:11 |
| tristan | right, I now sort of wish we had less details exposed there | 12:12 |
| juergbi | or could we have additional private/internal members of that flags enum? | 12:12 |
| juergbi | not sure what the policy on something like this is | 12:12 |
| juergbi | would be an issue for plugins consuming flags (which doesn't exist) but shouldn't be an issue for just using/constructing flags | 12:13 |
| tristan | Right now, sandboxes are only implemented by us | 12:13 |
| tristan | And shell is a feature that only we use | 12:13 |
| juergbi | exactly | 12:14 |
| tristan | The only part of SandboxFlags which really needs to be public is ROOT_READ_ONLY | 12:14 |
| tristan | afaics | 12:14 |
| juergbi | we don't want plugins to create sandboxes with holes | 12:14 |
| juergbi | (strictly speaking, this wouldn't be a hole but still) | 12:14 |
| tristan | yeah, but existing things allow plugins to poke holes | 12:15 |
| juergbi | right | 12:15 |
| tristan | ok so, regarding that; I'll file an issue for it and we can yank that part of the API, maintaining only ROOT_READ_ONLY for compatibility, and claiming that nobody should have been using those flags anyway | 12:16 |
| tristan | i.e. removing those from public view is technically API break, but only for plugins which were broken | 12:17 |
| tristan | but for now expedite fixing Michael's more pressing issues | 12:17 |
| tristan | sounds reasonable ? | 12:17 |
| tristan | Regarding optionality; I think default `bst shell` of whichever brand, is a weak sandbox and not a strong one | 12:18 |
| tristan | because that will be least frustrating | 12:18 |
| tristan | Any ideas for a `bst shell --option-name` to denote that the sandbox should be... strong/isolated ? | 12:22 |
| juergbi | sounds reasonable to me | 12:23 |
| skullman | `bst shell --sandbox=restrictive`? | 12:24 |
| skullman | or just --sandbox=strict | 12:24 |
| gitlab-br-bot | buildstream: issue #265 ("SandboxFlags API allows too much control, remove some") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/265 | 12:24 |
| tristan | skullman, do you envision more than 2 "modes" ? | 12:25 |
| juergbi | naming things, one of the hard problems in computer science ;) | 12:25 |
| tristan | I am thinking; either there is one mode, or specific things are controllable | 12:25 |
| tristan | while later we could add specific things if it ever became important (like --enable-network, --root-rw, etc) | 12:25 |
| juergbi | there might be a project-specific setup for passthrough | 12:26 |
| tristan | yeah, damn names | 12:26 |
| juergbi | but that would not be on the command line | 12:26 |
| juergbi | as hardcoding X and DBUS env variables is really bad | 12:26 |
| tristan | juergbi, I would really hope to not have project-specific passthrough if it touches the builds, rather an option to specify specific build uid | 12:27 |
| juergbi | this would be for the weak/interactive shell, not for the build process, of course | 12:27 |
| tristan | right | 12:27 |
| tristan | indeed, that can possibly make sense | 12:28 |
| juergbi | just something to consider when thinking about future CLI | 12:28 |
| tristan | juergbi, yes that hard code also has to turn into project defined shelling setup options | 12:28 |
| tristan | so anything similar could sit under a `shell:` dictionary in project.conf | 12:28 |
| juergbi | yes, makes sense | 12:29 |
| juergbi | but we might still want a CLI option | 12:29 |
| tristan | we still need at least one | 12:29 |
| juergbi | or maybe some sort of profiles in project.conf? and CLI just specifying the profile | 12:29 |
| juergbi | might be too fancy | 12:29 |
| tristan | if default is for weak sandbox with testing capability; we want something to debug the actual build environment | 12:30 |
| juergbi | we have --build | 12:30 |
| tristan | I know; but a really unfortunate thing; is that people like Michael, *want* to work inside a `bst shell --build` | 12:30 |
| juergbi | right | 12:31 |
| tristan | There is also a reasonable need for openness in the --build shell | 12:31 |
| tristan | i.e. `make distcheck` | 12:31 |
| tristan | for people who write a `make check` that assumes it will run in a full login session on a desktop | 12:32 |
| juergbi | --secure | 12:32 |
| tristan | an assumption which I think is wrong for `make check`, but; in fact I have to admit that is a matter of opinion | 12:32 |
| juergbi | --unshare | 12:32 |
| juergbi | i share your opinion given that this makes CI impossible | 12:33 |
| tristan | --secure/--unshare/--restrictive | 12:33 |
| tristan | juergbi, or on the other hand; it explains why most meta-build systems rarely run a module's `make check` | 12:34 |
| tristan | i.e. only some rpm spec files do it; probably same for debian | 12:34 |
| tristan | only the ones which are probably known to work in isolation | 12:34 |
| tristan | --isolated | 12:34 |
| tristan | --strict is off the table, that's a `bst` toplevel option | 12:34 |
| juergbi | --protect | 12:35 |
| tristan | --sealed | 12:36 |
| juergbi | --restrict / --isolate / --protect | 12:36 |
| tristan | I think I'm liking --isolate | 12:38 |
| tristan | --secure and --protect dont seem to convey the right thing | 12:38 |
| tristan | like damage can be done | 12:38 |
| juergbi | yes, the focus is more about the sandbox not being influenced by the host, not the other way round | 12:38 |
| juergbi | (although the latter is also important) | 12:38 |
| juergbi | --isolate works for me | 12:38 |
| * tristan has to make sure that that is also what is being used in the interactive debug session | 12:39 | |
| juergbi | btw: one thing i'm missing with bst shell is support for multiple targets | 12:39 |
| tristan | i.e. when a build fails; we want --isolate | 12:39 |
| juergbi | unfortunately, it's problematic to add this the syntax | 12:39 |
| juergbi | right | 12:39 |
| tristan | multiple targets... yes very difficult to express | 12:40 |
| tristan | it could theoretically leverage the -- separator | 12:40 |
| tristan | but lots of work | 12:40 |
| tristan | juergbi, Michael asked for something which struck me as pretty wild, too, but I think this is going to be a matter of educating people and managing expectations | 12:42 |
| tristan | wild thing: A.) I open a shell.... B.) I modify something else in the stack... C.) I build again... D.) How come my open shell does not reflect the built result ?? | 12:42 |
| tristan | Ha! | 12:43 |
| juergbi | ah, yes | 12:44 |
| tristan | Anyway, amusing - it could be interesting to set a PS1 dynamically for a non-isolated `bst shell`, to reflect the abbreviated cache key in the prompt | 12:44 |
| tristan | making it clear that this is the shell for *this* cache key | 12:44 |
| * tristan runs with --isolated, and should have a patch up in an hour or so | 12:45 | |
| tristan | --isolated or --isolate, yeah just --isolate | 12:45 |
| juergbi | right, could be helpful | 12:45 |
| juergbi | agree, i prefer the latter | 12:45 |
| juergbi | it might actually be possible to support something close to what Michael asked for | 12:46 |
| tristan | I think that educating people that "There is no constant sysroot" is the best path; because developers will later discover they have more power | 12:46 |
| juergbi | but it's very tricky | 12:46 |
| tristan | juergbi, yeah; I thought of something like that before :) | 12:46 |
| juergbi | could keep something like a workspace open | 12:47 |
| tristan | Keeping some local state around to track that | 12:47 |
| tristan | and keep a build dir updated, pretty wild though | 12:47 |
| juergbi | yes, i don't think we want/need to focus on this now but it could be interesting | 12:47 |
| tristan | ssam3, btw; what was your env var hack thing which made one of the integration commands run much faster ? | 12:48 |
| juergbi | for many things a sensible alternative could be to directly run bst shell some-command instead of really executing an interactive shell | 12:48 |
| tristan | ssam3, I think disabling fdatasync on every write or smth | 12:48 |
| *** cs_shadow has quit IRC | 12:49 | |
| *** cs_shadow has joined #buildstream | 12:51 | |
| tristan | eeesh | 12:52 |
| tristan | I just noticed one more flaw :-/ | 12:52 |
| tristan | well, maybe not | 12:52 |
| persia | *please* don't disable fsync by default. | 12:52 |
| tristan | juergbi, how about root read only unconditionally in the shell ? | 12:52 |
| persia | There's a reason the package to do that in Debian is called "eatmydata" | 12:53 |
| tristan | persia, I'm talking about `ENV_VAR=True specific-long-running-integration-command` to be set in gnome-build-meta | 12:53 |
| tristan | just for that command | 12:53 |
| tristan | just for that project, for that command | 12:54 |
| persia | As long as you understand why it is usually critically important to never do that, I won't tell you not to if you have a good reason. That you are asking about syntax suggests the former may not be the case. | 12:54 |
| tristan | juergbi, right now; we root-read-only for the isolated case only, and allow writing to `/` in testing mode; which would mean `make dist` of epiphany would break on too many open fds again | 12:56 |
| tristan | but if we root-read-only, you can no longer `echo "nameserver 8.8.8.8 > /etc/resolv.conf"` in the shell | 12:57 |
| juergbi | right. i so hate it that linux doesn't have sensible immutable files. so painful | 12:57 |
| tristan | Well, ok I'll handle this; and will make root-read-only across the board for now; and add the default nameserver to gnome-build-meta for now | 12:58 |
| juergbi | shouldn't it rather be mount-bind from the host? | 12:58 |
| tristan | people will want more exotic things; like knowing about your host, and knowing that your host has a file named `/etc/resolv.conf` | 12:58 |
| tristan | but that will have to go into `shell:` configuration in project.conf, along with env vars and other payload/host specifics | 12:59 |
| juergbi | https://gitlab.com/BuildStream/buildstream/commit/dcc23d64cfbc7c08dc468162116ef6ec8cd920da | 12:59 |
| juergbi | yes, we should make this all configurable in the end | 12:59 |
| tristan | I couldnt find that MR yesterday, when Michael said it existed | 13:00 |
| juergbi | tristan: btw: the uid thing is not sufficiently helpful without also having suitable passwd/group entry | 13:00 |
| juergbi | it's not in a MR | 13:00 |
| tristan | looking at that link, I still cannot find that MR | 13:00 |
| juergbi | it's the juerg/gnome branch | 13:00 |
| tristan | oh | 13:00 |
| juergbi | as it was just an experiment | 13:00 |
| tristan | I'll root-read-only across the board for shell, and then consider a separate option if anyone complains they cannot modify the rootfs there | 13:01 |
| juergbi | ok | 13:02 |
| juergbi | we should also add a shell script to gnome-build-meta that quickly spins up a VM of GNOME but without having to go through image generation etc. for really fast testing in proper environment | 13:02 |
| tristan | I think that is a long way off; we still dont depend on freedesktop-sdk from GNOME | 13:03 |
| tristan | we need a bootable payload | 13:03 |
| *** ernestask has joined #buildstream | 13:06 | |
| ssam3 | tristan, it was `PKGSYSTEM_ENABLE_FSYNC=0` | 13:09 |
| ssam3 | I think it only has an effect for `update-mime-database` | 13:09 |
| ssam3 | https://bugzilla.redhat.com/show_bug.cgi?id=1052173 and https://bugs.freedesktop.org/show_bug.cgi?id=70366 have lots of overheated debate on the subject | 13:10 |
| tristan | right that one | 13:15 |
| ssam3 | i also wonder about making fsync and fdatasync no-ops in the SafeHardlinks backend | 13:17 |
| ssam3 | which is not "dangerous" for sandboxes, as sandboxes don't get reused if anything goes wrong | 13:17 |
| ssam3 | could be risky if we use that filesystem for other stuff that can be storing precious data though ... | 13:18 |
| tristan | ssam3, right, the sandbox is throwaway if you lose power | 13:19 |
| tristan | ssam3, the conclusion I'm tempted to draw, is to disable that in gnome-build-meta; and sync the disks once at sandbox exit time | 13:19 |
| juergbi | yes and for anything that is committed to OSTree, OSTree is in charge | 13:19 |
| tristan | That too, if you get to ostree or collecting an artifact; it must either be synced or buffered and end up in the artifact | 13:20 |
| tristan | so maybe not even required to sync the disk; syncing the artifact might be interesting, but it's a separate issue | 13:21 |
| tristan | looks like main.py got chopped up into main.py/cli.py | 13:35 |
| tristan | Good idea ! | 13:35 |
| tristan | :) | 13:35 |
| * tristan hits 3% remaining disk space issue with 11GB free on his SSD | 13:41 | |
| ssam3 | just in time :-) https://gitlab.com/BuildStream/buildstream/merge_requests/285 | 13:43 |
| tristan | heh | 13:44 |
| tristan | ssam3, crap; I just made space though ! | 13:44 |
| tristan | I wont be able to use this opportunity to test it | 13:44 |
| tristan | but lets merge that anyway | 13:44 |
| ssam3 | oh no! | 13:44 |
| ssam3 | you could do `truncate --size=10GB` :-) | 13:45 |
| gitlab-br-bot | buildstream: issue #216 ("Spurious min-free-space-percent error") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/216 | 13:45 |
| gitlab-br-bot | buildstream: merge request (sam/disable-ostree-free-space-check->master: _ostree.py: Disable OSTree's minimum disk space check in our repos) #285 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/285 | 13:45 |
| tristan | ssam3, did you try that btw ? | 13:45 |
| ssam3 | the truncate test ? actually no | 13:45 |
| ssam3 | let me test it now | 13:46 |
| tristan | Yeah that would be nice :) | 13:46 |
| ssam3 | i verified that it added the config | 13:46 |
| tristan | should be effective | 13:46 |
| gitlab-br-bot | buildstream: merge request (tristan/isolate-shell->master: Tristan/isolate shell) #286 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/286 | 13:48 |
| gitlab-br-bot | buildstream: merge request (juerg/dbus->master: WIP: Inherit user id and group id for bst shell) #265 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/265 | 13:50 |
| tristan | juergbi, if you want to give !286 a glance... it's pretty straight forward though | 13:51 |
| tristan | juergbi, I am going to build GNOME with some modifications I made to gnome-build-meta and try some things | 13:51 |
| tristan | maybe I should try gnome-control-center | 13:51 |
| tristan | juergbi, any other ideas of exercising/testing the inherited uid/gid such that D-Bus works ? | 13:52 |
| ssam3 | ok, i built something with 99% full disk and it committed to the repo fine | 13:56 |
| juergbi | tristan: dbus-monitor is the simplest test | 13:57 |
| juergbi | it immediately fails if dbus doesn't work | 13:57 |
| juergbi | i can take a look at !286 later, not right now | 13:58 |
| *** tristan has quit IRC | 13:58 | |
| *** tristan has joined #buildstream | 14:01 | |
| tristan | gah, pages deployment is less stable than usual these days | 14:05 |
| *** xjuan has joined #buildstream | 14:13 | |
| *** mcatanzaro has joined #buildstream | 14:53 | |
| * tlater also noticed the occasional dnf failure | 15:01 | |
| tlater | juergbi: I'm struggling to figure out how I would do a diff with the OSTree internals | 15:06 |
| tlater | As far as I can see, there's an OSTree.walk of sorts, is your suggestion to use that to compare the shas of all files of two artifacts? | 15:07 |
| juergbi | i actually haven't checked yet whether this is (efficiently) possible with the library | 15:07 |
| tlater | Right | 15:07 |
| juergbi | walk sounds interesting | 15:07 |
| juergbi | as long as you can also get the directory SHA, this should be useful | 15:07 |
| tlater | I'm not entirely sure it does what we want, I'll have to see... | 15:07 |
| juergbi | as you could skip unmodified directories completely | 15:08 |
| tristan | Yes it's possible, but tricky; colin pointed me to some internal code which walks and looks at file metadata iirc | 15:08 |
| tlater | Any chance you could point me to colin's pointer? ;) | 15:08 |
| tristan | I would say, a walk in ostree, is not a walk in the park ? | 15:08 |
| juergbi | but there are trees in the park | 15:08 |
| tlater | I'll call some pigeon Merkel, I guess | 15:09 |
| tristan | for this case, tlater; maybe this is more helpful: https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-diff.c | 15:10 |
| tlater | Ah, tyvm tristan | 15:10 |
| tristan | maybe using https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-diff.c#L199 directly from python is possible | 15:10 |
| juergbi | yes, even public API | 15:11 |
| juergbi | i think | 15:11 |
| tlater | Yeah, it is, but it doesn't quite do what we want | 15:11 |
| tlater | It only compares within the same artifact in our case | 15:11 |
| tristan | ostree_diff_dirs looks public | 15:11 |
| tlater | At least, the python bindings don't seem to quite get there | 15:11 |
| tlater | Maybe I should look more into what they do in detail though | 15:12 |
| juergbi | just repeat it for every dep? | 15:12 |
| tristan | is there some context of this ? is this `bst diff ...` ? | 15:33 |
| tristan | if so, wouldnt you want recursion into dependencies to be an optional thing anyway ? | 15:34 |
| tristan | hrmmm, this was working before; what is the thing you have to do to get fonts working in guis again ? | 15:35 |
| tristan | fc-cache smth-or-other ? | 15:36 |
| tlater | it's something along those lines, yeah | 15:37 |
| * tlater has to google it all the time | 15:38 | |
| tlater | tristan: The context is #215, where we want to compare the files in the latest successful build artifact with the latest built artifact | 15:38 |
| tristan | yeah, ok I see, what I'm seeing is a different problem than fc-cache, though | 15:39 |
| *** xjuan has quit IRC | 15:39 | |
| tristan | tlater, ohhhhh that | 15:39 |
| tristan | I feel unsure if `PKGSYSTEM_ENABLE_FSYNC=0 update-mime-database /usr/share/mime` is really making a difference here | 15:56 |
| tristan | well, I'll leave it in because it should; and verified that's the right env var | 15:57 |
| * persia doesn't understand why update-mime-database is required at build time | 15:58 | |
| tristan | persia, it's probably not | 16:00 |
| tristan | persia, however we never yet implemented any split between what integration commands are, and which arent | 16:00 |
| tristan | so there is only one place so far for integration commands | 16:00 |
| tristan | probably worth opening an issue for | 16:01 |
| tristan | ldconfig is required, and things like, querying pixbuf loaders is, otherwise GTK+ wont build | 16:02 |
| tristan | the "safest" bet is to have all of them run before every build, however separation by domain is possible here | 16:02 |
| tlater | Gah, I think I misunderstand how OSTree works | 16:07 |
| tlater | So, the way I interpret this is that I need to use OSTree.Repo to access the repository, which contains all our artifacts | 16:07 |
| tlater | These artifacts are represented as trees | 16:07 |
| persia | update-mime-database is a critical integration command. My confusion is now gone. | 16:07 |
| tlater | And I should be able to traverse those trees individually, and then compare files between them | 16:08 |
| tlater | My problem is that there doesn't seem to be any way to access these trees through the api | 16:08 |
| tlater | So clearly my understanding must be wrong? | 16:08 |
| tristan | tlater, yep; but I think the ostree_dir_diff() can do the heavy lifting for you | 16:08 |
| tristan | oh there is | 16:08 |
| tristan | tlater, I think you might be missing the point that ostree uses abstract GLib data types to represent things | 16:09 |
| tristan | GFile and friends | 16:09 |
| tlater | As I understand them they aren't actually files, but simply represent paths that could be accessed? | 16:10 |
| tristan | I couldnt say without spending a fair amount of time reading the code | 16:10 |
| tlater | Ah, fair | 16:10 |
| tristan | but I think if you look at the ostree-repo-checkout.c, you will find a similar walk | 16:10 |
| tristan | which uses public API to achieve it | 16:11 |
| tlater | Ok, I'll try to read through it again, ta for the patience here... | 16:12 |
| tlater | Right, so an ostree repo is also a GFile? | 16:14 |
| * tlater thinks he's starting to understand | 16:14 | |
| tristan | an ostree repo is ostree-repo.c, I think it's an object on it's own | 16:15 |
| tristan | but it will give you GFile | 16:16 |
| * persia thought that an ostree was a merkle tree containing GFiles | 16:18 | |
| tristan | persia, in an abstract sense, yes; at the implementation level, well also, but how to navigate that isnt super intuitive and clear | 16:21 |
| * tlater has somehow managed to avoid ever coding with GLib, so I still get to build that intuition ;) | 16:24 | |
| tlater | Gah, I feel like I might need to go through a GLib 101 first or something... Frustrating given the targets I set myself for today | 16:29 |
| tristan | tlater, aaaand there is no way to use that convenience diff function that is public in the library for our purposes ? | 16:30 |
| tristan | I still dont understand why that's not an option, and you want to walk the tree manually | 16:30 |
| tlater | I think there is, I simply don't understand how to use it | 16:30 |
| tlater | Because it takes two GFiles, and I can't figure out how to give it trees | 16:31 |
| * tlater still has a disconnect because he doesn't quite understand the data structures | 16:31 | |
| tristan | ah | 16:33 |
| tristan | so you need to get the toplevel dir | 16:33 |
| tristan | or not that, but you need to get files/ | 16:33 |
| tristan | and then you can use that, yep | 16:34 |
| tristan | indeed | 16:34 |
| persia | tlater: You can use g_file_equal () to get course granulatity, if you write your own tree walker. diff seems harder. | 16:34 |
| * persia imagines something involving g_file_read() | 16:35 | |
| tristan | persia, I think you're missing the point; he needs to *get* the GFiles, then he can use the convenience function | 16:36 |
| tristan | persia, rather, he needs to understand how ostree works with respect to GFiles and how they actually fit into the picture | 16:36 |
| tlater | I think it's quite simple, as soon as I understand how to manipulate this :) | 16:36 |
| tristan | persia, it will be very very far from "easy" - and the code will be probably very "simple" | 16:37 |
| persia | tristan: Oh, yeah, that works too. I was thinking of avoiding actually instantiating the files except as memory structures. | 16:37 |
| persia | (but my way involves much uglier code) | 16:37 |
| tristan | persia, if you assume an extracted artifact, that would be much more easy; but very heavy in terms of resources, and far less simple code :) | 16:38 |
| tlater | Heh, yeah, I think I could do that ;) | 16:39 |
| jennis | tristan, https://gitlab.com/jennis/buildstream/blob/master/buildstream/plugins/elements/compose.py#L75 would you be able to explain to me why the you've made the 'integrate' and 'include' values tuples? | 16:39 |
| jennis | pls :) | 16:40 |
| persia | tristan: In that case, I think we're talking past each other and had similar ideas :) | 16:40 |
| tristan | jennis, I dont see tuples on that line | 16:42 |
| tristan | jennis, `integration` is bool, `include` is list | 16:42 |
| tlater | tristan: Note the ',' at the end of each line | 16:43 |
| tlater | -> makes them tuples | 16:43 |
| tlater | But should probably be a syntax error | 16:43 |
| jennis | tristan, here is what they return: https://paste.codethink.co.uk/?4073 | 16:44 |
| jennis | Hence the latter method does not work | 16:44 |
| jennis | hence probably wasn't the right word there... but implementing the latter method doesn't work | 16:44 |
| tristan | jennis, tlater; not to my knowledge; you can simply call that a total fuckup | 16:45 |
| tristan | I can explain how the fuckup happened | 16:45 |
| jennis | haha ok, well to make it more explicit I've done this: | 16:45 |
| jennis | key = {} | 16:45 |
| jennis | key['integrate'] = (self.integration, ) | 16:45 |
| jennis | key['include'] = (sorted(self.include), ) | 16:45 |
| jennis | key['orphans'] = self.include_orphans | 16:45 |
| tristan | It was originally key = { 'integrate': self.integration, ... } like that | 16:46 |
| tlater | tristan: Do tell | 16:46 |
| tristan | And then we added `exclude` | 16:46 |
| tristan | So we changed it to be all: key['foo'] = bar | 16:46 |
| tristan | and a final `if` | 16:46 |
| tristan | with the intention of remaining cache-key compatible | 16:46 |
| tristan | after adding a new config option | 16:46 |
| jennis | Can you not still have key = { 'integrate': self.integration, ... } with an if statement following it...? I'm sure you can | 16:47 |
| tristan | But `,` on the end of line, for some crazy unknown reason; did NOT produce a compilation error. | 16:47 |
| tristan | jennis, Yes, but a code style change should be fine too | 16:47 |
| * tlater finds this very amusing - especially because it means that we'll have to keep them as tuples to avoid breaking the cache key | 16:47 | |
| jennis | oh just a style issue | 16:47 |
| * jennis was just about to make tlater's point | 16:47 | |
| tristan | jennis, tlater; its technically pretty alright to fix this in master; it's not alright to backport to `bst-1.0` | 16:48 |
| tristan | I would favor fixing it especially if the linter found this | 16:48 |
| tristan | we've already broken cache keys since 1.0, it's not an immensely huge deal if it's worth the benefit | 16:49 |
| * tlater suspects the linter disagreed with the repeated key['something'] calls, and that jennis found this while debugging | 16:51 | |
| tlater | Might complain about both, though | 16:52 |
| tlater | Oh, it does :) | 16:52 |
| tlater | "trailing-comma-tuple" | 16:52 |
| jennis | yes tlater, hence the quick fix mentioned above | 16:53 |
| jennis | also makes it v. explicit that it's tuple packing to anyone else who comes across it and doesn't spot that comma ;P | 16:53 |
| gitlab-br-bot | buildstream: merge request (tristan/isolate-shell->master: Tristan/isolate shell) #286 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/286 | 16:54 |
| gitlab-br-bot | buildstream: issue #227 ("Allow building releases") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/227 | 16:54 |
| gitlab-br-bot | buildstream: merge request (tristan/isolate-shell->master: Tristan/isolate shell) #286 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/286 | 16:54 |
| tristan | oh crap | 16:54 |
| tristan | juergbi, fuck me ok I can revert it | 16:55 |
| * tristan wanted to solve AT LEAST the optionality problem, the inherit UIDs problem, and one more problem in one day | 16:55 | |
| * tristan can add to that the few fixes he has not yet pushed to gnome-build-meta | 16:56 | |
| *** aday has quit IRC | 16:56 | |
| tristan | but if we revert the the UID inheritance, then that's gonna wait way long time | 16:56 |
| tristan | like a whole week | 16:56 |
| tristan | whoami here happly reports that it does not know the username associated to 1000 | 16:57 |
| persia | As it shouldn't, without configured shadow (or equivalent) installed. | 16:57 |
| tristan | /etc/passwd | 16:58 |
| tristan | persia, and BuildStream should never know about that | 16:58 |
| tristan | and as far as I see, it's totally fine | 16:58 |
| tristan | surely some things will fail | 16:58 |
| persia | Oh, yes, I suppose BuildStream should not have information about the contents. | 16:58 |
| persia | But I can imagine BuildStream running a build with a build-time dependency on configured shadow, if someone wanted to do that. | 16:59 |
| tristan | So project.conf has to grow a `shell:` section describing all the customizations and commands you wanna run to prepare an open testable shell | 16:59 |
| tristan | what ? | 16:59 |
| tristan | build-time dependency *of BuildStream* ? | 17:00 |
| tristan | oh | 17:00 |
| persia | No. Build-time dependency of an element. | 17:00 |
| tristan | persia, no no no | 17:00 |
| tristan | persia, you can never know the UID | 17:00 |
| tristan | persia, so the build cannot have static info in it, knowing the outside users UID | 17:00 |
| persia | That worries me, as it is possble to construct a source element that contains /etc/passwd. | 17:00 |
| juergbi | tristan: we can also leave it in if it doesn't cause more issues than what we had before | 17:01 |
| persia | If the rule is "you must always function even if you don't know the ID", then I'm less worried. | 17:01 |
| tristan | persia, I think you are again mixing your subjects | 17:01 |
| tristan | persia, are you aware that we are talking about https://gitlab.com/BuildStream/buildstream/merge_requests/286 ? | 17:01 |
| persia | Yes. | 17:01 |
| tristan | persia, which is dup of https://gitlab.com/BuildStream/buildstream/merge_requests/265 | 17:02 |
| persia | I only thought it was related to 265, but I now see it is a dup. | 17:03 |
| tristan | persia, ok; so you cannot introduce user noise into the build; that is just a fact | 17:03 |
| persia | hence "Oh, yes, I suppose BuildStream should not have information about the contents" | 17:03 |
| persia | I suspect the confusing thing was that I used the word "contents", which doesn't actually mean anything in the context of BuildStream. I apologies. | 17:04 |
| persia | And, yes, introducing user noise into a build would be terrible, and should not ever be done. | 17:05 |
| tristan | Anyway, there is no way to "have a configured shadow" | 17:05 |
| tristan | as far as I can see | 17:05 |
| tristan | That needs customization in project.conf that sets up stuff in the build environment before you shell into it | 17:06 |
| tristan | so that one can know what to do | 17:06 |
| tristan | same kind of deal with copying in /etc/resolv.conf, or specifying special env vars to allow passing on to the build env | 17:07 |
| persia | Right (returning the conversation to the pre-confused state, from "what?" above). | 17:07 |
| tristan | or not the "build env" I should say the "shell env" | 17:07 |
| persia | Yes. "shell env": none of that influences the build at all. | 17:07 |
| *** jonathanmaw has quit IRC | 17:11 | |
| *** xjuan has joined #buildstream | 17:13 | |
| *** aday has joined #buildstream | 17:18 | |
| gitlab-br-bot | buildstream: merge request (239-use-pylint-for-linting->master: WIP: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/283 | 17:30 |
| *** toscalix has quit IRC | 17:36 | |
| *** dominic has quit IRC | 17:46 | |
| *** valentind has joined #buildstream | 17:49 | |
| *** aday has quit IRC | 17:57 | |
| gitlab-br-bot | buildstream: merge request (239-use-pylint-for-linting->master: WIP: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/283 | 18:10 |
| *** ssam3 has quit IRC | 18:16 | |
| gitlab-br-bot | buildstream: merge request (239-use-pylint-for-linting->master: WIP: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/283 | 18:19 |
| *** tm has quit IRC | 19:40 | |
| *** ernestask has quit IRC | 19:50 | |
| *** tm has joined #buildstream | 19:53 | |
| *** Prince781 has joined #buildstream | 20:47 | |
| *** Prince781 has quit IRC | 20:51 | |
| *** aday has joined #buildstream | 21:09 | |
| *** tm has quit IRC | 21:22 | |
| *** aday has quit IRC | 21:30 | |
| mcatanzaro | Buildstream's bash prompt changed to "I have no name!@victory-road:/buildstream/build$" (victory-road is my hostname) | 21:39 |
| mcatanzaro | Probably shouldn't include the username there | 21:39 |
| *** tm has joined #buildstream | 21:44 | |
| *** valentind has quit IRC | 22:09 | |
| *** tm has quit IRC | 22:29 | |
| *** tristan has quit IRC | 22:46 | |
| *** cs_shadow has quit IRC | 22:48 | |
| *** Prince781 has joined #buildstream | 23:15 | |
| *** xjuan has quit IRC | 23:25 | |
| *** Prince781 has quit IRC | 23:29 | |
| *** Prince781 has joined #buildstream | 23:30 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!