| *** jjardon[m] has joined #buildstream | 00:17 | |
| *** mattiasb has joined #buildstream | 00:56 | |
| *** jonathanmaw has joined #buildstream | 07:32 | |
| *** tristan has joined #buildstream | 08:05 | |
| jonathanmaw | morning tristan. the dependencies required to run `debian/rules binary` for a debian source are quite large. So far I've needed to pull in debhelper, dpkg-dev, dpkg, libdpkg-perl, libtimedate-perl, lsb-release, python-docutils and python-roman | 08:06 |
|---|---|---|
| jonathanmaw | and now it seems that I need non-busybox `du` | 08:07 |
| tristan | jonathanmaw, ok so, have you tried this against the debootstrap repo ? (https://gnome7.codethink.co.uk/repo/) | 08:10 |
| tristan | I suspect it would have everything you need, and we can leave the "providing the deps" part anyway as an exercise for the user | 08:11 |
| tristan | it already is for any other build element (users need a sysroot with automake and friends to use autotools builds) | 08:11 |
| *** ChanServ sets mode: +o tristan | 08:12 | |
| jonathanmaw | hmm, I haven't tried that | 08:18 |
| tristan | jonathanmaw, you could easily use https://gitlab.com/BuildStream/gnome-modulesets-base as a template for a base system | 08:19 |
| tristan | Raising the fact that there are many dependencies, points to also requiring many dependencies for other elements (autotools is typically available on any dev system, but still requires a lot of deps generally) | 08:20 |
| tristan | This makes me think it's even more interesting to let elements assert presence of tooling in the sandbox | 08:21 |
| tristan | but also another idea | 08:21 |
| tristan | Which is... looking at the very bottom of: https://buildstream.gitlab.io/buildstream/projectconf.html#projectconf | 08:21 |
| tristan | Note that it's possible for a project configuration to customize element configurations on a per-element-type basis | 08:22 |
| tristan | It's conceivable for a project like baserock (which builds _everything_ from scratch), to configure element types with implied project-wide / element-type-specific dependencies | 08:23 |
| tristan | So that that project, which only itself can know what element provides a specific host tool, can just simply assert that "Any autotools element implicitly depends on libtool, autoconf, m4, automake" etc | 08:24 |
| tristan | (although note that I doubt defining this is possible at this time, it would need some code to allow project config to append dependencies to all elements of a given type) | 08:29 |
| tristan | Still sounds interesting | 08:29 |
| *** ssam2 has joined #buildstream | 08:33 | |
| jonathanmaw | tristan: I'm not sure how to use https://gnome7.codethink.co.uk/repo/. It seems to be a git repo, but `git clone https://gnome7.codethink.co.uk/repo/` isn't working, I assume because it's not a git server. | 08:36 |
| ssam2 | isn't that an OSTree repo ? | 08:37 |
| jonathanmaw | it's got a refs and objects subdir, I assumed it's a git repo | 08:40 |
| *** juergbi has quit IRC | 08:45 | |
| *** juergbi has joined #buildstream | 08:46 | |
| jonathanmaw | it looks like the ostree repo that's in my buildstream artifact cache. huh, I didn't know ostree repos looked like that. | 08:53 |
| *** jude has joined #buildstream | 09:11 | |
| tristan | It's an ostree repo yes | 09:19 |
| tristan | The linked project shows a base-system.bst element which consumes that repo | 09:20 |
| tristan | jjardon[m], ssam2, systemd has a facility for assigning uids/gids at startup right ? not only running a service under a specified username ? | 09:32 |
| tristan | I think jjardon[m] had a link to that but I cant find it right now | 09:32 |
| ssam2 | systemd-sysusers ? | 09:32 |
| tristan | maybe that's it yeah | 09:33 |
| tristan | Is that "enough" to handle the case (at least for systemd systems) of needing to assign UID/GIDs to files in a disk image ? | 09:33 |
| ssam2 | i'm not sure that it changes any file permissions .. i think it just updates /etc/passwd | 09:34 |
| ssam2 | useful if you need an 'apache' user but don't want to hardcode a UID | 09:35 |
| tristan | I see | 09:35 |
| ssam2 | something like the manifest used by the baserock install-files extension can handle file permissions though | 09:35 |
| tristan | Something *can* be done at boot time yes | 09:36 |
| tristan | Ok so basically: If there is nothing for the UID/GIDs, then there is no real argument for the suid mode bits either ? | 09:40 |
| tristan | ssam2, thoughts ? | 09:40 |
| tristan | I'm looking for the simplest thing here | 09:40 |
| ssam2 | I don't understand the question, sorry | 09:41 |
| tristan | Well, we cant create files owned by arbitrary uid/gids, nor can we check them out from ostree as a regular user | 09:41 |
| tristan | This UID/GID is an annoying limitation to creating real bootable systems, but it would seem that it's not necessary for many cases (everything can be owned by root) | 09:42 |
| tristan | That is true only somewhat, though. | 09:42 |
| ssam2 | I'd be happy with an approach where user, group and setuid information is transmitted "out of bound" | 09:43 |
| tristan | ssam2, If I want to make the case that I can get "very close" and not need other things which are strictly forbidden for the regular user | 09:43 |
| ssam2 | it's definitely needed, but I think it's rare and often a bit of a legacy hack | 09:43 |
| tristan | But, still want to have SUID because it's very easy to achieve | 09:44 |
| tristan | Then, my case is bunk basically, if UID/GID is at least as important | 09:44 |
| tristan | The problem with "transmitted out of bound" is that every approach I can see so far is a horrible hack :-S | 09:44 |
| tristan | If we wanted to use something like yocto psuedo, we would have to have pseudo *in the sandbox* | 09:45 |
| ssam2 | just make it explicit in the definitions | 09:45 |
| tristan | Or, stop using bubblewrap | 09:45 |
| ssam2 | add an 'ownership' section to an artifact that deployment elements can implement however is best for them | 09:46 |
| ssam2 | for example if you're generating a tarfile you can make the permissions whatever you want as non-root | 09:46 |
| tristan | But making it explicit in the definitions means more unnecessary work, its a bit counter my personal philosophy, that we should be able to get most things done by improving virtualization and reducing busywork | 09:46 |
| ssam2 | fair enough. those are my thoughts though :-) | 09:47 |
| tristan | I.e. in other words: In the vast majority of cases "make install" should be taking care of everything the software wants | 09:47 |
| ssam2 | that's already true, though | 09:47 |
| ssam2 | it's only a very small minority of components that expect to be able to chown during `mkae install` | 09:47 |
| tristan | ssam2, I dont understand what is "already true" | 09:49 |
| ssam2 | '"make install" should be taking care of everything the software wants' | 09:49 |
| ssam2 | that's already true | 09:49 |
| ssam2 | in most cases | 09:49 |
| tristan | Right | 09:49 |
| tristan | That should include chown and ensuring setuid I think, no ? | 09:50 |
| ssam2 | not sure | 09:50 |
| ssam2 | for how many programs is that important? | 09:50 |
| ssam2 | I can only think of sudo | 09:50 |
| ssam2 | and bubblewrap I guess | 09:50 |
| tristan | I think that is not the real question, I think the real question is; can you really make a production ready full system without that ? | 09:51 |
| tristan | And also... | 09:51 |
| ssam2 | you're changing the question | 09:51 |
| ssam2 | my view is that we should mark such special cases explicitly in the definitions | 09:51 |
| ssam2 | and if that only needs to be done for sudo and bubblewrap, I don't think that's much burden on the user | 09:51 |
| tristan | Do upstreams usually take responsibility for doing that at make install time ? Or do they generally offload this responsibility to package maintainers to take care of in debian/rpm metadata instead ? | 09:51 |
| tristan | ssam2, you make a fair point, if it really us such a small amount | 09:53 |
| tristan | ssam2, on the other hand, this I think will mean; samba ? mysql... apache... also things like gdm I suspect | 09:53 |
| ssam2 | I have 21 setuid binaries in /usr/bin | 09:55 |
| ssam2 | at, change, chfn, chsh, crontab, fusermount, fusermount-glusterfs, gpasswd, ksu, mount, newgidmap, newgrp, newuidmap, passwd, pkexec, pkmount, pumount, schroot, su, sudo and umount | 09:55 |
| tristan | Right, my examples were about uids/gids | 09:55 |
| tristan | so that is also a fair amount | 09:56 |
| tristan | although those mostly come from 3 or 4 upstreams | 09:56 |
| * ssam2 runs `find . -exec stat -c '%U' \{\} \;|egrep -v root` | 09:57 | |
| tristan | ssam2, so in any case, my first conclusion is probably right: If we cant have arbitrary uids/gids (in checkouts), then there is not much point in trying to have setuid/setgid in checkouts | 09:58 |
| tristan | Agreed ? | 09:58 |
| ssam2 | yes | 09:58 |
| tristan | Right that was my first order of business, if that is the case, then we really shouldnt bother arguing this line with Colin any further | 09:58 |
| tristan | And it's better for security of users who build stuff, anyway. | 09:58 |
| tristan | So this fixes the issue mostly: https://github.com/ostreedev/ostree/commit/5913b2294490a1f85d4b617fdcc9455ed0bc63a2 | 10:06 |
| tristan | With the ugly permission modifications | 10:07 |
| tristan | Now user mode checkouts really only strip the suid, I *think* | 10:07 |
| ssam2 | FWIW my /usr contains only 3 files that aren't owned by root | 10:19 |
| ssam2 | one owned by 'abrt', one by 'polkitd', and one by me (probably a cockup :-) | 10:19 |
| ssam2 | even /etc only has a few files not owned by root, in fact | 10:21 |
| tristan | ssam2, right, I think ownership comes into play a bit more on servers | 10:27 |
| tristan | however most daemons have a very simple setup where they tell the OS that they are user "foo" and they have a home dir somewhere | 10:28 |
| ssam2 | true, I have some server stuff installed probably though | 10:28 |
| tristan | and then the OS takes care of creating that home dir | 10:28 |
| tristan | not sure if it's necessarily created at useradd time | 10:28 |
| tristan | or if it can be deferred | 10:28 |
| ssam2 | I think that's exactly what systemd-sysusers is aiming to solve | 10:29 |
| ssam2 | 1. create user; 2. create /var directory for daemon with correct permissions; 3. run daemon as correct user; 4. daemon initializes its directory in /var as correct user | 10:30 |
| ssam2 | (well sysusers just does step 1, a systemd unit does the rest) | 10:30 |
| ssam2 | any shell-based init system can do the same if it wants to | 10:30 |
| tristan | right, but I think *anyway* a lot of this is handled in /etc/passwd | 10:31 |
| tristan | this takes care of defining the uids *and* specifying the home dir | 10:31 |
| tristan | I think the init system takes care of setup based on this, it's not normally done at build time | 10:31 |
| tristan | although debian has some triggers (I think they have adduser and addgroup which they use instead of useradd and groupadd) | 10:32 |
| tristan | maybe those *also* do some of that special sauce | 10:32 |
| persia | adduser and addgroup are just wrappers for useradd and groupadd (to allow changing implementations later without impact) | 10:50 |
| persia | But they *do* do the special sauce, of which there are two typs defined by debian policy. | 10:50 |
| persia | Type A is static, where it was determined that it would be difficult to translate names to IDs reliably, and so there is a project-wide registry of the allocations so that different packages can reliably use the same IDs, etc. | 10:51 |
| persia | Type B is dynamic, where users/groups are created in specific ranges (depending on purpose): I believe that is suppsoed to happen in preinst (although I haven't delved into it in some time). | 10:52 |
| persia | One of the issues with type B is that this contributes to the issue that installing the same package on two systems does not cause the same result. | 10:52 |
| persia | When I was paying more attention, the Debian Project was trying to reduce the amount of Type A that existed (and removing an entry from that list was considered a significant win, if someone managed to do it). | 10:53 |
| *** tristan has quit IRC | 11:09 | |
| *** tristan has joined #buildstream | 11:20 | |
| *** ChanServ sets mode: +o tristan | 11:21 | |
| tristan | fwiw: https://mail.gnome.org/archives/ostree-list/2017-June/msg00025.html | 11:26 |
| tristan | ssam2, that also has a wild idea I think we might be able to eventually do, using a similar to 'pseudo' approach but backed by ostree | 11:27 |
| tristan | that would be really nice actually | 11:27 |
| ssam2 | a bit less hacky, but still requires everything to use the same C library and be dynamically linked | 11:36 |
| *** tristan has quit IRC | 11:37 | |
| *** tristan has joined #buildstream | 12:25 | |
| *** ChanServ sets mode: +o tristan | 12:25 | |
| tristan | ssam2, yeah, I would prefer something a bit different, hence that part: | 12:27 |
| tristan | I'm not sure how I could get that to work without requiring that the | 12:27 |
| tristan | runtime environment inside bubblewrap actually have the said LD_PRELOAD | 12:27 |
| tristan | library available, for now it's just a wild but I think interesting | 12:27 |
| tristan | idea :) | 12:27 |
| tristan | Basically, it sucks if the content of the bubblewrap container has to be custom in some way | 12:27 |
| ssam2 | using ostree as the backend is neat | 12:27 |
| tristan | Maybe another fuse mount approach could do it, but I'm not sure of the restrictions which apply | 12:28 |
| tristan | I.e. could accessing a fuse mount be told that things are suid, but not have it really be so ? | 12:28 |
| ssam2 | except you still need a way to go from "ostree repo with arbitrary UIDs and setuid binaries" to "system image" | 12:28 |
| tristan | Same for extended attrs and even dev nodes ? | 12:28 |
| ssam2 | most of the paths to do that require using the kernel, which will require privileges | 12:29 |
| tristan | ssam2, no, creating images is not a problem at all | 12:29 |
| tristan | creating images has to be 100% user space | 12:30 |
| tristan | There is no such nonsense as mounting a loopback involved | 12:30 |
| tristan | Thats what mkfs -d is for | 12:30 |
| tristan | So mkfs -d will create a filesystem image from the -d ${sysroot}, where ${sysroot} is a directory that reports basically whatever attributes it wants (hopefully) | 12:31 |
| tristan | Using the LD_PRELOAD, yocto people manage to get mkfs -d to see the filesys as they want it (as it is hacked by pseudo) | 12:31 |
| tristan | But I just wonder if it can be done with fuse, it should I think | 12:32 |
| persia | You might be able to change permissions with a combination of overlayfs and fuse, although the fuse mount would have to be done privileged. | 12:33 |
| tristan | Once you get to the real underlying filesys, then the kernel is involved and will complain if the fuse layer tries to create a real dev node ? | 12:33 |
| tristan | Why ? | 12:33 |
| tristan | fuse mounting doesnt require privileges afaik | 12:33 |
| persia | In order to have a file on a filesystem that exposes privileges, that filesystem has to be mounted with those privileges. | 12:34 |
| tristan | The only questions are really, if the fuse mount is able to report false UID/GIDs to the user and suchlike | 12:34 |
| persia | The alternative is to fuse-mount setuid0 shell and run that to gain root anytime you like. | 12:34 |
| tristan | Hmmm, not sure I follow, maybe you are answering my question, the part I'm not sure about | 12:35 |
| tristan | persia, i.e. the way I'm seeing it, is that fuse sits on *top* of a real filesys (or can be backed by anything) | 12:36 |
| persia | I may be mistaken, but my understanding was that a fuse mount could expose any data one wanted *except* that if the mount was performed by an unpriviliged user, then no privileged files are honoured. | 12:36 |
| tristan | persia, but that filesys itself does not need to hit the kernel | 12:36 |
| persia | fuse+overlay can sit on top. Fuse by itself is just a filesystem. | 12:36 |
| persia | Oh! Yes, if you never need to exercise the permissions, it ought be fine. | 12:37 |
| tristan | persia, if you implement a passthrough and implement mknod in fuse, it will be the fuse process which gets permission denied if it tries to really mknod (instead of save something to a DB) | 12:37 |
| persia | So, if you're setuid0 in a unprivileged container, you should be able to set a file as setuid0 on a FUSE, although that won't be honoured later. But maybe it doesn't work: I remember some folk complaining about not being able to access port79 from an unprivileged container in another context, which is a similar thing. | 12:38 |
| tristan | Right so you would have user mode checkout of ostree underneath, and a fuse fs on top, which reports the UIDs/GIDs from the real ostree checkout (that was never *really* checked out) | 12:38 |
| persia | And I thought you wanted it uncomplicated :) | 12:38 |
| tristan | But those UIDs and GIDs are *reported* by fuse, but hopefully never exercised on a real filesys | 12:38 |
| tristan | Right I'm not sure if it can work | 12:39 |
| tristan | persia, actually this is *quite* uncomplicated if it actually works | 12:39 |
| tristan | But, I think we at least dont need it for immediate future | 12:39 |
| tristan | GNOME can live easily without that, but it will be great later on for real production systems | 12:39 |
| tristan | persia, here is an example of complicated: https://mail.gnome.org/archives/ostree-list/2017-June/msg00018.html | 12:40 |
| tristan | (email I was replying to) | 12:41 |
| tristan | For the Fedora/Project Atomic space in practice we end up using | 12:41 |
| tristan | Anaconda-in-a-VM (via a tool called imagefactory) - and remember | 12:41 |
| tristan | one can create VMs unprivileged. | 12:41 |
| tristan | And then the installer reads content from the host via a temporary webserver, | 12:41 |
| tristan | although with some work we could inject a virtio channel. | 12:41 |
| tristan | " | 12:41 |
| tristan | Like, holy shit | 12:41 |
| tristan | And that of course already requires a lot of host installed stuff | 12:41 |
| persia | It's like they never heard of 9fs or something | 12:41 |
| tristan | On the other hand, while pseudo is a lot nicer, probably even more than fakeroot (but I dont know fakeroot well enough to judge), it *still* requires that that custom LD_PRELOAD be a part of your build environment | 12:42 |
| tristan | This works for yocto because the build environment is your host (multi sysroot madness) | 12:42 |
| tristan | But in a namespace, it's ugly because you dont want to impose anything on what is placed in the runtime | 12:43 |
| tristan | But, *if* that same pseudo stuff can be done with a fuse layer, that is a pretty small amount of code, and a clean / not hacky experience | 12:43 |
| persia | At least it provides a clean interface. I'm not sure it isn't a hack. | 12:44 |
| tristan | Right, I mean a not-hacky experience | 12:44 |
| persia | Given the age of the relevant technologies, I would have considered it an acceptable ObHack for the alt.hack hierarchy back when I read newsgroups. | 12:44 |
| persia | (as it isn't obvious, but pleasing in it's own way) | 12:45 |
| tristan | mouahahahaha | 13:31 |
| tristan | persia, http://bindfs.org/docs/bindfs.1.html | 13:31 |
| tristan | ssam2, ^^^ | 13:31 |
| tristan | Ok with this, we *really* can do that | 13:31 |
| ssam2 | oh yeah, I've used bindfs before | 13:33 |
| tristan | ssam2, that will do it, well; something similar, and it can all be done with python haha | 13:33 |
| ssam2 | is BuildStream still aiming to run on kernels other than Linux ? | 13:34 |
| tristan | Will fuse not be supported on other systems ? | 13:34 |
| ssam2 | i don't know that it is | 13:34 |
| tristan | Yes, we want to support other unixes | 13:34 |
| tristan | ssam2, *however* | 13:34 |
| tristan | it's already certain that we wont have ostree or bwrap on other platforms | 13:35 |
| tristan | So that is sort of moot | 13:35 |
| ssam2 | true, i guess we just require root there | 13:35 |
| tristan | That is worst case scenario yes, I'll want to have a chroot fallback | 13:35 |
| tristan | start with that, + tarball artifacts | 13:35 |
| tristan | But other platforms have special features to optimize and do sandboxing also | 13:36 |
| tristan | we looked into that last time in manchester | 13:36 |
| * tristan is currently putting together a fuse context manager thingy for the copy-on-write hardlinks, which is fairly straight forward I think | 13:37 | |
| tristan | from there it's actually pretty simple to do all that pseudo stuff | 13:38 |
| tristan | Well that's done; but I have to churn the Sandbox API and BuildElement code a bit, tomorrow should be all done (not the awesome spoofing of permissions, but at least copy-on-write hardlinks) | 15:26 |
| tristan | I guess elements will have a bit more control on how sandboxes are constructed, I'll try to keep the API surface for this small, though | 15:27 |
| *** jonathanmaw has quit IRC | 15:45 | |
| *** tristan has quit IRC | 15:59 | |
| *** tristan has joined #buildstream | 16:06 | |
| *** ChanServ sets mode: +o tristan | 16:06 | |
| *** ssam2 has quit IRC | 18:11 | |
| *** tristan has quit IRC | 21:06 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!