IRC logs for #buildstream for Friday, 2018-02-16

*** tristan has quit IRC00:00
*** persia has joined #buildstream00:54
*** Prince781 has quit IRC04:43
*** mercy_ has joined #buildstream04:44
mercy_________  ______ _____ __  ______  __________  _   ___________________  ____  ____  ______04:45
mercy_/  _/ __ \/ ____// ___// / / / __ \/ ____/ __ \/ | / / ____/_  __/ ___/ / __ \/ __ \/ ____/04:45
mercy_/ // /_/ / /     \__ \/ / / / /_/ / __/ / /_/ /  |/ / __/   / /  \__ \ / / / / /_/ / / __04:45
mercy__/ // _, _/ /____ ___/ / /_/ / ____/ /___/ _, _/ /|  / /___  / /  ___/ // /_/ / _, _/ /_/ /04:45
mercy_/___/_/ |_|\____(_)____/\____/_/   /_____/_/ |_/_/ |_/_____/ /_/  /____(_)____/_/ |_|\____/04:45
mercy_persia juergbi slaf zalupik cs_shadow jennis theawless[m] lantw44 csoriano adds68 skullman nexus rafaelff[m] cgmcintyre[m] pro[m] connorshea[m] mrmcq2u[m] m_22[m] jjardon[m] bochecha kailueke[m] abderrahim[m] ptomato[m] asingh_[m] inigomartinez mattiasb waltervargas[m] ernestask[m] gitlab-br-bot04:45
*** mercy_ has left #buildstream04:45
*** Prince781 has joined #buildstream04:50
*** Prince781 has quit IRC05:04
*** Prince781 has joined #buildstream05:32
*** Prince781 has quit IRC06:10
*** noisecell has joined #buildstream07:53
*** valentind has joined #buildstream08:49
*** tristan has joined #buildstream08:57
*** toscalix has joined #buildstream09:17
*** aday has joined #buildstream09:19
gitlab-br-botbuildstream: merge request (proper-build-shells->master: element.py: Create real build shell for `bst shell --build`) #277 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27709:27
tristanjuergbi, right now we only have 2 news items since 1.0 in master, they're pretty serious ones but; any thoughts on what we might have missed there ?09:30
tristanor anyone :)09:30
tristanI think the tests refactoring was a pretty big change09:30
*** jonathanmaw has joined #buildstream09:32
skullmanthe hacky sleep waiting for fuse to be mounted bothers me more than it should09:40
skullmanis it that we might be running in a container where setuid doesn't work, hence `fusermount -u` won't work?09:40
tristanthat depends on the site it seems09:42
tristanfrom what I recall, fusermount is needed and setuid but not always; and the C library calls out to fusermount -u itself, only as a fallback mechanism09:43
tristanskullman, but I think you are talking about the sleep at mount time no ? not at umount time09:44
tristanat mount time; it could *probably* be improved by reading off a pipe from the child process to synchonize startup09:45
skullmantristan: aye I'm talking about the sleep at mount time, but AIUI if we were to let it background we wouldn't need the sleep but we'd need to use `fusermount -u` to unmount it09:45
skullmantristan: yeah, would have to replace fuse_main with fuse_mount, fuse_new and fuse_loop09:45
skullmanand notify before calling fuse_loop09:46
tristanohhhhh "let it background" is a whole other can of worms; I investigated that and it is in that case *impossible* to synchronize as I recall09:46
skullmanah, my mistake for assuming it would do something sensible if backgrounding09:47
tristanskullman, what I vaguely recall; is that if you relinquish control to the fuse C library in it's own background mode, there is no way for it to tell you when it's done it's work09:48
tristanwhich yeah, seems we're working around with that ismount thing currently anyway09:49
* skullman investigated casync recently so could compare its fuse handling, so would be in a position to make changes to fuse.py if there was will and funding to make it nicer09:49
tristanbut syncrhonization of libfuse mounts is limited, would have to refresh memory09:49
*** dominic has joined #buildstream09:49
tristanthere is also syncrhonization of umount which is necessary (and still buggy)09:50
tristanI think we may have some very rare cases where we still fail to tear down a sandbox because of it09:50
skullmanouch09:51
tristan(errors happen when trying to remove the dev nodes that `bwrap` leaves behind, because things are still mounted, it's not clear)09:51
skullmansounds like the bwrap code is the next most interesting thing for me to look at while I try to get a handle on how things fit together09:52
tristanhttps://gitlab.com/BuildStream/buildstream/issues/16109:52
*** tiago has joined #buildstream09:53
*** ssam2 has joined #buildstream09:56
juergbitristan: another news item would be incremental workspace builds, although this has still issues09:57
skullmantristan: wow, that is a weird one. Sounds like the fusefs process was terminated but the mount point not yet removed.09:57
juergbiit looks like we're not using --die-with-parent for bwrap. that might be a good idea to add09:58
tristanAh right !09:59
tristanskullman, it seems that way yes; fuse is very hard to syncrhonize09:59
tristanHmmm, ok so in around 30min I can validate my fix10:00
*** ssam2 has quit IRC10:01
tristanWish the artifact server was up10:01
tristanit is but, hrm10:01
*** ssam2 has joined #buildstream10:01
tristanisn't this config supposed to work: https://bpaste.net/show/8a1268778de2 ?10:01
tristanssam2, have you noticed that even though gnome7 was brought back to life; it's still not functioning as an artifact server ?10:02
ssam2i haven't, but i haven't been doing anything that would pull from it10:02
ssam2that should either work, or give you a warning early on in the pipeline that it doesn't10:02
tristanFailed to fetch remote refs from ssh://artifacts@gnome7.codethink.co.uk:10007/artifacts: ssh: connect to host gnome7.codethink.co.uk port 10007: Connection timed out10:03
* ssam2 tries and fails to find sshd logs on gnome710:05
ssam2oh, it's ssh.service not sshd.service10:06
ssam2no login attempts for user 'artifacts' are even getting logged by sshd10:06
ssam2on gnome710:06
ssam2i'm not sure what does that port 10007 forwarding10:07
tristanssam2, oh jeez... mkay...10:07
ssam2i think gary has set up something else we can use as a cache though10:09
ssam2he was certainly in the process of doing that10:09
*** valentind has quit IRC10:09
tristanssam2, yeah, in between cache servers - old one should still be working while we setup the new10:10
ssam2yeah10:10
tristanI'm kind of upset about this whole thing being unstable, and would be wiser to just not bother with it and wait for the stable cache to come up10:11
ssam2yeah, we're lucky really I think that nobody is really consuming that cache in anger yet10:12
tristanexcept me10:12
ssam2yeah10:12
tristanwho has to build webkit, again10:12
gitlab-br-botbuildstream: merge request (juerg/die-with-parent->master: sandbox/_sandboxbwrap.py: Use --die-with-parent) #278 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27810:18
gitlab-br-botbuildstream: merge request (sam/233-push-after-pull->master: Avoid pushing things we just pulled) #276 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27610:30
ltutristan, what's your view wrt all those outstanding documentation MRs? i'm assuming you still want to review them yourself rather than juergbi10:31
gitlab-br-botbuildstream: issue #232 ("fd leak when generating epiphany tarball") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/23210:35
gitlab-br-botbuildstream: merge request (proper-build-shells->master: element.py: Create real build shell for `bst shell --build`) #277 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/27710:35
gitlab-br-botbuildstream: issue #257 ("workspace local state file keeps getting created") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/25710:40
tristanltu, frankly I have not had the time, review of those docs was costing me more time than it would cost to just write docs10:41
*** noisecell has quit IRC11:07
gitlab-br-botbuildstream: merge request (212-git-source-needs-a-way-to-disable-checking-out-submodules->master: Resolve "Git source needs a way to disable checking out submodules") #259 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25911:11
gitlab-br-botbuildstream: issue #258 ("Configurable log line formatting") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/25811:35
*** noisecell has joined #buildstream11:46
gitlab-br-botbuildstream: issue #245 ("Logging issues") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/24511:46
*** noisecell has quit IRC12:09
*** noisecell has joined #buildstream12:09
*** noisecell has quit IRC12:25
*** noisecell has joined #buildstream12:39
gitlab-br-botbuildstream: issue #166 ("bst sometimes modifies source `track` parameters") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/16612:39
tristanNote, I've reopened #166 (thanks for fixing this !!!) because we should technically have this in the stable branch too12:41
nexusnp12:42
persiatristan: Do you feel that fix is important enough to backport?12:48
* paulsherwood thinks kbas beats buildstream's cache approach, except for identity/security12:51
persiapaulsherwood: By what measure?12:52
ssam2the concept, or the specific implementation ?12:53
persiafeature list?  pull speed?  storage requirements?12:54
ssam2random thought: we currently allow /dev/random and /dev/urandom access in the sandbox12:56
ssam2which *seems* like a bad idea for reproducibility :-)12:56
ssam2i might file a bug to try removing them and seeing what breaks12:56
jmacWhat if an element needs to create a pair of cryptographic keys?12:57
persiassam2: I suspect there are runtime things that break if they can't initialise with them.  We could fake them with known replay sequences (e.g. a simple counter)12:57
persiaBut I think better is to just cause sources to be reproducible except when they should not be (such as jmac's example)12:58
*** noisecell has quit IRC12:58
ssam2jmac, that would be bad unless they are purely for testing or creating a dummy input for something12:59
ssam2jmac, here, have this opensshd artifact I built for you which ships with a ready-to-use private key  ...12:59
ssam2i promise i won't give it to anyone else or take a copy myself13:00
tristanpersia, when you run `bst track`; it munges your project, and incorrectly so; I'd say that's pretty damn severe yeah13:00
juergbiurandom would be needed for testing binaries/libraries that use it13:00
jmacWould it? If I built and deployed it myself I don't see the problem13:00
ssam2it seems conceptually wrong to do that at build time, rather than on first-use. i can't articulate exactly why though13:01
tristanssam2, there are various things which dont build at all without those devices13:01
ssam2perhaps it's just that buildstream is designed in favour of artifact reuse13:01
persiaFor private artifacts within a trust zone, I think it is better not to do on first-run, as it means that one can have a purely read-only instance of a service.13:02
juergbii don't think it buys us much to avoid them. we can't completely enforce reproducibility anyway. if a tool that normally uses /dev/urandom doesn't find it, it likely either fails or falls back to PRNG which doesn't help either13:02
tristanssam2, I did try some baserock full system builds without it; but also I recall reading on reproducible-builds.org that trying to build without those is a bit overkill13:02
ssam2fair enough13:02
jmacI'm imagining a system (which may not exist in the real world) in which I can't get any access to a box unless ssh is running, and I need to extract the public key before first run13:02
ssam2in trying to sum up issue #38, i've ended up trying to document our sandbox setup13:02
persiajmac: That's actually common in the world.  cloud-init(1) was designed in part to be able to seed data before first login for precisely that class of use case.13:03
ssam2that led me to wonder if the 5 devices we specify in _sandboxbwrap.py are "canonical" enough to list them in the docs13:03
tristanjmac, I recall juergbi and I rehashed this age old "I wanna produce an artifact with my custom thing in it" thing a lot, a while ago13:03
persia(pre-cloud sshd created server certificates at install time: that was modified to be at first-run to allow creation of images that would be reused without leaking key data, which then required out-of-band injection and inspection tooling to discover the key)13:04
tristanjmac, originally we thought there might be a different kind of "deployment" element for that; but then we nuked host tools to hell and decided that that sort of stuff belongs in post-processing after extracting reproducible artifacts from buildstream13:04
jmacThat sounds reasonable.13:05
persiapost-processing buildstream to generate keys, followed by out-of-buildstream deployment seems correct to me.13:05
jmacWe need to make sure someone doesn't do it anyway, though; if someone does what I'm describing and uses a predictable /dev/random, boom13:05
jmacPerhaps show some big warnings if anyone reads from it13:06
tristanmyeah that's problematic13:06
tristanmaybe we could spoof it somehow13:06
tristanI mean, we can live with a "dont do that crazy shit" for a long time13:06
jmacYes.13:06
tristanbut seems interesting to have a real /rev/random that only ever produces the random number 113:07
ssam2https://github.com/bmwiedemann/reproducible-faketools creates /dev/random and /dev/urandom files that are actually the same as /dev/zero13:08
juergbihttps://xkcd.com/221/13:08
tristan:)13:09
ssam2but maybe better not to do that, so that when we do reproducibility testing we actually catch things that are reading from /dev/random13:09
ssam2rather than hiding the actual problem13:10
persiaThat's probably better.  Producing the same result from a source that doesn't typically produce the same result strikes me as less than ideal.13:12
*** noisecell has joined #buildstream13:12
*** noisecell has quit IRC13:17
*** noisecell has joined #buildstream13:17
gitlab-br-botbuildstream: merge request (jmac/microsecond-timing->master: WIP: Optional microsecond timing for log messages) #275 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27513:18
jmac^ removed the documentation from that request13:19
gitlab-br-botbuildstream: merge request (jmac/microsecond-timing->master: Optional microsecond timing for log messages) #275 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27513:21
paulsherwoodaiui kbas is more reliable, uses less resources, is easier to setup, has less dependencies. i may be wrong13:21
paulsherwoodmoving swiftly on... could bst be used to create custom Windows/MacOS images13:25
paulsherwood?13:25
persiapaulsherwood: Yes, but currently those would have to be buildable on linux.13:26
persiaThere is some work underway that *should* be able to allow creating custom MacOS images on MacOS, but it very much depends on the sources chosen at the current time.13:26
paulsherwoodso no in parcice, then?13:26
paulsherwoodpractice13:26
persiaIt depends on the source.13:27
persiaFor example, freeciv's windows binaries *can* be built from linux, and so it is theoretically possible to create a Windows installer of freeciv with BuildStream13:27
persiaBut that's not universally true.13:27
persiaFor MacOS, the main issue today is that the UNIX backend depends on being run as root.13:28
persiaWell, and that some sources can't be built on the default filesystems, but that's a smaller issue.13:29
paulsherwoodgeneric usecase - i'm developing sw to run on both MacOS+Windows. ideally i want to validate my sw on controlled (but special) MacOS/Windows images deployed in CI for build and or test. can i use bst to create the custom images13:29
persiaIf you are starting software from scratch, absolutely.13:31
paulsherwood(maybe via bst in a docker container or vm operating on host directories?13:31
persiaIf you are starting from existing software, maybe: depends on the software.13:31
paulsherwoodnot from scratch13:32
persiaIf you want to run the build under windows, you'd have to use some sort of unix compatibility layer (there are a few to choose from) and the generic unix backend, and build as root.13:32
paulsherwoodroot would be an acceptable compromise i think, provided sandboxing is still happening13:32
persiaThen the software has to be known to a) build safely on case insensitive filesystems and  b) build correctly using tooling you can inject into buildstream and run in a unix-like context13:33
persiaThe generic unix interface doesn't have real sandboxing (uses chroot, no namespace protection, etc.).  You'd need a sandboxing backend for your build platform.13:34
persiaIf you can build the software in a linux environment, you can have real sandboxing, but you will need to inject tooling that can run in that enviornment and build for your target environment.13:36
*** noisecell has quit IRC13:36
gitlab-br-botbuildstream: issue #259 ("Start time for jobs in the log output") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/25913:37
paulsherwoodthe usecase i'm considering is to replace hand-crafted mac/windows build machines with (say gitlab runner) appliances that have bst at the top level, but running whatever tooling is on the machine13:37
persiaBuildStream doesn't work that way.13:37
paulsherwoodin what sense?13:38
persiaEvery effort has been taken to remove access to the host tooling.  This is weak in the case of the generic unix backend (as it is possible to break out of the chroot and access host tooling), but not weak enough for normal payloads to run "whatever tooling is on the machine"13:38
persiaUnless one embeds a chroot escape exploit into the build instructions, everything that runs has to run against an environment either constructed by buildstream or injected into buildstream.13:39
paulsherwoodright, and that is sensible.13:39
persiaIt is possible to inject an install of a proprietary operating system as a baseline, assuming the semantics are roughly POSIX.13:40
persiaAnd one could use that + buildstream to build images that depend on tooling only available in that environment.13:41
persiaBut that's not quite the same as what you describe asking gitlab runner to do.13:41
persiaAnd anything that depends on deep features of an OS probably requires a target-specific sandbox implementation.13:42
paulsherwoodright. that sounds ok. i shouldn't have said 'whatever tooling is on the machine' - i would want to control that, but via the 'inejct into bst' route13:43
persia(note that this is possible: BuildStream already has two sandbox backends enabled, but it's work if you want it for platforms not currently supported)13:43
paulsherwoodyup13:43
persiaThen, broadly, yes, but it's a long road of work to make something reliable and secure :)13:43
persiaMy recommendation would be to start by trying to build some hello-world thing with the generic unix backend.13:44
persiaThere might be some platform bugs, as I don't think running bst under either MacOS or Windows is tested much,.13:45
persiaOnce that works, then you'd want a platform sandbox backend, to make it a bit more secure.13:45
persiaDuring the build/test cycle for that, most of the wrinkles about injecting an OS baseline are likely to be resolved.13:45
persiaFrom that point you could start defining elements to put atop the baseline towards building your application.13:46
persiaAnd you'll probably want a custom output plugin that generates the sorts of artifacts you want to deliver at the end.13:47
persiaFor MacOS, I suspect this is relatively easy.  For Windows, I'd strongly suggest focusing on a WSL-based implemnentation, as you'll encounter fewer false assumptions along the way.13:48
*** noisecell has joined #buildstream13:50
juergbiinteresting, WSL allows running Win32 binaries. afaik, it also has at least partial support for namespaces and thus bwrap might work13:52
persiaThat would be very cool. ,but needs testing.13:58
persiaIf we can use bwrap under WSL, building Windows software on Windows should work smoothly.13:58
gitlab-br-botbuildstream: issue #259 ("Start time for jobs in the log output") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/25914:10
*** noisecell has quit IRC14:22
*** noisecell has joined #buildstream14:23
juergbipersia: injecting windows SDK into buildstream may require some manual work and may not be distributable but other than that, yes14:28
juergbii'm wondering in what way Win32 processes running in a WSL namespace are sandboxed, though. really odd combination14:29
juergbii suspect not at all (or maybe Windows will refuse running Win32 binaries in a WSL namespace/sandbox)14:30
persiaYes.  Getting a license to distribute a windows SDK (or windows chroot) is well outside the scope of this discussion, but shouldn't be needed if push is restricted to internal non-public cache.14:32
*** noisecell has quit IRC15:45
*** Prince781 has joined #buildstream15:47
*** Prince781 has quit IRC15:57
*** noisecell has joined #buildstream16:00
*** mcatanzaro has joined #buildstream16:01
*** Prince781 has joined #buildstream16:14
*** toscalix has quit IRC16:28
jjardon[m]Hi, I want to put some files in a specific folder in a generated system (for example, the /usr/lib/os-release file); is there recommended way to do this in buildstream?16:32
persiaMy recommendation would be to put the files in a git repo, create a Makefile that put them in the desired locations when called with `make install`, and write an element for it.16:34
ssam2there's an easier way16:35
ssam2import them with the 'local' element from the same git repo as your project16:35
ssam2see sdk/files/initramfs-scripts in the freedesktop SDK repo for example16:35
ssam2if they are going to change frequently or there are lots of them then maybe a separate Git repo would be worthwhile, though16:35
persiaMy main motivation for recommending a repo is to parallel the base-files package in Debian.  If one just needs a couple files for a local thing, using the "local" element is probably easier.  If one wants to maintain a set of core files (which I thought was implied by including /usr/lib/os-release), then I think it is good to use proper SCM to collaborate on them.16:39
ssam2i expect javier's buildstream project is also in a "proper SCM"16:39
ssam2if not, then he has a bigger problem :-)16:39
persiaAnd you've convinced me.16:41
persiaI've just realised this is the same as "native" and "non-native" packaging, essentially, and I agree there is a place for both, and in cases where code is not shared between different sets of build instructions, "native" (using local elements) is the saner way to go.16:42
gitlab-br-botbuildstream: merge request (sam/doc-sandbox->master: doc: Add 'sandboxing' section) #279 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27916:45
ssam2jmac, that ^ might be interesting to you related to issue #3816:45
jmacI will try to remember that for when I look at that issue next week16:46
ssam2i will mark it as related, in fact16:46
gitlab-br-botbuildstream: merge request (sam/doc-sandbox->master: doc: Add 'sandboxing' section) #279 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27916:47
gitlab-br-botbuildstream: merge request (sam/doc-sandbox->master: doc: Add 'sandboxing' section) #279 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/27916:49
*** dominic has quit IRC16:53
*** ssam2 has quit IRC16:58
jjardon[m]ssam2: persia thanks for your inputs16:59
*** adds68 has quit IRC17:02
*** adds68 has joined #buildstream17:02
*** jonathanmaw has quit IRC17:05
*** noisecell has quit IRC17:14
*** valentind has joined #buildstream17:25
*** tiago has quit IRC17:42
*** adds68 has quit IRC18:11
*** Prince781 has quit IRC18:57
*** Prince781 has joined #buildstream19:00
*** aday has quit IRC19:12
*** Prince781 has quit IRC19:34
*** Prince781 has joined #buildstream19:47
*** Prince781 has quit IRC19:51
*** valentind has quit IRC20:57
*** valentind has joined #buildstream20:57
*** mcatanzaro has quit IRC21:07
*** cs_shadow has quit IRC22:17
*** valentind has quit IRC22:59
*** mcatanzaro has joined #buildstream23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!