| *** mcatanzaro has joined #buildstream | 00:05 | |
| *** tristan has quit IRC | 00:30 | |
| *** Prince781 has joined #buildstream | 00:35 | |
| *** Prince781 has quit IRC | 01:16 | |
| *** mcatanzaro has quit IRC | 02:11 | |
| *** tristan has joined #buildstream | 07:08 | |
| tristan | tlater, actually, I never found out that I needed the `-m integration` integration part of that invocation; looks to me that it's unneeded and `--integration` is enough by itself | 07:14 |
|---|---|---|
| tristan | except for the current hard requirement of knowing about an env var | 07:14 |
| juergbi | hi tristan, i was recently thinking about staging order and don't remember exactly, is the only reason for sort_dependencies() to get a hopefully better planning order (elements with many reverse dependencies first for more parallel build job opportunities) or is there another reason? | 07:35 |
| juergbi | i think i'd actually prefer full control over staging order, i.e., stage in the declaration order (depth first, of course) | 07:37 |
| tristan | juergbi, first reason was because in absence of that we needed tsort at stage time | 07:38 |
| tristan | which is expensive | 07:38 |
| tristan | but declaration order is wrong I think, because I think there is no declaration order | 07:38 |
| juergbi | why would we need tsort at stage time? | 07:39 |
| juergbi | is depth first not sufficient? | 07:39 |
| juergbi | depends is a list, so i'd say there is a declaration order | 07:39 |
| tristan | i.e. in the sense of only one element; there is a declaration order of only the direct dependencies, below that all bets are off because common dependencies will disagree | 07:40 |
| juergbi | we could just stage indirect dependencies first time we see them | 07:40 |
| tristan | Also, I think we never wanted the order of the depends list to be meaningful | 07:41 |
| tristan | i.e. reorder of those is not something that ever effected cache key | 07:41 |
| tristan | affected (/me learns new english word last week) | 07:41 |
| juergbi | cache key calculation follows the usual depth first order | 07:42 |
| juergbi | self.dependencies(Scope.BUILD) is used for both cache key and staging in build elements | 07:42 |
| tristan | Right, which is currently presorted and will not be affected by order of deps specified in a .bst file | 07:43 |
| juergbi | right but without sort_dependencies() we'd still have no cache key issue | 07:43 |
| juergbi | if overlaps were never allowed, it wouldn't matter, of course. but now i consider it a bit odd that you can allow overlaps (which are directional) but you can't influence staging order at all | 07:44 |
| tristan | well, it's not an "issue", but it's rather inconvenient if the order of deps you specify in an element becomes meaningful | 07:44 |
| tristan | yes, sibling elements with overlaps is indeed weird | 07:44 |
| tristan | is that common ? | 07:45 |
| tristan | (when I say sibling in this sense, I mean not depending on eachother; sibling in dependency graph) | 07:45 |
| juergbi | i don't know but i don't like not being able to predict the staging order | 07:45 |
| tristan | heh | 07:46 |
| juergbi | why i'm thinking about it right now is that an overlay could be useful in some special cases | 07:46 |
| juergbi | i.e., where you allow overlaps and want to ensure that the overlay is on top | 07:46 |
| tristan | ummm, what is an overlay then ? | 07:47 |
| juergbi | e.g., overlay with cross compile environment | 07:47 |
| * tristan is thinking overlay = some fuse layer or equivalent | 07:47 | |
| juergbi | no, i meant for elements | 07:47 |
| tristan | yeah I dont know what this means | 07:47 |
| juergbi | i.e., just a special element staged on top of the other dependencies | 07:48 |
| juergbi | with staging in declaration order, one option would be to simply put that at the end of the depends list | 07:48 |
| tristan | and what would it do ? it would be special in the sense that for that element overlaps are always allowed / expected ? | 07:48 |
| juergbi | that functionality is already there, we can configure that | 07:49 |
| juergbi | but we'd need to be able to ensure it's really the last element staged | 07:49 |
| tristan | So an overlay is just an element that you expect to overlap other things ? | 07:49 |
| tristan | that is already achievable with explicit depends | 07:50 |
| tristan | i.e. if I depend on A, B and C, and I want to be sure C is staged over B | 07:50 |
| tristan | then make C depend on B | 07:50 |
| juergbi | but B build depends on C (bootstrapping) | 07:50 |
| juergbi | so for most elements B wouldn't be staged at all, but it may get pulled in via indirect dependencies | 07:51 |
| juergbi | in those cases, i want to ensure that C is on top | 07:51 |
| tristan | Uhhhh | 07:51 |
| tristan | ummmm, errrr | 07:52 |
| juergbi | i can also provide some concrete examples if you want | 07:52 |
| juergbi | or is it already clear? | 07:53 |
| tristan | no it's not really clear, there is something I'm missing entirely | 07:53 |
| juergbi | i have a cross compiler (+ coreutuils etc.) that i want to use to construct a target system | 07:53 |
| tristan | right, are we talking about things we currently support ? | 07:54 |
| tristan | or things we want to support in the future ? | 07:54 |
| juergbi | more or less currently supported. i already successfully cross built an aarch64 system... | 07:54 |
| tristan | I think baserock does it right, and freedesktop-sdk is cheating | 07:55 |
| tristan | because we dont understand arch in the buildstream core, yet, by design | 07:55 |
| juergbi | i build everything in one session on x86-64 | 07:55 |
| juergbi | in the target elements i build depend on the cross compiler | 07:56 |
| tristan | ok sorry, I'll let you explain first | 07:56 |
| juergbi | in normal cases that's all fine, it builds with the cross compiler as expected | 07:56 |
| juergbi | however, there is also an element for a target native compiler | 07:56 |
| juergbi | which is not used for the buildstream build but we want a compiler in the target system | 07:56 |
| tristan | right, the canadian cross; cross built native | 07:57 |
| juergbi | some target elements may runtime-depend on a compiler, so they obviously depend on the target native compiler | 07:57 |
| juergbi | if i now want to build an element that build depends on such an element (that runtime-depends on the target-native compiler), two compilers get pulled in | 07:57 |
| juergbi | the cross compiler and the target-native compiler | 07:58 |
| juergbi | and they conflict in my case. i want the cross compiler to 'win' because the other one would only work via qemu-user | 07:58 |
| juergbi | i.e., the cross compiler is C in your previous example. the target native compiler is B | 07:58 |
| juergbi | can also get more concrete with packages, might be easier | 07:59 |
| juergbi | valac should probably runtime-depend on (target-native) gcc | 07:59 |
| tristan | in practice this doesnt happen because the cross compiler will be sysrooted to something like /tools | 07:59 |
| juergbi | not in this case | 07:59 |
| juergbi | i use the same prefix to avoid path issues | 08:00 |
| juergbi | the elements that are cross-built don't know that they are using a cross compiler | 08:00 |
| tristan | which is explicitly used, while the cross built native one goes in /usr | 08:00 |
| tristan | but that doesnt invalidate what you're saying | 08:00 |
| tristan | So say, cross built glibc runtime depends on cross built gcc for parts of the runtime | 08:02 |
| tristan | and you need to link everything against cross built glibc | 08:02 |
| tristan | so you always need both compilers in the runtime | 08:02 |
| tristan | juergbi, correct ? | 08:03 |
| juergbi | the original cross compiler gcc already cross compiles libgcc and co | 08:03 |
| juergbi | they can directly be used | 08:03 |
| tristan | a lot of text came at once... | 08:03 |
| juergbi | yes, it was just meant to be an example use case. we don't have to discuss the details of the cross environment, but we can ;) | 08:05 |
| tristan | So, I in that specific case, I want both to win, and I want to probably error out if cross/native are overlapping | 08:05 |
| tristan | because I need both to be staged in different places, the cross compiler for build tooling, the cross built for linking the runtime lib bits from it | 08:06 |
| juergbi | you never need two compilers staged at the same time, no | 08:06 |
| juergbi | roughly you first build the cross compiler with your bootstrap native compiler and after that you build everything with the cross compiler and don't need the bootstrap compiler anymore | 08:07 |
| juergbi | (and the target native compiler is not needed at all) | 08:07 |
| tristan | well, unless you need libgcc.a | 08:08 |
| tristan | to link the cross built libstdc++ | 08:08 |
| juergbi | the details with libgcc are a bit tricky but you still never need two compilers staged as dependencies | 08:09 |
| juergbi | the way i do it is | 08:09 |
| tristan | you probably also need it to cross build glibc, cause glibc will want to have intimate knowledge of the compiler; cause it's on crack | 08:09 |
| juergbi | build minimal cross-gcc with (cross-built) libgcc.a. that's with 'host' compiler as dependency | 08:10 |
| juergbi | (libgcc.a will be built using the cross compiler in the same tree) | 08:10 |
| juergbi | then build glibc using the minimal cross-gcc in /usr. no host compiler needed | 08:11 |
| tristan | I'll clarify that the reason why I'm talking about this case, is because afaics, the safe/clean way to do it is to sysroot the compilers in different places and never overlap | 08:11 |
| juergbi | then build full cross-gcc. again using the host compiler as dependency | 08:11 |
| juergbi | during the two gcc builds, i use a mini-sysroot for cross-binutils | 08:12 |
| juergbi | and for the second gcc build also a mini target sysroot that just contains glibc (+ kernel headers) | 08:12 |
| juergbi | however, outside the gcc build, no sysroot is required for completely clean build without overlaps | 08:12 |
| juergbi | and that's good because many packages can't properly handle cross compilation via sysroot | 08:13 |
| juergbi | even though autotools does support it properly | 08:13 |
| tristan | outside of gcc build, you are hopefully using an explicit cross toolchain sysrooted into /tools | 08:13 |
| juergbi | no, it's in /usr then | 08:13 |
| tristan | where you have setup your cross toolchain | 08:13 |
| tristan | i.e. the cross toolchain that you can run | 08:13 |
| tristan | while cross building the rest of the runtime (busybox or coreutils) | 08:14 |
| juergbi | no, i always put it in /usr | 08:14 |
| tristan | I argue that you should not | 08:14 |
| tristan | I know that you *can* | 08:14 |
| juergbi | with my method i don't have to deal with cross compilation issues | 08:14 |
| juergbi | packages think they're being compiled natively | 08:14 |
| tristan | Or, at least configured with --prog-prefix="aarch64-" | 08:14 |
| juergbi | it's installed as /usr/bin/gcc | 08:15 |
| juergbi | (and aarch64-linux-gnu-gcc but that's not strictly needed) | 08:15 |
| tristan | Not strictly needed, but safer and cleaner is what I'm getting at | 08:15 |
| juergbi | i disagree | 08:16 |
| juergbi | in contrast to baserock and co. i don't need to adjust any gcc specs, linker/include paths etc. | 08:16 |
| juergbi | that's a source of a lot of issues in cross compilation environment | 08:16 |
| tristan | I would rather mimic the process that would happen on a single system where you did steps one by one on the same filesys | 08:16 |
| juergbi | and i never need to stage two compilers at the same time | 08:16 |
| tristan | than come up with some way to swap things along the way | 08:17 |
| juergbi | relocation can be painful | 08:17 |
| tristan | It doesnt really need relocation, though | 08:17 |
| juergbi | i don't see an issue with my approach. it's very clean. i don't see why i'd want to change it | 08:18 |
| tristan | Where is this approach ? | 08:18 |
| tristan | does it exist ? | 08:18 |
| juergbi | just a local test for now but it works | 08:18 |
| juergbi | i still use a slightly modified buildstream but not heavily | 08:18 |
| tristan | Ok well, I'm certainly not in favor of that approach | 08:19 |
| tristan | So lets talk instead about what in buildstream is blocking you from doing your own thing | 08:19 |
| tristan | And if there is really a rational reason to change things for the sake of this, and what would be the more elegant thing to change to allow you to do that | 08:20 |
| juergbi | i think it would actually build with the existing staging order in bst master. but with more complex dependency trees, it could cause issues in the future | 08:21 |
| juergbi | (i.e., i don't have any conflicts in my current build) | 08:21 |
| tristan | First, I think that being able to reorder staging such that what is deeper comes out on top would be bad | 08:21 |
| tristan | i.e. you have B depends on C but want C on top when A depends on B and C; weird | 08:22 |
| juergbi | well, B only build-time depends on C, not run-time | 08:22 |
| tristan | Rather, it would be better to explicitly drop something and not stage at all ? | 08:22 |
| tristan | Back in YBD days, I wanted a "replace" semantic | 08:23 |
| juergbi | yes, something like that could be interesting | 08:24 |
| tristan | I.e. so that something like coreutils, or a stack with it's direct dependencies; could replace busybox | 08:24 |
| tristan | I think we have solved that in other ways; but maybe not ? | 08:24 |
| tristan | I *thought* we did with build only dependencies; but maybe it's more work to setup those dependency chains than needed ? | 08:25 |
| juergbi | not sure what you're referring to | 08:26 |
| tristan | Being able to build busybox and use it "up to a point" and then stop staging it once we've built its replacements | 08:26 |
| tristan | is something I wanted to solve with build-only dependencies | 08:27 |
| juergbi | i think i would be happy if i could explicitly specify which out of two elements would be staged if both end up in dependencies(Scope.BUILD) | 08:27 |
| tristan | or I wanted that to be solved by build-only dependencies | 08:27 |
| tristan | but, maybe there is an argument to implement replace semantics instead ? | 08:27 |
| juergbi | you mean by manually specifying busybox as build dependency for coreutils and co? | 08:28 |
| tristan | Basically yeah; I would have the runtime and the busybox "stacks" separate at the lowest level | 08:29 |
| tristan | And when building coreutils and fileutils and friends, I would build-only depend on the minimal tooling, which is separate from the runtime | 08:30 |
| tristan | In a stack which builds those tools | 08:30 |
| tristan | Higher than that; I'd just depend on the "core" stack which built coreutils/fileutils/etc | 08:30 |
| tristan | and busybox goes away | 08:30 |
| juergbi | yes, this is certainly possible | 08:30 |
| juergbi | requires some manual effort but works | 08:31 |
| juergbi | it doesn't solve the cross compiler case, though | 08:31 |
| juergbi | where you actually want to keep it around for the whole build of the target system | 08:31 |
| tristan | Right, that was kind of the idea; I'm trying to see if there is commonality I can find | 08:31 |
| tristan | woosh, the *whole build* ? | 08:32 |
| juergbi | yes, i want to build everything on x86-64 | 08:32 |
| juergbi | so i want to use the cross compiler | 08:32 |
| tristan | Ok, well we never said you werent allowed to do it | 08:32 |
| tristan | but it's sort of a goal to not do that | 08:32 |
| juergbi | this works without having to patch packages for cross compilation | 08:33 |
| juergbi | for me that's a win-win | 08:33 |
| tristan | So, to add to the whole thing you were talking about before... you are *also* experimenting with qemu-user ? | 08:33 |
| juergbi | yes, it's great :) | 08:34 |
| tristan | ok, I really need a big, complete picture to be helpful | 08:34 |
| tristan | it sounds great yes | 08:34 |
| tristan | I dont particularly like the idea of restructuring dependencies; we should find another way | 08:35 |
| tristan | (or I think so for now) | 08:35 |
| juergbi | ok, so the basic idea is to select a few tools that behave identically whether they run on x86-64 or AArch64 | 08:35 |
| tristan | It would be more ideal to not stage what you dont want staged at all | 08:36 |
| tristan | juergbi, no no... | 08:36 |
| tristan | juergbi, hahahaha | 08:36 |
| juergbi | hm? | 08:36 |
| tristan | I rather think that this is the kind of thing I will open in my inbox and try to really let it soak in with my first coffee of the day | 08:37 |
| juergbi | ah ;) | 08:37 |
| tristan | Well, okay shoot if you like; I'm curious | 08:37 |
| tristan | but I dont expect to really "get it" in a quick irc exchange | 08:37 |
| juergbi | basic tools from coreutils and co. behave the same on different architectures | 08:38 |
| tristan | this sounds like understanding how aboriginal works, for a completely different aboriginal :) | 08:38 |
| juergbi | and a cross compiler also generally works the same as a native compiler for the same target with the same configuration | 08:38 |
| juergbi | so what i do is to compile these tools statically | 08:38 |
| juergbi | on x86-64 | 08:38 |
| juergbi | and place them at the normal locations, i.e., in /usr | 08:38 |
| juergbi | the rest of the system is from the target | 08:39 |
| juergbi | and then thanks to qemu-user the whole sandbox behaves the same as if it was running natively on aarch64 with native tools | 08:39 |
| juergbi | but the compiler a few other tools run much faster | 08:40 |
| juergbi | *and | 08:40 |
| tristan | I have to admit I'm uncertain how qemu-user really works, is it going to know all by itself what needs to be run under which arch ? | 08:40 |
| juergbi | that's the linux binfmt-misc magic | 08:41 |
| juergbi | you can register ELF aarch64 to be run via qemu-user | 08:41 |
| juergbi | (unfortunately requires root on the host) | 08:41 |
| tristan | Ah | 08:41 |
| juergbi | and then aarch64 ELF will work in addition to x86-64 ELF | 08:41 |
| juergbi | all libraries in the sandbox will be for the target system. same for headers. no possibility for confusion | 08:42 |
| tristan | So you can just put aarch64, i386 and mips binaries all over the place and things will still work | 08:42 |
| juergbi | yes | 08:42 |
| tristan | super crazy :) | 08:42 |
| tristan | right, and *programs* | 08:42 |
| tristan | but yeah I get it | 08:43 |
| juergbi | and except for the selected tools where i know that they are safe, everything is built as aarch64 | 08:43 |
| juergbi | so i don't have a constant risk with new packages or anything like that | 08:43 |
| tristan | So you will want to select replacements for things during a build | 08:43 |
| tristan | instead of running the native compiler under qemu-user, you want to stage the cross one | 08:43 |
| juergbi | right | 08:44 |
| juergbi | i use the cross compiler as the main devel build dependency, so normally that is all fine without extra features | 08:44 |
| tristan | juergbi, note that only now, with a bit more context, can I really think that using a cross compiler in /usr is anything close to a sane idea | 08:44 |
| tristan | this is all part of your bigger picture | 08:45 |
| juergbi | yes, sure, it definitely needed context | 08:45 |
| tristan | heh | 08:45 |
| tristan | so, this is something you will want to repeat beyond the compiler, too | 08:45 |
| tristan | the more you do this, the better your build will be | 08:45 |
| tristan | or *faster* | 08:46 |
| tristan | is more accurate | 08:46 |
| tristan | you will soon want to stage host perl instead of target perl | 08:46 |
| juergbi | no, i won't do that | 08:46 |
| tristan | perl and the shell are the bottlenecks to configure scripts, though | 08:46 |
| juergbi | i currently do it for gcc, binutils, make, {core,diff,find}utils, gawk, grep, m4, patch, sed, tar | 08:47 |
| juergbi | and bash | 08:47 |
| juergbi | yes, perl is indeed slow in the sandbox | 08:47 |
| tristan | Right | 08:47 |
| juergbi | however, perl supports loadable modules, so it's not safe for this case | 08:47 |
| juergbi | and in the big picture, this approach is still very fast | 08:47 |
| juergbi | and perl is not used that much anymore, especially if you don't autoreconf everything | 08:48 |
| tristan | But I mean, if you want to take your experiment and mainline a feature in BuildStream for it, even if *you* wont do this with perl, someone else will try to | 08:48 |
| juergbi | that's definitely possible, yes | 08:48 |
| tristan | And what I mean by this mostly is; what we do in BuildStream should probably make staging work for what people wanna do | 08:48 |
| juergbi | i think this could be a very solid cross build story, though | 08:49 |
| tristan | it sounds promising :) | 08:49 |
| tristan | some admin config before running build sessions as a regular user I think is not horrible | 08:50 |
| juergbi | i think on debian there is even a package for it that registers it | 08:51 |
| juergbi | and systemd has /etc/binfmt.d/ for this | 08:51 |
| tristan | I think it's needed anyway for sandboxing on some platforms (I think AIX had something like "jails" which need some admin stuff to be done first) | 08:51 |
| juergbi | i should definitely be faster than the qemu-system and distcc approach | 08:52 |
| tristan | so that's not really a concern to me | 08:52 |
| tristan | definitely | 08:52 |
| tristan | plus you get the linker, you can't reasonably do `distld` | 08:52 |
| juergbi | right, linker and shell probably help quite a bit | 08:53 |
| juergbi | and you don't have the distcc overhead. also, qemu-system is slower because the kernel is emulated as well | 08:53 |
| juergbi | although you could probably use the distcc approach also with qemu-user | 08:54 |
| juergbi | i use junctions to support bootstrap (imported binaries), stage1 (building up-to-date native host toolchain), stage2 (building cross compiler), and then the final target system | 08:55 |
| juergbi | the latter three are all the same repo, just different options | 08:56 |
| juergbi | i'll try to get something semi-clean out of it soon, so i can push it | 08:57 |
| *** tm has joined #buildstream | 09:02 | |
| tristan | juergbi, sounds exciting | 09:06 |
| tristan | I think we should find out what platforms we could potentially do this on | 09:06 |
| tristan | or if it's really only linux specific | 09:07 |
| tristan | and what feature we could have | 09:07 |
| juergbi | qemu-user has bsd support, not sure which bsd | 09:07 |
| juergbi | i don't think it supports anything else | 09:07 |
| tristan | something I've been thinking about too, is that maybe stacks can help in some way, they are currently quite feature-free | 09:08 |
| tristan | but I've been thinking sometimes that it would be helpful to infer things about the direct dependencies explicitly listed in a stack | 09:09 |
| tristan | it may not be helpful for this; not sure | 09:09 |
| juergbi | hm, not sure | 09:09 |
| tristan | when you depend on a stack, you actually mean to depend on all of it's dependencies | 09:10 |
| juergbi | one aspect that seems to be an issue for many approaches is that if you compose an element that runtime-depends on a native compiler, you normally want the native compiler, and not replace it with the cross one | 09:10 |
| tristan | which is an opportunity to express something as a batch | 09:10 |
| gitlab-br-bot | buildstream: merge request (juerg/sandbox->master: Sandbox improvements) #288 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/288 | 09:50 |
| gitlab-br-bot | buildstream: merge request (juerg/die-with-parent->master: sandbox/_sandboxbwrap.py: Use --die-with-parent) #278 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/278 | 10:11 |
| *** tm has quit IRC | 10:19 | |
| gitlab-br-bot | buildstream: issue #271 ("Assert minimum version of bwrap on host") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/271 | 10:45 |
| juergbi | tristan: not sure about the artifact version bump | 10:49 |
| juergbi | merging without bumping the version would still improve things. the question is whether we want to consider existing artifacts potentially contaminated because we didn't isolate it before | 10:50 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 10:50 |
| juergbi | policy decision, up to you :) | 10:50 |
| tristan | yeah, I'm not sure either | 10:50 |
| *** tm has joined #buildstream | 10:51 | |
| tristan | I think the question is: If something was not guaranteed constant before and now is guaranteed constant; does that constitute a version bump | 10:51 |
| juergbi | in general i'd say yes | 10:52 |
| juergbi | not sure whether we want to be strict in this case. i assume we'll have artifact version bumps in the not so far future | 10:52 |
| juergbi | while the hostname can influence bit-for-bit reproducibility, i don't expect any real issues | 10:52 |
| juergbi | and SysV IPC was unlikely a real issue in builds so far | 10:53 |
| tristan | True, it's a tradeoff of documenting things vs imposing rebuilds | 10:53 |
| juergbi | if we anyway fix this https://gitlab.com/BuildStream/buildstream/issues/266 i guess it doesn't matter | 10:54 |
| juergbi | but let's avoid bumping it too frequently | 10:54 |
| tristan | That does not constitute a core artifact version bump, though | 10:55 |
| juergbi | right, that's only compose | 10:55 |
| tristan | i.e. 266, is strictly related to the compose element and how it is buggy in generating a key | 10:55 |
| tristan | yeah, so that's what I've been thinking; I definitely want an artifact version bump for this change | 10:56 |
| tristan | But as you say, it's relatively minor and not nice to impose rebuilds just for this | 10:56 |
| tristan | So the thing is "I'm not in a hurry" | 10:56 |
| tristan | I'd be happy to know that one will eventually come, in advance of next stable | 10:57 |
| tristan | which is a long while | 10:57 |
| juergbi | yes, i expect it to be likely that we'll anyway need one before then | 10:57 |
| tristan | Okay, lets not do it | 10:57 |
| tristan | it's not a good time to require everything to rebuild; and the change mostly adds security where there was not before | 10:58 |
| juergbi | yes, | 10:58 |
| juergbi | let's merge this, then | 10:59 |
| juergbi | tristan: apropos small bwrap sandbox patches, does https://gitlab.com/BuildStream/buildstream/merge_requests/278 look fine now as well? | 10:59 |
| gitlab-br-bot | buildstream: merge request (juerg/sandbox->master: Sandbox improvements) #288 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/288 | 10:59 |
| tristan | juergbi, yup thats good to go, please rebase and merge :) | 11:02 |
| juergbi | ta | 11:02 |
| * tristan broke his own test in a way he doesnt understand | 11:03 | |
| gitlab-br-bot | buildstream: merge request (juerg/die-with-parent->master: sandbox/_sandboxbwrap.py: Use --die-with-parent) #278 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/278 | 11:03 |
| juergbi | btw: i quite like the 'replaces' idea. should be pretty straight forward. the question would be whether it's sufficiently useful in general to add it upstream | 11:06 |
| juergbi | i'll probably push a MR for this at some point | 11:06 |
| tristan | yeah I dont think we'll get the right solution in a hurry | 11:07 |
| juergbi | yes, we better think about this further | 11:08 |
| juergbi | but i might try it out with my test project to get some experience | 11:08 |
| tristan | So I have 2 tests, one passes and the other fails | 11:08 |
| tristan | But the two tests are exactly identical | 11:08 |
| juergbi | also have to write a .bst generator for my distro so i can try building the whole world ;) | 11:09 |
| juergbi | Schrödinger's test? | 11:09 |
| tristan | I dont really get it, am I constructing the argv array wrong ? adding two lists together ? | 11:09 |
| tristan | juergbi, wanna take a quick look here: https://gitlab.com/BuildStream/buildstream/blob/project-shell-customize/tests/integration/shell.py#L58 ? | 11:10 |
| tristan | there is this execute_shell() helper, which is used everywhere else; using it here makes it break | 11:10 |
| tristan | but.... it looks like it's doing the same thing either way | 11:10 |
| juergbi | how does it break? | 11:10 |
| tristan | Program output was: | 11:11 |
| tristan | ${ANIMAL} | 11:11 |
| juergbi | tristan: in the upper test you return before the asserts | 11:11 |
| juergbi | so you never check them | 11:11 |
| tristan | oops, looks like something got lost in translation | 11:11 |
| tristan | Ok I'll have to work on this :-S | 11:12 |
| tristan | juergbi, thanks for the fresh eyes ! | 11:12 |
| juergbi | np | 11:12 |
| tristan | mine are a bit foggy from too many soju bottles last night hehe | 11:12 |
| juergbi | tristan: btw: maybe we should go with environment-inherit instead of inherit-environment | 11:12 |
| juergbi | for consistency with environment-nocache | 11:12 |
| juergbi | haha | 11:12 |
| tristan | juergbi, yeah good idea, and will be nicer if we ever added constant settings in there (if the user wants something to be a specific value, only in shells) | 11:13 |
| gitlab-br-bot | buildstream: merge request (juerg/die-with-parent->master: sandbox/_sandboxbwrap.py: Use --die-with-parent) #278 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/278 | 11:15 |
| tristan | juergbi, I think you saw it wrong :-/ | 11:24 |
| tristan | oohhh no | 11:24 |
| juergbi | what? | 11:24 |
| tristan | you saw it right :) | 11:24 |
| tristan | hahahaha | 11:24 |
| tristan | return cli.blblblblb | 11:24 |
| juergbi | ah | 11:24 |
| tristan | I think it's because I'm only invoking echo, not the shell | 11:24 |
| juergbi | ah, yes, the shell takes care of variable replacement | 11:27 |
| juergbi | no shell, no replacement | 11:27 |
| juergbi | tristan: do we support a way to completely replace/overwrite a dict? e.g., if i want to be in charge of environment variables, how can i make sure i don't get any from the default project config? | 11:30 |
| tristan | We dont | 11:32 |
| juergbi | i don't even know whether we can unset env. variables individually. setting to empty string is not always the same | 11:32 |
| juergbi | i think we should | 11:32 |
| tristan | juergbi, current situation is that defaults are not allowed to change in stable series | 11:32 |
| tristan | I was considering some directive to delete keys | 11:33 |
| tristan | when implementing the list manipulation directives | 11:33 |
| juergbi | ok, that makes sense but is still not quite good enough, imo | 11:33 |
| juergbi | syntax is always tricky, of course | 11:33 |
| tristan | No, as a completely separate activity; I was thinking we may want a way to export/import defaults for a given project to be able to reuse at a later date with a new version of buildstream | 11:34 |
| tristan | not sure I recall why that sentence started with "No" | 11:34 |
| juergbi | that could definitely be helpful | 11:34 |
| tristan | but also; that activity was inline with an initiative for longer term stability of cache keys | 11:34 |
| juergbi | for myself i'd rather want the ability to start with a clean slate dict-wise | 11:35 |
| tristan | which we concluded was low priority enough to not be in immediate project scope | 11:35 |
| juergbi | yes, sure, it's not urgent, i just noticed it | 11:35 |
| tristan | yeah | 11:35 |
| juergbi | maybe we could also support (=): for dicts? or would this cause syntax issues? | 11:35 |
| tristan | Maybe we can | 11:35 |
| tristan | sounds reasonable | 11:36 |
| juergbi | i'll file an issue | 11:36 |
| tristan | The only issue I can think of off the top of my head is, if you really want a key that is '(=)' | 11:36 |
| tristan | which seems unlikely; but is not a problem at all for lists | 11:36 |
| tristan | well, actually maybe it's the same problem for lists | 11:37 |
| tristan | those directives are just not allowed heh | 11:37 |
| tristan | as keys | 11:37 |
| juergbi | right, we should probably simply reserve those | 11:37 |
| juergbi | another thing that i'm missing is support for staging dependencies at particular paths as part of regular build elements. i can work around it with helper elements but it's a bit odd | 11:38 |
| juergbi | nothing terribly critical | 11:39 |
| *** tristan has quit IRC | 11:44 | |
| gitlab-br-bot | buildstream: issue #272 ("Support overwriting complete dict") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/272 | 11:46 |
| *** ssssam[m] has joined #buildstream | 13:04 | |
| *** valentind has joined #buildstream | 13:18 | |
| *** tristan has joined #buildstream | 13:53 | |
| *** tristan has left #buildstream | 13:53 | |
| *** tristan has joined #buildstream | 13:53 | |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 14:09 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 14:13 |
| gitlab-br-bot | buildstream: issue #264 ("The build environment should set C.UTF-8 by default") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/264 | 14:16 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 14:24 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 14:26 |
| gitlab-br-bot | buildstream: issue #273 ("Cache keys not printed correctly with workspaced elements") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/273 | 15:10 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 15:16 |
| gitlab-br-bot | buildstream: issue #231 ("Bash completion broken inside bst shell") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/231 | 15:18 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 15:20 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 15:34 |
| gitlab-br-bot | buildstream: merge request (project-shell-customize->master: Allow customization of shell environment, and set custom prompt in PS1) #287 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/287 | 15:45 |
| juergbi | hm, it's suddenly very slow to load the pipeline. don't see any changes that look relevant | 17:21 |
| juergbi | tristan: what's your stance on per-element options? there are some sources that i want to build twice with same ref and mostly same config and depends | 17:31 |
| juergbi | one reason is bootstrapping. for one there is gcc, but other packages exist with dependency cycles (e.g., systemd/LVM2, pkg-config/glib) | 17:33 |
| juergbi | and then there is a whole set of packages that can be built as python2 or python3 | 17:33 |
| juergbi | maybe i should use a subproject for python packages, have a project option for python2 vs. python3 and then two junctions into the that subproject | 17:34 |
| juergbi | unrelated note, name: would have been nicer than filename: for depends. but i guess we don't want to add an alias for this now... | 17:35 |
| tristan | I think per element options and building the same element twice is two different subjects | 17:37 |
| tristan | I dont see what we dont support about per-element options right now, an option can be declared for use by a specific element | 17:38 |
| tristan | And building same element twice... well that requires a lot of thought/work | 17:39 |
| tristan | I should also note, in the very early iterations of the parser; I had `include` semantics | 17:39 |
| juergbi | i was thinking of the possibility of specifying options on the dependency but i'm not sure whether it's really worth it | 17:40 |
| tristan | which might suffice for the purposes you describe, i.e. composite the dict loaded from this file "right here" | 17:40 |
| juergbi | i was thinking about include as well recently | 17:40 |
| juergbi | it would require creating three .bst files for a source that is built twice but would avoid duplication | 17:40 |
| juergbi | on the other hand, tracking might be tricky | 17:41 |
| tristan | include is fairly easy to do; I think even if the include has a source; `bst track` will "just work" | 17:41 |
| juergbi | it might be a bit confusing if you try to track both elements at the same time | 17:41 |
| tristan | track re-dumps the toplevel node to the file which it was loaded from | 17:41 |
| juergbi | ok | 17:42 |
| tristan | which is not related to any assumption of a specific .bst file | 17:42 |
| juergbi | for gcc i currently have a separate gcc-source.bst and that works pretty well | 17:42 |
| tristan | but yeah; track both elements with same source, is weird | 17:42 |
| tristan | you might do it twice | 17:42 |
| juergbi | but i have the large gcc depends and config duplicated, which i don't like | 17:43 |
| juergbi | so include would definitely help for gcc for me | 17:43 |
| juergbi | for python packages it seems a bit cumbersome, though, with 3 files each | 17:43 |
| tristan | if you want to install a python package for both python2 and python3... then maybe enhancements to the distutils element would be more convenient | 17:44 |
| juergbi | well but i might also not want to pull in python2 for no reason | 17:45 |
| juergbi | for reverse dependencies | 17:45 |
| tristan | I'm not saying make the distutils element do that by default, or force it to happen | 17:45 |
| tristan | just saying it's a feature which could be offered by the element | 17:46 |
| juergbi | i meant even for packages that i want to build for both | 17:46 |
| juergbi | i might not want to have the reverse dependency pull in python2 | 17:46 |
| juergbi | maybe should just wait for python2 to die ;) | 17:47 |
| tristan | If you want to install a python2 package, you probably want to depend on python2 | 17:47 |
| juergbi | yes, but the same python package might have reverse dependencies that support python3 but also some that need python2 | 17:47 |
| juergbi | if i don't install the reverse dependency that needs python2, i don't want to pull in python2 at all | 17:47 |
| tristan | It depends what you're going to build | 17:48 |
| tristan | You have options | 17:48 |
| tristan | I.e. `--option foo value` | 17:48 |
| tristan | you have ways | 17:48 |
| juergbi | yes, if i don't need python2 in the whole project config | 17:48 |
| juergbi | for a typical full system setup this doesn't help | 17:49 |
| tristan | I mean, if you're writing a project; you know the outputs you want to generate | 17:49 |
| juergbi | anyway, it's not a huge concern right now, was just wondering | 17:49 |
| tristan | If you have one target that needs it, and another that doesnt, it's possible to author something which does what you want | 17:49 |
| juergbi | if the output is a distro with selectable packages, it's a bit trickier | 17:49 |
| tristan | We did a run down the implicit element demands something of another element path before | 17:50 |
| tristan | and broke some teeth on the wall we smashed our face on | 17:50 |
| juergbi | hehe | 17:50 |
| juergbi | yes, that's not what i'm after | 17:50 |
| juergbi | junctions should actually work pretty well for this | 17:51 |
| juergbi | put all python packages in a separate subproject and then use junction: python3.bst or python2.bst from outside that project | 17:51 |
| *** Prince781 has joined #buildstream | 17:51 | |
| juergbi | and internally you have the python version-independent dependencies | 17:51 |
| juergbi | the junction coalescing could get into the way as i also have the different bootstrap stages | 17:52 |
| juergbi | wondering whether we should allow some more flexibility with junction handling | 17:52 |
| juergbi | there could even be a generic bst project with generated .bst files from pypi for pure python packages? not sure | 17:53 |
| tristan | I'm... not all here btw, gonna sleep soon last night was toooo much and need to recover :) | 18:04 |
| juergbi | sleep well :) | 18:07 |
| gitlab-br-bot | buildstream: merge request (remove-absent-manifest-files->master: MANIFEST.in: remove files that are no longer in this repo) #289 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/289 | 18:27 |
| jjardon[m] | hi | 19:04 |
| jjardon[m] | https://gitlab.gnome.org/GNOME/gnome-build-meta/-/jobs/9429 | 19:04 |
| jjardon[m] | when the support for the new shell: syntax was introduced? our pipeline is failing even after trying to use current bst master | 19:05 |
| juergbi | hm, it works locally | 19:07 |
| juergbi | Requirement already satisfied: buildstream in /usr/local/lib/python3.6/site-packages | 19:09 |
| juergbi | jjardon[m]: is it possible that it's using some previously installed buildstream? i'm not familiar with that CI setup | 19:10 |
| juergbi | maybe we need to update the buildstream docker images | 19:10 |
| juergbi | however, it seems the local buildstream pip3 install in gnome-build-meta is useless if it anyway only uses the one from the docker image | 19:11 |
| jjardon[m] | yup, it works locally as well; CI problem sorry for the noise | 19:12 |
| jjardon[m] | ah, that can be it | 19:12 |
| *** tm has quit IRC | 19:16 | |
| *** cs_shadow has joined #buildstream | 19:47 | |
| *** Prince781 has quit IRC | 19:56 | |
| *** Prince781 has joined #buildstream | 20:11 | |
| *** Prince781 has quit IRC | 20:18 | |
| *** Prince781 has joined #buildstream | 20:44 | |
| *** xjuan has joined #buildstream | 21:03 | |
| *** tristan has quit IRC | 21:06 | |
| *** cs_shadow has quit IRC | 22:43 | |
| *** valentind has quit IRC | 23:30 | |
| *** Prince781 has quit IRC | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!