| *** tristan has quit IRC | 04:53 | |
| *** tristan has joined #buildstream | 05:21 | |
| *** ChanServ sets mode: +o tristan | 05:21 | |
| tristan | ironfoot, it's not committed yet, but I believe I found the best solution | 09:45 |
|---|---|---|
| tristan | juergbi, I might ask you for a little audit if you have a moment... my theory fwiw; is that we dont need to perform many topological sorts throughout the build process, if direct dependencies of every given element is sorted once at load time | 09:47 |
| ironfoot | tristan: :) | 09:54 |
| *** ssam2 has joined #buildstream | 10:11 | |
| tristan | ironfoot, do you recall which element specifically had problems ? usbutils ? | 10:19 |
| tristan | Also, what should I check here... ldd script yeah | 10:19 |
| ironfoot | ldd script for bash detection in glibc, yes | 10:19 |
| ironfoot | if that goes ok, linux-api-headers should atleast start building | 10:20 |
| tristan | has #! /usr/bin/bash now :) | 10:20 |
| * ironfoot wonders if /bin -> /usr/bin symlink will be present in the staging area | 10:21 | |
| tristan | ironfoot, it is | 10:22 |
| ironfoot | great | 10:22 |
| tristan | ironfoot, I just did bst shell --run build-essential.bst | 10:22 |
| tristan | and all of this, was about ensuring that both fhs-dirs occurrences are staged before anything else | 10:22 |
| tristan | the symlinks it puts in place must be there before other things get staged | 10:22 |
| tristan | so they must depend on fhs-dirs | 10:23 |
| ironfoot | ah, I see why you were looking at the dependencies order yesterday | 10:24 |
| tristan | right, well first I sorted out symlinks based on code I wrote this weekend for ybd | 10:24 |
| tristan | but then I got circumstances where staging was not in dependency order | 10:25 |
| tristan | e2fsprogs passed this time... | 10:30 |
| ironfoot | that was already fixed :P | 10:33 |
| tristan | yeah | 10:34 |
| tristan | ironfoot, alright pushed new buildstream master and new changes to build-essential branch of buildstream-tests to convert on top of | 10:36 |
| tristan | This is the wild sort: https://gitlab.com/tristanvb/buildstream/blob/master/buildstream/_loader.py#L622 .. for which my theory is, that the generator function Element.dependencies()... | 10:37 |
| tristan | Found here: https://gitlab.com/tristanvb/buildstream/blob/master/buildstream/element.py#L109 | 10:38 |
| tristan | Will always report dependencies in correct build & stage order | 10:38 |
| tristan | And I hope that with this, Element.dependencies() can be used to somewhat simplify the juergbi's original planner code here: https://gitlab.com/tristanvb/buildstream/blob/master/buildstream/_pipeline.py#L44 | 10:39 |
| tristan | juergbi, any thoughts on any of this stuff ? | 10:39 |
| juergbi | will take a look in a bit | 10:39 |
| tristan | ironfoot, ok so an autoconverted foundation.bst builds... except for groff | 10:55 |
| tristan | that is fixed by setting the max-jobs variable to 1 in groff.bst | 10:55 |
| tristan | however that also needs a buildstream change, because I want notparallel: True to control that | 10:55 |
| tristan | difference is that max-jobs is autoresolved and does not effect cache keys | 10:56 |
| ironfoot | i forgot about auto converting max-jobs, etc | 11:05 |
| tristan | ironfoot, no worry, I'm making a patch so if you see max-jobs: 1, make it notparallel: True | 11:06 |
| ironfoot | cool | 11:07 |
| tristan | this sed presents a problematic regex: http://git.baserock.org/cgit/baserock/baserock/definitions.git/tree/strata/tools/zip.morph | 11:44 |
| tristan | ironfoot, and now buildstream master supports notparallel correctly, i.e. see: https://people.gnome.org/~tvb/buildstreamdocs/elements/autotools.html#module-autotools for reference | 11:46 |
| tristan | it's valid on any build element actually, but only relevant to those elements which have parallelization support | 11:47 |
| ironfoot | tristan: regarding the sed is because defs2bst didn't handle it? | 11:48 |
| tristan | ironfoot right, defs2bst handled it, but I believe it did the wrong thing | 11:48 |
| tristan | I'm not sure what $$PREFIX is supposed to mean, but it ends up as $%{prefix} | 11:49 |
| tristan | So have to figure out what that means, I suspect it's escape sequence ? | 11:49 |
| ironfoot | you could ask jmacs in #baserock if needed | 11:49 |
| ssam2 | escaping $ inside the Makefile ? | 11:50 |
| tristan | So if so, its intended to place $PREFIX and $DESTDIR variables into the makefile | 11:50 |
| tristan | and it relies on PREFIX and DESTDIR being in the environment | 11:50 |
| tristan | I think the solution is to just also handle $$PREFIX and $$DESTDIR, will be harmless to sed the expanded variable directly into the makefile | 11:51 |
| ironfoot | I'd think so, yes | 11:51 |
| tristan | while that is true, the substitution might not _always_ be true, but I think we can assume this is corner case enough for the definitions repos we know | 11:52 |
| ironfoot | i wonder if others are relying in those variables being in the environment | 11:56 |
| ironfoot | in which case: would it be sensible to add these 2 variables to all the bst files during the conversion? | 11:58 |
| ironfoot | (instead of replacing the vars?) | 11:58 |
| tristan | ironfoot, I would much rather not if we can avoid it | 11:59 |
| tristan | although it is possible | 11:59 |
| tristan | it's a better approach to avoid dirtying the environment | 11:59 |
| ironfoot | i would prefer to avoid it, yes, but i'm not sure we will be able to trust the conversion otherwise | 12:00 |
| tristan | ironfoot, in other words, yes it may work better as a conversion, but the output quality of that conversion encourages not so great behavior | 12:00 |
| ironfoot | totally agree | 12:00 |
| tristan | I'd say, lets take the current approach, and if it's wrong for a handful of definitions out of hundreds, we'll mention it in the porting guide or such | 12:01 |
| ironfoot | better to fix a few broken things afterwards than stay with all that garbage | 12:01 |
| tristan | yes | 12:01 |
| tristan | There will be other things to note in the porting guide besides this, i.e. the build will fail on running integration commands of modules like gdm | 12:02 |
| tristan | which try to assign arbitrary uids | 12:02 |
| tristan | those should be easily portable to something else | 12:02 |
| tristan | (also we'll want to support encoding of static dev nodes somehow) | 12:03 |
| tristan | Or... say: you're out of luck, run bst as root | 12:03 |
| tristan | but rather not | 12:03 |
| ironfoot | hehe, I've been running bst as root all the time | 12:04 |
| ironfoot | is it not needed? | 12:04 |
| tristan | ironfoot, not at all no :) | 12:04 |
| tristan | hahaha | 12:05 |
| tristan | hahahahahaha | 12:05 |
| ironfoot | :P | 12:05 |
| tristan | Yeah, stop doing that ! | 12:05 |
| tristan | :) | 12:05 |
| ironfoot | well, running in docker means root all the time | 12:05 |
| tristan | yeah Ok | 12:05 |
| tristan | anyway it should not be needed | 12:05 |
| tristan | and if we want to setup CI for definitions building with bst, better to setup a docker with a regular user | 12:06 |
| ironfoot | but i believe that bwrap failed if i wasnt running docker as `--privileged` | 12:06 |
| tristan | to catch the cases of definitions which require root | 12:06 |
| tristan | hmmm, that is possible | 12:06 |
| tristan | ironfoot, it needs to be allowed to create a user namespace pretty much | 12:06 |
| tristan | (disclaimer: I dont know the details of how docker works, really unfamiliar with docker) | 12:07 |
| ironfoot | "When the operator executes docker run --privileged, Docker will enable to access to all devices on the host" | 12:07 |
| ironfoot | it looks like you can have fine grain control over the privileges you want to give to your docker container, that's handy | 12:08 |
| tristan | yes | 12:08 |
| ironfoot | don't worry, I'll be the bst in docker guy | 12:08 |
| tristan | sure, It will be useful for gitlab-ish thingies | 12:09 |
| tristan | but practically speaking, we do our own containerization with bwrap | 12:09 |
| ironfoot | yeah, this is only for users like me that use an OS that doesn't have python 3.5 | 12:11 |
| ironfoot | or that don't really like to install software in their laptops | 12:12 |
| *** xjuan has joined #buildstream | 12:22 | |
| tristan | xjuan, interestingly I ended up ditching the tsort | 13:07 |
| tristan | xjuan, in favor of sorting direct element dependencies once at load time | 13:07 |
| tristan | Now, I'm pondering what would happen if we were to use a fuse layer that implements copy-on-write hardlinks :) | 13:09 |
| * tristan wonders if he could do that in python | 13:09 | |
| tristan | or with a low dependency | 13:09 |
| * ssam2 hopes the answer isn't "terrible performance" | 13:10 | |
| tristan | ssam2, I doubt it, hopefully it would only effect performance in the off chance that a build writes to a linked file in the sandbox | 13:12 |
| ssam2 | i guess if it's only open() calls that go thru the fuse layer it shouldn't be too bad... | 13:17 |
| ssam2 | although there are still huge amounts of calls to open() from a configure script | 13:18 |
| tristan | indeed | 13:21 |
| tristan | we'd have to try it, but it would be an elegant solution to the staging / integration problem | 13:21 |
| *** tristan has quit IRC | 13:55 | |
| *** tristan has joined #buildstream | 14:15 | |
| *** ChanServ sets mode: +o tristan | 14:16 | |
| tristan | ironfoot, this seems to be getting quite far, fwiw: https://gitlab.com/tristanvb/defs2bst/tree/pedro-changes | 15:11 |
| ironfoot | yay | 15:12 |
| ironfoot | if we could sort the dump :P | 15:15 |
| tristan | ironfoot, there is prior art for that... | 15:16 |
| tristan | ironfoot, I think YBD even also has a dumper | 15:16 |
| tristan | since the latest commits | 15:16 |
| ironfoot | one day :P | 15:16 |
| tristan | leeming's spec2def has one | 15:16 |
| tristan | it's easy enough to copy/paste and give it a list of keys you want ordered | 15:17 |
| * leeming grasps context | 15:18 | |
| leeming | spec2def isn't public afaik. I asked for it to be open sourced.. but some commit history _might_ need cleaning | 15:18 |
| leeming | we talking about sorted key dumps for yaml? | 15:20 |
| ironfoot | yup, but that can wait | 15:28 |
| *** ssam2 has quit IRC | 15:47 | |
| *** ssam2 has joined #buildstream | 15:49 | |
| tristan | ironfoot, https://paste.gnome.org/p1be1td51 :) | 16:28 |
| tristan | getting there | 16:28 |
| tristan | I dont understand why wpa_supplicant.bst doesnt build | 16:28 |
| tristan | it may be the sandbox running is not handling the scripts properly. but, I doubt that because glibc.bst works and also does the same | 16:28 |
| ironfoot | looks great | 16:29 |
| tristan | for some reason, cat > file << EOF .... EOF, has the last EOF in the file it creates (for wpa_supplicant) | 16:29 |
| ironfoot | I just hit groff build issue | 16:29 |
| tristan | Oh ! | 16:29 |
| tristan | I have an idea | 16:29 |
| tristan | ironfoot, you wont hit groff issue using master buildstream and the defs2bst branch I linked above | 16:30 |
| ironfoot | the bst file looks as expected, although the "True" is quoted. Is that ok? | 16:31 |
| * tristan pushes fix to buildstream for wpa_supplicant case, all commands run in sandbox should have trailing \n | 16:32 | |
| ironfoot | right, the version of bst didn't have the nonparallel fix | 16:33 |
| tristan | ironfoot, either is OK, but conversion is prettier if we encode True | 16:33 |
| tristan | I think it's my mistake | 16:33 |
| tristan | I think I converted it stringified, but it can be dumped as bool | 16:33 |
| * ironfoot continues validating tristan's success | 16:33 | |
| ironfoot | gah, re-building core :P | 16:35 |
| tristan | good, I have an error up at mesa, which I suspect is a side-effect of not having explicit \n at the end of commands run in the shell | 16:42 |
| tristan | http://pastebin.com/raw/r7hiGUJk | 16:45 |
| tristan | weird | 16:45 |
| tristan | but I think if you have last buildstream commit and are rebuilding core, you may not get that error | 16:45 |
| tristan | ironfoot, and next on the conversion list, is system-integration commands -> public: bst: integration-commands | 16:47 |
| tristan | like this: https://people.gnome.org/~tvb/buildstreamdocs/format.html#public | 16:47 |
| ironfoot | i don't get the "bst" domain | 16:49 |
| ironfoot | things just executed when buildstream uses the element? | 16:50 |
| tristan | "bst" domain is the domain in which BuildStream searches for public data exposed on elements | 16:50 |
| tristan | public data is something that an element is allowed to read off of any other element that it depends on | 16:51 |
| tristan | ironfoot, for integration commands, we currently run them after staging, and before building | 16:51 |
| tristan | (hence the need for something like a fuse layer that does copy-on-write hardlinks) | 16:51 |
| tristan | (or, we can have an error pop up and nuke the tainted artifact extract) | 16:52 |
| tristan | ironfoot, at the base of the stack for instance, ldconfig is an integration command | 16:52 |
| ironfoot | yep | 16:53 |
| tristan | at higher levels, they may update font caches, pixbuf loaders, or even tinker with systemd symlinks | 16:53 |
| tristan | or sysvinit symlinks :) | 16:53 |
| ironfoot | in some cases, things like "ca-certificates" integration commands don't need to run every time | 16:53 |
| ironfoot | what i'm saying is that the system-integration-commands is something a bit different to these ones | 16:54 |
| tristan | ironfoot, I realize that, but unless it turns out to be a serious performance hit, I dont really care | 17:02 |
| ironfoot | yup | 17:02 |
| tristan | we can add something different in a deployment element and do optimizations after an auto-convert, too | 17:02 |
| ironfoot | so my other question, what other options there will be in "public" other than "bst" | 17:02 |
| ironfoot | (just want to understand multiple use-cases) | 17:03 |
| tristan | ironfoot, that all depends on the plugins which use them :) | 17:03 |
| tristan | So plugins in the pipeline can use extra decorated data in their domain | 17:03 |
| tristan | one example will be rpm metadata for packaging | 17:03 |
| tristan | another might be flatpak metadata | 17:03 |
| * ironfoot nods | 17:03 | |
| tristan | ssam2, fwiw, I realized that performance of fusepy is thankfully not an issue at all :) | 17:04 |
| ssam2 | nice | 17:04 |
| tristan | ssam2, because you only need that for the rootfs that is outside the build directory | 17:04 |
| tristan | configure scripts operate on data you've checked out | 17:05 |
| tristan | (so it will need a separate mount, something to figure out in the sandbox code) | 17:05 |
| ssam2 | i mean, configure scripts shouldn't touch anything on the source tree, I quite like that Morph had a read only mount that enforced that | 17:05 |
| ssam2 | hmm, a total tie! | 17:05 |
| ssam2 | total lie! You mean checked out from Git | 17:06 |
| tristan | you mean outside the source tree | 17:06 |
| tristan | configure scripts only touch stuff inside the source tree | 17:06 |
| ssam2 | yeah | 17:06 |
| tristan | ssam2, I think we will do the read-only mount _also_ | 17:06 |
| ssam2 | good | 17:06 |
| tristan | copy-on-write hardlinks is for the stage of running integration commands before running the build | 17:06 |
| ssam2 | that makes sense | 17:06 |
| ssam2 | I think we tried unionfs-fuse for that a couple of years ago | 17:07 |
| ssam2 | which worked pretty well, patches for Morph are still around | 17:07 |
| tristan | nice | 17:08 |
| ssam2 | oh, and overlayfs | 17:08 |
| ssam2 | which is now pretty common | 17:08 |
| ssam2 | you can store an OSTree commit + the overlayfs backing store | 17:08 |
| tristan | whatever it is, my concern is that it's easily obtainable | 17:08 |
| ssam2 | sure. Well all these options are Linux-only unfortunately | 17:08 |
| tristan | so I thought a python layer like fusepy, uses ffi to access the C interface of fuse directly | 17:08 |
| tristan | yes | 17:09 |
| tristan | ssam2, right now the one sandbox implementation we have is linux only | 17:09 |
| tristan | but I'd like to keep those restrictions in the sandbox implementations | 17:09 |
| tristan | for building on windows, it may not be an issue with WSL | 17:09 |
| ssam2 | overlayfs is built into the kernel since 3.18 so that may be easiest to obtain | 17:10 |
| tristan | but I think it would be very sweet, if the sandbox for windows itself automatically used the WSL layer | 17:10 |
| ssam2 | right, why not | 17:11 |
| tristan | might not be easy | 17:11 |
| tristan | I would like that, but I have a feeling WSL is restricted to using a vetted ubuntu thingy | 17:11 |
| tristan | not entirely sure if we can shove whatever runtime we want in there | 17:11 |
| ssam2 | well, the issue is whether they implement all the necessary syscalls right? | 17:11 |
| tristan | that too | 17:12 |
| ssam2 | unless they're signing the ubuntu image that's allowed to run somehow, but it seems a bit of a GPL violation if so | 17:12 |
| tristan | but as they are running ubuntu stuff, I am hopeful that it stresses their syscall interface quite a bit | 17:12 |
| tristan | worst case we say: Install this WSL thing and run buildstream inside it | 17:12 |
| tristan | ssam2, also has to do with filesys, we probably need a dedicated quota | 17:13 |
| * tristan hasnt looked deeply into that WSL stuff yet, obviously | 17:13 | |
| paulsher1ood | build first, world domination can wait? :) | 17:14 |
| tristan | yes | 17:14 |
| tristan | paulsher1ood, https://paste.gnome.org/p1be1td51 ... and chugging along :) | 17:15 |
| ironfoot | world domination plans being developed between builds :P | 17:15 |
| * paulsher1ood wants to see elapsed times | 17:15 | |
| tristan | didnt save that build log | 17:16 |
| paulsher1ood | nm | 17:16 |
| tristan | and have to clean up UI, it's cluttered especially with messages about staging each dependency | 17:16 |
| tristan | thats extra-extra verbose mode I guess | 17:16 |
| tristan | super-duper-verbose, I dont know, probably just yank it, keep it only in the saved logs | 17:16 |
| tristan | thats just the output of `bst show gnome.bst`, it shows only what is cached, what is ready to build, and what is waiting to build | 17:18 |
| tristan | so it gives a picture at least of what we managed to build so far :) | 17:18 |
| ironfoot | around 2/3 of gnome.bst | 17:23 |
| tristan | ya know we have an opening for anyone who wants to write Source implementations, so *cough* dont let me hold everyone back ! | 17:26 |
| tristan | hehehe | 17:26 |
| * tristan opens the proverbial floodgates | 17:26 | |
| ironfoot | to write Source implementations? | 17:27 |
| tristan | nod | 17:27 |
| ironfoot | what's that | 17:27 |
| tristan | https://people.gnome.org/~tvb/buildstreamdocs/buildstream.source.html#module-buildstream.source | 17:27 |
| tristan | ironfoot, currently we only have git, local and ostree | 17:27 |
| tristan | barely enough to only do baserock definitions | 17:27 |
| ironfoot | aahh i get it | 17:27 |
| tristan | we're going to need implementations for tarballs and other vcses | 17:28 |
| ironfoot | nod | 17:28 |
| tristan | before we can convert flatpak json | 17:28 |
| ironfoot | what is supported by the tool that currently builds these flatpaks? | 17:29 |
| tristan | https://paste.gnome.org/peyhl5e1h/egzwlk/raw | 17:30 |
| tristan | looks like only git,bzr and tarballs | 17:30 |
| tristan | and "file" | 17:30 |
| tristan | which is like our "local" | 17:31 |
| tristan | others "script" and "patch" can be converted with "local" as well | 17:31 |
| ironfoot | hm.. source-patch | 17:31 |
| ironfoot | is that what i think it is? | 17:32 |
| tristan | patches just need to be a "local" source and have a pre-configure-command | 17:32 |
| tristan | ironfoot, look at a flatpak json for reference, you'll see what I mean | 17:32 |
| tristan | some of those flatpak-builder sources have side effects | 17:33 |
| ironfoot | clever | 17:33 |
| tristan | like running a script or applying a patch | 17:33 |
| ironfoot | sources will have to run in order then | 17:33 |
| tristan | not necessarily | 17:33 |
| tristan | patches and scripts must be applied in the order specified in flatpak json | 17:34 |
| ironfoot | apologies, I mixed concepts in my head | 17:34 |
| tristan | yeah, can get confusing :) | 17:34 |
| tristan | anyway, should be easy enough to convert | 17:34 |
| tristan | biggest hurdle is investigating everything flatpak-builder does, but I've made provisions for it to be close enough | 17:35 |
| tristan | (the integration commands running before builds being one of those) | 17:35 |
| *** ssam2 has quit IRC | 18:41 | |
| *** tristan has quit IRC | 20:24 | |
| *** xjuan has quit IRC | 21:23 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!