| *** tristan has quit IRC | 04:53 | |
| *** ssam2 has joined #buildstream | 08:26 | |
| *** jonathanmaw has joined #buildstream | 09:14 | |
| *** tristan has joined #buildstream | 09:16 | |
| *** ChanServ sets mode: +o tristan | 09:16 | |
| tristan | jonathanmaw, run out of disk space yet ? | 09:17 |
|---|---|---|
| jonathanmaw | tristan: nope, still have 14G left, apparently :) | 09:19 |
| jonathanmaw | looks like my overnight build failed on gnu-toolchain/stage1-binutils, though. I'll shove it on pastebin. | 09:20 |
| jonathanmaw | https://pastebin.com/8cbvhJz1 | 09:21 |
| tristan | :-/ | 09:21 |
| tristan | Oh | 09:22 |
| tristan | Ok I see | 09:22 |
| tristan | jonathanmaw, that's not because of ldconfig (although there *was* such a bug a whiiile back) | 09:22 |
| tristan | thats because of "No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'." | 09:22 |
| tristan | jonathanmaw, normally that is solved because the `bwrap` program is supposed to be installed as setuid root | 09:23 |
| tristan | although perhaps an older package of bwrap is not setuid ? | 09:23 |
| tristan | jonathanmaw, first ensure that debian jessie installed bwrap as setuid root, and if it did not well... that's a bit sad for jessie as it will come up again :-/ | 09:24 |
| tristan | You can also do what the message says and allow unprivileged_userns_clone on your system, or, you can set it setuid on your system | 09:25 |
| jonathanmaw | tristan: looks like it was me building bubblewrap from source that had the problem | 09:26 |
| jonathanmaw | Installed bubblewrap from package manager (might have been from jessie-backports) seems to be setuid | 09:26 |
| tristan | Ah ! | 09:28 |
| tristan | Ok ! | 09:28 |
| tristan | jonathanmaw, just to confirm, that solved it for you ? | 09:35 |
| jonathanmaw | stage1-binutils seems to have succeeded now | 09:37 |
| jonathanmaw | so I think so | 09:37 |
| tristan | Ah, ok | 09:37 |
| jonathanmaw | on the other hand, there's an ugly stack trace in the middle of my stdout https://pastebin.com/0dKuV3BJ | 09:37 |
| tristan | Well, unfortunately, if you're still down at binutils, I dont think 14GB is going to be enough | 09:37 |
| tristan | oh that's horrible | 09:38 |
| tristan | I may have regressed that somehow, have been playing with exceptions lately and things tend to already be fetched once I've fetched them | 09:39 |
| jonathanmaw | there, cleared up to 32G | 09:39 |
| * tristan tries to reproduce | 09:39 | |
| *** tiagogomes has quit IRC | 09:40 | |
| *** tiagogomes has joined #buildstream | 09:40 | |
| * tristan provokes a fetch task | 09:42 | |
| tristan | oh, that just worked | 09:44 |
| tristan | jonathanmaw, so I think cntl-C and process termination is rough around the edges | 09:44 |
| tristan | may need some cleanup, how did you provoke that ? it didnt happen all on it's own did it ? | 09:45 |
| tristan | was a cntl-C ? | 09:45 |
| jonathanmaw | tristan: afaict it happened without user input. the stdout up until that point is https://pastebin.com/rJURYNNy | 09:46 |
| tristan | :-/ | 09:48 |
| tristan | although it seems to have received SIGTERM somehow | 09:49 |
| tristan | File "/home/jonathanmaw/workspace/buildstream/buildstream/buildstream/utils.py", line 570, in _terminator_handler | 09:49 |
| tristan | jonathanmaw, so basically, that "happened" but the build just continued, just that you have this in the middle of stdout ? | 09:50 |
| * tristan confused how this could have happened | 09:50 | |
| tristan | jonathanmaw, anyway let me know what happens, and if this is reproducible | 10:13 |
| jonathanmaw | ok, no new stack traces yet. Currently fetching linux-api-headers and python, by the looks of it. | 10:14 |
| tristan | ahhh, yeah linux fetching will take a long time :-/ | 10:16 |
| tristan | that'll block the build | 10:16 |
| * tristan hasnt built from a 'no sources exist yet' state in a very long time | 10:16 | |
| ssam2 | mirroring many git repos efficiently is still not a very well solved problem | 10:17 |
| ssam2 | Trove helped by serving tarballs of the whole repo instead of going through the git clone protocol, does buildstream make use of those? | 10:17 |
| ssam2 | really Git should be more efficient here | 10:17 |
| tristan | ssam2, well you can't really cheat bandwidth, you have to get them somehow | 10:17 |
| tristan | indeed | 10:18 |
| tristan | there is that | 10:18 |
| tristan | no we dont use the tar-server | 10:18 |
| tristan | ssam2, I'm hesitant to do that too, to be honest though, parallel fetching should mostly solve that issue | 10:18 |
| ssam2 | Depends.. some repos are pathalogically slow | 10:18 |
| ssam2 | like GCC where there's a big Changelog | 10:19 |
| ssam2 | downloading and extracting a tar shortcuts all that nonsense | 10:19 |
| tristan | ssam2, what I've noticed when we do serial fetches, is that there is a ramp-up time when communicating with git first (maybe that's just network), and then there is some computational overhead which may be server side | 10:19 |
| tristan | so, if you fetch many things in parallel, you mostly get to use up your bandwidth | 10:20 |
| tristan | what I dont like about adding the tar-server support is that it's another custom moving part | 10:20 |
| ssam2 | I agree | 10:23 |
| ironfoot | i believe that for the gcc case, downloading the tarball (for the real git mirror, and not gcc-tarball) was worse | 10:38 |
| ironfoot | it's a *huge* tarball | 10:39 |
| tristan | maybe it includes mpfr/mpc and that stuff ? | 10:39 |
| tristan | or maybe indeed it's a ChangeLog from hell | 10:39 |
| ironfoot | long long history, for gcc I think a shallow clone would be way faster | 10:41 |
| tristan | I dont think a shallow clone really exists, I spent over a day looking into git features for something like that | 10:42 |
| tristan | there is something in relatively new versions of git | 10:42 |
| tristan | however, it requires that the git repo you are pulling from is configured with special sauce | 10:42 |
| ironfoot | `git clone --depth 1` | 10:43 |
| tristan | Ah, theres a reason why I couldnt just use that | 10:45 |
| tristan | something to do with primarily relying on commit shas, and only having branch tracking as an optional feature | 10:45 |
| ironfoot | just wanted to point out that it exists :) | 10:46 |
| * ironfoot clones gcc with --depth 1 | 10:46 | |
| tristan | yeah, I did see that, and there is something that lets you also get "only this commit sha" too (the latter being something relatively new and tweaky) | 10:46 |
| ironfoot | cloned | 10:49 |
| ironfoot | that was fast | 10:49 |
| tristan | ironfoot, ok hotshot :) | 10:54 |
| tristan | now go play in https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/plugins/sources/git.py | 10:54 |
| tristan | See if you can apply that :) | 10:54 |
| ironfoot | of course not saying this is what you (or we) have to do, only that it looked interesting to me in the past | 11:13 |
| ironfoot | i wish I could spend some more time looking at this, I promise | 11:14 |
| tristan | ironfoot, I'm of course just teasing | 11:16 |
| tristan | ironfoot, but if it _is_ possible and I've overlooked something, feel free to find it :D | 11:16 |
| tristan | the constraints of the plugin are hard though, to make that shallow clone fit | 11:16 |
| ironfoot | just some more data about the quick investigation I've done: gcc.git tarball (produced by magic trick in trove) is 1.8G, shallow clone is 770M, being 670M the actual size of the gcc.git tree (100M of .git) | 11:21 |
| ironfoot | what i just wanted to point out is that *maybe* the tarball trick is not always faster/a good idea | 11:22 |
| paulsher1ood | good point | 11:46 |
| jonathanmaw | Another build failure: https://pastebin.com/NxtUB1YU | 12:06 |
| jonathanmaw | gnulib/gnulib-tool: no such file or directory | 12:07 |
| jonathanmaw | also, when I tried to drop into an interactive shell, it failed with "sh: can't access tty; job control turned off" | 12:07 |
| tristan | hmmm, there was an incident months ago this reminds me of | 12:07 |
| tristan | jonathanmaw, yeah that's normal | 12:08 |
| jonathanmaw | probably because you don't have a full shell in the sysroot when building coreutils, at my guess. | 12:08 |
| tristan | we didnt give it a /dev/tty | 12:08 |
| tristan | we could | 12:08 |
| tristan | jonathanmaw, check if the subdir exists and has content | 12:08 |
| jonathanmaw | where would I find that in buildstream? | 12:09 |
| tristan | in the shell you should be in /buildstream/build | 12:09 |
| tristan | that is the build directory | 12:09 |
| jonathanmaw | ah, so I am! | 12:09 |
| jonathanmaw | the "gnulib" dir is empty | 12:10 |
| jonathanmaw | I think it's a submodule | 12:10 |
| tristan | ironfoot, it finally happened again | 12:10 |
| tristan | this happened once to ironfoot | 12:10 |
| tristan | it would be really awesome to find out why... jonathanmaw can you check the full log in ~/.cache/buildstream/logs ? | 12:11 |
| tristan | and see if it says anything about checking out gnulib ? | 12:11 |
| tristan | jonathanmaw, this is something that... happened once and didnt happen again, you got lucky to trigger it | 12:11 |
| tristan | the build log should be in the log directory + build-essential/coreutils-common-coreutils/9028a2a8-build.16994.log | 12:14 |
| jonathanmaw | tristan: https://pastebin.com/FLQWh9DA | 12:16 |
| jonathanmaw | that's the references to gnulib in the fetch and build logs | 12:16 |
| ironfoot | oh i remember, something about staging the source but not the submodules | 12:16 |
| tristan | jonathanmaw, ok so I'm only concerned with the build log of coreutils | 12:19 |
| tristan | actually, can you paste that full build log ? | 12:19 |
| tristan | jonathanmaw, part of that build log will be the staging (clone + checkout) of coreutils into the build directory | 12:19 |
| tristan | it *should* be accompanied by the staging of gnulib | 12:20 |
| tristan | but I suspect it will not be | 12:20 |
| jonathanmaw | tristan: https://pastebin.com/uH6t6QPH | 12:20 |
| jonathanmaw | how do I check which sources have already been fetched? | 12:20 |
| jonathanmaw | I suspect it might be related to the fact that buildstream seems to defer fetching repos | 12:21 |
| tristan | hmmm | 12:21 |
| tristan | jonathanmaw, the submodule will have been fetched at the same time as coreutils | 12:21 |
| tristan | jonathanmaw, you can look for gnulib in ~/.cache/buildstream/sources/git | 12:22 |
| jonathanmaw | if I understand the logging format correctly, if gnulib was fetched, ther's be a <hex>-fetch.<number>.log | 12:22 |
| tristan | it will be there | 12:22 |
| jonathanmaw | ./git___git_baserock_org_delta_gnulib | 12:22 |
| tristan | what's going on I think is at stage time, the GitSource plugin *sometimes* forgets about there being a submodule | 12:22 |
| jonathanmaw | yep, it exists | 12:22 |
| tristan | yeah note that sources which belong to the same element are not fetched in parallel | 12:23 |
| tristan | scheduler is only in element granularity | 12:23 |
| tristan | and actually, the submodule is a detail of one source | 12:23 |
| tristan | so it's all done synchronously, that's why it'll end up in the fetch log of coreutils | 12:23 |
| tristan | if it was not already fetched by some other module | 12:24 |
| tristan | which also has a gnulib submodule (highly likely) | 12:24 |
| tristan | but the problem here is at stage time | 12:24 |
| tristan | Ah | 12:25 |
| tristan | ironfoot, I think it will happen... only the first time around | 12:26 |
| tristan | Ok, I will try to reproduce and fix | 12:26 |
| tristan | my guess is that in git.py, we need self.refresh_submodules() at the beginning of self.stage() | 12:26 |
| tristan | jonathanmaw, for right now, feel free to just terminate and restart, it will pick up where it left off, and you wont get that problem again for coreutils | 12:27 |
| jonathanmaw | ok. I'll restart my build and see what happens. | 12:27 |
| tristan | yeah | 12:27 |
| tristan | I'm pretty damn sure, because now that the mirror is there, when it loads up, it will be aware of the submodule already | 12:27 |
| tristan | what happened is that fetch() happens in a subprocess, and discovers/fetches the submodule in that process | 12:28 |
| ironfoot | aha, makes sense | 12:28 |
| tristan | when we call stage in the build subprocess, and it wasnt there in the master process at load time, it never did a cat-file on .gitmodules yet | 12:28 |
| ironfoot | using git is possible to see the contents of .gitmodules of remote files, i believe | 12:29 |
| tristan | sure, but still | 12:29 |
| ironfoot | but pointless if you want to make the build tool to not depend on internet connections | 12:29 |
| jonathanmaw | ironfoot: I think you need the server to do something special to do it remotely | 12:29 |
| tristan | ironfoot, we dont want the pipeline load to imply network activity first | 12:29 |
| tristan | only in fetch() | 12:29 |
| tristan | and we dont want network activity at build time either | 12:29 |
| tristan | exactly | 12:30 |
| * tristan pushes some patches which "freeze time" when the build is suspended | 12:31 | |
| tristan | that's a bit nicer | 12:31 |
| tristan | durations dont include stopped time | 12:31 |
| tristan | now, I think bwrap is acting up at SIGCONT time though, seems it does not consistently continue and needs a kick in the nuts every so often | 12:32 |
| *** tristan has quit IRC | 12:45 | |
| *** tristan has joined #buildstream | 12:49 | |
| *** ChanServ sets mode: +o tristan | 12:49 | |
| tristan | Nice, I reproduced it, lets try the fix | 12:53 |
| tristan | Fixed the submodule issue in master | 12:57 |
| tristan | 81G/home/tristan/.cache/buildstream/ | 12:58 |
| tristan | :-/ | 12:58 |
| tristan | that's a bit big I think | 12:59 |
| * tristan makes backup of his cache and runs a full build of gnome system with fetches and everything overnight | 13:02 | |
| tristan | ah, I think ostree plugin can use some love, seems I have some tmp directories that remain and are a big huge | 13:04 |
| jonathanmaw | hrm, failure building ostree, unable to access https://github.com/mendsley/bsdiff | 15:45 |
| jonathanmaw | probably because I didn't pull the new version of buildstream that fixes the submodule problem :/ | 15:46 |
| *** jonathanmaw has quit IRC | 17:08 | |
| *** ssam2 has quit IRC | 17:26 | |
| *** tristan has quit IRC | 22:11 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!