| *** mcatanzaro has quit IRC | 00:06 | |
| *** bochecha has quit IRC | 00:19 | |
| *** bochecha has joined #buildstream | 00:21 | |
| *** bochecha has quit IRC | 02:17 | |
| *** tristan has quit IRC | 07:02 | |
| *** tristan has joined #buildstream | 07:05 | |
| *** bochecha has joined #buildstream | 08:49 | |
| gitlab-br-bot | buildstream: merge request (cache_local_element_unique_key->master: Cache the local element's unique key) #193 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/193 | 09:03 |
|---|---|---|
| *** jude has joined #buildstream | 09:07 | |
| *** valentind has joined #buildstream | 09:19 | |
| gitlab-br-bot | buildstream: merge request (cache_local_element_unique_key->master: Cache the local element's unique key) #193 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/193 | 09:21 |
| *** noisecell has joined #buildstream | 09:39 | |
| *** ssam2 has joined #buildstream | 10:06 | |
| *** bethw has joined #buildstream | 10:13 | |
| *** sstriker has quit IRC | 10:15 | |
| *** jonathanmaw has joined #buildstream | 10:22 | |
| juergbi | gitlab CI is failing very early on (distcheck job): ERROR: Preparation failed: exit status 1 | 10:52 |
| juergbi | already retried, any ideas? | 10:52 |
| ssam2 | not sure, but if it's running on the baserock runners we might be able to ssh in | 10:53 |
| ssam2 | could you link to the log ? | 10:53 |
| ssam2 | the runners are deleted after the build runs, but adding a long sleep statement just after the failure would be enough for us to try and reproduce it on the runner itself | 10:54 |
| *** valentind has quit IRC | 10:57 | |
| * tlater noticed that the bst-here script doesn't have completion | 11:04 | |
| tlater | Should the docker image add completion? | 11:04 |
| juergbi | ssam2: https://gitlab.com/BuildStream/buildstream/-/jobs/44538627 | 11:07 |
| ssam2 | tlater, it would need to install it into the host | 11:08 |
| ssam2 | i think there's a tool that could help with that | 11:08 |
| ssam2 | juergbi, ah, definitely an issue with GitLab CI or the runners | 11:08 |
| ssam2 | tlater, https://stowage.org/ might be useful | 11:09 |
| tlater | Huh, their ssl cert is broken... | 11:09 |
| * tlater dismisses vivaldi's message | 11:09 | |
| tlater | ssam2: I suppose that could work, but it would be different from the bst-here script | 11:12 |
| ssam2 | yes | 11:13 |
| ssam2 | maybe that's a good thing, i don't know | 11:13 |
| tlater | Would we want to support both? It seems really trivial to support stowage... | 11:14 |
| ssam2 | i wouldn't want to support both long-term, but would be nice to try stowage and see if we can reduce our maintenance burden ... | 11:14 |
| tlater | Right, I see what you mean. | 11:14 |
| * tlater checks exactly how you make a docker image compatible with this. | 11:14 | |
| tlater | Hm, it seems like a bit more of a pain than I thought, documentation isn't great :| | 11:29 |
| tlater | -> todo list | 11:29 |
| *** mcatanzaro has joined #buildstream | 12:00 | |
| juergbi | ta ssam2, CI completed successfully | 13:03 |
| tlater | Integration tests are still slow with a different base platform and pytest :\ | 13:57 |
| ssam2 | interesting | 14:00 |
| ssam2 | i didn't expect pytest to gain anything, but i thought that reducing the size of the base dependency would definitely help | 14:00 |
| ssam2 | although "slow" isn't a useful metric, the question is whether they are the same speed as before ? | 14:00 |
| tlater | Haven't tested that properly yet, quite potentially faster, but I'm using local files which should get around a lot of the slowdown | 14:03 |
| tlater | Unexpectedly "slow" for that | 14:03 |
| tlater | We'll see when the full suite is converted and the docker image is up, I suppose | 14:04 |
| gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: Multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 14:07 |
| tristan | tlater, I believe we recommend launching `bst-here` without any arguments, so that it launches a shell in the docker image which *does* have completions that work | 14:31 |
| tristan | tlater, running `bst-here build [TAB]` will never get you completions, though | 14:31 |
| tlater | tristan: The launched shell does not have completions, that's my complaint ;) | 14:31 |
| tristan | that is a bug, then | 14:32 |
| ssam2 | tlater, ah, I misunderstood | 14:32 |
| ssam2 | pip must be failing to install the completions file somehow | 14:32 |
| tlater | It's not installed by pip afaik | 14:32 |
| tristan | it should be | 14:32 |
| ssam2 | RUN ["/usr/bin/pip3", "install", "/root/buildstream"] | 14:33 |
| tristan | by pip, but not in all circumstances | 14:33 |
| ssam2 | from the Dockerfile | 14:33 |
| tristan | so a few things can be happening | 14:33 |
| * tlater thinks the docker image doesn't have bash-completions installed | 14:34 | |
| tristan | it might be that it was never tested to work, and that fedora doesnt like completions being installed in the "regular place" | 14:34 |
| tristan | maybe we need a --tweak-me-for-fedora option in interim, the right way would be to depend on `bash-completions` and read back the completions dir from the bash-completions package | 14:35 |
| tristan | which provides a pkg-config file, without being bash | 14:35 |
| tristan | also, it can be much more hackable and easy without that, you only need to source the completion script when entering a shell | 14:36 |
| tristan | that's what bash-completions package helps with anyway | 14:36 |
| tristan | so you could just source it when entering the bst-here shell to work around that | 14:36 |
| ssam2 | `pkg-config --variable=completionsdir bash-completion` on fedora gives me /usr/share/bash-completion/completions | 14:38 |
| ssam2 | which is the location we have hardcoded, as i understand it | 14:39 |
| tristan | does it do that in the fedore that is in the docker image ? | 14:39 |
| ssam2 | let me crank up docker and check | 14:39 |
| tlater | It's installed to "$PREFIX/share/bash-completion/completions", which presumably is /usr/local? | 14:40 |
| tristan | ssam2, that is the problem | 14:40 |
| tristan | tlater nailed it | 14:40 |
| tristan | ssam2, for it to work properly in fedora, it needs to be installed as it would be for a distro package | 14:41 |
| tristan | without pip | 14:41 |
| ssam2 | so `pip3 --prefix=/usr` ? | 14:41 |
| ssam2 | how would we install it without using Pip ? | 14:41 |
| tristan | python3 setup.py install will do the correct thing for a distro | 14:41 |
| ssam2 | ok | 14:41 |
| tristan | or should | 14:41 |
| tlater | No, it looks like it is /usr on fedora | 14:41 |
| tristan | what what what ? | 14:42 |
| tlater | It's installed to the correct directory | 14:42 |
| tlater | Looking at the buildstream docker image rbn | 14:42 |
| tlater | *rn | 14:42 |
| tristan | tlater, buildstream is, with `pip install` ?? | 14:42 |
| tlater | Apparently, yes | 14:42 |
| tlater | And I can see the completions file in the correct directory | 14:42 |
| tristan | tlater, pip installs system-wide stuff in `/usr/local` | 14:42 |
| tlater | `ls /usr/share/bash-completion/completions/ | grep bst` gives me 'bst' | 14:43 |
| tristan | only the package manager is supposed to install stuff in `/usr`, which is why we have this gross /usr/lib/python vs /usr/local/lib/python thing in the first place | 14:43 |
| tlater | Same for /usr/bin, incidentally | 14:43 |
| tlater | It's down to distro configuration, though, possible they don't specify a prefix for pip | 14:43 |
| tristan | weird, I dont get how all this pip stuff is supposed to not work anyway | 14:44 |
| * tlater checks the pkg-config line | 14:44 | |
| tristan | I expect that maybe bash-completions isnt installed | 14:44 |
| tlater | Huh, no pkg-config | 14:44 |
| ssam2 | ah, that could be the case | 14:44 |
| ssam2 | didn't realise it was a separate package | 14:45 |
| tristan | and so the mechanics of sourcing the completions at shell startup time are just missing | 14:45 |
| tlater | That's possible, but there look to be a lot of scripts in the completions directory | 14:45 |
| ssam2 | `dnf install bash-completion` causes a package to be installed | 14:45 |
| tlater | Ah, alright, that's it then | 14:45 |
| tristan | having completions installed doesnt mean that bash-completions is installed, right | 14:46 |
| tlater | Yeah, the variable isn't set in pkg-config either | 14:46 |
| tlater | tristan: It *should* append the correct line to /etc/profile | 14:46 |
| tlater | On most distros installing it is enough | 14:46 |
| tristan | I dont know what appends what where | 14:46 |
| ssam2 | ah yes, now I get a nice hang when typing `bst sh<TAB>` | 14:46 |
| ssam2 | followed eventually by some completions :-) | 14:46 |
| tlater | tristan: The package manager appends the line to source bash-completions to the generic system profile :) | 14:47 |
| tristan | I know that there is this standard called "bash-completions", and the distros which decide to maybe use that standard, allow depending packages to read the completions dir from it | 14:47 |
| tristan | besides that, a million things can happen when you launch a shell, parsing /etc/profile might, or might not be one of those things | 14:47 |
| ssam2 | tlater, wanna do a patch for the docker-images repo to add the bash-completion package ? | 14:48 |
| *** semanticdesign has joined #buildstream | 14:48 | |
| tristan | tlater, ok I'll take your word for it :) | 14:48 |
| tlater | Yup, should just be a second | 14:48 |
| tristan | ssam2, are you sure the hang is not just needing a second <TAB> ? is it that horribly slow in docker ? | 14:49 |
| ssam2 | not sure | 14:49 |
| tlater | It's pretty slow outside of docker, too | 14:49 |
| * tristan noticed annoyingly in some cases he has to hit <TAB> an extra time, probably needs to be improved | 14:50 | |
| * tlater wonders if we incur the setuptools loading slowness for completion as well | 14:50 | |
| ssam2 | we will do, yes | 14:50 |
| tristan | that would be horrible yeah | 14:50 |
| tristan | we do some hacks to make that codepath quick | 14:50 |
| tristan | we should do more | 14:50 |
| tristan | like, we sys._exit() or such, so save time at shutdown | 14:51 |
| tristan | because completions should be snappy | 14:51 |
| tlater | They really are not atm, definitely could use some improvements - perhaps worth an issue? | 14:51 |
| tristan | sure | 14:52 |
| gitlab-br-bot | buildstream: issue #171 ("bash completions are slow") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/171 | 14:55 |
| tlater | ssam2: MR is up: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/13/diffs | 15:05 |
| * tlater is glad he isn't tempted to push these one liners to master | 15:05 | |
| ssam2 | thanks | 15:06 |
| * ssam2 secretly prefers the package list to be alphabetically sorted | 15:06 | |
| * tlater completely overlooked that as a possibility, and will now never be able to do without again | 15:07 | |
| ssam2 | :-) | 15:11 |
| *** nexus_ has joined #buildstream | 15:14 | |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-plugin-loading->master: Add plugin loading fixes) #195 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/195 | 15:19 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-plugin-loading->master: Add plugin loading fixes) #195 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/195 | 15:21 |
| *** noah has joined #buildstream | 15:34 | |
| *** noah has quit IRC | 15:39 | |
| *** noah has joined #buildstream | 15:40 | |
| tristan | integration_unix down to ~45min, integration_linux at ~15min | 16:11 |
| tristan | I think that's an improvement ? | 16:11 |
| tristan | should be much better after getting rid of the staging of org.freedesktop.Runtime weirdness | 16:13 |
| tlater | That's a lot quicker | 16:13 |
| tlater | I assume that's down to your hacking last night? | 16:13 |
| tristan | tlater, thats only a commit which changes all the checkouts to use `bst checkout --hardlinks` | 16:15 |
| *** bethw has quit IRC | 16:15 | |
| tristan | the bottleneck on the runners is I/O | 16:15 |
| tristan | so | 16:15 |
| tlater | What improved it then? | 16:15 |
| tristan | `bst checkout --hardlinks` vs `bst checkout` | 16:16 |
| tristan | I mean it *does* help | 16:16 |
| tristan | it's just very easy workaround | 16:16 |
| tlater | Does it actually halve the test time? | 16:16 |
| tristan | we dont modify the outputs in place anyway in those tests so should be safe | 16:16 |
| tristan | I dont know, how long was it before that commit ? | 16:16 |
| tristan | maybe | 16:16 |
| tlater | Last I remember was it taking 2 hours | 16:17 |
| tristan | this can easily be checked on the pipelines page | 16:17 |
| * tlater takes a peek | 16:17 | |
| tristan | https://gitlab.com/BuildStream/buildstream/commit/819645b230506cb0b2736d08e9c155f24c724cda | 16:18 |
| tristan | that's the commit which introduces it | 16:18 |
| tristan | it could be tested by creating pipelines with and without the commit ;-) | 16:18 |
| tlater | Hmm | 16:18 |
| *** bochecha has quit IRC | 16:19 | |
| tlater | Oddly enough, most pipelines were at ~1 hour before then | 16:19 |
| tlater | It saved 15 mins, at least | 16:19 |
| tlater | But a week or so ago we had a commit that removed a bunch of imports | 16:19 |
| tlater | Since then the tests have been taking 1/3 of the time | 16:19 |
| tlater | Hm, no, maybe that was just an outlier | 16:20 |
| tlater | Well, 15 mins is something :) | 16:20 |
| tristan | reducing runtime size, and avoidance of staging 2 runtimes, will help a lot | 16:21 |
| tlater | I've started moving the tests to an alpine-based base image, presumably there's no reason to run the tests with the gnome sdk? | 16:21 |
| * tlater only discussed this with ssam2 | 16:22 | |
| tristan | jonathanmaw, so looks like you have a pep8 problem on your MR... | 16:22 |
| tristan | jonathanmaw, I'd like to merge that otherwise... | 16:22 |
| tristan | tlater, I dont really know what alpine is, but... | 16:23 |
| tristan | tlater, if we're going to use a runtime, I think the one we should eventually be using is to build it from freedesktop-sdk project | 16:24 |
| * tlater hasn't really read up on that project yet | 16:26 | |
| tristan | tlater, as it seems to make sense to dogfood as much as we can in this case, I would like to see buildstream tested using images which buildstream produces | 16:26 |
| tristan | at least, as far as the runtime goes | 16:26 |
| tristan | for the docker in which we run the tests, it makes more sense of course to use specific versions of distros | 16:26 |
| * tlater is concerned with test time here | 16:28 | |
| tlater | I'm guessing that the freedesktop-sdk project will be very small though | 16:28 |
| tristan | tlater, in any case; I think that it's probably better to do something else, or even help whip the freedesktop-sdk project into shape, rather to make a change that will probably change again soon-ish | 16:28 |
| tlater | Yeah, I agree on that | 16:28 |
| tristan | the size of the output will be somewhat selective | 16:29 |
| tristan | we will take from it the smaller runtime I suppose, but we will depend on it as a project | 16:29 |
| tristan | I think | 16:29 |
| tristan | so post-landing of junctions | 16:29 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-plugin-loading->master: Add plugin loading fixes) #195 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/195 | 16:30 |
| tlater | Would it make sense to do the integration-test revamp after we can start depending on the freedesktop-sdk then? | 16:30 |
| tristan | I dont think it's worth holding off on that no | 16:30 |
| tristan | So looking at the details; the runtime produced by freedesktop-sdk will be flatpack-ish too | 16:31 |
| tristan | so it will want to be mounted into /usr of a "usermerge" rootfs | 16:32 |
| tristan | similar to what we *should* be doing with the flatpak Sdk | 16:32 |
| tristan | but are failing to do, we should change that right away | 16:32 |
| tristan | and reduce a lot of I/O thanks to that change | 16:33 |
| tristan | Then, when freedesktop-sdk is ready, post junctions, we can replace the base dependency once, and address any fallout from that change at the same time | 16:33 |
| tristan | once | 16:33 |
| tristan | That is an activity fairly orthogonal to the rest | 16:34 |
| tlater | Yeah, that seems reasonable | 16:34 |
| tristan | So in between, if we can make the other changes, moving to pytest, testing only the relevant details instead of "absolute sameness" | 16:34 |
| tristan | we can do those too | 16:34 |
| *** noisecell has quit IRC | 16:42 | |
| *** noah has quit IRC | 16:55 | |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 16:56 |
| gitlab-br-bot | buildstream: issue #132 ("Loading external plugins works without explicit requirement in project.conf") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/132 | 17:19 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-plugin-loading->master: Add plugin loading fixes) #195 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/195 | 17:19 |
| *** valentind has joined #buildstream | 17:37 | |
| *** xjuan has joined #buildstream | 17:38 | |
| *** bochecha has joined #buildstream | 17:39 | |
| bochecha | ssam2: I can't believe after all these years Python still doesn't have something for atomic writes in its stdlib, it's such a common need :x | 17:41 |
| bochecha | ssam2: would you send yours upstream? | 17:41 |
| ssam2 | i'm having enough trouble getting it merged into BuildStream, never mind cpython | 17:44 |
| ssam2 | i agree that it would be really useful, though | 17:44 |
| juergbi | i fear ssam2 will think twice before writing another utility function ;) | 17:46 |
| ssam2 | it's not nearly as bad as the last time i tried to submit something to cpython :-) | 17:48 |
| *** bochecha has quit IRC | 18:09 | |
| *** bochecha has joined #buildstream | 18:11 | |
| *** jonathanmaw has quit IRC | 18:42 | |
| gitlab-br-bot | buildstream: issue #172 ("pkg_resources module causes a delay of between 0 and many seconds every time `bst` starts up") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/172 | 19:00 |
| gitlab-br-bot | buildstream: merge request (sam/meson->master: WIP: use Meson instead of setuptools to install and distribute BuildStream) #196 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/196 | 19:08 |
| *** ssam2 has quit IRC | 19:15 | |
| *** semanticdesign has quit IRC | 20:25 | |
| *** bethw has joined #buildstream | 21:41 | |
| *** xjuan has quit IRC | 21:51 | |
| *** bethw has quit IRC | 21:57 | |
| *** bethw has joined #buildstream | 22:01 | |
| *** bethw has quit IRC | 22:10 | |
| *** mcatanzaro has quit IRC | 22:26 | |
| *** valentind has quit IRC | 22:40 | |
| *** bochecha has quit IRC | 22:53 | |
| *** bochecha has joined #buildstream | 22:53 | |
| *** bochecha has quit IRC | 23:09 | |
| *** bochecha has joined #buildstream | 23:15 | |
| *** bochecha has quit IRC | 23:18 | |
| *** bochecha has joined #buildstream | 23:18 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!