| *** hergertme has quit IRC | 04:22 | |
| *** hergertme has joined #buildstream | 04:28 | |
| *** tristan has joined #buildstream | 05:47 | |
| paulsher1ood | jjardon[m]: call me petulant, but i won't be attempting that :) | 06:18 |
|---|---|---|
| *** ssam2 has joined #buildstream | 07:29 | |
| *** LaurenceUrhegyi has joined #buildstream | 08:29 | |
| *** LaurenceUrhegyi has quit IRC | 08:34 | |
| *** LaurenceUrhegyi has joined #buildstream | 08:34 | |
| *** jonathanmaw has joined #buildstream | 08:45 | |
| tristan | violeta, so the buildstream-tests repo is just something I threw together so that people could "try stuff", there are various branches in there which build different things, but it would seem like a good name for a repo to hold an actual buildstream test suite, too | 09:35 |
| violeta | I see, I did think it didn't have a lot in it :-) | 09:36 |
| tristan | I guess, we could have different branches in there that we would mark as "protected" (kind of bunch of separate masters), each one of which holds a project that tests a certain aspect of the buildstream plugins | 09:36 |
| violeta | why does each aspect have to be tested on different branches? | 09:37 |
| tristan | yeah the "build-gnome" branch in there is most interesting I guess, if you want to look at it; that 'build-gnome' is, is a baserock conversion that builds a converted gnome system from the bootstrap all the way to an image deployment | 09:37 |
| violeta | if it makes sense to have this other repo separately, we could just create a test-suite one specifically for it | 09:37 |
| tristan | violeta, well, that is a good point, it could all be on the same "branch", but what I would like is to have a separate buildstream "project" for each plugin | 09:38 |
| violeta | ok, I'll have to look more into this before I have more questions :-) | 09:39 |
| tristan | The point is to have small samples which quickly identify something that went wrong, and to try to get decent coverage | 09:39 |
| tristan | Sure :) | 09:39 |
| tristan | violeta, are you familiar with the gitlab side of things, too ? | 09:40 |
| tristan | setting up .gitlab-ci.yml and suchlike ? | 09:40 |
| violeta | never used gitlab before | 09:40 |
| tristan | Ah, ok well, that is going to be core to this also, maybe I should outline a bit more clearly | 09:40 |
| * tristan doesnt want you to get lost :) | 09:41 | |
| violeta | thanks \o/ | 09:41 |
| violeta | I don't want to get lost either | 09:41 |
| tristan | currently buildstream CI is failing since a week (not the test suite, but the pages docs deployment, I have to look at that) | 09:42 |
| tristan | but here: https://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml | 09:42 |
| tristan | So that is the gitlab-ci.yml in the buildstream project root | 09:42 |
| tristan | It tells gitlab to run some things every time something is pushed to the repository | 09:42 |
| tristan | the format is pretty simple, and already takes care of setting up the environment to have buildstream's dependencies installed | 09:43 |
| tristan | So at this line we run the test suite: https://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml#L20 | 09:43 |
| violeta | is this like Travis? | 09:44 |
| * tristan has never met Travis :) | 09:44 | |
| violeta | hah, Github Travis CI :-P | 09:44 |
| violeta | ok ok | 09:44 |
| tristan | I guess its very similar then, i.e. you can see it's assumed that CWD is in a checkout of buildstream | 09:45 |
| tristan | In any case... we're going to want 2 different setups here | 09:45 |
| tristan | We'll want, after running the test suite, to also checkout buildstream-tests where the tests are, and run bst build on those projects in that repo, which we can checkout right there in the script | 09:46 |
| tristan | Also, we'll want a very similar .gitlab-ci.yml setup for buildstream-tests | 09:46 |
| tristan | Which just always runs against buildstream master | 09:46 |
| violeta | ok | 09:47 |
| tristan | That way, we can run the tests both when A.) Someone commits to a buildstream branch, it runs against the most recent test cases... and B.) Someone updates the test cases, they get rerun in a merge request against buildstream master before updating the mainline test cases | 09:47 |
| tristan | Makes sense ? :D | 09:47 |
| violeta | yes, makes sense, thanks :-) | 09:48 |
| tristan | great | 09:48 |
| tristan | Ok so... it looks like running a debootstrap of amd64 on an aarch64 machine works fine, at least for our purposes :) | 09:54 |
| tristan | Just because, we happen to have some aarch64 machines... | 09:54 |
| tristan | only problem now, is that it seems the apache reverse proxy thing is ... borked | 09:55 |
| tristan | and I am allergic to all things networking | 09:55 |
| tristan | well, all things which involve ips and routing tables and all of that nonsense | 09:55 |
| tristan | Nope ! | 09:59 |
| tristan | Good news: https://gnome7.codethink.co.uk/repo/ | 10:00 |
| tristan | Ok that currently serves a debian amd64 testing snapshot, which will be updated regularly ... once I fix/finish this script which does it on a cron job | 10:03 |
| *** tristan has quit IRC | 10:16 | |
| *** tristan has joined #buildstream | 10:16 | |
| *** LaurenceUrhegyi has quit IRC | 10:19 | |
| *** LaurenceUrhegyi has joined #buildstream | 10:23 | |
| *** ChanServ sets mode: +o tristan | 10:32 | |
| ironfoot | tristan: let me know if you need help setting up infra | 10:43 |
| tristan | ironfoot, actually I think I understand now... or not really but kindof | 10:57 |
| tristan | ironfoot, actually my setup of the bot was going to work, but I would have just had to wait around for 30 minutes or so | 10:57 |
| tristan | maybe | 10:57 |
| ironfoot | :) | 10:57 |
| tristan | anyway, in this case... I have these 8 aarch64 cartridges which were donated for building GNOME on arm | 10:58 |
| ironfoot | right | 10:58 |
| tristan | And they all run apache behind this reverse proxy thingamajigie | 10:58 |
| tristan | When I install one of them or wake them up after a long time, they get a new ip on the local dns | 10:58 |
| tristan | and I have to update the reverse proxy setup on the jumpserv middleman | 10:59 |
| tristan | So far so good | 10:59 |
| tristan | I did that last time, and this time | 10:59 |
| tristan | And then after restarting apache... the servers dont resolve | 10:59 |
| tristan | I think this was the same issue with the bot which expects to receive POST | 10:59 |
| tristan | ironfoot, so anyway... if I come back 30 min later and leave the config as is... it starts to resolve | 11:00 |
| tristan | So i'm thinking, some DNS somewhere has to slowly get updated and figure out what gnome7.codethink.co.uk resolves to | 11:00 |
| tristan | it "eventually" works, just not right away | 11:00 |
| tristan | anyway now https://gnome7.codethink.co.uk/repo/ is being served, and looks like my cron job setups are also working | 11:02 |
| tristan | So there will be a daily import of the ostree for both i386 and amd64 debian arches into that repo | 11:02 |
| tristan | nice, added armhf and arm64 to that list :) | 11:14 |
| tristan | ok next: Continuous auto migrations of jhbuild modulesets | 11:14 |
| * tristan tries to build against the gnome7.codethink.co.uk hosted runtime | 11:15 | |
| *** LaurenceUrhegyi has quit IRC | 11:25 | |
| *** LaurenceUrhegyi has joined #buildstream | 11:42 | |
| persia | tristan: I don't know the details of your setup, but a few solutions I've seen work before are: A) have a client that runs on startup on the machine that connects to the proxy and updates config immediately after boot, B) Using an overlay network internally, with each node advertising that it can route to a logical IP address thorugh the physical interface, and redirects pointing to logical IP addresses, and C) configuration of the DHCP server to | 11:48 |
| persia | provide static IP addresses to be associated with the MAC addresses for the machines behind the proxy. | 11:48 |
| persia | Making it work with pure DNS is complicated, and requires fairly tight timings, which can be frustrating if there is a long keepalive on some SOA further up in the hierarchy. | 11:50 |
| tristan | persia, right now it's not broke, so I wont spend time fixing it :) | 11:51 |
| tristan | heh | 11:51 |
| persia | excellent plan :) | 11:51 |
| tristan | It's rare enough that these machines get re-setup | 11:51 |
| *** jude has quit IRC | 12:47 | |
| jonathanmaw | hmm, when trying to build with my bzr plugin, I'm getting the error " bzr source at emacs.bst [line 4 column 2]: Failed to checkout revision 118363 from branch /home/jonathanmaw/.cache/buildstream/sources/bzr/savannah_emacs/trunk to /home/jonathanmaw/.cache/buildstream/build/emacs-4p14l0eo/buildstream/build" | 12:53 |
| jonathanmaw | it seems to be because emacs-$foo/buildstream/build doesn't exist. | 12:53 |
| jonathanmaw | I'm guessing that's because it's stage's responsibility to make sure that dir exists | 12:53 |
| *** jude has joined #buildstream | 13:01 | |
| ssam2 | when I run `setup.py test`, no tests are discovered | 13:02 |
| ssam2 | is that happening for anyone else? | 13:02 |
| jonathanmaw | ssam2: seems to be working for me | 13:03 |
| ssam2 | maybe i'm missing deps and it's swallowing the errors somehow | 13:04 |
| tristan | Well, my bad news is that cross bootstrapping is not working easily with debootstrap | 13:05 |
| tristan | But on the bright side | 13:05 |
| tristan | It was very easy to change for multistrap | 13:06 |
| ssam2 | what's multistrap? | 13:06 |
| ssam2 | oh https://wiki.debian.org/Multistrap | 13:06 |
| tristan | https://wiki.debian.org/EmDebian/CrossDebootstrap#Multistrap | 13:06 |
| tristan | Yep | 13:06 |
| ssam2 | so you could debootstrap and then multistrap in there | 13:06 |
| ssam2 | presumably this only for arches Debian is already ported to though so not much use to me | 13:07 |
| tristan | so its actually closer to what I want, it will unpack everything but not run the scriptlets, giving me a cross-arch created chrootable thing | 13:07 |
| ssam2 | but good for doing armv8l and such i guess ? | 13:07 |
| tristan | Which I could actually run dpkg --configure -a inside of on real hardware or qemu to make it bootable | 13:07 |
| tristan | ssam2, actually, its necessary for me right now because I *happen* to have my hands on an aarch64 machine I can cron some automated imports with | 13:08 |
| tristan | So I need cross in order to also produce/publish the x86 runtimes | 13:08 |
| tristan | Right not useful for your cross bootstrapping new arches thing | 13:08 |
| tristan | this is to produce base systems that satisfy the GNOME moduleset sysdeps which I can run builds of converted modulesets on top of | 13:09 |
| tristan | ssam2, not sure about your ./setup.py test problem, did you follow the docs on installing deps and such ? | 13:09 |
| ssam2 | I've done `pip3.4 install --user buildstream` | 13:10 |
| ssam2 | pip3.5 rather | 13:10 |
| ssam2 | so should have all the deps in my PYTHONPATH | 13:10 |
| tristan | right if that passed, it should be working really | 13:10 |
| ssam2 | to be fair, it failed with a weird error | 13:10 |
| ssam2 | but installed stuff first | 13:10 |
| tristan | well | 13:11 |
| ssam2 | https://wiki.debian.org/Multistrap | 13:11 |
| ssam2 | argh | 13:11 |
| ssam2 | https://pastebin.com/Few61mK0 | 13:11 |
| ssam2 | python packaging is dismal | 13:12 |
| tristan | jonathanmaw, so yes, we dont create the build directory in advance as that would cause issues sources | 13:12 |
| ssam2 | upgrading pip has fixed that | 13:13 |
| tristan | jonathanmaw, i.e. if you do git clone url <directory> and the directory exists, well git wont like that | 13:15 |
| tristan | looks like I'm not connected to the internet :-S | 13:15 |
| ssam2 | ok, upgrading py.test has fixed that | 13:15 |
| ssam2 | a test suite! | 13:15 |
| * tristan gets reconnected seemlessly, it seams | 13:15 | |
| tristan | ssam2, yeah so did you follow the advice ? | 13:16 |
| tristan | and try again ? | 13:17 |
| tristan | is that the full log too ? | 13:17 |
| ssam2 | it's working now | 13:17 |
| ssam2 | had to upgrade pip and py.test | 13:17 |
| ssam2 | i guess its an issue with `pip` packages that they don't get autoupdated, unlike distro packages | 13:18 |
| tristan | Nice | 13:20 |
| tristan | Yeah pip is garbage | 13:20 |
| tristan | But oh well | 13:20 |
| tristan | ssam2, so btw... ok with `pip install --user` you will get man pages, but... you probably want `pip install --user -e` | 13:49 |
| tristan | ssam2, the difference is that, with `-e` it will do a developer install, which means that any modification you make to buildstream will be effective immediately and not require a separate install step | 13:49 |
| tristan | just saying, in case you wonder "how come this change isnt working ?!" | 13:50 |
| tristan | that is in the HACKING file, but not in the user facing getting started docs | 13:50 |
| ssam2 | ok, ta | 13:51 |
| ssam2 | ok, ta | 13:51 |
| ssam2 | (oops) | 13:51 |
| ssam2 | I did `--develop` which is hopefully the same | 13:51 |
| ssam2 | yeah, seems to be | 13:52 |
| tristan | I think so, anyway it's what pip internally passes to setup.py | 13:59 |
| *** LaurenceUrhegyi has quit IRC | 14:07 | |
| *** LaurenceUrhegyi has joined #buildstream | 14:09 | |
| ssam2 | I got an initial Baserock system -> BuildStream conversion done pretty easily | 14:18 |
| ssam2 | defs2bst seems to work great | 14:18 |
| ssam2 | have found some teething issues with ctrl+c during Git fetches I think | 14:19 |
| ssam2 | my desktop started to seize up so I hit ctrl+c, and that fixed stuff | 14:19 |
| ssam2 | but on resume a bunch of fetches have failed | 14:19 |
| tristan | ssam2, I have opened this for that purpose: https://gitlab.com/BuildStream/buildstream/issues/30 | 14:20 |
| tristan | Not much we can do otherwise, I think | 14:20 |
| ssam2 | ok cool | 14:20 |
| ssam2 | yeah `git clone` is just a bit fragile | 14:20 |
| tristan | Well, note we have this context manager which handles suspend / resume | 14:21 |
| tristan | So if for example, a git source knew that git handles a specific unix signal to try to reconnect and resume... we could hook into that and send it | 14:21 |
| tristan | but... meh | 14:22 |
| ssam2 | urgh, high disk IO making my desktop unusable | 14:22 |
| ssam2 | Linux really not great at scheduling under pressure | 14:22 |
| tristan | Yes, that will happen with gnome shell | 14:22 |
| tristan | or with linux | 14:22 |
| tristan | indeed, io makes my life hard here too | 14:23 |
| *** ssam2 has quit IRC | 14:26 | |
| *** ssam2 has joined #buildstream | 14:27 | |
| ssam2 | that didn't take long... Linux got itself completely locked up and i had to reboot | 14:28 |
| ssam2 | i really haven't missed this from working on Baserock | 14:28 |
| tristan | What ?? | 14:28 |
| tristan | ssam2, well, you can configure your fetchers and builders | 14:28 |
| ssam2 | yeah, i'll do that | 14:28 |
| tristan | either on command line or in a config file | 14:28 |
| ssam2 | should be OK if I have just 1 or 2 fetchers | 14:28 |
| tristan | You think it's the fetch tasks ? | 14:28 |
| tristan | I wonder, well, it's possible | 14:29 |
| ssam2 | hmm, let me look at the logs | 14:29 |
| ssam2 | it did look like there were 8 fetches going on | 14:29 |
| tristan | I recently cranked them up to 10 by default | 14:29 |
| ssam2 | starting from scratch remember | 14:29 |
| tristan | since at least here, it makes little or no difference to me | 14:29 |
| ssam2 | yeah, only logs I have are fetch jobs | 14:29 |
| tristan | mostly blocking on network I/O | 14:29 |
| tristan | well, that'll be it then I guess | 14:30 |
| tristan | I think in normal usage though, it's really the ostree that is I/O intensive | 14:30 |
| ssam2 | ah, I think there was an ostree pull going on | 14:30 |
| tristan | When caching artifacts and the initial extractions/checkouts | 14:30 |
| ssam2 | for the gnome SDK | 14:30 |
| tristan | nod | 14:30 |
| tristan | Once that completes once, should be better | 14:31 |
| ssam2 | long term it would be nice to provide "starter pack" bundles, like a big tarball with some snapshot of the various Gits and OStree repos you're likely to need | 14:31 |
| tristan | I suspect though, with artifact cache shares and downloading artifacts, things will be similar | 14:31 |
| ssam2 | more of a burden for us to provide, but saves each new user going through this same initial hassle | 14:32 |
| tristan | ssam2, well, that's what artifact cache sharing should probably accomplish by itself, though | 14:32 |
| ssam2 | oh true | 14:32 |
| ssam2 | yeah I guess then you go back to only needing to clone a repo once you hack on something | 14:32 |
| tristan | depending, but yeah | 14:32 |
| tristan | i.e. depending on build mode, with build plans based on weak cache keys, then you dont even need to clone everything that depends on glib when hacking on glib | 14:33 |
| tristan | for example | 14:33 |
| * ssam2 tries again with 2 fetches | 14:34 | |
| ssam2 | *fetchers | 14:34 |
| tristan | I honestly think it wont make a huge difference, maybe ostree needs to be throttled or smth ? | 14:38 |
| tristan | anyway let me know if it changes much :) | 14:38 |
| tristan | ssam2, I'm guessing you have old fashioned hard disks, not SSD ? | 14:38 |
| tristan | maybe a good internet connection + slow disks will cause that | 14:39 |
| ssam2 | yeah, old HD | 14:41 |
| tristan | Ok, stayed a bit late, but now I think I nailed the server to import these debootstraps | 14:41 |
| tristan | with gpg signing of the ostree commits and all that | 14:42 |
| tristan | unfortunately looks like my download from gnome7.codethink.co.uk is very slow :-S | 14:42 |
| tristan | eek, average 375ms round trip | 14:43 |
| tristan | ok times up, I'll retry the build from home | 14:52 |
| * tristan out... | 14:52 | |
| *** tristan has quit IRC | 14:55 | |
| jonathanmaw | hrm, it looks like my bzr tests are failing in CI because they don't have bzr | 15:04 |
| jonathanmaw | for some reason the exception raised by preflight() isn't getting propagated when it's in a Setup class initializer | 15:05 |
| ssam2 | sounds like a bug | 15:06 |
| ssam2 | in something.. | 15:06 |
| jonathanmaw | seems like Setup is a fixture that comes from pytest. there's probably some internal magic that swallows the exception | 15:07 |
| jonathanmaw | I'm going to try moving the call out of setup to see if the problem persists | 15:07 |
| ssam2 | i see no reason pytest should ignore exceptions in fixtures... it might be something about the way that fixture is implemented | 15:09 |
| ssam2 | can't see anything obvious though | 15:09 |
| ssam2 | i'm eagerly awaiting buildstream to start actually building stage1-binutils, but it's been "Staging gnu-toolchain/base-sdk.bst/5aa1311b" for 5 minutes | 15:11 |
| ssam2 | ok, finally started building | 15:12 |
| paulsher1ood | ....just like the old days, watching moprh :) | 15:12 |
| ssam2 | so many memoriues | 15:12 |
| paulsher1ood | heh | 15:12 |
| ssam2 | it really shouldn't take 5 minutes to do an `ostree checkout` though, if this isn't just a one time thing then something must be wrong | 15:12 |
| ssam2 | oh it seems to have gone quicker the 2nd time | 15:14 |
| ssam2 | must be a one time thing the first time you do `ostree checkout` | 15:14 |
| jonathanmaw | hmm, nothing raised when preflight() is called outside the constructor, either | 15:27 |
| jonathanmaw | I'm expecting the exception to be raised because I use utils.get_host_tool(), which raises an exception | 15:27 |
| jonathanmaw | and I can see nothing wrong with get_host_tool() | 15:35 |
| jonathanmaw | sod it, I'm going to leave it alone and just try to ensure bzr is installed in the CI runner. | 15:36 |
| *** tristan has joined #buildstream | 15:44 | |
| jonathanmaw | >:( when I add in .gitlab-ci instructions to install bzr, the tests that use bzr still fail | 15:57 |
| jonathanmaw | latest litany of failure at https://gitlab.com/jonathanmaw/buildstream/builds/16982830 | 15:57 |
| jonathanmaw | setup.source.track(), setup.source.stage() and setup.source.get_consistency() all lead to calling 'bzr' | 15:58 |
| jonathanmaw | other than the tests failing when they oughtn't to in CI, the bzr plugin seems to be functional | 16:08 |
| jonathanmaw | emacs from bzr is failing to build, but the generated stage looks okay, and I have tests that the output of stage is expected. | 16:09 |
| *** ssam2 has quit IRC | 16:24 | |
| *** jonathanmaw has quit IRC | 17:01 | |
| *** ChanServ sets mode: +o tristan | 17:24 | |
| *** LaurenceUrhegyi has quit IRC | 19:36 | |
| *** violeta_ has joined #buildstream | 21:00 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!