| *** ChanServ has quit IRC | 02:32 | |
| *** ChanServ has joined #baserock | 02:34 | |
| *** sinisalo.freenode.net sets mode: +o ChanServ | 02:34 | |
| *** gtristan has quit IRC | 07:03 | |
| *** gtristan has joined #baserock | 07:04 | |
| *** toscalix has joined #baserock | 08:42 | |
| *** jude_ has joined #baserock | 09:07 | |
| *** toscalix has quit IRC | 09:21 | |
| *** noisecell has joined #baserock | 09:39 | |
| *** ssam2 has joined #baserock | 10:06 | |
| *** locallycompact has joined #baserock | 10:09 | |
| *** jonathanmaw has joined #baserock | 10:22 | |
| ssam2 | juergbi`, jjardon1, benbrown_ : looks like something is going really wrong with the CI runners | 11:21 |
|---|---|---|
| ssam2 | `docker-machine ls` on the manager shows loads of this: | 11:21 |
| ssam2 | runner-f45ec36a-machine-1511738480-bffee10e-baserock - digitalocean Error Unknown dropletID is invalid because cannot be less than 1 | 11:21 |
| *** juergbi` is now known as juergbi | 11:22 | |
| ssam2 | and this is happening: https://gitlab.com/BuildStream/buildstream/-/jobs/44513272 | 11:22 |
| ssam2 | we have 5 actual runner VMs active | 11:24 |
| ssam2 | all of which show up in the `docker-machine ls` logs with that "dropletID is invalid" error | 11:24 |
| ssam2 | i have no idea what to do really .... maybe reboot the manager ? | 11:24 |
| * ssam2 will try that | 11:27 | |
| ssam2 | that didn't seem to help | 11:29 |
| ssam2 | https://github.com/docker/machine/issues/2548 | 11:34 |
| ssam2 | trying to get into one of the broken runners is impossibly frustrating | 11:44 |
| ssam2 | digitalocean reset the root password to a 20 character hex string, which i have to enter into a VNC console that doesn't support pasting | 11:44 |
| ssam2 | twice | 11:44 |
| ssam2 | then the machine asks me to choose a new password, but it rejected my choice and now i have to retype this 20 character hex string for the 5th and 6th time | 11:45 |
| ssam2 | ok, i am finally in | 11:46 |
| ssam2 | the machine doesn't seem to even have docker installed, which seems a bit wrong | 11:47 |
| ssam2 | perhaps something is going wrong with the base image or the docker-machine init process ... not sure how it's meant to work | 11:47 |
| ssam2 | i'm surprised that each runner VM gets a public IP | 11:50 |
| ssam2 | that explains why the journal is full of attempts to log in as random users over SSH, though | 11:51 |
| ironfoot | bad | 11:51 |
| ironfoot | well, good luck figuring out the password, though | 11:51 |
| ssam2 | i guess it's fair that it was enforcing a strong root password if its actually internet facing | 11:52 |
| ssam2 | seems that docker-machine messed up its own config for that machine | 11:57 |
| ssam2 | if I manually edit the config.json file on the manager and add the correct droplet ID and IP address, I can get `docker-machine ssh` to work for accessing the machine | 11:57 |
| ssam2 | i wonder why it's doing that, though | 11:58 |
| ssam2 | i've deleted every stale machine, let's try a CI job now | 12:00 |
| ssam2 | that seems to have failed | 12:01 |
| ssam2 | the journal shows it creating the machine, immediately trying to remove it for some reason, and then failing to remove it because it "doesn't exist" | 12:02 |
| ssam2 | and there don't seem to be any new droplets in the DigitalOcean web interface, so I guess it's on to something about the machine not existing ... | 12:02 |
| ssam2 | ah, invalid image | 12:03 |
| ssam2 | I think the issue is that we're using Fedora 25 still, and Fedora 25 recently hit end-of-life | 12:03 |
| ssam2 | that should be easy to fix | 12:03 |
| ssam2 | this looks more promising, let's see if https://gitlab.com/BuildStream/buildstream/-/jobs/44565555 succeeds | 12:05 |
| ironfoot | that worked | 12:13 |
| ssam2 | happy days | 12:17 |
| ssam2 | so basically once a year someone has to update the image we boot on the runners :-) | 12:17 |
| ironfoot | or we move to a LTS distro :P | 12:28 |
| persia | Once a year is probably easier to remember than once every two years. | 12:28 |
| paulsherwood | or self host :) | 12:29 |
| paulsherwood | i'll get my coat... | 12:29 |
| ironfoot | heh | 12:31 |
| *** toscalix has joined #baserock | 15:08 | |
| *** noisecell has quit IRC | 16:42 | |
| *** locallycompact has quit IRC | 17:33 | |
| *** paulwaters_ has joined #baserock | 18:25 | |
| *** jonathanmaw has quit IRC | 18:42 | |
| *** paulwaters_ has joined #baserock | 19:09 | |
| *** ssam2 has quit IRC | 19:15 | |
| *** toscalix has quit IRC | 19:25 | |
| *** ChanServ has quit IRC | 22:17 | |
| *** ChanServ has joined #baserock | 22:24 | |
| *** sinisalo.freenode.net sets mode: +o ChanServ | 22:24 | |
| *** ChanServ has quit IRC | 22:28 | |
| *** ChanServ has joined #baserock | 22:31 | |
| *** sinisalo.freenode.net sets mode: +o ChanServ | 22:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!