IRC logs for #baserock for Thursday, 2017-12-14

*** ChanServ has quit IRC02:32
*** ChanServ has joined #baserock02:34
*** sets mode: +o ChanServ02:34
*** gtristan has quit IRC07:03
*** gtristan has joined #baserock07:04
*** toscalix has joined #baserock08:42
*** jude_ has joined #baserock09:07
*** toscalix has quit IRC09:21
*** noisecell has joined #baserock09:39
*** ssam2 has joined #baserock10:06
*** locallycompact has joined #baserock10:09
*** jonathanmaw has joined #baserock10:22
ssam2juergbi`, jjardon1, benbrown_ : looks like something is going really wrong with the CI runners11:21
ssam2`docker-machine ls` on the manager shows loads of this:11:21
ssam2runner-f45ec36a-machine-1511738480-bffee10e-baserock   -        digitalocean   Error                 Unknown   dropletID is invalid because cannot be less than 111:21
*** juergbi` is now known as juergbi11:22
ssam2and this is happening:
ssam2we have 5 actual runner VMs active11:24
ssam2all of which show up in the `docker-machine ls` logs with that "dropletID is invalid" error11:24
ssam2i have no idea what to do really .... maybe reboot the manager ?11:24
* ssam2 will try that11:27
ssam2that didn't seem to help11:29
ssam2trying to get into one of the broken runners is impossibly frustrating11:44
ssam2digitalocean reset the root password to a 20 character hex string, which i have to enter into a VNC console that doesn't support pasting11:44
ssam2then the machine asks me to choose a new password, but it rejected my choice and now i have to retype this 20 character hex string for the 5th and 6th time11:45
ssam2ok, i am finally in11:46
ssam2the machine doesn't seem to even have docker installed, which seems a bit wrong11:47
ssam2perhaps something is going wrong with the base image or the docker-machine init process ... not sure how it's meant to work11:47
ssam2i'm surprised that each runner VM gets a public IP11:50
ssam2that explains why the journal is full of attempts to log in as random users over SSH, though11:51
ironfootwell, good luck figuring out the password, though11:51
ssam2i guess it's fair that it was enforcing a strong root password if its actually internet facing11:52
ssam2seems that docker-machine messed up its own config for that machine11:57
ssam2if I manually edit the config.json file on the manager and add the correct droplet ID and IP address, I can get `docker-machine ssh` to work for accessing the machine11:57
ssam2i wonder why it's doing that, though11:58
ssam2i've deleted every stale machine, let's try a CI job now12:00
ssam2that seems to have failed12:01
ssam2the journal shows it creating the machine, immediately trying to remove it for some reason, and then failing to remove it because it "doesn't exist"12:02
ssam2and there don't seem to be any new droplets in the DigitalOcean web interface, so I guess it's on to something about the machine not existing ...12:02
ssam2ah, invalid image12:03
ssam2I think the issue is that we're using Fedora 25 still, and Fedora 25 recently hit end-of-life12:03
ssam2that should be easy to fix12:03
ssam2this looks more promising, let's see if succeeds12:05
ironfootthat worked12:13
ssam2happy days12:17
ssam2so basically once a year someone has to update the image we boot on the runners :-)12:17
ironfootor we move to a LTS distro :P12:28
persiaOnce a year is probably easier to remember than once every two years.12:28
paulsherwoodor self host :)12:29
paulsherwoodi'll get my coat...12:29
*** toscalix has joined #baserock15:08
*** noisecell has quit IRC16:42
*** locallycompact has quit IRC17:33
*** paulwaters_ has joined #baserock18:25
*** jonathanmaw has quit IRC18:42
*** paulwaters_ has joined #baserock19:09
*** ssam2 has quit IRC19:15
*** toscalix has quit IRC19:25
*** ChanServ has quit IRC22:17
*** ChanServ has joined #baserock22:24
*** sets mode: +o ChanServ22:24
*** ChanServ has quit IRC22:28
*** ChanServ has joined #baserock22:31
*** sets mode: +o ChanServ22:31

Generated by 2.15.3 by Marius Gedminas - find it at!