IRC logs for #baserock for Monday, 2014-11-17

*** JPohlmann [~jannis@m65s13.vlinux.de] has joined #baserock00:15
*** JPohlmann [~jannis@m65s13.vlinux.de] has quit [Changing host]00:15
*** JPohlmann [~jannis@xfce/core-developer/JPohlmann] has joined #baserock00:15
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock03:55
*** thecorconian [~jte@75-27-44-31.lightspeed.orpkil.sbcglobal.net] has quit [Remote host closed the connection]04:32
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]04:42
*** aananth [~caananth@74.112.167.117] has joined #baserock07:18
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock07:56
*** radiofree_ [radiofree@unaffiliated/radiofree] has quit [Read error: Connection reset by peer]07:58
*** fay_ [~fay@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]07:59
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:02
*** radiofree [radiofree@unaffiliated/radiofree] has joined #baserock08:06
*** fay [~fay@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:10
fay is now known as Guest1258908:11
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]08:29
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has joined #baserock08:43
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:45
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]08:46
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock08:55
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock08:58
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Quit]09:16
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:17
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:18
*** tiagogomes [~tiagogome@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:23
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:26
richard_mawstraycat: If you update your system to include a newer version of yarn, then it will be quicker and require less storage, as it doesn't snapshot directories it doesn't need to09:37
Guest12589 is now known as fay_09:37
straycatrichard_maw, okay cool i'll do that09:38
Mode #baserock +o pedroalvarez by ChanServ09:40
* Kinnison congratulates the project -- my baserock-dev folder just hit 10,000 entries :-)09:53
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock09:54
Mode #baserock +v ssam2 by ChanServ09:54
pedroalvarezKinnison: yay!09:54
pedroalvarezpaulsher1ood: wrt oFono; I'v just realized that 1.11 >= 1.9 09:55
pedroalvarez:)09:55
Kinnisonheh09:55
KinnisonWhat algorithm were you using for version comparison before?!09:55
pedroalvareznot sure, but executed by an  human brain09:56
Kinnisonheh09:56
* Kinnison strongly recommends using the Debian version number comparison algorithm if you can09:56
DavePagepedroalvarez: Always the weak point :)09:59
pedroalvarezAlso, I realized on friday that we need `morph edit` for now10:06
pedroalvarezwe are using it in our licensecheck script10:06
aananthpedroalvarez: Good Morning! I just checked http://136.18.233.152/cgi-bin/cgit.cgi/delta/linux.git/ of my trove server, the directory is still empty. The server ran for 2 complete days. :-)10:11
aananthAny test to see if things go well?10:11
pedroalvarezhmm that's odd10:11
ssam2aananth: I think the problem Aananth has is that trove-setup.service failed10:11
pedroalvarezssam2: but I thought that some of the lorries worked10:11
ssam2ah, ok, hmm10:12
ssam2I remember seeing a log from Aananth where trove-setup.service failed because it tried to read SSH host keys from git.baserock.org and couldn't because it lacked proxy config10:12
ssam2which I was hoping to make a fix for today10:12
pedroalvarezah yeah, you missed all the fun last week :)10:12
*** Krin [~mikesmith@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock10:13
aananthssam2: I am sorry, the proxy issue got resolved last week. I failed to add a ',' between username and password.10:13
ssam2so it's a specific lorry (linux.git) that isn't working10:13
aananthapart from the above, I had to set up corkscrew, .gitconfig, ssh proxy etc.10:14
ssam2it could be that it's timing out, but really we need to see the logs for that lorry to see what's going on10:15
aananthssam2: yes, today I think all linux*.git repo are empty. Ok I will send the log. Output of journalctl?10:15
ssam2aananth: we should be able to filter the output of journalctl a bit so we only see what's needed10:16
ssam2I'll try to work out the correct commandline10:16
pedroalvarezaananth: If you run this: `ssh -L 12765:localhost:12765 root@136.18.233.152` and you keep that ssh connection open, you will be able to  open "localhost:12765/1.0/status-html" in your browser10:16
*** petefoth_ [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock10:18
*** petefoth [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 245 seconds]10:18
petefoth_ is now known as petefoth10:18
pedroalvarezaananth: once you are there we can work out how to get some logs :)10:21
aananthssam2, pedroalvarez: http://fpaste.org/151383/14162197/10:23
aananthpedroalvarez: "localhost" did not resolve, hence i replaced it with my trove-server IP. I got lorry controller status only.10:24
pedroalvarezaananth: but you have the button "See separate list of all jobs that have ever been started"10:26
pedroalvarezfrom there you will access to the list of all the lorry jobs (mirroring attempts) of your trove10:27
aananthpedroalvarez: its a link?10:27
pedroalvarezaananth: it is, can't you open it?10:28
ssam2<http://localhost:12765/1.0/status-html> should work10:28
ssam2if not try <http://127.0.0.1:12765/1.0/status-html>10:29
aananthpedroalvarez: Yes, I am opening, I could see a long list, with last column 0, 1, 127. I am pasting it.10:29
pedroalvarezgreat,10:29
pedroalvarezthe ones that are working have the 010:29
pedroalvarezyou should be worried about these with 1, 127, etc10:29
pedroalvarezIf you click in the Job ID of a job that failed, you see a log10:32
pedroalvarezyou will se a log*10:33
aananthpedroalvarez, ssam2: Ok, I am trying to paste it, but it is a huge text, hence it is not working.  Ok I will look into those non zero items and report those logs.10:34
pedroalvarezaananth: delta/linux would be a good start10:34
aananthpedroalvarez, ssam2: Thank you, here it is: http://fpaste.org/151389/20801141/10:39
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has quit [Ping timeout: 258 seconds]10:40
pedroalvarezeverything looks ok in that log, but it stopped before finishing10:43
radiofreenew systemd patches work a treat10:44
ssam2radiofree: is that a +1 ?10:45
radiofreeyes i have +1ed in the list as well10:45
radiofreeerm apart from one thing actually!10:45
radiofreelet me respond on the list10:46
ssam2ok, sweet10:46
pedroalvarezI wonder how long is the default lorry timeout10:47
aananthFew other failures: http://fpaste.org/151391/21427141/ . Everytime linux or linux-rt ran/executed, it failed. What is the meaning of return code '-9'?10:50
pedroalvarezradiofree: I'm interested in knowing if it improves the network connectivity on them, and if you can unplug the ethernet cable and plug it again and get connection10:51
pedroalvarezaananth: I think that -9 means that the process was killed, an probably because it was taking too long10:51
radiofreepedroalvarez: yes that was the first thing i tried and yes that works!10:52
paulsher1oodpedroalvarez: i can't reproduce it here, because for some reason it's *not public* but in the doc I saw it appears that Bluetooth Handsfree requires 1.14 or greater.... is that a typo, do you think?10:52
pedroalvarezaananth: to solve this you can add "lorry-timeout" in your lorry-controller.conf file, like we do in our trove: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/local-config/lorries.git/tree/lorry-controller.conf10:52
aananthpedroalvarez: The system is too slow and only 117M free (from free command). Is it worth restarting?10:52
richard_mawI'm merging "Use same kernel for jetson genivi and devel systems", just in case anyone else is attempting it and I would tread on toes10:53
pedroalvarezpaulsher1ood: ah! I thought it was 1.9, I'll look again10:53
pedroalvarezradiofree: amazing10:53
radiofreeit is yes!10:54
*** petefoth [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]11:01
*** petefoth [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock11:02
aananthpedroalvarez, ssam2: I rebooted after adding timeout on both lorries. But up-on reboot, the system reported disk error and asking me to check (fsck) manually... May be this could be the reason? 11:12
JPohlmannHi folks! Did anyone ping me in here during the last few days? (The IRC tab was highlighted)11:13
ssam2aananth: I'd expect to see an error in the log for delta/linux if disk corruption caused the delta/linux lorry to fail11:14
paulsher1oodJPohlmann: i think you were mentioned in passing. we have logs, now ... http://testirclogs.baserock.org/ 11:14
ssam2aananth: definitely run fsck anyway though!11:14
ssam2jpohlmann: hi! I don't remember needing to summon you for anything :)11:14
JPohlmannpaulsher1ood: Ok :)11:15
JPohlmannYup, found it. Something about a script I wrote that extracts the Bison version from NEWS.11:19
pedroalvarezoh yeah :)11:20
pedroalvarezI had to use again that patch11:20
*** locallycompact [~lc@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock11:23
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has joined #baserock11:27
aananthssam2, pedroalvarez: fscked, added timeout and restarted. Another interesting info: http://fpaste.org/151398/24191141/ clearly say timeout is the reason. 11:38
aananthAny idea, why the run queue in lorry controller status is still 893? Is that normal?11:39
pedroalvarezaananth: yes, that is the normal status. They will be there running every 2 hours (you can configure it) to fetch new changes11:40
aananthpedroalvarez: Any reason why the disk corruption happened? Slow CPU / network speed?11:40
aananth... second delta/linux started before the first one got completed?11:41
pedroalvarezaananth: no Idea about why the corruption happened11:41
pedroalvarezaananth: there is only one delta/linux11:42
pedroalvarezaananth: why do you think there are 2?11:44
pedroalvarez"Richard Moore pointed out that the URLs aren't terminated with quotes"11:49
pedroalvarezwas this Richard Maw?11:49
pedroalvarez:)11:50
richard_mawyes11:50
rdaleoops sorry11:50
richard_mawno worries11:50
aananthpedroalvarez: I assume that the delta/linux will be queued every 2 or 6 hrs and I meant the 2nd instance of the queue started before the first job was completed... etc :-)11:53
pedroalvarezah! no, that can't happen. delta/linux will appear in the queue again once it finishes, but not before11:54
aananthpedroalvarez: whether such scenario can happen, in case of slow network & cpu?11:54
aananthOk11:55
pedroalvarezaananth: btw, I think you don't need to reboot the machine everytime you want to change the configuration11:57
pedroalvarezit was needed when trove-setup was failing, but not anymore :)11:57
aananthOk11:57
aananthpedroalvarez: So, if I change the timeout value and push the config, will the system will take the new timeout value immediately or after I perform "systemctl start lorry-controller-readconf.service". 11:59
pedroalvarezyou can run that, and you will force the lorry-controller to read the configuration 12:00
pedroalvarezbut that is going to happen anyway12:00
aananthOk, so the lorry check its configs before it starts loading/unloading work! Very intelligent lorry :-)12:01
pedroalvarez:D12:01
jjardonpedroalvarez: hi! seems /usr/share/zoneinfo is not there anymore, I think its related with the glibc transition12:03
pedroalvarezjjardon: I've no idea about what's that and why is that needed12:03
jjardondo you remember to touch anything related with the zoneinfo or the tzdata?12:03
pedroalvarezjjardon: nope12:03
jjardonok, I will take a look12:04
pedroalvarezjjardon: can this be related to "cd o && make localtime=UTC" ?12:05
pedroalvarezthis is how we build glibc12:05
jjardonpedroalvarez: see 6.9.2  in http://www.linuxfromscratch.org/lfs/view/development/chapter06/glibc.html12:05
jjardonthis used to be there with eglibc, so maybe now its diferently configured, not sure yet12:06
pedroalvareznow I'm tempted to install the locales also in glibc12:07
jjardonah, seems the zoninfo is now provider by tzdata12:08
jjardonpedroalvarez: yes please! so I can see my name correctly ;)12:08
paulsher1oodjjardon: i vaguely remember tripping over this ages ago12:09
pedroalvarezyou can install them manually in your system12:09
jjardonpaulsher1ood: the zoneinfo problem?12:09
paulsher1oodyup12:09
* paulsher1ood can't remember why12:10
jjardonpedroalvarez: I think it would be better if baserock supports UTF8 by default ;)12:10
paulsher1ood+112:11
pedroalvarezjjardon: for locales: `mkdir -p /usr/lib/locale/ &&  localedef -v -c -i en_GB -f UTF-8 en_GB.UTF-8`12:11
ssam2pedroalvarez: how did we fix Aananth's issue with trove-setup.service last week?12:14
ssam2would http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/trove-setup.git/commit/?h=sam/fix-init-behind-proxy&id=bd80ed9a1690accd8d5dcb964ce387a27f6b014b be useful ?12:14
ssam2(i've not tested that patch yet and I won't bother if we don't need it :)12:14
jjardonpedroalvarez: is there a place where I can file a bug or not yet?12:16
pedroalvarezjjardon: on the mail list  :/12:16
pedroalvarezssam2: nice!12:19
pedroalvarezI really like it12:22
pedroalvarezjjardon: so is this a bug?12:25
jjardonpedroalvarez: Its a regression, yes12:25
* pedroalvarez awaits the BUG email :)12:26
*** dabukalam [~quassel@ec2-54-69-244-150.us-west-2.compute.amazonaws.com] has quit [Quit: dabukalam]12:32
*** dabukalam [~quassel@ec2-54-69-244-150.us-west-2.compute.amazonaws.com] has joined #baserock12:33
*** petefoth_ [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock12:37
*** petefoth [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 265 seconds]12:37
petefoth_ is now known as petefoth12:37
jjardonpedroalvarez: sure ;) probably a first step is to lorry ftp://ftp.iana.org/tz/tzdata-latest.tar.gz , ok to push this? http://paste.baserock.org/nucaqasewo.sql12:39
pedroalvarezjjardon: I'd prefer ftp://ftp.iana.org/tz/releases/tzcode2014j.tar.gz12:43
paulsher1oodisn't there a git repo?12:43
paulsher1oodhttps://github.com/tzinfo/tzinfo12:44
paulsher1oodhttps://github.com/tzinfo/tzinfo-data12:44
paulsher1oodpedroalvarez, jjardon ^^12:44
pedroalvarezisn't it that for ruby?12:45
paulsher1oodyes. maybe i'm completely off-topic here12:46
pedroalvarezpaulsher1ood: but you raised a good point :)12:46
jjardonpaulsher1ood: I think that has a copy of tzdata inside12:46
straycatgood news, looks like upstream setuptools might accept our patch12:47
pedroalvarezjjardon: isn't this also tzdata? http://git.baserock.org/cgi-bin/cgit.cgi/delta/glibc.git/tree/timezone12:47
jjardonpedroalvarez: why would you dont want to lorry the latest tzdata? does lorry not support changes in the tarball automatically and make a new commit?12:48
pedroalvarezstraycat: that's good!12:48
pedroalvarezjjardon: I really don't know, and also I don't know from where is going to get the version number to create a tag12:48
jmacsIt would be useful to have a description of what "morph branch" and "morph edit" do on http://wiki.baserock.org/contributing/12:48
ssam2straycat: cool12:49
paulsher1oodjjardon: i'm just generally against tarballs if there's a vcs source12:50
straycatI say might because they haven't merged it yet, they seem to want to update some of the docs associated with the change12:50
jjardonpaulsher1ood: sure12:50
robtaylorjjardon: i don't think tarball inport can import a series of tarballs and recreate history12:50
robtaylorjjardon: would be nice if it did =)12:51
jjardonpaulsher1ood: https://people.gnome.org/~walters/Tarballs.jpg ;)12:51
jjardonpedroalvarez: mmm, looks like it, yes; but at least fedora and arch depends on tzdata to build glibc12:52
paulsher1oodpetefoth: not sure where this fits in with the current site-structure http://wiki.baserock.org/traceability-and-reproducibility/12:52
straycathttp://sprunge.us/BTFE so i could do with this now12:53
paulsher1oodstraycat: +112:54
petefothpaulsher1ood: neither am I yet. Let me have a think. It maybe that it should be linked from a top-level story in StoryBoard or in a hypothetical ‘Roadmap’ document. 12:55
pedroalvarezI still don't like the -bitbucket suffix12:55
jjardonrobtaylor: btw, about the status of GNOME: its currectly broken because this tzdata problem: use to build up to gnome-shell but failed to run (after fixing some problem with the glib schemas not compiled  and dbus services not installed in the corect location)12:55
franredstraycat, +112:56
pedroalvarezjjardon: is this bug also going to be a problem in our devel systems?12:56
pedroalvarezor just in your gnome system>12:56
pedroalvarez?12:56
paulsher1oodpetefoth: -1 for pages which are in your head but don't exist in real life :-)12:57
petefothpaulsher1ood: creating it is on my to-do list12:57
petefothI’d like to do it in StoryBoard but….12:57
straycatmerged thanks12:58
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Ping timeout: 240 seconds]12:59
pedroalvarezI still think that there wasn't an strong reason to use the bitbucket suffix, and I hope we don't start doing this for new lorries13:01
petefothpaulsher1ood: add a readable version of the Traceability document in the misc-docs directory13:03
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock13:20
persia__ is now known as persia13:27
*** persia [quassel@2400:8900::f03c:91ff:feae:3452] has quit [Changing host]13:27
*** persia [quassel@ubuntu/member/persia] has joined #baserock13:27
*** Guest77313 [quassel@2400:8900::f03c:91ff:feae:3452] has quit [Changing host]13:27
*** Guest77313 [quassel@ubuntu/member/persia] has joined #baserock13:27
Guest77313 is now known as persia_13:27
straycatnot sure what's going on with gbo atm but there's a lot of cvsimport's running that don't look like they're associated with any currently running lorry job13:34
KinnisonPossibly killed jobs, cvs might have lost its controlling process group :-(13:35
paulsher1oodpetefoth: are you saying that page is unreadable?14:07
petefothpaulsher1ood: no, but it’s a Google Doc not accessible to non-Codethinker. So it needs to be moved out of Google Docs (or maybe shared publically if you want to do that), and put in a format that users of w.b.o can read14:08
straycatradiofree, didn't you say you were going to fix vim? :p14:10
* richard_maw wants to move all configuration file generation out of the install commands, and into deployment time configuration extensions, so that you can change how future vim instances will be configured without affecting the cache key and requiring a rebuild14:12
paulsher1oodpetefoth: http://wiki.baserock.org/traceability-and-reproducibility is not a googledoc14:13
jjardonstraycat: what problem do you have?14:13
paulsher1oodnor was it last time i sent the link :-)14:13
straycatjjardon, copy pasting14:13
straycatcan we move webtools into the devel system? or else move pip further down?14:14
straycatjjardon, radiofree mentioned the same problem a few weeks ago and i thought he said he was going to fix it :)14:14
pedroalvarezstraycat: so you need pip for the import tool14:14
straycatyeah14:15
pedroalvarezmakes sense to me14:15
pedroalvarezis there anything huge in the webtools?14:15
jjardonstraycat: you have to comment the "set mouse=a" line in /etc/vimrc but a patch to change this by default would be indeed helpful14:16
paulsher1oodi don't think so, pedroalvarez 14:16
paulsher1oodbut maybe it's time to tidy this anyway14:16
pedroalvarezin that case, you can go ahead with thad, but bear in mind that GNU tar doesn't build 14:17
straycatoh? why not?14:17
petefothpaulsher1ood: sorry! Too much fliting between jobs today :(14:18
pedroalvarezstraycat: because of the glibc change. I sent a patch last week, not sure if it can be accepted14:18
straycatjjardon, that's awesome thanks14:18
pedroalvarezI'll take a look at the discussion14:18
*** aananth [~caananth@74.112.167.117] has quit [Quit: Leaving]14:23
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has quit [Ping timeout: 264 seconds]15:02
*** genii [~quassel@ubuntu/member/genii] has joined #baserock15:03
pedroalvarezfranred, ssam2, how are you today? are you still ok of having a quick meeting here to talk about the present of our infra? Can we do it at 16:00 utc?15:03
pedroalvarezs/of having/with having/15:04
jmacsIs there anything better for making Baserock videos with than kdenlive?15:04
pedroalvarezkdenlive? is that a tool for recording>15:05
pedroalvarez?15:05
DavePagekdenlive is a tool for video editing15:05
DavePageActually it's a tool for eating all your RAM and crashing but hey15:05
jmacsYes, it's a video editor that crashes all the time15:06
jmacsHence me asking if there's anything better15:06
DavePagejmacs: Not taht I've found. pitivi eats all your RAM and hangs rather than crashing, but that's not a great improvement.15:07
franredpedroalvarez, I can do at 16:00 utc15:08
ssam2pedroalvarez: sure15:13
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has joined #baserock15:17
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has quit [Ping timeout: 250 seconds]15:36
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has joined #baserock15:46
pedroalvarezReminder: Baserock Infrastructure meeting in 12 minutes15:48
*** petefoth [~petefoth@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: petefoth]15:48
paulsher1oodpedroalvarez: what is that, where?15:52
pedroalvarezhere15:52
paulsher1oodooh, cool :)15:52
pedroalvarezyou are welcome to join if you have points to raise15:53
pedroalvarezeverybody is15:53
pedroalvarezbut the meeting is to talk about the present, not the future15:53
paulsher1oodSotK: did you actually put code or an instance of your stuff somewher?15:53
paulsher1oodpedroalvarez: is there an agenda? sorry - i've been in my own little world15:54
* paulsher1ood guesses this may all be on a trello board somewhere :)15:54
pedroalvarezactually there is not an agenda, and is not in the trello board15:55
pedroalvarezand I guess I should do it the next time15:55
SotKpaulsher1ood: I put the code for my prototype stuff here: http://git.baserock.org/cgi-bin/cgit.cgi/delta/openstack/turbo-hipster.git/log/?h=baserock/adamcoldrick/mason-plugin-prototype15:55
pedroalvarezis the first time I want to do a meeting on IRC on an opensource project :)15:56
SotKpaulsher1ood: there isn't an instance of it running somewhere accessible yet though15:56
pedroalvarezInfrastructure meeting in 1 minute, please don't interrupt the meeting if is not related.15:59
pedroalvarezall right, Baserock Infrastucture meeting starts16:01
pedroalvarezfranred, ssam2, are you around?16:01
ssam2hi16:01
franredhi16:01
pedroalvarezanybody else wants to join?16:01
paulsher1oodhi16:01
straycateh? wha?16:02
pedroalvarezOk, I'd like to discuss the following points:16:02
* straycat hides16:02
pedroalvarez* [1] Migration of our Havana instances to Icehouse.16:02
pedroalvarez* [2] baserock-clone and cache.baserock.org.16:02
pedroalvarez* [3] git.baserock.org. Datacentred Migration.16:02
pedroalvarez* [4] Prioritizing the work and check who has time to start with it.16:02
pedroalvarezDoes any of you have more points to discuss?16:02
ssam2a plan for setting up future infrastructure16:03
ssam2basically to see if you guys like the approach I've been taking in https://github.com/ssssam/test-baserock-infrastructure and want to adopt it or not16:03
pedroalvarezcool, that we the [5]16:03
pedroalvarezMigration of our Havana instances to Icehouse.16:04
ssam2is there an easy way to do that?16:04
pedroalvarezI've been talking with the datacentred guys and we should start moving our infra to the new tenant that we have16:04
ssam2actually, I know using 'nova' you can download an image16:05
pedroalvarezthe way to do it I believe is: create a snapshot, download it, and upload it to the other tenant16:05
ssam2right16:05
ssam2one issue is that all the public IPs we've been using need to change16:05
ssam2and we have less floating IPs available in the new tenant16:05
ssam2I looked on Friday at how to set up HAProxy, and I think it'd make sense to use that16:06
paulsher1oodcan't the download/upload be somewhere at dc, to save time?16:06
paulsher1ood+1 for haproxy16:06
pedroalvarezpaulsher1ood: yes, we should do the migration from dc to dc16:06
ssam2paulsher1ood: good idea, better than downloading them to the office!16:06
pedroalvarezssam2: do you think that 10 Ips is not enough for now?16:06
ssam2pedroalvarez: we're using 12 right now :)16:07
ssam2I think setting up HAProxy will be simple enough, and we can then point all our subdomains at one IP and just update the HAProxy config when we want to add/remove/change pieces of infrastructure16:07
ssam2it can forward requests to the correct instance based on subdomain or even matching bits of the URL16:07
pedroalvarezsounds like the right thing to do16:08
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]16:08
paulsher1oodhaproxy on a baserock system should be easy enough... has anyone done that?16:08
ssam2I've been avoiding using Baserock for the infrastructure so far, because all this infrastructure work is unfamiliar territory for me anyway16:08
paulsher1oodnod16:08
ssam2I'd like to get it working first, then start moving it to Baserock16:09
pedroalvarezI agree16:09
ssam2I can finish setting up HAproxy on Friday, or maybe earlier16:09
radiofreestraycat: yes i did, for now set mouse-=a16:09
franredcan not we do some networking trick in neutron to use NAT?16:10
pedroalvarezfranred: not sure about that, want to research?16:10
franredI can have a look yes16:10
ssam2so the main things to migrate are:16:11
ssam2paste. and testirclogs.16:11
radiofreewhat would be the correct patch for that vim thing?16:11
ssam2and the mason-x86-64 and its trove16:11
pedroalvarezradiofree: please, can you wait until the meeting ends?16:11
pedroalvarezssam2: yes16:11
radiofreeoh sorry i had no idea!16:11
pedroalvarezradiofree: np16:11
pedroalvarezdo we agree that we can start moving these things asap?16:11
ssam2could we just redeploy the Mason and the Trove ? should be easy enough16:11
ssam2and just move the Trove's volume to the new instance?16:12
pedroalvarezredeploying them makes sense16:12
ssam2I think we agree that we caen start moving them ASAP, and I can look at doing some of the work on Friday16:12
pedroalvarezabout the trove, it was my second bullet point16:12
ssam2I'll leave paste. and testirclogs. to you as you deployed them originally16:12
pedroalvarezgood16:12
franredssam2, we will have to copy the volume so we can not avoid to create an snapshot of it?16:12
ssam2franred: i'm not sure if a snapshot of an instance includes any volumes, to be honest16:12
ssam2i'd have thought that the volume needs to be migrated separately anyway? or am I wrong?16:13
pedroalvarezI think you are right16:13
ssam2i'm not sure if we can redeploy the Trove and reuse the volume. I think we can if we have the original cluster morph for the trove kept somewhere16:13
franredssam2, yes, you are right, but could the volumes be copied or we need to create an snapshot of them?16:13
pedroalvarezI think we should move to my second point now16:14
pedroalvarez[2] baserock-clone and cache.baserock.org.16:14
pedroalvarezshould it be the same thing? Do we still want baserock-clone? 16:14
ssam2baserock-clone is useful I think, but not valuable16:14
ssam2i.e. it doesn't matter if there's downtime16:14
pedroalvarezIt is good to use the jetsons we have in DC16:14
ssam2oh, that's true16:15
franredI use baserock-clone for my testing and I still cloning  repos  from there16:15
pedroalvarezI've been thinking about that, because I want to create a mason instance to test things in armv7lhf, and I found that this mason will be different than the others16:16
franredI think we should use as a test baserock-lorry and use it in the new instance too, but not sure if someone is using it at me moment16:16
pedroalvarezhm.. I'm still unsure, a trove just to test lorries sounds like too much16:16
pedroalvarezbut, I think we need it anyway to use the jetsons16:17
paulsher1oodi thought we'd end up with lorry in devel so users could test themselves?16:17
franredwell, problem that we have is, that if we want a clean g.b.o or a g.b.o with mess16:17
ssam2paulsher1ood: i hope that happens, yeah16:17
paulsher1oodclean gbo16:17
pedroalvarezfranred: but not having baserock-clone doesn't mean that we will have a g.b.o with mess16:18
franredbut we don't have the lorry in devel, so for the moment have a test-trove is the clean solution, I though16:18
paulsher1oodpedroalvarez: we should look closer at what SotK is proposing for maason i think16:18
pedroalvarezpaulsher1ood: true, but I want discuss the present in this meeting, not the future16:18
pedroalvarezonce we have sorted out our current situation we can start looking forward16:19
ssam2we need baserock-clone anyway for Jetsons, so I think we are decided that we will keep it and migrate it as soon as possible, which may lead to some downtime for it16:19
paulsher1oodpedroalvarez: ok16:19
franredssam2, ok16:20
pedroalvarezthen, I think that cache.baserock.org should be baserock-clone16:20
paulsher1ood+116:20
ssam2that works for now16:20
pedroalvarezcool16:20
franredfor now, ok16:20
paulsher1oodcan we rename baserock-clone to something better?16:20
ssam2in fact, that works forever as long as all our masons upload artifacts to baserock-clone16:20
pedroalvarezpaulsher1ood: yes, any suggestion?16:20
paulsher1ood(mirror?)16:20
ssam2point 3 is about migrating git.baserock.org to datacentred, so perhaps in the end we rename it to 'git.baserock.org' :)16:21
pedroalvarezmirror as trove-id?16:21
paulsher1ood:)16:21
paulsher1oodgbo-mirror as trove id perhaps16:21
paulsher1oodcan be discussed out of meeting16:21
pedroalvarezworks for me16:21
pedroalvarezfair16:21
pedroalvarezssam2: is a possibility16:22
richard_mawunless baserock-clone also has copies of the lorry state, it's not valid to just rename it to git.baserock.org16:22
pedroalvarezrichard_maw: yeah sure, I know that16:22
ssam2richard_maw: that's true. I just mean that eventually we might not need baserock-clone to exist16:22
ssam2I guess baserock-clone has multiple roles and it's hard to pick a name that reflects all of them16:23
richard_mawfair enough, I just dipped into the conversation and was concerned that things would just be renamed16:23
ssam2the instance is currently named 'mason-artifact-cache-server-plus-git.baserock.org-mirror' :)16:23
pedroalvarezanything else to raise regarding this point?16:23
ssam2how about baserock-2 ?16:23
ssam2i'll stop, we can discuss that outside this meeting16:24
pedroalvarez[3] git.baserock.org. Datacentred Migration.16:24
pedroalvarezI just wanted to raise this point, we should move it to DC16:24
ssam2this reminds me of a 6th point we should discuss: backups16:24
pedroalvarezbut I think we should move other things first so we can test  it16:24
pedroalvarezssam2: point added16:25
ssam2we need a lot more capacity for volumes before we can migrate g.b.o, too16:25
ssam2it won't fit in 200GB16:25
paulsher1oodi can ask16:25
pedroalvarezssam2: it will, but we need also space for cache.b.o16:26
DavePageIs that because g.b.o serves many functions?16:26
DavePageIt might be worth trying to split that out as part of the migration.16:26
pedroalvarezyeah, splitting the cache is being part of the migration16:26
ssam2just VCS imports and hosting is still going to take up a lot of space16:26
ssam2it's currently 115GB of artifacts + Gits16:26
ssam2but we will keep adding more stuff, so we'll hit 200GB soon enough16:27
pedroalvarezpaulsher1ood: I'll appreciate that16:27
pedroalvarezok, so this migration is going to be more complex and it has to wait until we have resources and we have tested DC16:27
pedroalvarezand we have a plan for backups16:28
ssam2also, g.b.o is working fine, unlike e.g. Storyboard :)16:28
pedroalvareztrue, but I know that DavePage is not confident about its security situation16:28
DavePageWell, to be specific I'm not confident about the host it's running on either :)16:29
ssam2why? is that not more to do with the lack of a formal security process for the OS it runs, than which VM hosting service it's hosted on?16:29
DavePageFor starters I could do with rebooting the VM host for a kernel security update. For another thing the host is running kvm/qemu with no security support.16:30
franredwhat is the difference between the host is running now and the one it will be runnig in DC?16:30
* paulsher1ood wonders about the expected duration of this meeting16:30
ssam2ok, so it would be good to migrate anyway16:30
pedroalvarezpaulsher1ood: I expect we can finish it in 10 minutes16:30
DavePagefranred: One is my problem, the other is not ;)16:31
pedroalvarezshould we move to [4] Prioritizing the work and check who has time to start with it.?16:31
ssam2yes16:31
ssam2we need a todo list for this, I guess16:32
ssam2should we use the existing Trello for now ? or a wiki page with a list of tasks ?16:32
paulsher1ood+1 for wiki :)16:32
pedroalvarezI was going to say trello for simplicity, but ok wiki16:33
pedroalvarezthings to do:16:33
paulsher1oodi'll drop my +1 if others prefer16:33
pedroalvarez* Migrate irclogs16:33
pedroalvarez* Migrate paste.baserock16:33
pedroalvarez* Migrate mason and trove16:33
ssam2I guess mason and trove can be done independently, which makes it slightly less daunting16:34
ssam2just need to update mason with the new IP of the trove16:34
pedroalvareztrue16:34
pedroalvarezI guess I'm missing things16:35
ssam2i still plan to set up HAProxy, an OpenID provider, and Storyboard16:35
ssam2in that order16:35
pedroalvarezright, I can do the paste.baserock, and the mason migration tomorrow16:35
pedroalvarezand irc logs I guess16:35
pedroalvarezand with this we can move to [5] plan for setting up future infrastructure16:36
franredpedroalvarez, I can give you a hand16:36
pedroalvarezfranred: thanks16:36
ssam2I can look at the Trove then16:36
ssam2will be on Friday16:36
pedroalvarezssam2: cool16:36
pedroalvarez[5] I think we should follow what ssam2 is doing to setup the infra16:37
pedroalvarezhttps://github.com/ssssam/test-baserock-infrastructure16:37
pedroalvarezI'll bear this in mind when doing the migration16:37
ssam2my idea is that Packer maps reasonably closely to 'morph deploy', so there should be fairly clear migration paths when moving stuff to Baserock16:38
pedroalvarezmakes sense16:38
ssam2we should decide where that repo lives permanently, then. On g.b.o makes sense except then everyone will be mirroring it16:39
ssam2but I suppose it's only ever going to be small16:39
pedroalvarezI think that for now that location is ok16:39
ssam2ok16:39
pedroalvarezshould we move then?16:39
ssam2move on to [6]? ok16:40
pedroalvarezbackups in DC16:40
pedroalvarezI don't know anything about the possibilities yet, but yes, we should ask, get information about our current backups plan and decide what are we going to use in DC16:40
pedroalvarezi volunteer to do that16:40
ssam2I was thinking for future infrastructure we should set up one database server shared by all the infrastructure16:41
ssam2Storyboard seems to need MySQL so I guess it'll have to be a MySQL server16:41
ssam2then Gerrit, Storyboard and whatever else can use that and we only need to back that up16:41
ssam2everything else can be redeployed16:41
* pedroalvarez nods16:41
ssam2we should also find out what DC's backup policy is16:41
paulsher1oodi wonder if we've properly established why SB needs mysql. they used to support pg - maybe we could re-animate that16:42
ssam2hopefully they can take care of physical backups, and we just need to worry about the logical (database) backup16:42
ssam2paulsher1ood: I've not investigated why, I'll ask them16:42
franredssam2, sounds like a good plan16:42
pedroalvarezok, anything else?16:42
pedroalvarezI think this is more than enought to start :)16:43
ssam2thanks for running the meeting Pedro16:43
pedroalvarezI declare this meeting finished16:43
straycatirc meetings are cool16:43
* paulsher1ood notes that pg works on baserock out of the box...mysql is more work16:43
franredthanks Pedro :)16:43
straycatradiofree, not sure, can't it be a patch to vim?16:44
ssam2straycat: my poor eyes beg to differ16:44
straycatpedroalvarez, so what's the deal with tar?16:46
pedroalvarezstraycat: I believe it can be merged :)16:47
straycat"it" ?16:48
ssam2paulsherwood: sounds non-trivial to use PostgreSQL for Storyboard --all the migrations are MySQL-specific16:48
ssam2there are about 30 migrations and we'd need to fix and maintain them and any future ones16:48
wdutch:w16:49
straycatNo file name16:50
*** CTtpollard [~tom@82-70-136-246.dsl.in-addr.zen.co.uk] has joined #baserock16:50
paulsher1oodurgh :/16:50
ssam2all of https://github.com/openstack-infra/storyboard/tree/master/storyboard/db/migration/alembic_migrations/versions16:51
straycatf16:51
straycatheh >.>16:51
ssam2there doesn't seem to be much MySQL specific stuff in there, actually16:51
pedroalvarezstraycat: "it" = the patch I sent to fix tar16:52
rdalehaving somethink like MYSQL_ENGINE = 'InnoDB' in a database migration seems a bit broken to me16:54
paulsher1ood:-)16:57
straycatpedroalvarez, is this with upstream?16:59
pedroalvarezstraycat: no, I just had to  upgrade it to a newer version glibc compatible17:00
pedroalvarezI'll merge it soon17:00
straycatwhy can't we merge it now?17:01
pedroalvarezWe can, I'm just in the middle of something. If you want to merge it, i'll appreciate it :)17:01
radiofreestraycat: i was thinking it would be easier to just create a /root/.vimrc file with "set mouse-=a" and install that in the chunk17:04
radiofreerather than having to modify the vim source code17:04
radiofrees/source code/repo17:04
straycatpedroalvarez, it has two +1s i'm fine merging it if all we're doing is effictively disabling -werror17:04
richard_mawradiofree: I don't think vim as root allows .vimrc17:04
richard_mawsecurity reasons17:04
pedroalvarezstraycat: please :)17:05
richard_mawthough I may be mixing that up with the `vim: foo` lines17:05
ssam2richard_maw: I use a .vimrc in Baserock all the time, so it must work as root17:05
radiofreerichard_maw: works here17:06
paulsher1oodSotK: is there some reason your patches for mason are authored by 'Mason Test Runner'?17:06
pedroalvarezstraycat: and also you have my +1 to move webtools to devel, although I prefer if you send a patch to see if anybody disagrees17:06
straycathave a system integration thing that modified the /etc/vimrc ?17:06
straycat*s17:06
pedroalvarezs/move /include in/g17:07
* richard_maw has come to the conclusion that if we can't do atomic runtime fs updates, then we can't do runtime updates at all, as the real value in package-based distributions is that it encodes the logic to safely remove a bunch of files from the filesystem. delta-based application can result in the filesystem being in states not viable for running applications17:07
*** wdutch [~william@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Quit]17:07
richard_mawthat's the real value of packages17:08
richard_mawbut if you can do an atomic fs update then you don't need them17:08
robtaylorprojectatomic17:08
robtaylorbut I still want to do apps with baserock, but that's something rather different17:09
ssam2robtaylor: does OSTree pivot OS version without rebooting?17:09
richard_mawaccording to the docs I could find it runs `systemctl reboot`17:10
ssam2I had a feeling it requires a reboot to upgrade, but I might be wrong17:10
robtaylorssam2: yep, in project atomic you updates17:10
robtaylorumm bad paste17:10
robtayloryou systemctl reboot17:11
ssam2right. So it provides a similar thing to what we currently have with Btrfs subvolumes (except with the implementation in userspace instead of in the kernel)17:11
robtayloryep17:11
* richard_maw needs runtime atomic updates and has been informed that containerising the applications is not an option17:12
richard_mawit's doable with clever use of pivot_root17:12
robtaylorrichard_maw: is it the applications that need updates or the whole system?17:12
richard_mawboth17:13
robtaylorwhole system updates without restart? ouch17:13
DavePagekexec? :)17:14
richard_mawif init or the kernel changes it's permissible to kexec, but for everything else I need it to stay up17:14
robtayloryep, pivot root is your approach there, and i guess its up to the sysadmin to figure out ehen they need to reboot17:14
robtaylorall very old school17:15
paulsher1oodreally? that sounds hard :-) would super-fast boot not be an option?17:15
robtaylorpaulsher1ood: that would be the modern container-oriented way, indeed17:15
richard_mawpaulsher1ood: not possible with the class of hardware involved17:15
*** franred [~franred@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:16
richard_mawkexec is the fastest reboot semantics possible, and I hope the hardware supports kexecing17:16
robtaylormm. sounds big irony ;)17:16
richard_mawI couldn't possibly comment.17:16
robtaylorrichard_maw: hah, just had an evil thought17:17
richard_mawrobtaylor: do tell >:-D17:17
robtaylorrichard_maw: you could always boot systems in a pid namespace. Then you can boot up a new system, and use gcroups to close down the old system when its stopped doing things17:18
robtaylori guess you could end up with some weird behaviour for apps that think they're the only people talking to their store17:19
robtaylorhell, just always boot in a container =)17:19
* paulsher1ood likes it17:19
richard_mawrobtaylor: I've been informed that containers impose too much overhead17:19
robtaylorprobably could just add support to systemd to do this17:19
paulsher1oodeven better :-)17:20
robtaylorrichard_maw: um, they don't know what they're talking about then17:20
paulsher1oodrobtaylor: careful... :)17:20
robtaylorrichard_maw: there's only overhead when you use VETH devices to bridge network namspaces17:20
robtaylorhmm, which you would probably have to do in this model17:21
richard_mawrobtaylor: that was my first thought, but it means that there's more copies of binaries and libraries around, which means there's more pressure on the caches, so things keep dropping out of it all the time17:21
robtaylornot if you do things right17:21
robtaylorthe pressure on the caches will be the same for the pivot_root approach, actually17:21
robtayloryou'll have the old sharedlibraries and executables mmaped in when you load the new system,17:22
robtaylorand no managed way to get rid of them17:22
robtaylorif you containerise, you can add systemd commands to query the state17:22
robtaylorhmm17:22
robtaylormaybe just use a cgroup17:22
richard_mawrobtaylor: not exactly. With the container approach you need to keep the outer system's libraries pristine when you update the inner one. But with the pivot root, you re-exec _all_ your binaries in the new version17:23
richard_mawso there's no processes left using the old binaries17:23
robtaylorthat's just a reboot17:23
richard_mawif you do it right, it's a reboot with no service interruption, since you can have processes gracefully re-exec and keep the connections open17:24
richard_maw`systemctl daemon-reexec` is an example of this17:24
robtaylorrighty17:24
robtaylorso you kinda want to do a full-system daemon-rexec?17:24
richard_mawyeah17:24
richard_mawafter migrating all the processes to a new mount tree17:25
robtaylorit would then make a lot of sense to put everything in a cgroup, and start a new gcroup when you reexec17:25
robtaylorthen you can easily track what hasn't restarted17:25
robtaylor(and warn/debug if its haveing problems)17:25
robtaylordoes that make sense?17:26
richard_mawinteresting idea, and if it maintains the systemd state you can track it back to the service, and have a nice interface to be able to tell the service to gracefully re-exec before forcing it to17:27
robtayloryep17:27
*** tiagogomes [~tiagogome@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:27
robtaylor(can you tell i've recently spent a lot of time trying to understand containers? ;) ;))17:27
richard_mawfirst class support for this in systemd would be the best place to put this pivot and gracefully re-exec logic I think17:28
*** jonathanmaw [~jonathanm@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:28
robtayloryep, i expect it'll be gladly accepted and walters will love you17:28
richard_mawcurrently there's not a race-free way of doing this, unless you can freeze all the processes17:28
robtaylorwell, you can17:28
robtayloryou can freeze a cgroup17:28
robtaylorand be told when everything is quiesent17:29
robtaylorhttps://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt17:29
robtaylorbut if everything providing a service is a well behaved socket-activated unit, you could totally do it race free17:30
robtaylorold active connections carry on using the old service, new connects startup and use the new service17:31
richard_mawnice17:31
robtaylorand you use the freezer subsystem to tell when the old services have finished17:31
robtaylorhmm, maybe17:33
robtayloryou may need some more than freezer17:34
richard_mawso you'd only worry about ensuring systemd is properly migrated, and over time the system eventually migrates itself17:34
robtayloroh17:34
robtayloruse notify_on_release rather than freezer17:34
robtaylorand maybe have a way to manually freeze the old system if its being bad about restarting17:35
robtaylor(or maybe some units need some special handling)17:35
richard_mawif I'm understanding your freezing suggestion correctly, then if it behaves that badly, then you're better off killing it and accepting that some connections get dropped17:36
robtayloras long as you can a) tell what is actiually happening easily and b) tell stuff to go away if its being an arse and c) rollback if its not working17:36
robtaylorrichard_maw: yeah, probably17:36
robtaylorrichard_maw: i think the freezer may be useful in terms of checkpointing17:36
richard_mawah, so you're suggesting _process_ rollback may be possible?17:36
robtaylorwith freezer you can rollback a cgroup17:36
* robtaylor suddenly feels very evil17:37
* richard_maw feels like he's a child, it's christmas and he's been given lots of new toys, some of which have warning stickers17:37
robtayloryou probably don't want that though, as you can only get a consistent checkpoint by forcing a cgroup to be queisecnt17:38
DavePage"Some of these are toys, some of them are beartraps that will take off your hand. Have fun!"17:38
robtaylorrichard_maw: you probably want to take a look at CRIU17:39
robtaylorDavePage: about right17:39
robtaylorhere be shiny shiny dragons17:39
richard_mawrobtaylor: probably, I had assumed it required that checkpointed processes need to be restored in an identical filesystem, but I could be wrong17:41
robtaylorrichard_maw: well you can do that with btrfs17:41
robtaylorrichard_maw: you can even send/receive a given state to a new system17:42
* robtaylor all kinds of evil17:42
richard_mawyes, but I don't want to restore it in an identical filesystem, I want to restore it in a different filesystem because it contains updatees17:42
robtayloroh, yeah, i'm not suggetsing you do that17:42
robtaylorthat would be bad17:42
robtaylorjust checkpointing for rollback on failure17:42
robtaylorso new system blows up, freeze that, make a chroot with your old snapshot and restart your old cgroup17:43
robtaylorand then kill off the borken new cgroup17:44
richard_mawthat's assuming the services are continuing then re-execing, rather than us freezing the old ones and starting new versions17:44
robtaylor(if that makes sense)17:44
richard_mawI'm assuming that checkpointing isn't quick, because there's a lot of process data that needs to be serialised17:44
robtaylordepends, when you snapshot and send/receive17:45
robtaylorbut i'm suggesting this sequence -> 1) start up new system in a new cgroup, handnd over as per daemon reexec. 2) wait for all the old system to stop doing stuff 3) snaphot 4) kill it17:46
robtaylorif that makes sense17:46
robtaylor2) is the hardest bit. I don't think you can assume the services will exec, you'll just have to montor the cgroup17:47
robtaylors/exec/exit/17:47
richard_mawpivot_root has some limitations a) it works per-namespace, so I'd need to pivot in each namespace, rather than all at once. b) chrooted processes don't get thier root changed c) unless the working directory of the process is /, it also won't be chdir'd. b) and c) can be dealt with by chrooting before pivoting, but without a pivot_root that can take fds, you can't always refer to the mount_points17:47
richard_maws/mount_points/new mount point/17:48
richard_mawplus, openat means old processes can still see the old state until they re-open those files17:48
richard_mawbut that was from when I was thinking I needed to do some magic without cooperating from systemd17:49
robtaylorin this sceme you don't really pivot root, you're really doing soemthing like nspawn --share-system17:49
robtaylorhmm, you may also want to worry about sytemd upgrades =)17:50
richard_mawnot exactly though, as I need systemd to also do the transition to the new system, which may require systemd being backwards compatible with its serialised state when doing a `systemctl daemon-reexec`17:50
robtayloryep17:50
robtaylorthat sounds about right17:50
richard_mawdaemon-reexec is only supported for being able to reload the libraries it depends on and re-execing for debugging17:51
richard_mawre-execing for debugging is basically just so you can compile a version with print statements in17:51
robtaylorinteresting http://www.freedesktop.org/wiki/Software/systemd/SystemUpdates/17:51
robtaylor(not what you want but a little informative)17:52
richard_mawit appears to miss the point for me, as it's doing offline update when you have packages available, when one of the advantages of packages is that you can do online updates17:53
robtaylorhmm, actually does any state really need to be passed?17:53
robtaylorbetween new and old sytemd?17:53
robtaylorits just really atomic socket activation handover17:54
richard_mawonly if we're allowing services to see the new version of the system, which may be allowable, since it's likely to work, as you see the same transition with non-atomic updates17:54
pedroalvarezjjardon: I still want to know how critical is the bug you have found with glibc and tzdata.. :P17:54
richard_mawas services may be used to being still alive when packages are being updated17:55
robtayloryeah, anything that breaks in this model would have broken before17:55
robtaylorindeed, astually the model you're replacing never had atomic handover17:55
robtayloryou'd always have downtime unless your service specifiucally had graceful restart17:56
robtaylor(e.g. apache2)17:56
richard_mawalso irssi according to one source :-)17:56
richard_mawyou run /update or something17:56
richard_mawah /upgrade17:57
robtaylorso you just need a way to say how to graceful in the unit, and if that isn't there, you stop and restart17:57
robtaylorand of course, that's already there17:57
robtaylorExecReload=/usr/sbin/apache2 -k graceful $APACHE2_OPTS17:57
richard_mawI thought reload was just for config17:58
*** ssam2 [~ssam2@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Leaving]17:59
robtaylormm, yes, you'd probaly need a new unit line17:59
*** mariaderidder [~maria@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Ex-Chat]18:00
richard_mawso, we want graceful restarts for everything, but systemd can make that easier to implement for services by them just having a graceful shutdown and being socket activated18:00
robtaylorumm, well i'm saying you proably can't have graceful restarts for everthing18:01
robtaylorbut you can where services *support* graceful restarts18:01
robtaylorand that can be indicated in the unit18:01
robtaylorif they don't, you just stop and start them18:02
robtaylorand you';ll be at parity with current systems, but a lot more controlled18:03
*** CTtpollard [~tom@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Quit: Ex-Chat]18:03
richard_mawif we had a magic pivot_root that rebased all the paths in all the processes (cwd, root and all open fds) we could get away without special logic in systemd, but it sounds like systemd support would be easier to achieve18:05
robtaylori think bad things would probably happen if you did that18:06
robtaylorimaging not all your shared library is paged in18:07
robtaylor(or your mmaped data set)18:07
*** Krin [~mikesmith@82-70-136-246.dsl.in-addr.zen.co.uk] has quit [Remote host closed the connection]18:07
richard_mawhm, so not all fds, but fds referring to directories might be ok18:08
robtaylor(actually shared library is probably a red herring, as that'll all be in ram anyhow)18:08
robtaylorit'd be certainly very tricksy and probably hit a lot of unexercised code paths18:09
richard_mawthe processes are going to be gracefully restarted soon anyway, so keeping files open wouldn't be too problematic18:09
jjardonpedroalvarez: sorry, no time to investigate now, I will take a look when arrive home18:09
robtaylorsystemd approach would of course be much cooler and give you more kudos ;)18:09
richard_mawrobtaylor: yeah, and it's unlikely to be accepted as a kernel patch, since it feels like pivot_root is probably already a security vulnerability waiting to happen18:10
robtayloryep18:10
robtaylorjust realised one issue - actaully nothing will support graceful restarts on this kind of system out of the box18:11
richard_mawI'll probably still have to knock up a racy prototype, just to show that it's possible, but the systemd root looks plausible18:11
robtaylorif you apachctl --graceful, its expecting to shutdown and do the restart18:11
richard_mawrobtaylor: oh?18:11
robtaylorwhere as we want to have two halves18:11
robtaylor--graceful-shutdown and --graceful-restart18:12
richard_mawI suppose ExecStop is what you want for graceful shutdown anyway18:12
robtaylorgraceful-stop18:13
robtayloryep18:13
robtaylori was just thinking you could basically just get the old sysytemd to command the new systemd18:13
richard_mawyou need systemd to re-exec itself and be PID1 though18:14
robtaylorbindmount the new sytemd's socket from the new ns into the host18:14
robtayloruse pid namespace18:14
robtaylor(and user namespace)18:14
richard_mawcan the parent pid and user namespace go away when that happens?18:15
robtaylori'm still interested to hear why someone thinks containers have a perfomance impact and where they think that performance impact is18:15
robtaylorrichard_maw: i think you'd always keep real uid0 and pid 1 pristine18:15
robtaylorhmm18:16
richard_mawthis would have been worth discussing at the Linux Plumbers conference18:17
robtaylorwe wouldn't have thought of it then18:18
robtaylori'm sure i could arrange some beers with lennart though18:18
robtaylorprobably after a poc ;)18:18
richard_mawpoc?18:18
robtaylorproof of concept18:18
richard_mawyeah, as I said, I'll need to make one anyway for the deadline of my current chunk of work18:19
robtaylorcool18:19
robtaylormaybe i can finish my app sandboxing stuff in the same timeline ;)18:20
richard_mawit's likely to be racy as hell if I want to be able to change everything of importance18:20
robtaylorwell, you can probably just make everything actually quiesce18:21
robtaylorthe tricky bit will be handing over the sockets between the systemds18:21
richard_mawbut if we can get online atomic updates going, then it could be the nail in the coffin for packages18:22
robtayloryep, this would get widely used, i'm sure18:22
robtaylorit solves the real problem you always had with upgrades, old cr*p hanging around and you haveing no realy way to know what was what18:22
richard_mawplus you don't need the complication of packages needing to leave the system in a runnable state after every installation or removal, you can get away with just applying a delta18:23
robtaylormaybe a first poc would be best done with just working within the same systemd instance18:23
richard_mawyeah, pivot_root in all mount namespaces that are just for private mounts, rather than full containers (shared PID namespace probably)18:25
robtaylori'd just use nspawn tbh18:26
robtaylorhmm18:26
* robtaylor is running out of brain. maybe we could pick this up again tomorrow18:27
robtaylorone thng. If the conatiner concern is just cache pressure, you can easily still use containers in this scheme but mitigate that concern18:29
robtayloryou have a top level (real) pid 1 thats a systemd. The 'current system' would be a container under that systems, as would be your 'new system'18:30
richard_mawprobably, I've got a planning meeting to decide who's doing what to get us closer to the proof of concept level for a whole bunch of stuff, but if I immediately start on picking through the atomic online update stuff a face to face chat about this would probably be of immense value18:30
robtaylorI could probably do a face to face tomorrow if you'd like18:31
robtayloroh no i can't , i won't be in mcr18:31
robtaylorcan do a call/online whiteboard if you'd like18:31
robtayloranyhow, lets catch up tomorrow here first ;)18:32
richard_mawsure18:32
* robtaylor drives home18:32
* richard_maw would still like a variant of pivot_root that took fds and had flags for future expansion18:34
* richard_maw is a little amused that he might be able to call himself a Linux Plumber in the future18:37
dabukalamrichard_maw: Is a linux plumber anyone that's committed code to the kernel?18:52
*** cosm [~Unknown@host-78-150-56-250.as13285.net] has quit [Ping timeout: 265 seconds]19:02
*** cosm [~Unknown@host-78-150-56-250.as13285.net] has joined #baserock19:02
*** cosm [~Unknown@host-78-150-56-250.as13285.net] has quit [Ping timeout: 264 seconds]19:19
*** rdale [~quassel@9.Red-83-45-185.dynamicIP.rima-tde.net] has quit [Ping timeout: 256 seconds]19:40
*** cosm [~Unknown@cspc154.cs.man.ac.uk] has joined #baserock19:54
*** zoli_ [~zoli_@linaro/zoli] has joined #baserock19:56
*** zoli_ [~zoli_@linaro/zoli] has quit [Remote host closed the connection]19:58
* jjardon merges the systemd 217 branch \o/21:47
paulsher1oodw00t! :)21:50
* paulsher1ood kicks off a build21:50
robtaylordabukalam: linux 'plumbing' is the lower levels of userspace and the user land interfaces of the kernel21:52
robtaylordabukalam: linux plumbers is a confernce for this http://www.linuxplumbersconf.org/21:53
jjardonpaulsher1ood: :) if you get bored: baserock/jjardon/gstreamer14 is available as well ;)22:00
* paulsher1ood stops his build, merges the above, and restarts :-)22:05
*** genii [~quassel@ubuntu/member/genii] has quit [Read error: Connection reset by peer]22:15
cosm@robtaylor do you know what's the plan for the kernel hacking workshop at UoM?22:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!