IRC logs for #baserock for Friday, 2015-04-17

jjardonRelated: https://storyboard.baserock.org/#!/story/1900:07
*** zoli__ has quit IRC00:11
*** bashrc_ has quit IRC01:26
*** mauricemoss_ has quit IRC01:26
*** CTtpollard has quit IRC01:26
*** sambishop has quit IRC01:26
*** fay_ has quit IRC01:26
*** flatmush has quit IRC01:26
*** nowster has quit IRC01:26
*** tiagogomes_ has quit IRC01:26
*** paulw has quit IRC01:26
*** fay_ has joined #baserock01:26
*** bashrc_ has joined #baserock01:26
*** mauricemoss_ has joined #baserock01:26
*** paulw has joined #baserock01:27
*** nowster has joined #baserock01:27
*** CTtpollard has joined #baserock01:27
*** franred has quit IRC01:27
*** sambishop has joined #baserock01:27
*** franred has joined #baserock01:27
*** flatmush has joined #baserock01:28
*** tiagogomes has joined #baserock01:28
*** zoli__ has joined #baserock04:27
*** zoli__ has quit IRC05:26
*** zoli__ has joined #baserock05:46
*** Albert_ has joined #baserock07:09
*** a1exhughe5 has joined #baserock07:16
*** mdunford has joined #baserock07:41
*** rdale has joined #baserock07:49
*** mariaderidder has joined #baserock08:00
*** jonathanmaw has joined #baserock08:12
KinnisonHmm, someone needs to update storyboard to use the new SSL cert08:13
*** edcragg has joined #baserock08:21
pedroalvarezyes, we should :)08:28
*** ssam2 has joined #baserock08:32
*** ChanServ sets mode: +v ssam208:32
*** gary_perkins has joined #baserock08:34
ssam2the root file system on my Jetson keeps becoming readonly for some reason08:56
ssam2dmesg is just full of errors from systemd-journald being unable to write to it, nothing useful08:56
ssam2rebooting seems to fix each time, but there's clearly a bug somewhere08:57
DavePageIs smartctl useful here? Could be a dodgy disk?09:03
rjekssam2: Can you disable systemd-journald from logging to dmesg, or increase the size of the dmesg buffer so you can see what was logged when the error occured?09:05
ssam2DavePage: the rootfs is a partition on the built-in flash MMC in the Jetson. I don't know if there's an equivalent to SMART for those09:09
ssam2rjek: that's a good idea09:09
rjekssam2: There are lots of reasons why a file system might get automatically remounted read-only, but it's normally an assertion failure or similar in the file system implementation.09:10
rjekWhich would suggest file system corruption, but this might be caused by a controller or drive failure, which would hopefully be also logged in dmesg allowing you to distinguish between a software or a hardware fault.09:10
ssam2[  730.708070] mmcblk0: timed out sending r/w cmd command, card status 0x90009:17
ssam2[  730.715427] blk_update_request: I/O error, dev mmcblk0, sector 244371209:17
ssam2[  730.722364] BTRFS: bdev /dev/mmcblk0p2 errs: wr 18, rd 0, flush 0, corrupt 0, gen 009:17
ssam2that doesn't look good09:17
*** lachlanmackenzie has joined #baserock09:17
tiagogomesmmm, how do I drop patches in gerrit?09:17
tiagogomesthat is, abandon09:17
ssam2there should be a button, to the right of the commit message09:18
*** Krin has joined #baserock09:19
tiagogomesI just see a reply button09:19
ssam2down a bit09:20
tiagogomesI was not signed in09:21
tiagogomesI found it09:21
dabukalamcan anyone tell me the exact date of the initial Baserock release? I haven't yet been able to fathom it from the mailing list or the git logs09:22
rjekdabukalam: https://wiki.baserock.org/releases/09:25
rjek"Rebel Mountain was released on the 10th July 2012"09:25
KinnisonAnd it was the first release09:25
KinnisonCf <20120710152037.GX31576@somnambulist.local>09:25
rjekI'm sure there was an Accelerated Vegetable release once, and that's not listed.09:25
richard_mawthe naming scheme was such that you could get a phrase by suffixing the name of one of the release artifacts09:25
dabukalamThe inital release was 2?09:25
richard_mawwe did internal releases first09:26
Kinnisonone internal release, then rebel mountain09:26
* richard_maw is just disappointed that he wasn't allowed to have a release called Buttery Biscuit09:26
dabukalamcool, thanks09:26
*** wdutch_ is now known as wdutch09:26
* rjek sneers at release names.09:26
richard_mawwas Secret Volcano the third release or the first internal one?09:27
rjek"Secret Volcano was released on the 20th August 2012"09:27
Kinnisonaccelerated vegetable was the first release09:27
Kinnison(internal)09:27
* Kinnison can find internal discussions from late may 2012 which suggest accelerated vegetable had been released before then internally09:28
* rjek wanted it to be Vegetable Accelerator, but apparently that didn't fit in with the selected scheme.09:28
radiofreessam2: i'm under the impression that's file system corruption09:28
radiofreei have a jetson here that does that occasionally (maybe after a day.. maybe after a few hours) after i did the "resize max"09:29
radiofreereboot fixes it, fsck.btrfs doesn't seem to do anything09:29
rjekradiofree: ssam2: A timeout sending a r/w command to an MMC block device sounds like a hardware problem, not a software one?09:29
radiofreerjek: i believe it is09:29
radiofree(hardware)09:29
rjek(Could be a /driver software/ problem, obv, but not a btrfs problem)09:29
rjekAh, misunderstood what you said, sorry09:30
ssam2right. remind me why I can't just cross-compile from my laptop again :(09:30
radiofreei don't think the emmc likes btrfs taking up everything09:30
rjekssam2: gobject-introspection! :)09:30
radiofreessam2: if you need to build i have a jetson here with cache from master the other day09:30
radiofreeit's cache from a devel-system + all of the weston image09:30
ssam2thanks, but I'm ok for builds right now09:31
franredgood morning guys, there are several patch-series that would be nice if someone can have a look at them today: http://paste.baserock.org/ojiyazuvuw09:54
straycatyes, buttery biscuit would have been good09:54
tiagogomespedroalvarez, jjardon can you please +1 again my Ironic changes again? I had to re-submit the changes because I wanted them merged to master instead of openstack-on-baserock and cherry-picking to another branch with Gerrit didn't work09:56
jjardontiagogomes: I was about to ask why did you abandon them :) let me take a look10:01
jjardontiagogomes: Did you make any changes?10:02
tiagogomesjjardon, I had to fix some conflicts. In openstack-no-configure, some code was moved into another section: https://gerrit.baserock.org/#/c/357/1/openstack-nova.configure instead of https://gerrit.baserock.org/#/c/337/1/openstack-nova.configure10:07
tiagogomesjjardon, that should be the only change10:07
*** mariaderidder has quit IRC10:27
*** mariaderidder has joined #baserock10:28
franredtiagogomes, could you rebase https://gerrit.baserock.org/#/c/356/ ? it is getting me a weird error when I've tried it10:38
tiagogomesfranred,10:39
tiagogomesfranred, sure10:39
tiagogomesthe rebase button didn't work for me though10:39
franredtry to rebase manually and send the 2 remaining patches again, I will merge them10:39
tiagogomes'kay10:40
tiagogomesfranred, can you try again10:54
franredtiagogomes, sure10:57
franredtiagogomes, ironic merged10:58
richard_maw\o/10:59
tiagogomesfranred \o/10:59
pedroalvarez\o/11:02
ssam2does anyone have an idea why a rootfs for Jetson might fail to mount an external SATA drive?11:07
ssam2a previous build on the same Jetson successfully mounts it11:08
ssam2kernels appear to be both 3.18.0, although I've not checked the exact commit SHA1s yet11:08
paulsherwood/etc/fstab ok?11:08
ssam2this is not the same Jetson that had rootfs corruption errors, by the way11:08
ssam2/etc/fstab is the same in both versions of /etc11:08
ssam2i've different the whole of /etc and not found much of interest11:08
ssam2I'm down to comparing SHA1s in /baserock/*.meta, but that will take a while...11:09
paulsherwoodssam2: if there's anything diff at all, please could you paste both?11:09
ssam2*diff'ed, not different11:09
ssam2sure11:09
ssam2http://sprunge.us/FRMB11:10
ssam2it's a bit of a dull read :)11:10
ssam2clearly systemd has changed, maybe that is the issue11:11
paulsherwoodyou think? :) :-)11:12
paulsherwoodssam2: i have a jetson you could experiment on if that would help?11:13
ssam2i've got 2 here11:13
paulsherwoodheh11:13
ssam2same issue with both11:14
paulsherwood:(11:15
ssam2i've now verified the kernel is identical in both systems, and will try reverting the last change to the systemd ref in master11:15
ssam2i wonder if the actual problem I'm trying to solve (.py files being overwritten with random hex strings on a Calxeda distbuild node) can be put down to systemd as well!11:16
paulsherwoodthat would be spooky11:17
DavePagepaulsherwood: First draft of a VET one-pager for Genivi at https://docs.google.com/document/d/107rS_hwlUOklxFHKgggM3zziHYIYxgs0D_8hwbfSkqE/edit12:21
DavePageOops, wrong window, sorry12:27
*** DavePage has left #baserock12:27
radiofreejonathanmaw: seems the demo platform uses a different weston?12:43
radiofreehttp://git.projects.genivi.org/?p=meta-genivi-demo.git;a=tree;f=recipes-graphics/wayland/weston;h=aa322319e3dd0b4cd00106703e71de6dd4a65c6f;hb=HEAD12:43
radiofreethere's things like http://git.projects.genivi.org/?p=meta-genivi-demo.git;a=blob;f=recipes-graphics/wayland/weston/browser_poc_hack.patch;h=8714db19c7a0a2e1a5fc1d109e0f849bccc1d7e6;hb=HEAD12:43
radiofreethough i have no idea how they turn http://git.projects.genivi.org/?p=meta-genivi-demo.git;a=blob;f=recipes-graphics/wayland/weston_1.5.0.bbappend;h=5f820b11e4b2344d4fbd9df5b1ab2801a794a5b4;hb=HEAD in to where to get the source from12:44
jonathanmawradiofree: my best guess is that it's overlaid on top of meta-ivi's weston_1.5.012:45
radiofreejonathanmaw: i *think* https://github.com/ntanibata/weston-ivi-shell/tree/genivi-1.3.0 + ivi-extension 1.3.012:48
radiofree+ ivi-shell-click-event.patch + 0001-Enable-disable-default-virtual-keyboard.patch on top of weston12:49
radiofreebut that's a total guess really12:49
radiofreebut they say they use ivi-extension 1.3.0, and the tarnyko branch was just weston 1.5.0 with whatever was around at the time from tanibata's repo patched on top12:50
radiofreethis is going off http://git.yoctoproject.org/cgit/cgit.cgi/meta-ivi/tree/meta-ivi/recipes-graphics/wayland/weston_1.5.0.bbappend12:50
radiofreei can't turn that into a source repo + branch12:50
radiofreethere's a comment at the top Branch: genivi-1.2.0, but there is no genivi-1.2.0 branch in that tarnyko branch12:52
radiofreecan anyone read yocto? do you know where the source for this is actually cloned from?12:52
radiofreejonathanmaw: http://git.yoctoproject.org/cgit/cgit.cgi/meta-ivi/commit/meta-ivi/recipes-graphics/wayland/weston_1.5.0.bbappend?id=0628e4d2a9bc07eb0c7cdc9eda5ba508ea1fb5ac12:57
radiofreei can't find "the one from yocto" though, probably just stock weston 1.5.0?12:58
Zararadiofree: I'm currently struggling with something similar. I thought the source just referred to the stuff in the 'weston' folder here, but maybe I'm misinterpreting the question13:00
radiofreeaha,"the one from yocto" is in poky13:00
radiofreejonathanmaw: it's just a release tarball, either 1.5.0 or 1.6.0 (depending on the version of poky they are using)13:01
radiofreehttp://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-graphics/wayland/weston_1.6.0.bb13:01
radiofreeZara: there's a SRC_URI for things13:02
radiofreebut in the meta-ivi case that's not in that file, it inherits it from the yocto release13:02
ssam2here's an exciting new bug (possibly caused by the systemd update) -- /etc/resolv.conf on a Baserock system lists 'nameserver 2001:4860:4860::8844' as its first entry, and DNS resolution doesn't work13:04
ssam2it lists 8.8.8.8 as its second entry, and can ping 8.8.8.813:04
ssam2and if I put 8.8.8.8 at the top, DNS resolution works13:04
ssam2this might be a Busybox limitation that we've not hit before13:05
bjdooksresolver libraries try hard to use entry #113:05
Zaraah, okay, glad I've been warned about that because I haven't come across it yet. (It took me a long enough to work out what was going on when the SRC_URI pointed to files in a directory in the same repo as the recipe...)13:06
ssam2bjdooks: to the extent of ignoring entry #2 completely? seems a bit dumb..13:06
ssam2not that dumb software should come as a surprise13:07
bjdooksit takes a long time to try #213:07
persiassam2: Can you ping 2001:4860:4860::8844?13:08
radiofreeZara: for http://git.projects.genivi.org/?p=meta-genivi-demo.git;a=blob;f=recipes-graphics/wayland/weston_1.5.0.bbappend;h=5f820b11e4b2344d4fbd9df5b1ab2801a794a5b4;hb=HEAD i think that inherits meta-ivi, which inherits yocto13:08
ssam2persia: PING 2001:4860:4860::8844 (2001:4860:4860::8844): 56 data bytes13:08
ssam2ping: can't create raw socket: Address family not supported by protocol13:08
radiofreesince there's no way it could ever work with the version from yocto13:08
persiaGet a better ping, or try `ping6 ...`13:08
ssam2ping6 gives same error. This is on Calxeda, the ipv6 driver may just not work ?13:09
Kinnisonssam2: Are you expecting ipv6 support?13:09
bjdooksit /should/ work, however do we have ipv6 support on office network?13:10
KinnisonOf course we don't13:10
persiassam2: Or there is something else not working.  Sounds like a system problem, rather than no working IPv6 routing (although that may also be the case, once you saolve the system problem)13:10
ssam2Kinnison: I'm just expecting name resolution to work. I don't care about the protocol in this case13:10
Kinnisonssam2: aah13:11
persiaSadly, the resolver depends on having transport that works to the addresses in resolv.conf13:11
persiaMaybe try not to use IPv6 there if IPv6 isn't working?13:12
ssam2persia: /etc/resolv.conf is being created by systemd-network, so I suspect I need to do something about that13:13
ssam2http://git.baserock.org/cgi-bin/cgit.cgi/delta/systemd.git/commit/?id=822db23cfa98a9fbc48f41e11caafb6f1017e052 may be the reason...13:15
* persia looks13:16
persiaI don't blame that at all, and like that commit.13:17
Zararadiofree: is there an explanation of how the inheritance works anywhere? (or how to know if something is being inherited) (I'm looking at openbmc, so the specifics are different)13:17
persiaIf you're receiving a resolver list from DHCP, you should very much preserve the order, or face the wrath of the network maintainers.13:17
persiaIn your specific case, I'd recommend asking the folk who provide your DHCP to not provide an IPv6 resolver when providing an IPv4 address.13:18
ssam2I've noticed that another machine on the same network uses an internal nameserver13:18
persiaSince DHCP isn't used to provide IPv6 addresses, it is almost never a good idea to provide IPv6 nameservers in DHCP.13:18
ssam2so this machine is probably doing something special by using 8.8.8.8 and that IPv6 address13:19
ssam2it is on a different subnet. perhaps that's why we've not seen this issue before now13:19
persiaIt is worth tracking down the source of those values.13:19
persiaIf they are DHCP, the DHCP server is likely misconfigured.13:20
persiaIf it is not DHCP, something else interesting is happening that would be good to understand.13:20
ssam2i'll try to track down what serves DHCP to this machine13:20
persiassam2: A quick search suggests dhcping and dhquery might be useful tools for that, if you have to debug the protocol13:25
persiahttps://github.com/CyberShadow/dhcptest may also work, although it was originally developed for Windows13:27
paulsherwood2015-04-17 13:25:21 [openstack-system-x86_64] Now cached as openstack-system-x86_64@ab44d6c512e382cd2df0401b2ea71c7d62dd5a5874528f97ae639138d42eab2814:04
*** zoli__ has quit IRC14:13
straycatif anyone's got a moment to check out https://gerrit.baserock.org/#/c/368/ it's upstream's unit file modified so it restarts on-failure and comes after the swift setup. it only gets installed with swift14:17
straycatmerged thanks14:24
*** zoli__ has joined #baserock14:33
radiofreeZara: i have no idea about documentation14:34
radiofreeand i may be totally wrong about it, i'm just guessing that's how it works14:34
*** zoli__ has quit IRC14:40
Zararadiofree: no worries. fwiw I think you're right about it; I found something that looks similar in openbmc. (ie: https://github.com/facebook/openbmc/tree/master/meta-facebook/meta-wedge/recipes-wedge/lm_sensors seems to add some extra config to https://github.com/openembedded/meta-openembedded/tree/master/meta-oe/recipes-support/lm_sensors .)14:44
pedroalvarezAnybody fancies a bit of reviewing? https://gerrit.baserock.org/#/q/topic:baserock/openstack-in-baserock-3-nodes14:50
*** cosm has joined #baserock14:52
pdarIf anyone has time to review my patches to integrate openstack's ceilometer service into the baserock openstack stuff I'd be most greatful. These are they https://gerrit.baserock.org/382, https://gerrit.baserock.org/383, and https://gerrit.baserock.org/384.14:55
straycatjjardon, i appreciate https://gerrit.baserock.org/#/c/385/1 but this is not a good time15:05
straycatgive us a few days to let openstack settle and we can consider dropping the conf ext15:06
straycatit will require patching ntpd really15:06
jjardonstraycat: sure, Ive just put it there before I forgot15:06
paulsherwood2015-04-17 15:56:45 [genivi-baseline-system-x86_64-generic] Now cached as genivi-baseline-system-x86_64-generic@3addbab1d9f32c354e5e1431c517eea325a98927b4894a4e9045e6d0d4d40eb415:09
radiofreepdar: 383 and 384 already have +2?15:20
radiofreesorry, 382 and 38315:20
ssam2I don't have time to review anything this afternoon but I'm happy for anything that needs merging to be merged15:23
pdarradiofree: great! they were fresh off the press when I posted. Apologies.15:26
pdarthanks for the speedy reviews15:26
*** mdunford has quit IRC15:34
*** mdunford has joined #baserock15:48
*** a1exhughe5 has quit IRC16:02
*** mdunford has quit IRC16:06
ssam2for those following the saga of the .py files being overwritten by random hex strings, I've managed to reproduce the problem with an x86 rootfs hosted on the same NFS server16:32
ssam2so we can rule out random ARM / Calxeda issues as the cause16:33
ssam2i guess now I'll try deploying the nfs root to my laptop16:34
ssam2and see if the Trove nfs server is at fault16:34
ssam2deploying the system to a rawdisk image didn't show the problem, so it is pointing to NFS16:34
* pedroalvarez sends https://gerrit.baserock.org/388 16:39
pdarhow do I push revisions in gerrit?16:41
ssam2if you use git-review, just git-review again (new commit with same change-id)16:41
ssam2without git-review, push to wherever you pushed before (again, push a new commit with the same change-id as the change you're updating)16:42
*** CTtpollard has quit IRC17:01
pdarThanks ssam217:01
*** bashrc_ has quit IRC17:02
*** mdunford has joined #baserock17:03
*** Krin has quit IRC17:04
*** sambishop has quit IRC17:06
*** tiagogomes has quit IRC17:12
*** mariaderidder has quit IRC17:14
*** jonathanmaw has quit IRC17:18
*** gary_perkins has quit IRC17:20
*** edcragg has quit IRC17:24
*** paulw has quit IRC17:40
*** zoli__ has joined #baserock17:40
pedroalvarezWow! Openstack on Baserock!!17:41
pedroalvarezhttp://wiki.baserock.org/guides/openstack-on-baserock/17:41
*** franred has quit IRC17:41
rdaleare there any guidelines to using 'linux32' with a morph build?17:42
jjardonpedroalvarez: \o/17:42
ssam2rdale: if you're using a chroot, run 'linux32 enter-baserock' so the whole chroot sees 32-bit arch17:43
ssam2i've never tried 'linux32 morph build xxx', but that should probably work too17:43
rdalethats what i'm doing17:43
rdaleand stages 1 and 2 of gcc worked, but it is as though stage3 of gcc didn't get run under linux3217:44
ssam2weird. I don't know why, i'm afraid17:44
jjardonseems the ./native-bootstrap > build.log 2>&1 trick worked for ARMv5 as well17:44
rdalessam2: i'll try starting a chroot under linux32 and see if that is any different17:44
*** zoli__ has quit IRC17:45
*** sambishop has joined #baserock17:45
*** lachlanmackenzie has quit IRC18:21
ssam2so, booting a system off NFS from my laptop (when I finally got it to work) shows the same 'files being overwritten with random hex strings' bug18:29
ssam2but the same system deployed to rawdisk did not18:29
ssam2this doesn't make the slightest bit of sense.18:30
*** ssam2 has quit IRC18:30
paulsherwooderk19:07
persiaI wonder if it might be related to https://bugzilla.redhat.com/show_bug.cgi?id=15630719:10
paulsherwoodthat's a *long time ago* ? :)19:11
persiaEssentially, the .pyc files are being written at runtime in many cases, and closely-timed writes and reads can cause issues with NFS.19:12
persiaFaster networking, closer timesync, and some delays between compilation and read might help, but I'm not sure python works that way.19:12
persiaAlso, from the bug description, it sounds like it is really hard to reliably reproduce.  Maybe also related to load on the NFS server, or something.19:12
persiaThe bug is old, but remains unsolved.  Most projects (including RHEL) remove old bugs if they cannot be reproduced, assuming they were fixed upstream somehow.19:13
persiaThat this was closed "cantfix" suggests to me that it is an architectural issue with NFS.19:14
ratmice__doesn't look like the .pyc compiler attempts to append to .pyc files afaict, NFS seems to have known issues with an inability to do atomic appends19:41
persiaI thought the pyc compiler did overwrites.19:44
persiaThe bug I linked was about a race condition, where the overwrite is still in-process to the backing store, but the NFS server seems to get confused.19:45
*** Albert__ has joined #baserock19:48
*** Albert_ has quit IRC19:48
ratmice__doesn't look like PEP-0304 ever happened, to allow you to specify where to generate/search for bytecode, and 3147 doesn't appear like it'd be any help20:02
*** cosm has quit IRC21:17
*** cosm has joined #baserock21:34
*** Albert__ has quit IRC22:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!