IRC logs for #baserock for Tuesday, 2015-03-10

*** flatmush has quit IRC03:11
*** flatmush has joined #baserock03:13
*** zoli__ has joined #baserock05:04
*** zoli__ has quit IRC06:43
*** robtaylor has quit IRC07:22
*** mike has joined #baserock07:27
*** mike is now known as Guest4731607:28
*** robtaylor has joined #baserock07:28
petefothThe build I left running last night failed with this . When I restarted it this morning, it got on with building chunk stage2-glibc. Any ideas what may have caused the failure? The VM uses the WiFi adapter on the host - would that going down cause the failure?07:34
*** a1exhughe5 has joined #baserock07:53
*** gfinney has joined #baserock08:11
*** gfinney has joined #baserock08:11
*** CTtpollard has joined #baserock08:26
paulsherwoodpetefoth: was this *after* getting everything cached?08:28
paulsherwoodthat looks like your vm didn't have connectivity, which i know is what you are trying to test08:29
petefothpaulsherwood: no - it's the build that is pulling everything in08:29
petefothI left it to cook last night, and it fell over 45 minutes after I left :)08:30
paulsherwoodweird, thn08:30
paulsherwoodif you re-run, same problem?08:30
petefothpaulsherwood: no it's runnign fine now, so I guess a break in connectivity when it was trying to clone was what lkilled it08:31
petefothbtw - to ''tweak stage1-binutils.morph' i started by doing `morph branch baserock:baserock/definitions my-new-system' then editing stage1-binutils.morph. Was the git branch necessary or is there an easier way (such as morph edit baserock:baserock/definitions' or...?08:34
radiofreeIs morph branch the correct way to do things? I always just morph checkout ... And work mostly using git08:36
petefothradiofree: I have no idea :) I'dlike tio find out for the sake of getting the 'Working with BAserock offline' story straight08:38
*** gfinney has quit IRC08:38
radiofreepetefoth: i can't remember the last time i actually used morph branch... however maybe there's a reason i should be using it?08:39
radiofreei tend to just morph check master, cd into definitions, then if i wanted to "tweak stage1-binutils.morph" i'll just git checkout -b james/wip and mess about there08:40
petefothradiofree: I guess that would do it08:40
paulsherwoodthe trouble with morph checkout master, is that later morph edit foo can break, because it will try to create a branch master for foo08:42
paulsherwoodi have once-and-for-all done 'morph branch b:b:d reference ; cd reference/b/b/d' then i use normal git process in my reference branch (eg git fetcg, git reset --hard origin/master)08:43
petefothso starting with 'morph branch' will avoid that problem?08:43
petefothpaulsherwood: thanks08:44
paulsherwoodi should make a tutorial on this too, i guess08:44
*** mdizzle has joined #baserock08:44
petefothThe UI BoF at the meetup asked for 'How I use Baserock' stories - your story would be very useful I think08:45
paulsherwoodi stayed out of the UI conf... i'm a commandline guy these days :)08:50
petefothpaulsherwood: command line *is* a UI - just not a G one :)08:50
paulsherwoodas radiofree famously said in a meeting once 'our ui *is* the commandline' :)08:51
jjardonI too have used morph checkout, morph branch exactly one time:  what are the benefits of using them? Is there any docs where I can take a look?08:54
petefothpaulsherwood: how do you go about updating the local clones of the git repos? Do you use 'morph for-each'?08:56
paulsherwoodpetefoth: first issue is to enusre you actually have the git clones of the repos08:57
paulsherwoodsorry, i can't help on this now08:58
petefothpaulsherwood: np08:58
*** bashrc_ has joined #baserock09:02
*** rdale has joined #baserock09:14
*** jonathanmaw has joined #baserock09:15
* straycat thought we were going to get rid of morph checkout/branch and just let people use git for that stuff09:19
*** tiagogomes_ has joined #baserock09:20
*** gfinney has joined #baserock09:39
*** gfinney has joined #baserock09:39
*** edcragg has joined #baserock09:44
wdutchI've submitted a patch to lorries which hassn't got 2 +1s yet but I've noticed a couple of mistakes, should I patch that patch or patch from current HEAD?09:50
Kinnisonfollow up the thread indicating you're -1ing your own patch09:51
*** gary_perkins has joined #baserock09:51
Kinnisonthen provide a new patch series which has been corrected09:51
Kinnison(is how I'd do it)09:51
wdutchokay thanks09:55
*** ssam2 has joined #baserock09:57
*** ChanServ sets mode: +v ssam209:57
richard_mawsystemd will optionally depend on libacl, which we currently have in foundation, but not as a dependency of systemd. Unfortunately systemd wants to be able to make shared objects, which doesn't work because libacl hasn't been compiled to produce shared objects, or build the .a files with -fPIC11:24
paulsherwoodstraycat: yup, we were/are. but it's quite a big change, i think... removal of workspace, and documenting the result properly11:25
* straycat nods11:26
* richard_maw discovered the libacl problem when rebuilding systemd so he could add more debugging information11:27
* richard_maw continued with --disable-acl11:28
pedroalvarezI had to use --disable-acl in libarchive too11:30
*** zoli__ has joined #baserock11:47
*** edcragg has quit IRC11:49
jjardonrichard_maw: whats the context? Or you mention that as an informative comment?11:50
*** edcragg has joined #baserock11:51
richard_mawmostly informative, though if I have time tomorrow, I'm going to have a poke at it11:51
richard_mawwe currently only intentionally use it for btrfs-progs, which is fine, as it doesn't require a shared object there11:51
*** CTtpollard has quit IRC12:03
simonh_Hi, can someone from the Baserock ops trigger a sync of the lorries? I'd like to use the openwrt lorries that have just been added.12:14
*** simonh_ is now known as mauricemoss_12:14
persiamauricemoss_: There's lots of them: a sync is hard.  What you probably want is to have one or more jobs promoted.12:19
persiamauricemoss_: I'd recommend asking for some specific jobs, if you have a list, as that is more likely to be something someone can do.12:19
persiamauricemoss_: is the current queue12:20
pedroalvarezmauricemoss_: new lorries go to the top of the queue, so they might be lorried already12:20
persiapedroalvarez: Oh, excellent!12:21
mauricemoss_pedroalvarez: ah nice, ta!12:21
pedroalvarezmauricemoss_: if you find out that some of them are not lorried, poke us and we will investigate12:22
bashrc_is the definitions format going to change?12:23
pedroalvarezI think there is an ongoing discussion about it; Yes12:25
bashrc_well, I hope it's not going to xml12:26
radiofreeyou don't have to worry abou tthat12:28
tiagogomes_I think that new lorries don't go to the top, at least they didn't 1/2 months ago12:29
ssam2no, it'd be good if they did. would need someone to make a change to lorry-controller12:30
pedroalvareztiagogomes_: new lorries go to the top. Lorry changes (modify an exsisting lorry) may not12:33
tiagogomes_pedroalvarez ah, may that then12:33
pedroalvarezbut I may be wrong :/12:33
ssam2oh, i didn't realise they went to the top12:35
ssam2i don't see code that'd do that12:36
mauricemoss_seems like they don't, search for openwrt here: "in 2 h 2 min"12:37
SotKI think thats because they've already been run once though?12:38
mauricemoss_Oh, true I had upstream:procd instead of upstream:openwrt/procd in my morph file.12:44
jjardonhi, why baserock fail because there is an error in a system completely different from the one Im trying to build? Is this the intended behaviour?12:47
radiofreecould you clarify that a bit?12:48
radiofreeoh i think i know what you mean, there's some issue with systems/bar.morph that's being flagged, even though you're trying to build systems/foo.morph?12:49
jjardonnope, I get an error about armv7lhf-cross-toolchain... but Im not building that in my system12:50
ssam2jjardon: morph parses all morph files when it checks if you have any local changes12:50
ssam2i think12:50
jjardonoh, ok, so its intended ;)12:51
SotKmorph parses all morph files all of the time (in some commands anyway) I think12:51
jjardonis that really what we want?12:52
SotKI don't think so, but others do12:52
jjardonyes, I do not want other to have errors building their systems if I make a mistake in mine12:56
ssam2we shouldn't have any invalid .morph file in definitions.git12:59
ssam2once we have pre-merge CI we probably won't have any12:59
persiaWell, assuming we CI everything.  Current CI is only a subset, isn't it?13:00
SotKyes, but the build will fail if the repo contains any broken definitions13:01
jjardonssam2: agree, but is still possible. think you have several teams managing different systems. Even if CI detects the problem I do not want to stop working because there is a problem in a system I do not nothing about13:03
ssam2if it's pre-merge testing, nothing is going to stop you working -- the breakage will never be merged13:04
SotKI just don't like the way that we have the root of a tree (the system morph), and know where to find all the other nodes in the tree (we reference the needed morphologies by path), but ignore that and just load everything in the repo and look through it13:15
ssam2--local-changes=ignore might bypass that code, I think it's the temporary build branch stuff that does it13:16
SotKssam2: looks like it does :)13:23
ssam2constructing this Gerrit system has made me really want components to identify their runtime dependencies in advance13:34
ssam2lots of small, unexpected issues. like you can't use lorry-controller without the morph-utils stratum because it uses cliapp and cliapp lives in morph-utils13:35
gfinneyIt doesn't NORMALLY take an age to run the build commands for stage1-gcc does it?13:35
SotKI had a similar experience last year when trying to make a system based off the Genivi baseline with loads of stuff on top13:36
SotKgfinney: how long is the age? :)13:36
gfinney36mins so far13:36
gfinneyThis is being run from YBD btw13:37
ssam2gfinney: building GCC is not a quick operation13:37
ssam236 mins is slower than i'd expect though13:38
SotK36 minutes doesn't seem too outlandish to me13:38
SotKI've seen it take about that long sometimes, if my system is having a slow day13:38
gfinneyI thought so myself, I just don't remember it being quite so slow on previous occasions13:38
gfinneyOh well13:38
ssam2motivation for you to implement artifact caching in ybd at least :)13:38
gfinneyIt is implemented, but I cleaned it :(13:39
gfinneyIt was being odd13:39
mauricemoss_rdale, I'm still having the `make` segmentation fault problem in a chroot, unfortunately this doesn't help: Did you change something additionally?14:00
rdaleno that's the script i use14:01
straycatanyone up for bug squashing on saturday? a possible flaw is i'm not sure if there's shared knowledge of what there is to fix14:01
bashrc_is there a bug list?14:02
mauricemoss_Hmm weird, I'll copy strace to the chroot.14:02
petefoth!/project/list is the bug tracker, which is a start14:03
* straycat thought he read something from ssam2 saying not to fill it with issues just yet14:06
ssam2it's just that it's not backed up yet14:06
petefothindeed, but it's nowher near full yet :)14:07
*** Blacksnow has quit IRC14:15
mauricemoss_ This might be the issue: 'access("/etc/", R_OK)      = -1 ENOENT (No such file or directory)'14:17
*** CTtpollard has joined #baserock14:19
*** CTtpollard has quit IRC14:26
bashrc_do chunks have an enable flag?14:27
pedroalvarezwhat for?14:28
pedroalvarezenable or disable the chunk?14:28
Kinnisonmauricemoss_: It's rare for to exist, no?14:28
pedroalvarezbashrc_: if you don't want a chunk in a system, you have to remove from the stratum of that system that is installing it14:28
bashrc_seems kinda crude14:29
bashrc_what if you disable some stratum but then want to reenable it a few deployments later?14:30
straycatcommit your removal14:30
straycatthen revert14:30
bashrc_I guess so14:30
pedroalvarezI was going to say that14:30
tlsaKinnison: about 8 months ago, we did a release together, and there was a "pre-release tests" bit of the release process.  IIRC you wrote a script for that that checked that all the chunk refs were sha1s and not branches (maybe other checks too), but I can't find any reference to it any more.14:46
tlsaKinnison: do you still have that script?14:47
KinnisonI think I probably just grepped stuff14:47
* Kinnison will look14:47
tlsapretty shire you had a script/or python which generated a report14:48
tlsabut I could be wrong :)14:48
KinnisonIf there was such a beast it wasn't me14:48
tlsahmmm, ok14:49
mauricemoss_Kinnison: True, just learned that. Any idea what could cause the seg fault?14:49
KinnisonSadly without a bunch more debugging effort I can't suggest a reason -- it's 'make' segfaulting right?  What architecture is this?14:50
Kinnisonmauricemoss_: do you have ltrace available to you at this point?14:51
pedroalvarezI've seen this in x86_6414:51
Kinnisonpedroalvarez: Okay, so it's not an obscure platform only issue14:52
mauricemoss_Kinnison, I'm afraid not :(14:52
KinnisonIs this a release version of 'make' or the contents of master or something?14:52
pedroalvarezI believe this is make 4.1 from a  tarball lorry. But the issue is not building make, or using make to build other things. The issue is that make can't run in a staging area, segfaulting14:55
pedroalvarezMaking it dificult to debug when integrating new software14:55
mauricemoss_yep, I'm using make 4.1 ref f75919b14:55
mauricemoss_exactly, I need this integrating some openWrt packages.14:56
pedroalvarezI believe that it used to work before without having to bind-mount some partitions14:56
pedroalvarezAlso rdale says that mounting those partitions was enough to make it work14:57
pedroalvarezalthough you say that it's still not woring14:57
mauricemoss_rdale, could you post the output of a working `strace make`?14:57
Kinnisonmake most likely expects things like /proc and /dev to be sane14:58
rdaleyes, i've just been trying that - but i'm not seeing it traverse the /dev directory14:58
*** Krin has joined #baserock15:01
*** CTtpollard has joined #baserock15:07
*** CTtpollard has quit IRC15:13
*** gfinney has quit IRC15:45
*** gfinney has joined #baserock16:18
ssam2i've removed mention of pylru from the wiki, now that it's present in the latest release16:24
pedroalvarezmakes sense16:27
jjardonsome days ago was commented the builds are now faster, but we were not very sure about the cause: is this building a specific chunk or entire systems?16:47
jjardonI said that because I removed the max-jobs:1 from several components when I updated them, the biggest one systemd16:48
radiofreegcc is now much faster to build, i don't know about about systems16:51
radiofreeas far as i was aware the only major components that used max-jobs were systemd and qtwebkit?16:52
jjardonok, so its gcc on its own. Then its probably thanks to tiagogomes_ patches16:55
jjardonradiofree: yeah, systemd and qtwebkit were the big ones16:55
radiofreejjardon: i'm leaning towards that.. i checked the linux from scratch docs and newer versions of gcc tend to be slower to build16:58
*** a1exhughe5 has quit IRC16:59
jjardonanyone here know about pam/login? AFAIK debian system uses login binary provided by the shadow package, but other distros uses the "login" that comes with util-linux. Do anyone know what baserock uses? and why?16:59
tlsa <-- a few chunks using 'master' as ref and one using a personal branch17:01
straycathow did we miss that17:04
tlsathe tooling didn't check for us17:04
pedroalvarezI'll be really happy once we have these checks pre-merge :)17:05
pedroalvareztlsa: another requirement,17:06
straycatit did indeed get 2 +1s somehow17:07
straycat*shrug* okay whatever pre-merge check thing would be nice too17:08
tlsaat the moment I'm just making `morph build` report the issues as warnings17:09
tlsaI guess pre-merge check could be added as another test job in mason-v217:09
tlsabut I'm not working on that right now17:09
* straycat nods17:09
*** CTtpollard has joined #baserock17:11
pedroalvareztlsa: I was going to say, that another requirement would be to point to g.b.o in the repo fields17:15
pedroalvarezthen tried to remove it, to not bother you with more work, but I failed :)17:15
tlsapedroalvarez: yep, I need to add repo checking too17:16
tlsanot thought about how yet17:16
tlsajust looking at the python warnings module instead of using print :)17:17
*** zoli__ has quit IRC17:17
mauricemoss_How can I write all the build output to morph.log when using CMake? I tried changing the log-level but that didn't work.17:21
rdalemake VERBOSE=117:21
rdaleand then you need the morph option to write stdout to the log, which i've forgotten17:22
*** tiagogomes_ has quit IRC17:23
mauricemoss_I did the latter but forgot about VERBOSE=1, thanks17:24
pedroalvarez(morph --help)17:30
*** tiagogomes_ has joined #baserock17:30
*** mdizzle has quit IRC17:35
jjardonpedroalvarez: is -v and --verbose already taken for other stuff? --build-log-on-stdout seems not very standard17:38
nowsterverbose tells you what morph is doing17:41
nowsterbuild-log-on-stdout makes the actual "make" phase visible17:41
* paulsherwood would prefer that -v showed the make phase stuff as well, tbh. 17:42
paulsherwoodafaik morph is already logging its -v stuff in morph.log, irrespective of the -v switch?17:43
ssam2me too17:43
pedroalvarezansible have various levels of verbosity so you can say: -v -vv -vvv or -vvvv17:43
ssam2build-log-on-stdout is an internal flag used by distbuild, which happens to be actually useful17:43
paulsherwoodand so far, in debugging a *lot* of morph builds, i don't think i've gained much insight from the verbose content in morph.log17:44
paulsherwoodthe build-log output is definitely useful17:44
ssam2there's still quite some nonsense goes into morph.log. it used to be a lot worse!17:45
ssam2build-log output doesn't really belong in morph.log, it's chunk metadata17:46
* jjardon files a storyboard story17:46
ssam2it's bad that we hide build-logs in the artifact cache with no easy way to get to them, though17:46
rdalecouldn't you put the build log somewhere in /baserock for a chunk?17:47
jjardonfell free to add more in!/story/2217:48
paulsherwoodssam2: well, ok... so 1) stick the chunk name in the build-log name, so folks have a chance of parsing it by eye, 2) the build log should be provided as an artifact during fetch17:48
* paulsherwood can't log in to that from here17:48
rdaleah, "build-log output doesn't really belong in morph.log, it's chunk metadata" +117:49
paulsherwoodi was requesting it on the visible output, with -v set17:50
paulsherwoodmorph.log is not much use imo... but maybe i'm missing something17:50
straycatuseful to debug morph, less useful to debug builds perhaps17:51
ssam2paulsherwood: agreed on both17:51
* straycat notes that --build-log-on-stdout was only ever meant to be an internal option for distbuild's use17:52
paulsherwoodstraycat: i've had to debug a lot of morph. the info i need has not really been in morph.log17:52
ssam2i've added that comment to storyboard17:52
straycati guess that doesn't matter though17:52
ssam2or have I? storyboard's 'click a button and nothing happens' interface is maddening17:52
paulsherwoodstraycat: understood. my point is that users need the info, and morph doesn't give it by default17:52
ssam2oh, it appears if I reload the page17:52
jjardonssam2: use the return key instead17:52
paulsherwoods/users need/users sometimes need/17:52
straycatpaulsherwood, for debugging morph i guess we could be more liberal with logging.debug17:53
* jjardon wants an option that shows _everything_ in stdout17:53
straycatpaulsherwood, i agree too, i think i was about to add build-log-on-stdout=True to my morph.conf17:54
paulsherwoodjjardon: +117:54
* jjardon thinking maybe we can use the journald api for this17:54
jjardonso we can select what exactly we want to see in the log17:55
paulsherwoodjjardon: i think this sohuld not depend on systemd17:55
perryli've noticed there's an issue with the morph version error message displaying incorrectly; can someone give this a quick once over before i send to the ML to check that the error should display properly?
paulsherwoodbut i may be wrong17:55
paulsherwoodon ybd, i'm trying to have as few dependencies as possible17:55
jjardonpaulsherwood: sure, it can be an optional dep17:56
straycatperryl, imo it might be better to have those values interpolated, just to be consistent with the rest of morph17:57
perrylstraycat: interpolated?17:58
straycat'like %s' % this17:58
straycat* 'this'17:58
*** Guest47316 has quit IRC17:58
perryli must have done it wrong in the reason, as the output the current message gives is "ERROR: Failed to build baserock:baserock/definitions 5e09ce28da6bddfc75661837058743447474f185 systems/build-system-x86_64-chroot.morph: ['Protocol version mismatch between server & initiator: distbuild network uses distbuild protocol version %i, but client uses version %i.', 1, None]17:58
perrylprobably the commas actually!17:59
ssam2yes, you have to use %17:59
ssam2 /except/ some functions will collect all arguments after the first one and pass them to % for you18:00
ssam2logging.debug(), info(), error() etc. do this18:00
ssam2which is nice, but a source of confusion18:00
perryli think this is just a custom message construct as opposed to logging.*18:01
ssam2yeah, which is why the interpolation wasn't happening18:02
* perryl shall give it another look in the morning and send the patch on a fresh mind18:02
ssam2morph ./check fails in master of the reference systems on 'morph foreach runs command in each git repo' yarn, I'm guessing it's due to the busybox upgrade18:03
ssam2I mention it here because I don't have time to fix it (although I could send a patch to remove it, I guess workspaces are going the way of the dodo anyway)18:03
jjardonok, so I guess baserock uses the shadow login because is build after util-linux, rigth?18:03
*** jonathanmaw has quit IRC18:05
*** bashrc_ has quit IRC18:05
* jjardon will try to find the differences between the "login" command shipped with shadow and the one from util-linux18:05
jjardonAFAIK, debian/ubuntu usees the one from shadow, but Arch uses the one from util-linux18:05
jjardonrichard_maw: coincidentally I found an issue building tar today because a bug in acl! (fixed in curent acl master)18:07
straycatso -v would be just an alias for --build-log-on-stdout or an alias for --verbose --log-level=DEBUG --build-log-on-stdout, or something inbetween?18:09
ssam2i'd like it if -v did what it does now, but also output the build logs18:10
jjardonId say it should be what normally -v means: "show me all the info in stdout", wherever that means in implementation level18:11
ssam2you definitely don't want *all* the info :)18:12
ssam2try --log=/dev/stdout sometime18:12
* jjardon scared now :)18:12
*** edcragg has quit IRC18:13
* pedroalvarez likes ansible verbosity levels (-v -vv -vvv -vvvv)18:13
straycatpart of me thinks that if we expect the user to want the build logs on stdout by default then it might be better to put the option that enables that in the example morph.conf, that way people don't need to type -v all the time, or just make it default in morph18:14
jjardonpedroalvarez: yes, that's quite standard as well (try lspci -vvvv, for example)18:15
jjardonstraycat: +118:15
pedroalvarezjjardon: oh! then more reasons to follow something similar?18:16
*** tiagogomes_ has quit IRC18:20
*** CTtpollard has quit IRC18:20
*** ssam2 has quit IRC18:22
*** gary_perkins has quit IRC19:03
*** nowster has quit IRC19:11
paulsherwoodi'd be happier to type -v for the extra output, than have to type something else to turn it off.20:40
straycati think it's fair for it to be configurable21:03
straycati need to make it as few characters as possible, in fact morph build is about to be aliased to 'mb'21:04
straycatmaybe just j21:04
straycatj and k21:04
straycat(so i'm in support of everything, fwiw)21:05
*** zoli__ has joined #baserock21:25
*** gfinney has quit IRC21:35
*** zoli__ has quit IRC22:12
* paulsherwood wonders what's up with the mason connection to gbo23:47

Generated by 2.14.0 by Marius Gedminas - find it at!