IRC logs for #baserock for Tuesday, 2014-07-29

*** fay_ [] has joined #baserock07:14
rjekNope, that didn't work:
rjek(Still no bin symlink)08:06
Kinnisonrjek: hrf08:08
Kinnisonrjek: could you please delete your stage2-fhs-dirs artifacts from your artifact cache and also the unpacked chunk directories in your tmpdir, and then run:08:08
Kinnison$MORPH --log=/src/log-making-fhs-dirs.txt --log-level=debug build my-system08:08
Kinnisonand then put that log file somewhere we can review it to see if anything odd happens during the fhs-dirs chunk build08:09
* rjek thinks about this for a moment08:11
rjekIt would be nicer if the hashes came after the names in these caches and chunk directories; it means you can more usefully employ tab completion08:13
KinnisonThank you.  I can't guarantee I'll be able to work it out from that, but hopefully a colleague will come along soon who can.08:20
persia+1 on renaming everything to be easier for humans to parse.08:20
persia(note that this requires rebuilding the world, and repopulating every baserock artifact cache in existence, so is rather invasive: best to do sooner, rather than later)08:21
* Kinnison strongly believes the naming scheme right now is correct08:21
persiaKinnison: Why?08:21
Kinnisonbecause normally you're more interested in finding an artifact by cache ID08:21
Kinnisonrjek's situation is unusual08:21
persiaI've never known the cache ID *except* by parsing the filename.08:22
persiaPlease explain this "usual" situation :)08:22
Kinnisonmorph's output tells you it at various points08:22
* Kinnison is usually interested in $cacheid.*08:22
persiaAt a speed I can't follow, as part of a volume that makes it uninteresting.08:22
*** jonathanmaw [] has joined #baserock08:22
* rjek has never wanted to find something by hash, tbh.08:23
persiaAh, $cacheid.* is an interesting point.08:23
persiaBut still, the current model is mostly user-frustrating.08:23
rjekOh yesl08:25
* persia vaguely wonders what nifty punctuation lives near the 'l' key on rjek's keyboard08:26
KinnisonUnfortunately I'm not doing my degree course in user interaction design until 201608:26
rjekpersia: . is drectly below l08:26
rjekGod, it gets worse.08:27
persiaKinnison: Which means this is an excellent time to start working on the concepts, so that you don't need to think so much during the course :)08:27
persiarjek: How?08:27
* Kinnison chose a distributed/concurrent systems course to do next08:27
rjekpersia: My typing.08:28
persiaYou might try a Handmaster Plus, or similar product: tends to help keep one's fingers from binding.08:31
* persia has trouble deciding if Baserock tooling needs integration with distributed systems or UX more08:32
Kinnisonpersia: Whatever it needs more, the distributed/concurrent systems course is in its last cycle so I gotta take it first08:33
rjekI am not interested in an experience; I simply want an interface with which I can describe to the computer what it is I want it to do.  The less experiences I have doing it the better.08:34
* persia hopes "in it's last cycle" doesn't mean "scheduled for review because the material is outdated"08:34
persiarjek: Have you tried any of the brain-computer-interfaces yet?08:34
rjekI have not.08:34
rjekBut I don't think I'd want to :008:34
* persia likes to set focus-follows-thought in the window manager, but hasn't explored too much further yet08:34
Kinnisonpersia: I think it's just not popular enough (which I hope means it's interestingly hard)08:35
persiaDistributed systems is indeed interestingly hard.08:35
persiaIf the course doesn't meet your expectations, it's trivial these days to spin up 10-100 instances somewhere to play with the ideas.08:36
* Kinnison might indeed have to play with EC2 or similar08:37
KinnisonI think for a few 10s of $s I can do some fun stuff08:37
persiaThere's free clouds for folk playing, and some of the pay clouds have free space if you can show you're taking a course.08:37
* persia doesn't like the signup conditions for that, but it's the first hit for search terms08:40
Kinnison"For access to the x86 zone, join our Facebook Group"08:40
Kinnisonerm no.08:40
* persia did say something about not liking the signup conditions08:46
persiaThere are alternatives.  I can't find the page right now, but I know HP has an academic program.08:46
persia(or had, or something)08:46
Kinnisonfor a few 10s of $s I can spool up some tiny instances in EC2 with little effort08:47
*** stetaylor [] has joined #baserock08:57
benbrown1 is now known as benbrown_08:57
Kinnisonrjek: I haven't forgotten you, I was looking at stuff.  Is there a build log for stage2-fhs-dirs in your artifact cache directory?  If so, could you pop that somewhere for us too?09:27
paulsherwoodKinnison: EC2 running baserock?09:28
Kinnisonpaulsherwood: down boy.09:28
persiaBaserock is actually a nice platform for distributed system work, because morph deploy makes it easy to deploy several things together.09:28
paulsherwoodKinnison: i'm old enough to be your father, i believe ;)09:29
Kinnisonnaah, my dad is, erm, err.09:29
* Kinnison does maffs09:29
KinnisonYou're not that old09:29
rjekSecond request: build logs have the artifact name in like everything else does10:05
persiaThat has been requested: apparently there's some annoying obstacle to implementation.10:06
richard_mawrjek: it would be more plausible to have the source name in there10:06
persiaripsum had some ideas about a workaround, but then got distracted and solved several more annoying problems that he noticed.10:06
Kinnisonrjek: Okay, so there's nothing showing in that log around the ln line which would suggest it had failed in some non-detected manner10:07
* Kinnison guesses the next step is to edit that chunk so we can fail it at the end of install-commands and see if the symlink is on the filesystem10:07
Kinnisonif it's on the filesystem and not in the chunk then we've narrowed the scope of where the problem can be, quite a lot10:08
richard_mawKinnison: if you're going to add a command to make it fail, quote "false", otherwise it turns it into a boolean False10:08
Kinnisonrjek: Could you please 'morph edit stage2-fhs-dirs' go into the checked out directory, and add: - "false" as a command to the end of the install commands in the stage2-fhs-dirs.morph file?10:09
Kinnisonrjek: then retry your build and we should get a failed/ dir10:09
Kinnisonrjek: in that failed/ dir, check stage2-fhs-dirs.inst for the presence of the bin symlink10:09
rjekThere's only one thing left in failed/ and that's the linux-api-headers10:11
rjek(And it has no bin)10:11
Kinnisondid you do the edit as requested?10:12
rjekI'm still trying to work out what you mean.10:12
rjekI'm normally wary of ever using morph edit because it doesn't ever seem to do the right thing...10:12
persiaYou can do it with git, if you prefer.10:13
rjekAlso: ERROR: morph edit needs the names of a system, a stratum and optionally a chunk as parameters10:13
Kinnisonrjek: use your $MORPH please10:13
rjekOh yes10:13
persiaWhich morph are you running?10:13
*** franred [] has joined #baserock10:14
Kinnisonrjek's system installed morph is wrong, he has a /src checkout which he sometimes forgets to use10:14
rjekKinnison: I assume your request is missing some steps.  Do I need to commit and push this change somewhere, or edit a hash somewhere?10:15
rjek(the "- false" change)10:15
richard_mawif you use `morph edit` it will make the changes to your stratum morphologies, and you just need to make the change to the morphology in ../fhs-dirs/stage2-fhs-dirs.morph10:17
Kinnisonrjek: morph *should* pick up the change (and since it's temporary, we don't care about keeping it)10:17
rjekHow confident are you that it will, though? :)10:17
* rjek kicks off anotgher build10:17
richard_mawrjek: normally I'd be 100% sure, but you attract software spectres10:18
liw-orcrichard_maw, are you saying you're James Bond and rjek is Blofeld? :)10:19
rjekTypeError: sequence item 2: expected string, bool found10:19
richard_mawliw-orc: no, I'm saying rjek is haunted10:19
* rjek laughs10:19
rjekI forgot the ""10:19
rjekA big fat stack backtrace when I fumble my YAML isn't very friendly :)10:20
liw-orcrjek, agreed; we should catch the exception and produce a nicer error message10:20
KinnisonWe could, perhaps, use Rx10:21
* richard_maw thinks we should hook into the yaml parser to make it create objects, so we have textual context when it goes wrong10:22
rjekUse Lua! >:)10:22
richard_mawrjek: no, we don't want anything turing complete10:22
richard_mawit makes it more difficult to cache10:22
richard_mawand reason about for users reading it10:23
rjekKinnison: stage2-fhs-dirs.inst has a symlink10:24
* persia mumbles about how most of morph could be replaced by a make library, and morphologies as makefiles, easing the transition for folk familiar with make10:24
rjek/src/tmp/failed/tmp7JSTVt/stage2-fhs-dirs.inst # ls -l10:24
rjektotal 6810:24
rjeklrwxrwxrwx    1 root     root            10 Jul 29 10:21 bin -> /tools/bin10:24
Kinnisonrjek: thank you10:24
persiaSo that means it is created, but not included in the chunk?10:24
rjekpersia: I have noticed the similarity with Make before.10:24
Kinnisonrichard_maw: so, we have a pair of boundaries as close as we can expect rjek to find... the symlink is in the .inst but not in any of the chunks10:24
persiarjek: What similarity?10:25
KinnisonHaving written distributed build systems in make before, I wouldn't want to do it again10:25
rjekpersia: That they're basically Makefiles, except in a pickier format.10:25
richard_mawKinnison: then it has to be in the artifact splitting logic, but it's working for x86_6410:25
persiarjek: I take it you've never written an executable in make.10:25
Kinnisonrichard_maw: is there any logging in there rjek can turn on, or is it time for someone to write some investigatory tools to see if it's our filesystem interface squiffing on a combination of MIPS64 and ext3 ?10:28
richard_mawKinnison: morphlib.builder2.ChunkBuilder.assemble_chunk_artifacts line 488 is where we get the file listing10:31
Kinnisonyeah that was where I was10:33
* richard_maw is currently pondering if the `os.path.islink` is failing10:34
Kinnisoncan you write a quick test for rjek?10:34
richard_mawalready working on it10:34
rjek>>> os.path.islink("bin")10:35
richard_mawrjek: that assumes the path is being checked in the same directory10:37
rjek>>> os.path.islink("../stage2-fhs-dirs.inst/bin")10:38
richard_mawI mean, the problem may be that the morph build code is checking whether the wrong "bin" is a symlink10:38
rjekAh, right10:38
richard_mawthe only chdir we do is for subprocesses10:41
richard_mawrjek: Can you strace the build, looking for stat calls?10:41
richard_mawwithout the "false" command to make it fail10:41
rjek/bin/sh: strace: not found10:42
rjekAlso, I'm not sure strace is even supported on MIPS64 if I did have it.10:42
richard_mawhmm, then we'd have to instrument the call so we know its result, or take a leap of faith and change it to `if os.path.islink(os.path.join(dirname, x))`10:45
richard_mawand I'm not convinced we've found the culprit yet, since this link turns up on x86 builds10:45
persiainstrumentation sounds safer, as it it doesn't work, it explains why, and if it does, it helps lead toward a real fix, rather than just something that happened to work today.10:46
richard_mawwait, the sumllink doesn't end up in the subdirs list anyway10:47
richard_mawit's part of basenames10:47
richard_mawrjek: in the stage2-fhs-dirs.inst direcctory, can you run the following10:48
richard_mawimport os10:48
richard_mawfor dirname, subdirs, basenames in os.walk('.'):10:48
richard_maw  for bn in basenames: print bn10:48
rjekOh, that's interesting10:49
Kinnisonerm, that's the etc10:50
Kinnisonwhich we'd expect to see from the walk10:50
richard_mawbut no "bin"10:50
richard_mawthat's the interesting part10:50
rjekShouldn't that be waling ./ not ./etc/ ?10:51
richard_mawwalk is recursive10:51
rjekThere are a lot more files in that directory than are listed there10:51
KinnisonIf the symlink points at something which *works* then os.walk seems to not list it10:52
rjekI don't understand what that python snippet is doing10:52
rjekIt appears to be listing only the files in ./etc10:53
richard_mawhm, links that point to an existing directory end up in the subdirs list, and links that are dangling end up in basenames10:55
richard_mawI think it's failing for you because you have a /tools/bin, so the link ends up in subdirs, and the os.path.islink check is failing10:56
richard_mawwhich explains why it worked for us, as we don't have a /tools/bin10:56
richard_mawand why it failed for you10:56
rjekSo, this would still fail for x86_64 if it were beign cross bootstrapped?10:56
richard_mawso the proper fix is to change it to os.path.islink(os.path.join(dirname, x))10:57
richard_mawbut you should be able to continue your builds by removing /tools10:57
* rjek tries that10:57
Kinnisonjust in case10:57
rjekHow do I unedit?10:57
rjekie, the morph edit stage2-fhs-dirs I did.10:58
Kinnisondelete the checked out branch and if you had made no other changes, git reset --hard in your definitions10:58
* rjek types the magic runes in10:58
Kinnisonthere's no 'morph unedit' :-(10:58
rjekOK, I don't think it's going to be safe to rename/remove /tools10:59
rjekBecause the system's /bin and /lib64 etc are still symlinks to /tools10:59
rjekI need lunch.11:01
*** straycat [] has joined #baserock11:03
*** stetaylor [] has quit [Remote host closed the connection]11:36
*** stetaylor [] has joined #baserock11:37
*** stetaylor [] has quit [Ping timeout: 240 seconds]11:41
* rjek has had lunch.12:13
KinnisonI believe richard_maw has been working on a patch for the problem rjek exposed12:13
richard_maw should fix it12:22
*** stetaylor [] has joined #baserock12:23
*** fay_ [] has quit [Ping timeout: 240 seconds]12:33
*** fay_ [] has joined #baserock12:38
richard_mawSotK, persia: Sorry I haven't responded to e-mails yet, I'm needed to dig through the code to locate strange bugs13:12
persiarichard_maw: No worries.  On IRC, I expect replies in a day or so, but I in symmetry, so if people respond to my email within three or four months, I count myself lucky13:13
KinnisonGoodnews, richard_maw's branch worked and we fixed the problem rjek was enduring.  sotk will merge it13:19
KinnisonBadnews, rjek's back to stage2-eglibc -- yay interp. paths13:20
SotKmerged, and definitions updated13:25
Kinnisonthanks sotk13:26
KinnisonSotK: Did you update both tools.morph and cross-bootstrap.morph ?13:26
SotKI updated the ref in both tools.morph and cross-bootstrap.morph so we don't get that problem again13:26
jjardonHi, today I had problem compiling another module because seems baserock version of busybox "ln" command doesnt support --relative. Any idea what is the plan forward for this? upgrade busybox? or maybe switch to use coreutils?13:29
KinnisonIn the short-term I'd say "Upgrade busybox"13:30
persiajjardon: Is newer busybox free of these issues?  If so, you might want to upgrade it (which would be useful anyway).13:30
KinnisonBut if that won't solve your problem, then we may have to look at the harder-to-do coreutils switch13:30
persiajjardon: But given the number of issues you're finding, I wonder if you wouldn't do better to change your system to use coreutils.13:30
* persia decides to stop typing, because Kinnison is saying the same things more concisely and faster13:31
jjardonsure, only wanted to know if there was a prefered path to fix this.13:34
jjardonCan I ask why busybox is currently used instead coreutils?13:35
KinnisonBecause busybox is one thing which gives us coreutils, findutils, util-linux etc13:36
Kinnisonand has significantly less in the way of build-dependencies13:36
Kinnisonbecause it's not autotoolised13:36
jjardonrigth, thanks for the explanation13:37
liw-orcwe do have coreutils, but we prefer busybox in build-essential13:44
persiaIs there a strong reason not to use coreutils in build-depends for GNOME?13:51
jjardondo not think this should be GNOME specific (the compilation problem is in systemd, for example)13:52
richard_mawjjardon: it's systemd specific, if you read back upstream were told by debian maintainers that this doesn't work on their build systems, and were told to either patch it, or update coreutils14:02
richard_mawso I think there's a patch floating around somewhere14:02
richard_mawIIRC we _have_ built coreutils by this point, so you could see if the version of coreutils we're using is too old to support --relative, or if we've told it not to provide ln14:03
jjardonrichard_maw: ok, thanks. But AFAIK coreutils is in tools, and foundation doesnt depend on that strata14:05
richard_mawhm, I thought it was in core14:06
jjardonrichard_maw: also we disable the build of all commands that are already in busybox14:16
persiaReviewing backscroll here, I wonder if we're being a bit overambitious to have a single "core", etc.14:21
persiaWhile it makes thinking about these things convenient, I suspect that as we seek to support a wider variety of systems, it makes sense to have multiple paths.14:22
persiaSo that we can use the same build-essential suite over (slightly) different platforms to support variants from tiny little systems suitable for monitoring solar panel performance through to complex desktop environments.14:23
persia(MPI applications and distributed systems being somewhere between those extremes in terms of "system" compelxity for the semantic value of "system" used in Baserock, despite being massively more complicated than desktop environments for the semantic value of "system" used by HPC folk)14:24
persiaIs this possible to achieve without either forking or massive duplication with current morph?14:24
rjekI think the only other "Linux distro" that aims to be universal in use (from embedded to supercomputer) is Debian, and it's not great at it.14:25
persiaI'm not convinced "Baserock" and "Linux distro" are really the same category: I think of "Baserock" as being a set of tools to build distros.14:26
persiaBut that niggle aside, I entirely agree :)14:26
persiaPlus, as my parenthetical note was intended to imply, desktops are harder than supercomputers.14:27
persia(To make a supercomputer, wire up as much of the newest, coolest kit one can afford via the fastest available interconnect, install a kernel, busybox, an MPI environment, and drivers for any co-processors, and create an account on a head node for a researcher.  Sit back and pay cooling and electric bills.  Simple.)14:28
jjardon FYI, our coreutils are indeed new enough to support --relative. Would it be ok to move them to foundation?15:18
persiaI don't think that question can be answered here, but suspect there would be concerns about the minimal system at least.15:21
persiaFor your local definitions.git, I think it makes perfect sense to do the move: that will significantly reduce your build issues.15:21
persiaFor master, it probably needs discussion on the mailing list.15:21
radiofreei don't think the minimal system uses foundation?15:22
pedroalvareziirc they dont'y use even core15:23
pedroalvarezdaniel! :D15:29
rjekAww, sweet love.15:31
pedroalvarezbtw, I can now confirm that a devel system running on OpenStack works with a Trove running on the same cloud15:33
richard_mawminimal systems don't even use all of build-essential15:47
*** fay_ [] has quit [Remote host closed the connection]15:48
*** stetaylor [] has quit [Quit: Leaving...]16:26
*** jonathanmaw [] has quit [Quit: Leaving]17:00
*** franred [] has quit [Remote host closed the connection]17:38
*** flatmush [] has quit [Ping timeout: 260 seconds]17:53
*** flatmush [] has joined #baserock19:03

Generated by 2.14.0 by Marius Gedminas - find it at!