IRC logs for #baserock for Monday, 2015-09-21

*** Stanto has quit IRC02:15
*** Stanto has joined #baserock02:19
*** paulw has joined #baserock06:15
*** fay_ has joined #baserock07:36
*** bashrc_ has joined #baserock08:02
*** mariaderidder has joined #baserock08:07
*** toscalix__ has joined #baserock08:26
*** rdale has joined #baserock08:27
*** jonathanmaw has joined #baserock08:34
*** ssam2 has joined #baserock09:11
*** ChanServ sets mode: +v ssam209:11
Kinnisontiagogomes_: when you did the armv8l64 support / bootstrap, do you recall ever having an issue where the stage 2 compilers, or the stage 3 libc would miscompile and thus break the build?09:44
paulsherwoodwhat i'm seeing is gcc failing09:44
KinnisonThat's the manifestation of it, yes09:45
KinnisonI'm unsure if it's the stage 2 compilers, or the stage 3 libc which is broken, leading to the failure to compile the stage 3 compiler09:45
KinnisonI suppose it could be something else, but if the ordering wasn't it, then I'm stuck for other options09:45
tiagogomes_Kinnison no, the issues that I had were only due bad configuration. The compiler itself was working fine09:46
Kinnisontiagogomes_: we have an issue right now, where gcc (stage 3) failed to build, because it claims it can't find libz.so.109:46
Kinnisonthough that is present in /usr/lib and in the ld.so.cache09:46
Kinnisontiagogomes_: however, unlike my expectations, the libc is in /lib64 not in /lib which surprises me09:47
Kinnisontiagogomes_: though I can't remember how it looked by the time we'd finished the initial armv8l64 work09:47
tiagogomes_I can't recall either. Are you trying to build it with ybd? Can you try  it with morph to take out the build tool of the equation?09:52
KinnisonSadly we don't have an easy way to use morph on the units right now09:53
Kinnisonthey're running ubuntu and we don't have chroots to hand09:53
tiagogomes_have you tried to compile a test.c with the stage2-gcc and stage2-glibc to do a basic test to the toolchain10:04
KinnisonWe're currently rearranging the systems, once that's done, if we encounter the issue again, I'll take a look10:04
KinnisonIt managed to compile the stage 3 libc10:05
Kinnisonand to compile zlib against the stage 3 libc10:05
KinnisonBut somehow fails when compiling something against that zlib10:05
tiagogomes_mm, but then it finds zlib?10:06
KinnisonWell the error message is that it fails to find zlib10:07
Kinnisonbut zlib is in place10:07
KinnisonI personally think it's failing to find libc10:07
Kinnisonbut I can't be sure10:07
jjardonssam2: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/log/?h=jjardon/gnome10:10
ssam2hoooray!10:12
tiagogomes_Kinnison looking at the comments that I wrote when ugrading bintutils and gcc, it is true that libc should be on /lib10:13
Kinnisontiagogomes_: I wonder why it's not going into /lib then on these builds10:13
* paulsherwood notes also that this is only failing *sometimes* which is depressing.10:16
Kinnisonindeed10:17
Kinnisonintermittent errors make us sad10:17
tiagogomes_:'( That makes things much harder10:17
*** flatmush has quit IRC10:18
paulsherwoodwell if the machines haven't been reimaged, i did capture the build area10:22
* Kinnison thinks tlsa may have dropped at least one of them on the floor to make room10:23
tlsayeah, I dropped one10:23
paulsherwoodnever mind, then10:23
tlsaciat-large-0 is still there10:24
paulsherwoodso /src/gcc.build.fail and /src/gcc.succeed10:25
Kinnisonpaulsherwood: are those the assembly build areas?10:25
paulsherwoodyup10:25
Kinnisonhandy10:25
Kinnisonpaulsherwood: could you grant me access so I could take a look?10:25
paulsherwoodKinnison: i don't know how to do that?10:25
Kinnisonpaulsherwood: Either give me a password if there is one, or append my ssh public key (I can mail it to you) to .ssh/authorized_keys10:26
paulsherwoodi'll need to do the latter10:27
Kinnisonokay, stand by10:27
Kinnisonbiff :-)10:27
paulsherwoodwhich directory is that file in?10:28
Kinnisonthe ~ of the user you ssh in as10:28
Kinnison~/.ssh/authorized_keys10:28
Kinnisonit will have one key per line10:28
paulsherwoodKinnison: tvm. please try now10:30
Kinnisonpaulsherwood: did you let me know the user and address?10:30
paulsherwoodelsewhere, yes10:30
Kinnisonaah so you did10:30
Kinnisonyep I can get in10:30
* Kinnison has a poke around10:30
*** toscalix__ has quit IRC10:31
*** flatmush has joined #baserock10:33
KinnisonOkay, so the critical difference I can see is that in the failed build, the ld-linux-aarch64.so.1 points at the stage2 libc10:41
Kinnisonwhereas in the succeeded build, it points at the stage 3 libc10:41
Kinnisonwhich implies an inconsistency in unpack ordering for the assembly area10:42
Kinnisondespite what paulsherwood mentioned to me in person this morning10:42
* Kinnison wonders what could cause such a discrepancy10:42
* paulsherwood too10:42
Kinnisonpaulsherwood: when you said to me that you didn't think it was ordering related this morning; does that mean you believed the failde and succeeded builds to have the same unpack order?10:43
paulsherwoodKinnison: i believe that i have seen failed and succeeding builds after fixing the the unpack order. i'm not sure whether the unpack order in those two directories is the same11:00
paulsherwoodwin 5711:00
paulsherwood:/11:00
Kinnisonpaulsherwood: well, the fundamental difference between those two dirs is that in the failed one the stage 3 libc was not unpacked after the stage 2 one11:00
Kinnisonpaulsherwood: whether that's a red-herring or not, I can't easily say11:01
paulsherwoodi can re-run, assuming that machine is not going to die11:02
Kinnisontlsa: We'll not kill ciat-large-0 for now, will we?11:02
tlsaI don't plan to11:04
Kinnisonexcellent11:04
Kinnisonpaulsherwood: Have at it11:04
Kinnisonpaulsherwood: if you can reproduce a success and failure with the unpack ordering the same in both, then I'll investigate further11:04
Kinnisonpaulsherwood: I would note however, that the build-depends list in the stratum is to be considered a strictly ordered list for unpack purposes11:05
Kinnisonpaulsherwood: so it's worth making sure ybd always unpacks in the right order anyway, to not run into issues in future11:05
paulsherwoodack11:10
ssam2is the Baserock on Moonshot armv8 work complete?11:39
*** toscalix__ has joined #baserock11:43
*** petefoth has quit IRC11:47
*** petefoth has joined #baserock11:49
persiassam2: I've not tried it for many months, but it was working well in the past, and there are other reports of working aarch64 builds recently.12:05
Kinnisonssam2: AIUI the armv8l64 port was working; moonshot deployment was a tad scrunchy which is partly resolved by the partitioned deployment stuff still in gerrit12:11
Kinnisonssam2: I believe armv8b64 was in a similar state.12:11
Kinnisonssam2: there had been little to no work on armv8l32 or armv8b3212:11
persiaI had armv8l32 working at one point, but got distracted by trying to get armv7l running on the hardware.12:20
paulsherwoodKinnison: it occurs to me that i may have mis-tested previously12:27
paulsherwood(ie correct order for gcc, after building other components with random order)12:28
KinnisonThat wouldn't help, certainly12:28
paulsherwoodi remain confused about why this would not have happened on other architectures though12:29
KinnisonAt least x86 is incredibly forgiving12:30
Kinnisonalso we manage to push /lib64 to /lib on x86, but for some reason that fails on aarch6412:30
KinnisonIn general "You have been lucky so far"12:30
paulsherwoodpossibly :)12:32
*** tiagogomes_ has quit IRC12:52
*** tiagogomes_ has joined #baserock12:53
*** toscalix__ has quit IRC13:09
*** bashrc_ has quit IRC13:26
*** bashrc_ has joined #baserock13:28
ssam2cool. I guess we can close the armv8 story then: https://storyboard.baserock.org/#!/story/1214:07
ssam2edcragg: do you think there is more to do on https://storyboard.baserock.org/#!/story/12 ?14:07
jjardonssam2: hey, do you plan to  test the "python3 by default" stuff (thanks for the review btw) as is or do you want me to update the patch first? I ask because change the patch will require quite a lot of work14:13
ssam2jjardon: i don't have a plan to test it right now14:19
ssam2i think that change is quite important before we can merge it. but it's not urgent at all14:19
ssam2maybe it would be quicker to write a 'rename-stratum' tool than to do it manually :-)14:20
*** zoli__ has quit IRC14:21
*** brlogger has joined #baserock14:37
*** zoli__ has joined #baserock15:01
*** zoli__ has quit IRC15:01
*** zoli__ has joined #baserock15:02
*** zoli__ has quit IRC15:06
richard_mawssam2: did you merge a change to remove support for older definitions versions recently?15:15
pdar_I think so15:16
*** pdar_ is now known as pdar15:16
*** ssam2_ has joined #baserock15:19
*** ssam2 has quit IRC15:19
pedroalvarezthat was merged on friday15:19
pedroalvarezhttps://gerrit.baserock.org/#/c/1001/15:19
*** bashrc_ has quit IRC15:20
ssam2_richard_maw: yes15:23
ssam2_i'm having some weird issue with Morph master, in fact, where Bison fails to build15:23
ssam2_fails to configure, in fact15:23
ssam2_but it builds in the CI, so not sure what's wrong15:23
richard_mawgot a log?15:24
*** zoli__ has joined #baserock15:27
*** ssam2_ has quit IRC15:29
*** ssam2 has joined #baserock15:30
*** ChanServ sets mode: +v ssam215:30
ssam2richard_maw: http://sprunge.us/WPgW15:32
ssam2I can't reproduce the error in the failed staging area, I get a different error instead15:32
richard_mawssam2: it's failing to determine its own version15:33
ssam2indeed15:34
richard_mawI think bison was one of those components where we were supposed to manually inject the version, rather than let it use git describe15:34
ssam2we switched back to letting them all use git describe15:34
ssam2but anyway, it builds in the CI. so I think this goes a bit deeper15:34
ssam2I'm going to compare with the build log from mason-x86-64.baserock.org15:34
richard_mawis CI using current morph?15:34
ssam2no, it's using commit 60c378c55d5d0ef89184b49ae95e445f8de422e3 I think15:35
tiagogomes_If it builds in one place and not in another, have you tried to set max-jobs to 1?15:35
ssam2tiagogomes_: it's not getting as far as running 'make', though15:35
tiagogomes_that's true15:36
ssam2another fun part is Linux seems to basically freeze whenever I try to build Bison locally at the moment. Not sure what's going on there15:36
ssam2seems to be dying a swap death, I managed to catch it that time. Which is weird because for one thing i'm pretty sure I disabled overcommit and swapping15:37
ssam2and also of course it shouldn't need 4GB + of RAM to configure Bison15:38
ssam2this is with commit 60c378c55d5d0ef89184b49ae95e445f8de422e3 of Morph, too15:38
ssam2I guess I can try the same thing in an OpenStack instance. could just be that my laptop is hosed in some crazy way15:39
ssam2or the distro running on it is broken somehow15:39
richard_mawssam2: we did see some weirdness earlier on your laptop15:40
ssam2i'll try to reproduce the issue on a clean VM15:42
*** bashrc_ has joined #baserock15:42
*** bashrc_ has quit IRC15:44
*** bashrc_ has joined #baserock15:44
*** paulw has quit IRC15:45
*** zoli__ has quit IRC16:05
*** mariaderidder has quit IRC16:17
*** zoli__ has joined #baserock16:28
*** zoli__ has quit IRC16:38
*** zoli__ has joined #baserock16:40
*** zoli__ has quit IRC16:42
*** paulw has joined #baserock16:48
*** zoli__ has joined #baserock16:50
ssam2i managed to get the horrible hanging issue to reproduce under strace16:50
ssam2seems that the Bison pre-configure-commands somehow produce a fork bomb16:51
ssam2not very clear why though16:51
ssam2the clue must be somewhere in this 6.5 million line log file ...16:52
*** zoli__ has quit IRC16:52
ssam2I think `git` might be to blame, somehow16:54
pedroalvarezyou always find the weirdest bugs...16:54
ssam2heh16:56
ssam2from these logs, it does look a lot like `git status --porcelain` keeps forking and reexecuting itself16:56
*** bashrc_ has quit IRC16:57
ssam2it seems to do stat("gnulib/.git") just before16:57
ssam2aha: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=75068717:11
ssam2which also raises the question of why that dir isn't readable... it explains why in other situations, the process works fine but fails to read its version number from .git17:12
ssam2I guess the root cause is some file system weirdness that makes the permissions of the .git directories in the staging area be messed up, and the various errors I'm seeing are symptoms of that17:13
*** ssam2 has quit IRC17:16
*** zoli__ has joined #baserock17:16
*** zoli__ has quit IRC17:18
*** brlogger has joined #baserock17:48
*** jonathanmaw has quit IRC18:23
*** paulw has quit IRC18:40
*** zoli__ has quit IRC19:17
*** zoli__ has joined #baserock19:22
*** zoli__ has quit IRC20:27
*** zoli__ has joined #baserock20:36
*** radiofree has quit IRC20:41
*** radiofree has joined #baserock20:43
*** paulw has joined #baserock21:14
*** paulw has quit IRC21:14
* paulsherwood manages to build bison from definitions master using ybd21:37
*** zoli__ has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!