| *** Stanto has quit IRC | 02:15 | |
| *** Stanto has joined #baserock | 02:19 | |
| *** paulw has joined #baserock | 06:15 | |
| *** fay_ has joined #baserock | 07:36 | |
| *** bashrc_ has joined #baserock | 08:02 | |
| *** mariaderidder has joined #baserock | 08:07 | |
| *** toscalix__ has joined #baserock | 08:26 | |
| *** rdale has joined #baserock | 08:27 | |
| *** jonathanmaw has joined #baserock | 08:34 | |
| *** ssam2 has joined #baserock | 09:11 | |
| *** ChanServ sets mode: +v ssam2 | 09:11 | |
| Kinnison | tiagogomes_: when you did the armv8l64 support / bootstrap, do you recall ever having an issue where the stage 2 compilers, or the stage 3 libc would miscompile and thus break the build? | 09:44 |
|---|---|---|
| paulsherwood | what i'm seeing is gcc failing | 09:44 |
| Kinnison | That's the manifestation of it, yes | 09:45 |
| Kinnison | I'm unsure if it's the stage 2 compilers, or the stage 3 libc which is broken, leading to the failure to compile the stage 3 compiler | 09:45 |
| Kinnison | I suppose it could be something else, but if the ordering wasn't it, then I'm stuck for other options | 09:45 |
| tiagogomes_ | Kinnison no, the issues that I had were only due bad configuration. The compiler itself was working fine | 09:46 |
| Kinnison | tiagogomes_: we have an issue right now, where gcc (stage 3) failed to build, because it claims it can't find libz.so.1 | 09:46 |
| Kinnison | though that is present in /usr/lib and in the ld.so.cache | 09:46 |
| Kinnison | tiagogomes_: however, unlike my expectations, the libc is in /lib64 not in /lib which surprises me | 09:47 |
| Kinnison | tiagogomes_: though I can't remember how it looked by the time we'd finished the initial armv8l64 work | 09:47 |
| tiagogomes_ | I can't recall either. Are you trying to build it with ybd? Can you try it with morph to take out the build tool of the equation? | 09:52 |
| Kinnison | Sadly we don't have an easy way to use morph on the units right now | 09:53 |
| Kinnison | they're running ubuntu and we don't have chroots to hand | 09:53 |
| tiagogomes_ | have you tried to compile a test.c with the stage2-gcc and stage2-glibc to do a basic test to the toolchain | 10:04 |
| Kinnison | We're currently rearranging the systems, once that's done, if we encounter the issue again, I'll take a look | 10:04 |
| Kinnison | It managed to compile the stage 3 libc | 10:05 |
| Kinnison | and to compile zlib against the stage 3 libc | 10:05 |
| Kinnison | But somehow fails when compiling something against that zlib | 10:05 |
| tiagogomes_ | mm, but then it finds zlib? | 10:06 |
| Kinnison | Well the error message is that it fails to find zlib | 10:07 |
| Kinnison | but zlib is in place | 10:07 |
| Kinnison | I personally think it's failing to find libc | 10:07 |
| Kinnison | but I can't be sure | 10:07 |
| jjardon | ssam2: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/log/?h=jjardon/gnome | 10:10 |
| ssam2 | hoooray! | 10:12 |
| tiagogomes_ | Kinnison looking at the comments that I wrote when ugrading bintutils and gcc, it is true that libc should be on /lib | 10:13 |
| Kinnison | tiagogomes_: I wonder why it's not going into /lib then on these builds | 10:13 |
| * paulsherwood notes also that this is only failing *sometimes* which is depressing. | 10:16 | |
| Kinnison | indeed | 10:17 |
| Kinnison | intermittent errors make us sad | 10:17 |
| tiagogomes_ | :'( That makes things much harder | 10:17 |
| *** flatmush has quit IRC | 10:18 | |
| paulsherwood | well if the machines haven't been reimaged, i did capture the build area | 10:22 |
| * Kinnison thinks tlsa may have dropped at least one of them on the floor to make room | 10:23 | |
| tlsa | yeah, I dropped one | 10:23 |
| paulsherwood | never mind, then | 10:23 |
| tlsa | ciat-large-0 is still there | 10:24 |
| paulsherwood | so /src/gcc.build.fail and /src/gcc.succeed | 10:25 |
| Kinnison | paulsherwood: are those the assembly build areas? | 10:25 |
| paulsherwood | yup | 10:25 |
| Kinnison | handy | 10:25 |
| Kinnison | paulsherwood: could you grant me access so I could take a look? | 10:25 |
| paulsherwood | Kinnison: i don't know how to do that? | 10:25 |
| Kinnison | paulsherwood: Either give me a password if there is one, or append my ssh public key (I can mail it to you) to .ssh/authorized_keys | 10:26 |
| paulsherwood | i'll need to do the latter | 10:27 |
| Kinnison | okay, stand by | 10:27 |
| Kinnison | biff :-) | 10:27 |
| paulsherwood | which directory is that file in? | 10:28 |
| Kinnison | the ~ of the user you ssh in as | 10:28 |
| Kinnison | ~/.ssh/authorized_keys | 10:28 |
| Kinnison | it will have one key per line | 10:28 |
| paulsherwood | Kinnison: tvm. please try now | 10:30 |
| Kinnison | paulsherwood: did you let me know the user and address? | 10:30 |
| paulsherwood | elsewhere, yes | 10:30 |
| Kinnison | aah so you did | 10:30 |
| Kinnison | yep I can get in | 10:30 |
| * Kinnison has a poke around | 10:30 | |
| *** toscalix__ has quit IRC | 10:31 | |
| *** flatmush has joined #baserock | 10:33 | |
| Kinnison | Okay, so the critical difference I can see is that in the failed build, the ld-linux-aarch64.so.1 points at the stage2 libc | 10:41 |
| Kinnison | whereas in the succeeded build, it points at the stage 3 libc | 10:41 |
| Kinnison | which implies an inconsistency in unpack ordering for the assembly area | 10:42 |
| Kinnison | despite what paulsherwood mentioned to me in person this morning | 10:42 |
| * Kinnison wonders what could cause such a discrepancy | 10:42 | |
| * paulsherwood too | 10:42 | |
| Kinnison | paulsherwood: when you said to me that you didn't think it was ordering related this morning; does that mean you believed the failde and succeeded builds to have the same unpack order? | 10:43 |
| paulsherwood | Kinnison: i believe that i have seen failed and succeeding builds after fixing the the unpack order. i'm not sure whether the unpack order in those two directories is the same | 11:00 |
| paulsherwood | win 57 | 11:00 |
| paulsherwood | :/ | 11:00 |
| Kinnison | paulsherwood: well, the fundamental difference between those two dirs is that in the failed one the stage 3 libc was not unpacked after the stage 2 one | 11:00 |
| Kinnison | paulsherwood: whether that's a red-herring or not, I can't easily say | 11:01 |
| paulsherwood | i can re-run, assuming that machine is not going to die | 11:02 |
| Kinnison | tlsa: We'll not kill ciat-large-0 for now, will we? | 11:02 |
| tlsa | I don't plan to | 11:04 |
| Kinnison | excellent | 11:04 |
| Kinnison | paulsherwood: Have at it | 11:04 |
| Kinnison | paulsherwood: if you can reproduce a success and failure with the unpack ordering the same in both, then I'll investigate further | 11:04 |
| Kinnison | paulsherwood: I would note however, that the build-depends list in the stratum is to be considered a strictly ordered list for unpack purposes | 11:05 |
| Kinnison | paulsherwood: so it's worth making sure ybd always unpacks in the right order anyway, to not run into issues in future | 11:05 |
| paulsherwood | ack | 11:10 |
| ssam2 | is the Baserock on Moonshot armv8 work complete? | 11:39 |
| *** toscalix__ has joined #baserock | 11:43 | |
| *** petefoth has quit IRC | 11:47 | |
| *** petefoth has joined #baserock | 11:49 | |
| persia | ssam2: I've not tried it for many months, but it was working well in the past, and there are other reports of working aarch64 builds recently. | 12:05 |
| Kinnison | ssam2: AIUI the armv8l64 port was working; moonshot deployment was a tad scrunchy which is partly resolved by the partitioned deployment stuff still in gerrit | 12:11 |
| Kinnison | ssam2: I believe armv8b64 was in a similar state. | 12:11 |
| Kinnison | ssam2: there had been little to no work on armv8l32 or armv8b32 | 12:11 |
| persia | I had armv8l32 working at one point, but got distracted by trying to get armv7l running on the hardware. | 12:20 |
| paulsherwood | Kinnison: it occurs to me that i may have mis-tested previously | 12:27 |
| paulsherwood | (ie correct order for gcc, after building other components with random order) | 12:28 |
| Kinnison | That wouldn't help, certainly | 12:28 |
| paulsherwood | i remain confused about why this would not have happened on other architectures though | 12:29 |
| Kinnison | At least x86 is incredibly forgiving | 12:30 |
| Kinnison | also we manage to push /lib64 to /lib on x86, but for some reason that fails on aarch64 | 12:30 |
| Kinnison | In general "You have been lucky so far" | 12:30 |
| paulsherwood | possibly :) | 12:32 |
| *** tiagogomes_ has quit IRC | 12:52 | |
| *** tiagogomes_ has joined #baserock | 12:53 | |
| *** toscalix__ has quit IRC | 13:09 | |
| *** bashrc_ has quit IRC | 13:26 | |
| *** bashrc_ has joined #baserock | 13:28 | |
| ssam2 | cool. I guess we can close the armv8 story then: https://storyboard.baserock.org/#!/story/12 | 14:07 |
| ssam2 | edcragg: do you think there is more to do on https://storyboard.baserock.org/#!/story/12 ? | 14:07 |
| jjardon | ssam2: hey, do you plan to test the "python3 by default" stuff (thanks for the review btw) as is or do you want me to update the patch first? I ask because change the patch will require quite a lot of work | 14:13 |
| ssam2 | jjardon: i don't have a plan to test it right now | 14:19 |
| ssam2 | i think that change is quite important before we can merge it. but it's not urgent at all | 14:19 |
| ssam2 | maybe it would be quicker to write a 'rename-stratum' tool than to do it manually :-) | 14:20 |
| *** zoli__ has quit IRC | 14:21 | |
| *** brlogger has joined #baserock | 14:37 | |
| *** zoli__ has joined #baserock | 15:01 | |
| *** zoli__ has quit IRC | 15:01 | |
| *** zoli__ has joined #baserock | 15:02 | |
| *** zoli__ has quit IRC | 15:06 | |
| richard_maw | ssam2: did you merge a change to remove support for older definitions versions recently? | 15:15 |
| pdar_ | I think so | 15:16 |
| *** pdar_ is now known as pdar | 15:16 | |
| *** ssam2_ has joined #baserock | 15:19 | |
| *** ssam2 has quit IRC | 15:19 | |
| pedroalvarez | that was merged on friday | 15:19 |
| pedroalvarez | https://gerrit.baserock.org/#/c/1001/ | 15:19 |
| *** bashrc_ has quit IRC | 15:20 | |
| ssam2_ | richard_maw: yes | 15:23 |
| ssam2_ | i'm having some weird issue with Morph master, in fact, where Bison fails to build | 15:23 |
| ssam2_ | fails to configure, in fact | 15:23 |
| ssam2_ | but it builds in the CI, so not sure what's wrong | 15:23 |
| richard_maw | got a log? | 15:24 |
| *** zoli__ has joined #baserock | 15:27 | |
| *** ssam2_ has quit IRC | 15:29 | |
| *** ssam2 has joined #baserock | 15:30 | |
| *** ChanServ sets mode: +v ssam2 | 15:30 | |
| ssam2 | richard_maw: http://sprunge.us/WPgW | 15:32 |
| ssam2 | I can't reproduce the error in the failed staging area, I get a different error instead | 15:32 |
| richard_maw | ssam2: it's failing to determine its own version | 15:33 |
| ssam2 | indeed | 15:34 |
| richard_maw | I think bison was one of those components where we were supposed to manually inject the version, rather than let it use git describe | 15:34 |
| ssam2 | we switched back to letting them all use git describe | 15:34 |
| ssam2 | but anyway, it builds in the CI. so I think this goes a bit deeper | 15:34 |
| ssam2 | I'm going to compare with the build log from mason-x86-64.baserock.org | 15:34 |
| richard_maw | is CI using current morph? | 15:34 |
| ssam2 | no, it's using commit 60c378c55d5d0ef89184b49ae95e445f8de422e3 I think | 15:35 |
| tiagogomes_ | If it builds in one place and not in another, have you tried to set max-jobs to 1? | 15:35 |
| ssam2 | tiagogomes_: it's not getting as far as running 'make', though | 15:35 |
| tiagogomes_ | that's true | 15:36 |
| ssam2 | another fun part is Linux seems to basically freeze whenever I try to build Bison locally at the moment. Not sure what's going on there | 15:36 |
| ssam2 | seems to be dying a swap death, I managed to catch it that time. Which is weird because for one thing i'm pretty sure I disabled overcommit and swapping | 15:37 |
| ssam2 | and also of course it shouldn't need 4GB + of RAM to configure Bison | 15:38 |
| ssam2 | this is with commit 60c378c55d5d0ef89184b49ae95e445f8de422e3 of Morph, too | 15:38 |
| ssam2 | I guess I can try the same thing in an OpenStack instance. could just be that my laptop is hosed in some crazy way | 15:39 |
| ssam2 | or the distro running on it is broken somehow | 15:39 |
| richard_maw | ssam2: we did see some weirdness earlier on your laptop | 15:40 |
| ssam2 | i'll try to reproduce the issue on a clean VM | 15:42 |
| *** bashrc_ has joined #baserock | 15:42 | |
| *** bashrc_ has quit IRC | 15:44 | |
| *** bashrc_ has joined #baserock | 15:44 | |
| *** paulw has quit IRC | 15:45 | |
| *** zoli__ has quit IRC | 16:05 | |
| *** mariaderidder has quit IRC | 16:17 | |
| *** zoli__ has joined #baserock | 16:28 | |
| *** zoli__ has quit IRC | 16:38 | |
| *** zoli__ has joined #baserock | 16:40 | |
| *** zoli__ has quit IRC | 16:42 | |
| *** paulw has joined #baserock | 16:48 | |
| *** zoli__ has joined #baserock | 16:50 | |
| ssam2 | i managed to get the horrible hanging issue to reproduce under strace | 16:50 |
| ssam2 | seems that the Bison pre-configure-commands somehow produce a fork bomb | 16:51 |
| ssam2 | not very clear why though | 16:51 |
| ssam2 | the clue must be somewhere in this 6.5 million line log file ... | 16:52 |
| *** zoli__ has quit IRC | 16:52 | |
| ssam2 | I think `git` might be to blame, somehow | 16:54 |
| pedroalvarez | you always find the weirdest bugs... | 16:54 |
| ssam2 | heh | 16:56 |
| ssam2 | from these logs, it does look a lot like `git status --porcelain` keeps forking and reexecuting itself | 16:56 |
| *** bashrc_ has quit IRC | 16:57 | |
| ssam2 | it seems to do stat("gnulib/.git") just before | 16:57 |
| ssam2 | aha: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687 | 17:11 |
| ssam2 | which also raises the question of why that dir isn't readable... it explains why in other situations, the process works fine but fails to read its version number from .git | 17:12 |
| ssam2 | I guess the root cause is some file system weirdness that makes the permissions of the .git directories in the staging area be messed up, and the various errors I'm seeing are symptoms of that | 17:13 |
| *** ssam2 has quit IRC | 17:16 | |
| *** zoli__ has joined #baserock | 17:16 | |
| *** zoli__ has quit IRC | 17:18 | |
| *** brlogger has joined #baserock | 17:48 | |
| *** jonathanmaw has quit IRC | 18:23 | |
| *** paulw has quit IRC | 18:40 | |
| *** zoli__ has quit IRC | 19:17 | |
| *** zoli__ has joined #baserock | 19:22 | |
| *** zoli__ has quit IRC | 20:27 | |
| *** zoli__ has joined #baserock | 20:36 | |
| *** radiofree has quit IRC | 20:41 | |
| *** radiofree has joined #baserock | 20:43 | |
| *** paulw has joined #baserock | 21:14 | |
| *** paulw has quit IRC | 21:14 | |
| * paulsherwood manages to build bison from definitions master using ybd | 21:37 | |
| *** zoli__ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!