IRC logs for #baserock for Tuesday, 2017-10-24

*** juergbi has quit IRC01:19
*** juergbi has joined #baserock03:45
*** gtristan has joined #baserock04:37
*** locallycompact has joined #baserock06:26
*** gtristan has quit IRC06:53
*** jude_ has joined #baserock06:55
*** jude_ has quit IRC07:18
*** jude_ has joined #baserock07:20
*** gtristan has joined #baserock07:30
*** locallycompact has quit IRC07:54
*** tiagogomes has joined #baserock08:07
*** locallycompact has joined #baserock08:44
*** tiagogomes has quit IRC09:09
*** ssam2 has joined #baserock09:10
*** ChanServ sets mode: +v ssam209:10
ssam2it didn't work, of course :
ssam2something must have broken recently in the conversion process... no way that should take 8 hours09:10
*** tiagogomes has joined #baserock09:17
*** tiagogomes has quit IRC09:20
*** tiagogomes has joined #baserock09:20
ssam2ouch, i've spotted at least one issue09:38
ssam2which is that ybd stats every file in the current working directory09:38
ssam2and since we have to have the cache in the build directory, that means it'll stat every file in the cache09:38
ssam2paulsherwood: could we make ybd ignore hidden directories when looking for .morph files09:44
ssam2then we could call the cache dir `.cache` and solve that09:44
ssam2i guess ybd is also pointlessly traversing the `.git` directory at the moment09:44
paulsherwoodmakes sense09:44
ssam2ok, i'll do a patch09:44
benbrown_ssam2: that'd be os.walk09:48
ssam2it seems like the code checks for hidden dirs, but does so *after* os.walk has touched every subdir09:49
*** tiagogomes has quit IRC10:05
*** tiagogomes has joined #baserock10:19
ssam2morph-cache-server was broken on after the migration, i've fixed that now. YBD CI is a lot faster :-)10:36
ssam2could someone review the patch please ?
gtristanssam2, added comment, paulsherwood would probably know best how safe it is to revert that10:49
paulsherwoodssam2: what if directory != './' ? is that no longer possible in any code path?10:52
ssam2paulsherwood, it's inside a `chdir(directory)` context manager11:16
ssam2so that should still work fine (not sure how I test that though, is there a flag I can pass to YBD to change the directory?)11:16
ssam2i've updated the commit message to make it clearer that we now ignore all hidden dirs11:20
benbrown_ssam2: lgtm11:26
benbrown_paulsherwood, ssam2: Yeah, the chdir should ensure that11:28
benbrown_ssam2: the .def suffix did pick up a Makefile.def in a git repo in definitions, which confused me momentarily11:31
benbrown_but I renamed it so it was hidden and it was ignored :)11:32
ssam2oh hang on, i've reintroduced the .def suffix11:32
ssam2that wasn't deliberate11:32
benbrown_ah, wasn't sure if it was11:32
ssam2if it was, i should have said so in the commit message anyway :-)11:33
ssam2doesn't seem to have fixed
ssam2it seems like YBD is stuck waiting for a zombie child process12:14
ssam2oh, it's moved on now12:14
ssam2i do wonder what's going on though12:15
paulsherwoodssam2: none of that output is ybd?12:21
ssam2its defs2bst12:21
ssam2which calls YBD internally12:22
*** tiagogomes has quit IRC12:22
paulsherwoodcan you paste the ybd output somewhere?12:30
ssam2i don't know if it's recorded anywhere12:30
ssam2that's probably something that should be fixed in defs2bst12:30
ssam2 is finally building with buildstream and pushing to the cache ...12:36
ssam2it's still way too slow to get started because there's 13GB of sources in the cache, so it takes like 15 minutes to fetch and unpack everything12:36
benbrown_that is rather large12:37
ssam2it includes 2.4GB worth of failed linux clones in tmpdirs12:38
ssam2that can at least be removed12:38
ssam2other than that... very little we can do, we are building the world after all12:39
*** gtristan has quit IRC13:03
*** laurenceurhegyi is now known as ltu13:06
*** tiagogomes has joined #baserock13:08
ssam2a new breakage in the CI!
ssam2thankfully i can reproduce this one locally14:01
ssam2must be a buildstream regression; nothing has changed in definitions.git14:02
ssam2could be that there's nothing at /bin/sh14:05
ssam2`bst shell` manages to find /tools/bin/sh though14:05
ssam2maybe the issue is PATH not being set when integration commands run14:05
*** CTtpollard has quit IRC14:05
paulsherwoodlooking at that log... am i right that there's no way of distinguishing the various SUCCESS messages?14:05
* paulsherwood expressly put unique words in ybd logs containing 'Elapsed' to make it easy to grep for build times, for example14:06
ssam2to the left of the word 'SUCCESS' is the name of the task14:07
paulsherwoodright, but there are several build: SUCCESS messages for (say) stage2-fake-bash.bst]14:09
benbrown_grep for "SUCCESS *-build.*.log"?14:10
benbrown_though, the tasks are prefixed by 'build: ' anyway14:11
*** gtristan has joined #baserock14:23
ssam2gtristan, are you about to quickly explain something in the new world order of YAML composition ?14:25
ssam2in gnu-toolchain/fhs-dirs.bst we do something like: `environment: { PATH: /usr/bin:/tools/bin}`14:25
ssam2that fails to override the default PATH set in the project defaults14:26
ssam2seems like a bug actually, the docs say default behaviour is to replace the list14:27
* ssam2 will try and make a test case14:27
gtristanthat aint a list14:56
gtristanssam2, but it can be a bug indeed, this popped up today too:
gtristanI'm quite surprised though, if that is a bug; I tried something similar just this weekend and I thought it worked14:58
ssam2hmmm true that it's not a list14:59
gtristanstill the string should overwrite the other for that element14:59
ssam2it seems like the test cases already cover this kind of thing, and it works in those14:59
ssam2i tried to create a new test case with the exact setup, but that also passes14:59
gtristanReally ?!15:00
gtristanwtf ?15:00
ssam2so, possibly some pecularity in Element.__extract_env()15:00
ssam2or i've missed something, of course -- i've not really looked at this code before15:00
gtristanAh, you constructed a test case without involving the project / element compositing mechanics15:00
ssam2yeah, i put it in tests/yaml/yaml.py15:00
ssam2i'll try putting one in tests/variables/ somewhere that's a bit more realistic15:00
gtristanyeah, I want to keep the yaml tests around, but otherwise, most/all new tests should use testutils/ stuff15:01
gtristanand test real things15:01
gtristanssam2, do it in tests/format I think15:01
gtristanssam2, there we mostly use `bst show` to test ... right now thats where the option tests live, but eventually the loader tests should be migrated to that too15:01
gtristanalways test from the whole cli, dont bother bone picking with internals, not a good approach15:02
ssam2i agree in general15:02
gtristan(still, I wanna keep tests/yaml/, and maaaybe the plugin loader stuff)15:02
* gtristan notes, it should be very easy to add a test to tests/format too, just `cp -a a_test_directory new_test_directory`, do this override in the element.bst and `bst show --format %{env}`15:05
gtristan(most of the test directories around there I think, just have a project.conf and an element.bst)15:07
* gtristan has a feeling the bug is somewhere in indeed, it smells like the same test for issue 127 would probably not be reproducible in tests/yaml/ either)15:08
ssam2interestingly, `bst show --format '%{env}' --deps none gnu-toolchain/fhs-dirs.bst` shows the correct behaviour15:10
ssam2it seems to only be wrong when running integration commands15:10
gtristanmaybe it's not picking up the integration commands from the right place15:11
ssam2that's using element._Element__environment15:11
gtristanalthough, not sure where else it could come from15:12
gtristanI mean, the env15:12
ssam2where element.get_environment() uses self.__environment15:12
gtristanAh, you found the bug !15:12
ssam2did i? good :-)15:12
gtristansounds like - so wait, which one is using which ??15:12
gtristanNah, that is the same thing15:13
ssam2yeah, i thought it was15:13
gtristanelement._Element__environment should be replaced with the accessor, just adding noise15:13
ssam2I added print() statements around the composition, and that seems to be done wrong15:13
ssam2but given that, how does the format codepath work ??15:13
gtristannode_sanitize ?15:14
gtristanoh strange, it's called by both places15:15
ssam2i just traced through the format codepath, and final_env actually ends up with the correct value15:16
gtristanbut not when running bst build, specifically for integration commands15:16
ssam2same with the build codepath ...15:16
ssam2but that must be being lost somewhere15:16
gtristanso `bst build`... has the right environment when running the sandbox in ->assemble() ?15:17
gtristanBut not at integration command time ?15:17
ssam2this could be something much more obvious15:17
ssam2such as the integration command is on a different element15:17
gtristanIn which case, there's no bug :)15:18
ssam2it did use to work though15:18
ssam2so maybe there was a bug before :-)15:18
ssam2perhaps stack elements would previously inherit environment from their children, or something weird15:18
gtristanQuestion is, do you think it works the way it should ?15:18
ssam2if I can set environment: in the gnu-toolchain/stage2.bst element, that seems fine15:19
ssam2and correct15:19
gtristanNah, that sounds very unlikely15:19
gtristanand stack elements dont run integration anyway, but yeah they might provide commands for depending elements15:19
ssam2it does seem to work15:19
gtristanso yeah, elements run the stack integration commands15:20
gtristanbut, the question remains: Is the environment used for building something, the same as the environment used for integrating *against* that thing, using the commands it provides ?15:20
gtristanthis seems unclear to me, and worth thinking on15:20
ssam2well, it seems like the behaviour is now that the stack runs the integration commands15:21
gtristanssam2, i.e. in your specific case, you are working around this with a stack15:21
gtristannormally what happens, is something like; you build GTK+, which installs gtk-query-immodules, then things which depend on GTK+, have the input method modules queried before they build15:22
gtristan(or gdk-pixbuf loaders, etc, that kinda thing)15:22
ssam2this is a strange case i guess, where the stack decides that ldconfig needs to be run15:22
ssam2actually i guess this is working around the fact that we need stage2 to be cross-buildable15:23
ssam2otherwise we could have ldconfig as an integration command of stage2-glibc15:23
gtristanI also added stage1, stage2 stacks because they made the thing more comprehensive15:23
ssam2yeah, i like them15:23
gtristanbefore you changed it, and also used similar stacks15:23
gtristanIt allows one to easily drop them behind with build type dependencies15:24
gtristanI think was the original point15:24
gtristan(without verbosely build depending on dozens of elements)15:24
gtristanwell - I kind of feel like; the element which built something (like gdk-pixbuf for instance), is the one who knows the environment which should be used15:25
gtristanfor integrating itself on behalf of a reverse dependency15:25
gtristanBut, feel free to raise the issue on the ML if you like, it might be worth exploring15:25
* gtristan thinks it's behaving right - BUT - I cant explain how this behavior could have *changed* in buildstream15:26
ssam2i think this behaviour makes sense15:26
gtristanthis is highly unlikely, too15:26
ssam2it's definitely new, though15:27
gtristanthat said, I think we're doing things better now - I dont understand how the behavior could have changed, *however*, it's possible we fixed some edge cases in dictionary compositing in general15:27
gtristane.g. the new ChainMap derived copy-on-write dictionary fixes some things; and the code is a bit less spaghetti around now15:28
ssam2there was a successful build with bst a580a4a032c2e492c5287a976ced1db17846754915:28
ssam2which is before 'Implement array composition directives' landed15:29
ssam2maybe that fixed some edge case, indeed15:29
ssam2it was also before the new ChainMap15:30
*** locallycompact has quit IRC16:00
* paulsherwood wonders if buildstream on converted definitions is 'ready for primetime' yet?16:07
ssam2first we need to fix this CI16:07
ssam2then there are some outstanding branches to merge from the cross-bootstrap work myself and jonathanmaw did16:08
ssam2then we need to update for some changes in buildstream16:08
ssam2then we need to figure out how to host binary sysroots ('staging fillers' :-)16:09
ssam2then it's ready!16:09
ssam2hopefully that'll coincide with adds68 incorportating the important bits into the freedesktop base runtime16:09
ssam2all hopes pinned on this build!
* benbrown_ crosses fingers16:10
* paulsherwood vaguely remembers that he first got into gitlab ci as it was clocking through 10M builds16:10
adds68ssam2, i should hopefully have some bst files produced tomorrow morning i can push into the freedesktop-sdk repo :)16:11
adds68Then it's just fine-tuning and bug fixing in order to get the CI stood up!16:11
ssam2CI failure  again :-(16:15
ssam2i guess i'll just build locally first to find out how much more stuff has bitrotted16:15
ssam2HMMM... another chunk failing because /bin/sh is missing... which can't ever have worked16:19
ssam2but clearly did16:19
ssam2oh... maybe overlaps behaviour has changed16:22
ssam2and ruined that horrible thing we do with /bin symlinks16:22
ssam2building with buildstream commit a580a4a032c2e492c5287a976ced works, so *something* has changed in buildstream ... i wonder what16:28
*** jude_ has quit IRC16:29
ssam2fhs-dirs requires use of the (>) operator now :-)16:36
ssam2as does stage2-fhs-dirs... that no doubt explains the issue before as well16:38
ssam2as usual i spent hours barking up the wrong tree16:39
ironfootwoof woof16:39
ironfootwell, is good to find the fix for an issue just before beer time :)16:39
ssam2climbing time!16:42
gtristanfunny, we didnt *remove* post-/pre- commands yet17:06
gtristanbut anyway, should be using (<) and (>) instead yes17:06
ssam2it's not pre-commands, but an arch conditional that adds extra install-commands17:06
ssam2but of course, now it just overwrites the install-commands :-)17:06
ssam2simple to fix anyway17:07
*** ssam2 has quit IRC17:07
gtristanright, that behavior did change17:14
*** gtristan has quit IRC17:23
*** gitlab-br-bot has joined #baserock17:27
*** gitlab-br-bot has joined #baserock17:29
*** jude_ has joined #baserock18:34
*** gtristan has joined #baserock18:35
*** jude_ has quit IRC18:57
gitlab-br-botdefinitions: merge request (pedro/pbr-fix->master: Fix pbr builds for python3-core) #1 changed state ("closed"): undefined22:21

Generated by 2.15.3 by Marius Gedminas - find it at!