IRC logs for #buildstream for Monday, 2018-01-22

*** noisecell has joined #buildstream08:37
*** valentind has joined #buildstream08:55
*** bethw has joined #buildstream09:19
*** tiago has quit IRC09:36
*** tiago has joined #buildstream09:45
*** valentind has quit IRC09:46
*** jonathanmaw has joined #buildstream09:48
*** bethw has quit IRC09:52
*** ssam2 has joined #buildstream10:16
gitlab-br-botbuildstream: merge request (jonathan/bzr-source-method->master: Jonathan/bzr source method) #242 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24210:59
*** cs_shadow has joined #buildstream11:00
ssam2juergbi, is there a way to have a 'junction' element work when the project.conf is not at the top level of the repo ?11:06
ssam2i wanted to try building gnome-build-meta against the sdk project from https://gitlab.com/freedesktop-sdk/freedesktop-sdk11:06
ssam2using the sam/use-recursive-pipelines branch did for that project11:06
ssam2but seems impossible to use it because the project that I need is in the sdk/ subdirectory11:07
juergbihm, no, i don't think this is currently possible11:07
juergbiwe can't specify a subdirectory in the git source11:07
juergbiand there is no junction-specific way to specify a patch11:07
juergbiadding a junction config for this would likely be fairly simple, though11:08
ssam2ah right, so like a 'project-subpath' option that would go in the `config` section ?11:08
juergbimight be useful as alternative to the ../ paths we're about to disallow11:08
ssam2seems logical11:08
juergbiyes, unlike build elements where you can handle such thing via commands, we need explicit config option in the case of junctions11:09
ssam2i can have a go at a patch11:11
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23311:11
juergbithanks11:20
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24511:34
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23311:47
tlaterssssam: Why were your tests not possible using the old-style integration tests?12:09
tlaterWas it down to symlinks not being tracked correctly?12:10
tlaterSorry, ssam2 ^12:13
ssam2it's because i wanted to detect the error raised by symlinks pointing outside the sandbox12:22
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23312:22
ssam2the old test framework didn't support 'expected fail' tests12:22
tlaterAh, right, yeah, that would have been difficult12:22
*** tlaxkit has joined #buildstream12:23
jmacssam2: https://gitlab.com/BuildStream/buildstream/merge_requests/248 is the MR I mentioned to you earlier. It appears to have a pipeline failure which I'm unsure how to interpret.12:30
tlaterjmac: the 'source_dist' job may be failing due to the changes to the cache that are about to happen12:31
jmacRight.12:32
tlaterSorry if that's disrupting, clearing the runner cache should solve these issues - maybe I should look into using a separate cache12:32
jmacI noticed when I fork a repo on gitlab, it still tried to run automated tests on my local copy - is that normal?12:32
ssam2what do you mean by 'local copy' ?12:32
tlaterYup, when you clone it you copy the .gitlab-ci.yaml12:32
ssam2ah, the copy in your personal gitlab account ?12:33
jmacYes12:33
jmacBut presumably the existing runners won't pick that up12:33
ssam2right. what tlater said :-)12:33
ssam2but yes, runners are enabled per-project12:33
ssam2so you will get the tests running on the GitLab CI free-to-use runners12:33
ssam2which are slower and have 1 hour timeouts, so things might fail due to that12:33
ssam2although you can configure the timeout12:34
* tlater remembers he can do that and avoids testing with the main cache from here on12:34
jmacGitLab gives out free test running VMs? That's generous12:34
ssam2indeed12:34
jmacMy DogeCoin mining code needs lots of testing12:36
ssam2yes, i don't know how they deal with that12:36
ssam2presumably just manual review of projects using high CPU12:36
*** xjuan has joined #buildstream12:47
tlaterHm. I'm not sure why my cache files aren't being uploaded12:54
ssam2juergbi, should i do a merge request for the junction element 'path' option or do you want to review and cherry pick it directly into your branch ?12:54
ssam2it's pushed as sam/recursive-pipelines-in-subdir either way12:54
tlaterI've checked a few times now, there should be something in the directory I'm trying to cache12:55
juergbissam2: i think the latter makes more sense, ta12:55
tlaterIs it possible that the gitlab runner clears /tmp if it gets too full or something?12:57
ssam2are you trying to cache stuff from /tmp ?12:58
tlaterssam2: Yup, trying to change that now13:00
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23313:09
tlaterssam2: About the docker images... Do you want to keep with the '-' names? I.e., something like buildstream/testsuite-debian-8?13:11
ssam2yeah13:12
ssam2although, testsuite-debian13:12
ssam2with tag :813:12
tlaterRight, yeah, that makes sense13:13
ssam2i guess we could do testsuite:debian8; but it would be inconsistent with the existing buildstream-fedora image13:14
ssam2and also the UI for managing tags isn't that great, so if we have one image with loads of different tags it'll probably be confusing13:14
* tlater wonders how to do this without exploding the .gitlab-ci.yaml file13:14
ssam2you should only have to add a couple of lines13:14
ssam2one to call `docker build`, and two to call `docker push`13:15
ssam2although i wonder if the 'fixed tag, moving tag' concept is even relevant here13:15
tlaterYeah, but if we're going to have more platforms this will get a bit messy13:15
ssam2the idea of having a 'moving tag' for the buildstream-fedora image was so that people could pull 'latest'13:15
tlaterThis seems entirely pointless for a testuite image13:15
ssam2but the testsuite image will only be used by gitlab-ci, where we'll never want to do that13:15
ssam2yeah13:15
ssam2i don't think gitlab-ci.yml will get too messy if we have a few more distro bases13:16
tlaterI suppose we can cross that bridge when we get there anyway13:16
ssam2i'd be as worried the effort of maintaining all those different bases every time we have a change in our dependency set13:16
ssam2*about the effort13:16
ssam2long term distros should just package buildstream !13:17
tlaterWell, even then we'd have to install the dependencies rather than buildstream itself13:17
tlaterI guess most package managers will have neat ways to express that though13:17
ssam2at that point one would assume the distros would run our testsuite for us. although i guess it'd still be good for us to also do taht13:18
ssam2*that13:18
persiaPackaged buildstream will just have the dependencies, automated by distro packaging.13:18
tlaterpersia: I was considering our testing environments, in case we're doing stuff like solaris testing13:19
persiaBut upstream, we'll still want to test things raw, possibly on a few platforms, so that if we have a change that breaks on platform X, we'll know that before the distro folk get around to checking upstream (or maybe decide not to update because it would be hard).13:19
persiatlater: For testing environments, I think we never want to use packaged buildstream and I think we need to verify dependencies: even if they are installed, the versions might be off.13:19
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24513:20
tlaterThat in turn would be a bit of a maintenance overhead, as ssam2 says, if we start testing on many platforms - but I suppose we'll see where that goes13:20
tlaterI don't think we'll change our dependencies that often from here on out13:21
ssam2does the new testsuite image install buildstream? i think it shouldn't do13:21
paulsherwoodfamous last words13:21
* tlater was thinking that typing it out13:21
tlaterssam2: It doesn't, having an installed buildstream has bitten us before13:21
ssam2great13:21
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24513:24
tlaterHm, thinking about it, what if we change our testsuite images?13:28
tlaterShould we move the tag ahead, or do we want to keep a log of testsuite images?13:28
ssam2right13:28
ssam2it's probably better to reuse the 'fixed tag' logic used for the other images13:29
ssam2but maybe prepend the distro version too13:29
tlaterAnd then have multiple moving tags that specify the "current" version?13:29
ssam2so we have e.g 8-${number of commits in repo}-${sha1 of HEAD}13:29
ssam2where we're going, we don't need moving tags13:29
tlaterWell, the CI will only want to specify 'this tests debian-8'13:30
ssam2no, the CI will want to specify "use this exact version of our debian-8 base image"13:30
tlaterIn fact, I think that will be necessary unless we want to update buildstream's ci config every time we make a new commit13:30
ssam2that's exactly what tristan wants13:30
ssam2so that the only thing that can possibly break the CI is commits to buildstream itself13:30
tlaterThat will make using the CI very cumbersome :|13:31
ssam2this is how it already works13:31
ssam2and i agree that it would be nice to avoid having to update a bunch of tag names each time we need to change something in the base images13:32
ssam2but i have already discussed this with tristan in the past; and i do see his point13:32
* tlater forgot that the images don't have buildstream baked-in 13:32
tlaterYeah, that seems alright then13:32
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24513:38
* tlater wonders if we should enable per-branch caching13:46
tlaterOn the one hand that would slow down CI in some cases13:46
tlaterSpecifically, the first commit to a branch13:47
tlaterBut on the other hand, it would keep these branch problems in check13:47
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23313:55
tlaterssam2: I put an MR for my debian-8 image up: https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/1713:57
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24514:03
*** saxa has joined #buildstream14:06
ssam2tlater, per-branch caching might be an idea14:18
ssam2my main worry would be that as long as we're using a needlessly enourmous sysroot for the integration-tests, that will add up to many GBs of caches14:18
ssam2and the cache server is only of limited size14:18
ssam2but you'd expect that it would have some way of expiring artifacts when the disk is full14:19
ssam2that said, you'd also expect that from buildstream caches but actually they don't do that yet :-)14:19
ssam2i'll look at your MR14:19
tlaterta :)14:19
* tlater is still struggling to get gitlab to accept his cache :|14:20
* jmac wonders when he'll stop typing "BuildSteam"14:21
tlaterjmac: That's a nice name for the optimization branch14:21
*** mcatanzaro has joined #buildstream14:36
* tlater assumes gitlab is broken since it complains about files not existing that never should exist.14:47
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23314:48
ssam2tlater, you need to create the buildstream/testsuite-debian image on Docker Hub14:48
* tlater created that file14:48
ssam2the CI is failing because it can't push there because it doesn't exist14:48
tlaterAh14:48
ssam2it's not a file, you have to go on the web interface on hub.docker.com14:48
tlaterLet me check why that's failing14:48
tlaters/file/image/14:49
ssam2hmm, yes you did14:49
ssam2maybe that's an old build ? this is weird14:49
ssam2ah, i know the issue14:50
ssam2initially the image has nobody allowed to push to it14:50
ssam2if you go on the image and then the "collaborators" tag14:50
ssam2you can give write access to certain "Teams"14:50
ssam2the CI is a member of the "Automation" team so give that write access14:50
ssam2s/tag/tab/14:50
tlaterAh14:50
tlaterShould be set now14:51
tlaterI assume read/write is enough?14:51
ssam2yeah14:51
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23314:56
ssam2this is annoying; my machine spent 90 minutes building LLVM, failed during install-commands because my disk got full14:57
ssam2now i have to spend another 90 minutes building it again from scratch14:57
tlaterPerhaps we could work around that issue by pausing a build if we run out of disk space, allowing the user to deal with whatever15:09
tlaterThen again, it's run in the sandbox, so not actually trivial :|15:09
ssam2yeah, the out-of-disk-space error was from 'make install', so buildstream just saw a non-zero exit code15:09
ssam2what would have helped would be a 'retry but starting from install-commands' option15:10
ssam2there's a possible reproducibility issue there though ... it's fine if we assume that all buildsystems are free from bugs, but if we accept that they are not then we have to accept that the results of rerunning 'make install' after an initially failed 'make install' may not be the same as running a clean 'make install'15:10
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:11
tlaterHm, yeah, that's annoying...15:11
tlaterI suppose this wouldn't be fixed by artifact cache management either, since we can't predict how much space a build will need15:12
ssam2yeah, LLVM is humoungous15:12
ssam2hence why i ran out of space15:12
tlaterI guess this is "not buildstream's problem" - it's that no build systems manage to work around this issue nicely15:13
ssam2what might have been useful is a 'warning: your failed builds directory is taking up 35GB of space, do you want to sort that out"15:13
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:15
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:28
gitlab-br-botbuildstream: issue #202 ("Build of gnome-build-meta.git is broken") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/20215:30
ssam2juergbi, any ideas about https://gitlab.com/BuildStream/buildstream/issues/202 ?15:32
juergbissam2: your patch looks good to me, will add it to the MR15:32
ssam2thanks (after i finally fixed the style errors :-)15:33
juergbilet me take a look15:33
ssam2it seems something element-state related, I haven't dug into it yet15:33
nexusFor issue 191 https://gitlab.com/BuildStream/buildstream/issues/191 Would it be an option to always return a relative path? whether it is a subdirectory or not?15:33
ssam2I might need a bit more context to be able to answer that15:34
ssam2but if this is about recording workspace path relative to project dir ... what happens if the project dir moves but the workspace does not ?15:34
juergbissam2: commit 43bc41b from the log is missing my two fixes that i pushed to master last week15:35
juergbican you retry with that?15:35
juergbialthough, it also doesn't have my source state patch, so it shouldn't actually need those fixes15:36
ssam2oops, yeah i had the wrong branch checked out15:37
ssam2it happened on master too though15:37
ssam2just reproduced with f490343d98e40f9c9b651aa02f351d0bbee9afd915:37
juergbissam2: does it only happen with --no-strict?15:39
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:39
juergbii can take a look if it's reproducible. i'll be boarding in a few minutes, though15:40
ssam2juergbi, yes15:40
ssam2no worries, i can dig into it15:40
juergbita15:40
ssam2it's only with --no-scrict15:40
ssam2just wondered if you had any pointers15:40
juergbieither _update_state() is not called between the point where the cache key became calculatable and when bst attempts to use the cache key for pulling15:41
juergbior it's called and for some reason fails to calculate the cache key. i.e., one of the prereqs for that calculation are not available15:41
ssam2ok15:42
juergbimight need some debug prints in that function15:42
ssam2it seems like it shouldn't pull base.bst at all until it can calculate the cache key15:42
ssam2but, the refs for the base/ elements are actually all there15:42
ssam2it's weird that it shows them all as ?????15:42
ssam2it doesn't do that in strict mode15:42
juergbiwell, with --no-strict, it's sufficient to have the weak cache key15:43
ssam2right; but it actually has both for those elements15:43
juergbiit won't know the strong cache key until it has actually pulled the artifact15:43
ssam2ah I guess it doesn't bother to calculate strong keys in --no-strict mode, even if it can ?15:43
juergbithe strange thing is that it should only be pulling if it knows that the remote server actually has the artifact15:43
juergbiand that can only be the case if the (weak) cache key is known already15:44
ssam2right ... it can be known though, because the refs for the base/* elements are committed in the repo15:44
ssam2so maybe something is going wrong when it tries to calculate the cache key15:44
ssam2for base.bst15:44
ssam2i'll start by tracing that15:44
juergbior maybe it tries to use the strong cache key too early even when it should only just use the weak one15:45
juergbimaybe our test coverage of non-strict mode is insufficient15:45
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:45
ssam2looks that way15:45
* tlater did notice that looking at coverage reports15:46
juergbiboarding has started...15:47
ssam2safe trip15:47
nexuso/15:48
nexusssam2: did you see my question? :)15:48
ssam2yes, did you see my answer ?15:48
nexusno15:50
nexushmm15:50
ssam2the 2 lines after you spoke were directed at you15:50
nexusah15:50
nexushmm15:52
* nexus goes back to the drawing board15:52
nexusi was hoping to find a way of dealing with all workspaces in the same way, but that probably isn't possible15:54
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:55
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23315:56
nexusTo me then, that would make a combination of os.path.abspath and os.path.relpath the best option. Something like https://paste.codethink.co.uk/?386616:00
*** noisecell has quit IRC16:07
gitlab-br-botbuildstream: merge request (skip-configure-tests->master: Workspaced builds only run configure commands once) #191 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/19116:17
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: WIP: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24516:37
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23316:49
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23316:55
* tlater apologizes for the noise, gitlab CI is a fickle beast16:55
* nexus knows the feels16:55
ssam2this logic for loading strict cache key from the artifact is pretty broken17:06
ssam2the first thing it does is call OSTreeCache.extract()17:06
ssam2and the first thing that does is call Element._get_strict_cache_key()17:07
ssam2which is exactly the thing we don't know until we extract the artifact17:07
ssam2oh, seems i can just make the call to Element._get_strict_cache_key() conditional17:07
ssam2we are definitely missing test coverage here ...17:07
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23317:08
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23317:15
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23317:27
gitlab-br-botbuildstream: merge request (issue-166_yaml_removing_underscores->master: Issue 166 yaml removing underscores) #245 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/24517:37
tlaterGah, I screwed up and forgot to test if an sdist works with the debian image17:53
gitlab-br-botbuildstream: merge request (sam/fix-202->master: Fix breakage when running in no-strict mode with --track-all and an existing remote cache) #250 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25017:57
ssam2tiago, https://gitlab.com/BuildStream/buildstream/merge_requests/250 hopefully fixes that issue17:58
ssam2juergbi, i will be interested in your thoughts on that fix as well17:58
gitlab-br-botbuildstream: merge request (175-refactor-integration-tests->master: WIP: Resolve "Refactor integration tests") #233 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/23318:01
gitlab-br-botbuildstream: merge request (issue-191_relative_workspaces->master: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25118:04
gitlab-br-botbuildstream: merge request (issue-191_relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #251 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/25118:04
nexusoops18:04
*** ssam2 has quit IRC18:13
*** tlaxkit has quit IRC18:29
*** jonathanmaw has quit IRC18:41
*** valentind has joined #buildstream18:46
*** valentind has quit IRC21:57
*** cs_shadow has quit IRC22:27
*** noah has joined #buildstream23:03
*** mcatanzaro has quit IRC23:05

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!