IRC logs for #buildstream for Friday, 2019-01-18

*** bochecha has quit IRC00:04
*** nimish has joined #buildstream01:02
*** tristan has quit IRC01:15
*** nimish has quit IRC02:07
*** nimish has joined #buildstream02:07
*** xjuan has quit IRC02:10
*** nimish has quit IRC02:56
*** nimish has joined #buildstream03:14
*** nimish has quit IRC03:27
*** tristan has joined #buildstream04:10
*** kapil___ has joined #buildstream04:35
*** juergbi has quit IRC06:29
*** juergbi has joined #buildstream08:07
*** juergbi has quit IRC08:17
*** juergbi has joined #buildstream08:32
*** toscalix has joined #buildstream08:54
*** alatiera has joined #buildstream08:55
Kinnisonjennis: Do you have a pipeline pass?09:27
Kinnisonjennis: if so, merge it :-D09:28
gitlab-br-botjennis merged MR !1082 (jennis/profiling_outputs_binaries->master: Make the profiler output binaries (which can be used to visualise the data)) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108209:31
jennisKinnison :)09:31
*** raoul has joined #buildstream09:44
*** bochecha has joined #buildstream09:47
*** benschubert has joined #buildstream09:54
*** rdale has quit IRC10:08
*** jonathanmaw has joined #buildstream10:15
*** rdale has joined #buildstream10:25
*** nimish has joined #buildstream10:28
*** nimish has quit IRC10:33
*** nimish has joined #buildstream10:33
*** lachlan has joined #buildstream10:35
gitlab-br-botjjardon opened issue #874 (overnigth test (no cache) are failing: memory exhausted) on buildstream https://gitlab.com/BuildStream/buildstream/issues/87410:39
gitlab-br-botjjardon closed issue #865 (Overnigth tests are failing: Seems runners doesn't have enough memory) on buildstream https://gitlab.com/BuildStream/buildstream/issues/86510:42
*** nimish has quit IRC10:48
*** lachlan has quit IRC10:48
*** lachlan has joined #buildstream10:48
*** nimish has joined #buildstream10:49
*** nimish has quit IRC10:54
*** nimish has joined #buildstream10:54
*** nimish_ has joined #buildstream10:58
*** nimish has quit IRC10:58
*** nimish_ is now known as nimish10:58
valentindjjardon, since it is llvm that fails on the no cache tests, it seems to me this is also an issue with memory.10:59
jjardonYeah but 16GB of ram should be enough , right?11:00
valentindThat depends on the number of cores.11:02
gitlab-br-botalex-broad opened issue #875 (Workspace cache continuously invalidated by IDE metadata files) on buildstream https://gitlab.com/BuildStream/buildstream/issues/87511:02
*** nimish has quit IRC11:04
*** nimish has joined #buildstream11:04
valentindjjardon, I think we should add the logs directory in the artifacts for those builds.11:12
*** nimish has quit IRC11:14
jjardonvalentind: rigth, that maxime have 6 cores / 16GB RAM11:14
*** nimish has joined #buildstream11:15
*** nimish has joined #buildstream11:15
jjardonvalentind: also, did you see https://gitlab.com/BuildStream/buildstream/issues/873 ; you have fixed that one before; maybe something change in the CI and we are not using your fixes anymore?11:16
valentindjjardon, that looks like a disk full.11:17
valentindjjardon, maybe docker images do not get cleaned up correctly.11:20
jjardonthey do not11:20
jjardongitlab-runner doesn't do it automatically11:21
jjardonthere is a bug upstream , sec11:21
jjardonin the meantime having a cron job doing "docker system prune -a" and "docker volume prune" periodically helps11:21
jjardonhttps://gitlab.com/gitlab-org/gitlab-runner/issues/298011:22
valentindOK, I will make a timer that does that and deploy it today.11:22
gitlab-br-botvalentindavid opened MR !1084 (valentindavid/overnight-tests-logs-artifacts->master: .gitlab-ci.yml: Add overnight tests logs to artifacts.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108411:23
valentindjjardon, !108411:24
*** nimish has quit IRC11:25
*** nimish has joined #buildstream11:27
*** nimish has quit IRC11:27
*** nimish has joined #buildstream11:28
*** lachlan has quit IRC11:28
*** lachlan has joined #buildstream11:41
valentindjjardon, we are already pruning containers on the sleds.11:43
valentindThough we do "docker container prune -f"11:44
*** lachlan has quit IRC11:47
*** nimish has quit IRC11:48
jjardonvalentind: -a removes more things11:48
valentindI will try that.11:48
*** nimish has joined #buildstream11:48
jjardonalso "docker volume prune"11:48
valentindYes. I am deploying it now.11:49
*** nimish has quit IRC11:53
*** nimish has joined #buildstream11:53
*** nimish has joined #buildstream11:54
gitlab-br-botjjardon approved MR !1084 (valentindavid/overnight-tests-logs-artifacts->master: .gitlab-ci.yml: Add overnight tests logs to artifacts.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108412:01
gitlab-br-botjjardon merged MR !1084 (valentindavid/overnight-tests-logs-artifacts->master: .gitlab-ci.yml: Add overnight tests logs to artifacts.) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108412:01
*** nimish has quit IRC12:04
*** nimish has joined #buildstream12:04
*** lachlan has joined #buildstream12:07
valentindjjardon, I think the volumes were the problem.12:31
*** nimish has quit IRC12:34
*** nimish has joined #buildstream12:35
*** nimish has quit IRC12:40
*** nimish has joined #buildstream12:40
*** nimish has quit IRC12:45
*** nimish has joined #buildstream12:46
cs-shadowvalentind: you can try `docker system prune --all --force --volumes` to do it in one step13:12
valentindcs-shadow, good to know.13:12
*** nimish has quit IRC13:16
*** nimish has joined #buildstream13:16
*** nimish has quit IRC13:31
*** nimish has joined #buildstream13:31
gitlab-br-botvalentindavid approved MR !1083 (tristan/fix-bzr-race->master: Fix bzr race conditions) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108313:42
*** tristan has quit IRC13:47
*** nimish has quit IRC13:51
*** nimish has joined #buildstream13:52
*** nimish has joined #buildstream13:52
*** nimish has quit IRC13:57
*** nimish has joined #buildstream13:58
*** nimish has quit IRC14:03
*** nimish has joined #buildstream14:03
*** nimish has joined #buildstream14:04
raoulI'm doing some cache directory reshuffling as part of #870, is there any reason integration tests have their artifact cache in the integration cache instead of the temp folder for the test? From what I can tell it just creates a new one for each test anyway, so it seems redundant14:07
gitlab-br-botIssue #870: Root cache directory https://gitlab.com/BuildStream/buildstream/issues/87014:07
jennisraoul, I thnk it's because many of the artifact tests do share/reuse a cache14:13
jennisperhaps you're looking at some examples where they're spinning up a new one each time, but I could be wrong14:13
jennisthe integration tests*14:13
* jennis has been battling with artifacts too much14:13
raoulThe integration cache context manager explicitly deletes the artifact cache though, so I'm not sure it does14:14
raoulit keeps the source dir, but not artifacts14:14
*** sambishop has quit IRC14:14
jennisperhaps that then, to save re-fetching?14:15
raoulyeah that bit makes sense, it's just the artifacts bit I'm questioning14:15
*** nimish has quit IRC14:19
*** nimish has joined #buildstream14:19
*** nimish has quit IRC14:24
*** nimish has joined #buildstream14:24
*** nimish has quit IRC14:29
*** nimish has joined #buildstream14:30
*** nimish has quit IRC14:55
*** nimish has joined #buildstream14:56
*** nimish has quit IRC15:01
*** nimish has joined #buildstream15:01
*** phil has quit IRC15:02
*** phil has joined #buildstream15:03
*** raoul has quit IRC15:03
*** nimish has quit IRC15:15
*** nimish has joined #buildstream15:16
*** nimish has quit IRC15:21
*** nimish has joined #buildstream15:22
gitlab-br-botrichardmaw-codethink opened MR !1085 (richardmaw/centos-oldgit-test-fixes->master: Fix CentOS) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108515:36
*** nimish has quit IRC15:47
*** nimish has joined #buildstream15:48
*** raoul has joined #buildstream15:48
*** phil has quit IRC15:49
*** nimish has quit IRC15:53
*** nimish has joined #buildstream15:54
*** tristan has joined #buildstream15:54
*** phil has joined #buildstream15:58
tlater[m]raoul: Back when our base image was gnome-sdk (2GB large!) we needed to keep the artifacts around for whole test modules, or it would just be atrociously slow.16:15
tlater[m]Pretty sure that that is why the artifacts are stored like that.16:15
* tlater[m] incidentally just noticed that this also breaks the remove_artifact helper for integration tests16:16
tlater[m]I'll fix that helper, if you like, would you mind testing if the test suite breaks in horrible ways if we remove that indirection?16:17
raoulthe remove_artifact method in the Cli class?16:18
tlater[m]Yep16:18
tlater[m]The artifact cache dir is hardcoded16:18
tlater[m]I've already written a fix so I can use it for a test I'm writing16:19
raoulah yes it is16:19
raoulwhich indirection do you mean btw?16:19
tlater[m]The thing where we send artifacts to integration-cache when it should really be part of the test-specific tmp dir16:20
tlater[m]I don't think this is relevant anymore since the test image is really small these days.16:20
tlater[m]And keeping artifacts around can cause tests to succeed when they shouldn't, so we should really make this safe.16:20
raoulright right, I've somewhat started work on that as part of my #870 stuff, trawling through the test failures atm16:21
gitlab-br-botIssue #870: Root cache directory https://gitlab.com/BuildStream/buildstream/issues/87016:21
tlater[m]Well, was to be expected, I suppose. Damn, would have been nice if we could just change a few lines.16:21
tlater[m]In any case, thanks for looking at it :)16:22
raoulif only :P16:22
raoulnw :)16:22
*** nimish has quit IRC16:29
*** nimish has joined #buildstream16:29
gitlab-br-bottristanvb closed issue #868 (bzr plugin doesnt parallelize well) on buildstream https://gitlab.com/BuildStream/buildstream/issues/86816:35
gitlab-br-bottristanvb merged MR !1083 (tristan/fix-bzr-race->master: Fix bzr race conditions) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108316:35
*** nimish has quit IRC16:44
*** nimish has joined #buildstream16:45
benschubertThe bug were BuildStream waited until it had fetched everything before building has been fixed on master already right?16:48
*** ChanServ sets mode: +o tristan16:51
tristanbenschubert, Yes16:51
tristanbenschubert, I've been pondering the additional enhancement but I'm worried it could be a performance killer, and I'm not 100% certain of that change16:52
benschuberttristan: ok perfect thanks!  Do you ahve a branch for this potential improvement?16:53
tristanI.e. I should be able to explain exactly why the order of element processing would be better, if we were to change the loop to (A) Skip all skippable elements and promote those through queues in one loop first... (B) Process queues in reverse order, harvesting jobs as much as resources permit16:53
tristanI don't, it's a fairly simple change to make16:53
tristanbenschubert, I did write up a test case, but then sadly found that it passes anyway16:54
tristanI think provoking the behavior to change will be tricky16:54
tristanbenschubert, However... there is something that I *am* certain about16:54
tristanbenschubert, and that is... the initial promotion of elements through queues would require a *full* iteration over the elements of each queue, and each element would need to be asked if it is skippable16:55
benschuberttristan: we know if an element is skippable when we put it in the queue, correct?16:56
*** nimish has quit IRC16:56
tristanand then those lists would need to be queried *again*, that time asking each element if they are ready to process, but stopping as soon as we fill up the resources for that queue16:56
tristanbenschubert, An element can *become* skippable as a result of the completion of processing of some other element16:56
tristanas the data model is linked16:56
tristanI mean yes, we *initially* place elements directly on the skip queue16:57
*** nimish has joined #buildstream16:57
benschubertRIght, but if it was not skippable before, it meant it was waiting and not ready. Correct? That means, if we only add elements to a queue when they are _ready_, we would fix the problem and remove the need of querying the queue. And knowing when an element is ready for a queue, is a matter of updating reverse dependencies everytime we change. Or am I missing something? (I'm not saying this is simple to implement, just16:58
benschubertconceptually)16:58
tristanbenschubert, In any case that would be pretty expensive... some background is that the original design was based on data which ... slightly changed16:58
tristanI.e. we rely on sort order of elements to ensure that the iterations stay small enough16:59
tristanwe just "grab as much as we can" from the end of a queue and put it on another when it's done, and hope sort order makes that ready stuff always near the tip16:59
tristanBut now that we have this dynamic checking for pulling elements, that data has a bit of a different shape16:59
benschuberttristan: I have a few idea on how to "fix" those problems, but that would require a huge refactoring and I'm not sure I understand the entire state transition/pipeline well enough. Would you be happy if we catch up at the gathering and try to design a plan to refactor the pipeline? :)17:00
benschubertit might be easier in person17:00
tristanbenschubert, I'm not sure exactly what you mean about adding elements to a queue only when they are ready17:00
tristanbenschubert, the order needs to be preserved, first of all - so you dont improve things from splitting waiting and ready elements into separate lists17:01
tristanYou want the next ready element in the list, *at the time* a job has completed17:01
*** nimish has quit IRC17:01
tristanbenschubert, Normally, the fact that a job completed means that something in waiting state becomes ready... but it can potentially *also* mean that something goes from waiting state to skipped state17:02
tristanI suspect there is a case of that which revolves around discovering a strong artifact key of a pulled artifact in non strict mode17:02
benschuberttristan: we currently rely on checking the status whenever a state change. This tells us what needs to happen (skip/buildable/etc). We do know exactly which events change a state and the state of reverse dependencies. We could therefore do without those waiting queues, and whenever an element is finished processing somewhere, we go through its reverse deps and update them accordingly (putting them in a queue when needed)17:02
tristanbenschubert, I'm not on the same page as you, but I am in a similar book17:03
benschubertIt might require thinking more thoroughly about those transition states, but that would remove lots of unneeded operations we are currently doing I think17:03
benschuberttristan: :D where am I not explaining well enough? (I've been thinking about this for a while, so I'm not sure what is clear or not)17:04
tristanbenschubert, I.e. don't tangle the data model and the scheduler together - instead have the data model handle caching and cache invalidation itself17:04
*** nimish has joined #buildstream17:04
tristanbenschubert, In this way, any question the scheduler asks the data model is only ever expensive when it needs to be17:05
tristannormally any question is answered with a return self.__cached_something17:05
benschuberttristan: I agree with that, I don't think we need the pipeline and the data model needs to be too tangled, if we rework what information the data model gives (like reverse dependencies)17:07
gitlab-br-bottlater opened (was WIP) MR !1031 (tlater/message-lines->master: Avoid "showing 0 lines" messages when we're asked to show no lines) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/103117:07
*** nimish has quit IRC17:07
benschubertand by moving more restrictively on the datamodel we would be able to optimize the pipeline/scheduler a lot17:08
tlater[m]Would someone mind giving this a final review? https://gitlab.com/BuildStream/buildstream/merge_requests/103117:08
gitlab-br-botaevri opened MR !1086 (aevri/bst_track_guidance->master: Fixup refs to 'bst track' and 'bst fetch') on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108617:09
benschubertThis would require a more or less complete rewrite of those pieces. and might require us to have more things in memory (and that is bad since we need to reduce our memory footprint)17:09
tristanbenschubert, honestly I am confident that Consistency interrogations are correctly cached and inexpensive, but I don't know that that is the case for yet-to-be-resolved cache keys17:09
benschuberttristan: that is correct. However, no matter how inexpensive it is, when you have 50k elements and you reinterogate each waiting one when any element stage changes, this is not sustainable17:10
tristanbenschubert, Sure, but those are separate problems which belong in separate places - for now the scheduler uses queues and lists of elements17:10
tristanthat could change with a totally fancy expensive rewrite which processes graph data instead of lists for instance; but that operation would be isolated in the scheduler17:11
tristanbenschubert, essentially, I have little faith that the scheduler, as written, will perform acceptably by doing more interrogation in the loop under question, i.e. to get skip elements over the wall quicker17:12
tristanbenschubert, and I have little evidence that doing that will improve the outcome order significantly either17:12
tristanI'm not currently considering an expensive scheduler rewrite either :)17:13
benschuberttristan: ah! I got you. Agreed!17:13
tristanbut if that happens it's not a bad thing :)17:13
benschuberttristan: Ok, I will try to have a rewrite plan for it for the gathering, might be simpler to discuss it in one of the performance sessions :)17:14
tristanyeah that does sound interesting :)17:14
tristanit's certainly a fun topic17:14
gitlab-br-bottristanvb opened MR !1087 (tristan/cas-cleanup-improve->master: _cas/cascache.py: Cleanup directories when removing refs) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108717:20
gitlab-br-botjennis opened MR !1088 (jennis/add_new_profile_topic->master: Add new 'scheduler' and 'load-selection' profiling topics) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108817:23
gitlab-br-bottlater closed issue #281 (Make the testutils API public) on buildstream https://gitlab.com/BuildStream/buildstream/issues/28117:37
tristanjuergbi, any problem with cleaning up the directory entries when doing CASCache.remove() ?17:46
tristanmaybe there is a reason we weren't doing that already ?17:46
*** nimish has joined #buildstream17:46
tristan!1087 cleans it up and adds a regression test for it17:46
tristanAlso wondering what's with the files I have in ~/.cache/buildstream/artifacts/tmp17:47
tristanMaybe that is residue from an older version ?17:47
*** tristan has quit IRC17:49
* tlater[m] wonders if we can blanket skip integration tests when not on Linux+Bwrap18:04
tlater[m]Seems kinda odd that we'd need to add that mark to every integration test.18:04
*** nimish has quit IRC18:04
*** raoul has quit IRC18:05
*** jonathanmaw has quit IRC18:06
*** nimish has joined #buildstream18:06
*** tristan has joined #buildstream18:09
*** nimish has quit IRC18:14
*** ChanServ sets mode: +o tristan18:16
tristanjjardon, So BuildStream master makes about ~90G with freedesktop-sdk(18.08) all.bst18:16
tristanHave a fresh build and know that cleanups have removed anything not in my build18:17
tristanThat will be the expensive cached builddirs18:18
*** kapil___ has quit IRC18:44
*** toscalix has quit IRC18:47
gitlab-br-botaevri merged MR !1078 (aevri/shell_separator_hint->master: cli.py: add a hint about '--' to 'bst shell' help) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/107818:56
*** nimish has joined #buildstream19:17
*** nimish has joined #buildstream19:17
jjardontristan: in the positive side; good test case? :)19:30
tristanjjardon, Yeah looks like gnome-build-meta isn't building out of the box for me :-/19:37
jjardonMmm, try gnome-3.30 branch; that one should build as is building spacious sha, not master19:42
tristanI see19:43
tristanI will give it a shot19:43
*** nimish has quit IRC19:44
alatieratristan: is it currently possible to point bst to a project.refs in a remote location?19:47
alatieratristan: like another git repo, or hosted in a server19:47
alatierawe were thinking the other day how to make gnome-build-meta always build while still closely tracking master19:48
tristanjjardon, btw, that's what I get with master: https://bpaste.net/show/0e01d683030f19:48
tristanbut doesnt gnome-build-meta inherit the strip binaries stuff from freedesktop-sdk ?19:49
alatieraone idea was that after each successful build in the master branch we push the project.refs file somewhere19:49
alatierabut then for local development we would need to add extra instruction or a wrapper to fetch the file19:49
*** tristan has quit IRC19:52
*** tristan has joined #buildstream19:56
*** lachlan has quit IRC20:09
gitlab-br-bottristanvb merged MR !1087 (tristan/cas-cleanup-improve->master: CASCache cleanup improvements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108720:57
gitlab-br-botcs-shadow opened MR !1089 (chandan/import-is-not-buildelement->master: Derive import plugin from Element instead of BuildElement) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108921:25
gitlab-br-bottristanvb opened MR !1090 (tristan/fix-pip-source-test->master: tests/testutils/python_repo.py: Use subprocess to run sdist) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/109021:28
cs-shadowtristan: mind having a look at !1089 ^^ when you get a chance?21:29
cs-shadowIt's a small change but I'd like to make sure I am not missing some subtle details here :)21:29
*** alatiera has quit IRC21:32
*** ChanServ sets mode: +o tristan21:33
tristancs-shadow, I already did !21:33
tristancs-shadow, nice spot21:33
cs-shadowtristan: thanks!21:34
tristancs-shadow, if you've been seeing the same thing in CI as me, will be good to get this through first: https://gitlab.com/BuildStream/buildstream/merge_requests/109021:34
* tristan has been playing whack a mole with that spurious pip test failure21:35
cs-shadowtristan: sure, I've cancelled my automatic merge. Will wait for !1090 to go through first21:36
cs-shadowI did see that issue at least once but I chalked it up it as a random failure but it's great that you are looking into it21:37
gitlab-br-botcs-shadow approved MR !1090 (tristan/fix-pip-source-test->master: tests/testutils/python_repo.py: Use subprocess to run sdist) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/109021:39
tristancs-shadow, I think thats the last spurious failure :)21:39
tristanI fixed another one yesterday21:39
tristanand both were more-or-less explainable, which is good21:39
tristantoday's was a bit cryptic, but we can see the general area where things are going wrong and just avoid it21:40
* tristan gotta change location21:40
cs-shadowhaha, as they say only thing bad than no tests is a flaky test21:41
cs-shadowso many thanks for fixing them all21:42
*** tristan has quit IRC21:43
*** benschubert has quit IRC21:57
*** bochecha has quit IRC21:58
gitlab-br-bottristanvb merged MR !1090 (tristan/fix-pip-source-test->master: tests/testutils/python_repo.py: Use subprocess to run sdist) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/109022:07
*** tristan has joined #buildstream22:10
gitlab-br-botcs-shadow merged MR !1089 (chandan/import-is-not-buildelement->master: Derive import plugin from Element instead of BuildElement) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/108922:46
gitlab-br-bottristanvb closed issue #779 (It should be possible to disable error and message lines completely) on buildstream https://gitlab.com/BuildStream/buildstream/issues/77923:23
gitlab-br-bottristanvb merged MR !1031 (tlater/message-lines->master: Avoid "showing 0 lines" messages when we're asked to show no lines) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/103123:23

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!