IRC logs for #buildstream for Monday, 2019-12-02

*** qinusty has joined #buildstream08:19
*** ikerperez has joined #buildstream08:24
*** traveltissues has joined #buildstream08:25
benschuberttristan: oh that seems a good point, but I struggle to understand how the test would actually be testing that :) I'll investigate more, thanks !09:04
benschuberttlater[m]: sorry, was out on friday09:05
*** adds68 has joined #buildstream09:27
*** rdale has joined #buildstream09:28
*** tme5 has joined #buildstream09:34
*** santi has joined #buildstream09:34
*** tiagogomes has joined #buildstream09:46
*** tiagogomes has quit IRC09:51
*** tiagogomes has joined #buildstream09:53
coldtomhi, does anyone know if bst works with python 3.8?09:55
tpollardnot quite09:57
tpollardhttps://gitlab.com/BuildStream/buildstream/merge_requests/164709:57
coldtomack, ta tpollard10:00
*** cs-shadow has joined #buildstream10:03
*** jonathanmaw has joined #buildstream10:27
*** lachlan has joined #buildstream10:35
*** lachlan has quit IRC10:51
*** santi has quit IRC10:53
*** santi has joined #buildstream10:56
*** lachlan has joined #buildstream11:09
tlater[m]juergbi: Think we can land !1645 with the band-aid around the repeated digest thing?11:23
gitlab-br-botMR !1645: Refactor casserver.py: Stop relying on the buildstream-internal `CASCache` implementation https://gitlab.com/BuildStream/buildstream/merge_requests/164511:23
tlater[m]I'm just not sure why my branch suddenly duplicates a digest in the request, when master doesn't and none of that code changed.11:24
*** lachlan has quit IRC11:25
juergbitlater[m]: it might be a bug in buildbox-casd BatchReadBlobs11:27
juergbibefore we had: FetchMissingBlobs from bst to local casd. BatchReadBlobs from local casd to bst-artifact-server. bst-artifact-server handling that internally in the Python implementation11:28
juergbinow the last part is replaced by proxying BatchReadBlobs to the remote casd11:29
juergbii.e., we didn't hit the casd BatchReadBlobs code path before11:29
tlater[m]The duplication is definitely in the request sent from BuildStream - perhaps bst-artifact-server de-duplicated that?11:29
juergbino, but it was a simple implementation that didn't fall over with duplicated blobs11:30
juergbiit simply read blobs one by one and added responses11:30
tlater[m]Oh, right11:31
tlater[m]I guess in that case the band-aid is the best option on the BuildStream side for now11:31
juergbiI'm not particularly happy about the workaround. if we can quickly fix it in buildbox-casd, maybe that would be better11:32
tlater[m]I suppose. I guess I'm used to dependency updates taking a bit too long for that11:32
juergbiif it's a simple fix in casd, it shouldn't be an issue per se to update the casd dependency. although such an update still requires a bit more than just a click, unfortunately11:35
juergbihave to update casd anyway for my buildbox-run MR, though11:35
juergbiso maybe we could do this together11:35
tlater[m]Better than a temporary commit, I suppose :)11:36
juergbiit's the only blocker for your MR, so we can still otherwise consider this MR ready. and afaik, nobody is blocking on this MR, except my buildbox-run MR, which also requires a casd update11:36
juergbi(one reason I don't like the workaround is that it's O(N^2), afaict, although that would be easy to fix)11:38
tlater[m]juergbi: Yeah, it's not particularly efficient11:39
tlater[m]And I suppose we make a fair number of requests, so that's rather annoying.11:39
*** lachlan has joined #buildstream11:39
*** lachlan has quit IRC11:42
gitlab-br-bottpollard opened MR !1741 (tpollard/failedsource->master: element.py: Explicitly ensure failed build sources are not pushed) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174111:45
benschuberttpollard: ^ why would we want that? I don't think not being able to push a failed build should be correct :) What if I need to push the sources and the build to show a colleague that it's failing? (Or did we have a discussion somewhere that I missed?)11:48
tpollardI agree, but that's the current behaviour11:48
tpollardthe source push will never happen, because the build failed11:49
tpollardand there's no source push command11:49
benschubertif I have a '--on-error=continue' that still isn't pushed?11:49
tlater[m]I also think the problem is that a build failure may still be temporary and can't be repeated11:50
tlater[m]So if a failed build is pushed to a cache, currently anyone using said cache will get that failure, even if, say, the host ran out of memory11:50
benschuberttlater[m]: that's another problem, around artifacts, but for sources, I don't think we should avoid pushing them :)11:52
tlater[m]Ah, that's a different failure case :D11:52
juergbiI agree with benschubert, I'd rather consider the not pushing a bug/limitation11:52
juergbifor artifacts it's actually also considered a bug: #53411:52
gitlab-br-botIssue #534: Failed builds not pushing artifacts on quit https://gitlab.com/BuildStream/buildstream/issues/53411:52
gitlab-br-botBenjaminSchubert opened MR !1742 (bschubert/update-requirements->master: Update BuildStream requirements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174211:52
juergbihave an xfail for this11:52
tlater[m]We really should have a `source push` command some day.11:52
benschubert^ +111:53
tpollardyep11:53
juergbiyes, I think we agreed on that in the mailing list UX discussion11:53
tpollardI also thinks there's merit for default behaviour to not push the source if it failed, that could be overridden11:53
juergbinot sure whether we actually need that11:54
tpollarddo you want to push potentially gigabytes by default if somethings never going to build due to broken source?11:54
juergbiby default, in interactive mode, the user could anyway decide by choosing terminate/quit or continue11:54
juergbisources are pushed as individual blobs11:54
juergbiif you fix the bug in the source later, only the fixed source files will be updated11:55
juergbi(assuming you don't do strange things like importing a huge tarball as a single source file and then extract as part of the build commands)11:55
juergbii.e., I'd argue this would normally not waste lots of resources11:56
tpollardyes fair enough11:56
tpollardalthough that strange usecase is definitely a possibility....11:56
*** lachlan has joined #buildstream11:57
juergbimaybe but let's focus on the recommended workflows11:57
tpollardyep11:57
* tlater[m] creates a BuildStream project with only `manual:` elements and `local:` sources to complain about this11:58
tpollardI'm happy for the change to get closed, I didn't agree with the current assumptions (I only found the issue when doing the multiprocessing work)11:59
benschubertI think opening an issue saying it should get fixed might be better12:00
*** lachlan has quit IRC12:00
benschubertbecasue if we 'cement it' in place, we might not rememeber why and not have the desired behavior in the future12:00
tpollardi.e, the source push queue would execute before the mp queues had time to process the build failure between processes12:00
persiaCan the source push block on that processing completing?12:01
persia(yes, this makes it all a bit slower in the success case, but avoids pushing potentially failed things before processing)12:01
tme5I think if a build fails, the source should be pushed because caching a source is convenient and expected regardless of result. However i'm not sure a failed artifact should be automatically pushed. I think that goes against what the user wants in most cases -- I tried to build an element and it failed: the result is not what i want, so don't push it12:02
tme5as far as I understand, pushing a failed artifact can still be done manually12:03
tlater[m]persia: Probably, but scheduling that becomes a fun problem because it all happens on different processes.12:03
*** phildawson has quit IRC12:03
tlater[m]It's not a trivial fix12:03
benschuberttme5: I'd love a '--push-only-success' for now that I think about it12:03
gitlab-br-bottpollard closed MR !1741 (tpollard/failedsource->master: element.py: Explicitly ensure failed build sources are not pushed) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174112:04
tme5sources or artifacts?12:04
tlater[m]Could probably be both, don't really think you'd want to push a failed workspace12:05
traveltissuestpollard, left a comment there just as it was closed12:05
tme5pushing a failed artifact is, imo, ignorant of the user's intent12:05
*** phildawson has joined #buildstream12:05
tme5pushing automatically*12:05
tme5so in some sense i believe this issue is reversed: from failed builds, sources should push automatically, and artifacts shouldn't12:06
tpollardtraveltissues: yep I just saw it, will create an issue that links back to the MR so it's not lost12:06
benschuberttme5: in some CIs, you'll want only failing, all, or only succcess, depending on what's the purpose of the CI :) and there is no good way currently12:06
traveltissuesagree with benschubert though12:06
benschubertWhat we do it build everything with no push configuration, then push only the packages we want12:07
persiaI also believe it should be configurable.  There's lots of cases where you'd want failed CI builds to push (e.g. to speed local debugging of the failure).12:07
persiaOn the other hand, I suspect this is a knotty problem to solve (and doubt the current behaviour is either well-specified or intentional).  Are there other benefits from multiprocess that make it worth landing a change now (still with an issue to resolve the bug later) so that we at least can guess what will happen?12:08
tme5benschubert, interesting, hadn't encountered this kind of use case in my buildstream usage. I suppose having buildstream manage pushing itself is the only solution for that in CI12:10
benschubertYep :)12:11
gitlab-br-botcs-shadow approved MR !1742 (bschubert/update-requirements->master: Update BuildStream requirements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174212:11
gitlab-br-bottpollard opened issue #1225 (Formalise the pushing of failed sources) on buildstream https://gitlab.com/BuildStream/buildstream/issues/122512:17
tpollardjuergbi: do you think there's any merit for changing the logic of is_main_process to is_job_process in master, disregarding the wider multiprocessing changes?12:24
tpollardthere is in my head, but I release I've been working in different environment for quite a few months :_12:25
tpollard* :)12:25
tpollard*realise12:25
juergbiI'm not sure12:26
juergbiI would probably leave it as is in master12:27
juergbias there shouldn't really be any confusion about what the main process is there, afaict12:28
*** lachlan has joined #buildstream12:31
*** lachlan has quit IRC12:35
*** lachlan has joined #buildstream12:37
*** lachlan has quit IRC12:41
*** lachlan has joined #buildstream12:42
*** lachlan has quit IRC12:48
gitlab-br-bottlater approved MR !1742 (bschubert/update-requirements->master: Update BuildStream requirements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174212:59
tpollardjuergbi: ack13:03
tlater[m]juergbi: Looks like the casd bug has somehow become accross all branches.13:03
*** akvilebirgelyte__ has joined #buildstream13:14
*** santi has quit IRC13:15
tlater[m]benschubert: I think marge is refusing to merge things with orange pipelines13:16
tlater[m]We might need to revert to merging manually until the buildbox-casd thing is fixed13:16
tlater[m](Regarding !1742)13:16
*** akvilebirgelyte__ has quit IRC13:22
*** akvilebirgelyte__ has joined #buildstream13:22
benschubert:( poor Marge, Thanks I'll merge myself :)13:24
KinnisonIs she dead inside again?13:24
tlater[m]Kinnison: Nah, she just gets confused and tells us things take too long to be "marked as merged"13:25
tlater[m]But it happens consistently on anything with an orange pipeline, so I figure that's the problem13:25
KinnisonAah13:26
gitlab-br-botBenjaminSchubert merged MR !1742 (bschubert/update-requirements->master: Update BuildStream requirements) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174213:34
*** akvilebirgelyte__ has quit IRC13:35
*** akvilebirgelyte__ has joined #buildstream13:36
*** akvilebirgelyte__ has quit IRC14:02
*** akvilebirgelyte__ has joined #buildstream14:04
*** santi has joined #buildstream14:09
*** lachlan has joined #buildstream14:27
*** lachlan has quit IRC14:33
*** lachlan has joined #buildstream14:49
*** lachlan has quit IRC14:53
*** lachlan has joined #buildstream14:54
juergbitlater[m]: just saw this now. this would indicate that the issue started happening because of a change in either buildgrid or buildbox-* master14:56
juergbiso maybe it's an issue in the BuildGrid CAS server after all, bug in BatchReadBlobs14:57
*** lachlan has quit IRC14:59
*** lachlan has joined #buildstream15:21
*** lachlan has quit IRC15:26
tlater[m]juergbi: Best way to test would be to revert to whatever version of it we were using when the test suite last passed and see :D15:29
*** akvilebirgelyte__ has quit IRC15:36
juergbitlater[m]: the issue started Wednesday or Thursday, right?15:42
tlater[m]juergbi: I saw it Thursday for the first time15:42
juergbiit seems BuildGrid's nightly docker image build wasn't running in the last few weeks. a new build was produced last Thursday noon and since then it's failing15:43
juergbisee https://gitlab.com/BuildGrid/buildgrid.hub.docker.com/pipelines15:43
juergbipipeline #87140778 was good15:44
juergbiand I suspect the next pipeline #99215570 started breaking things in BuildStream15:44
tlater[m]Ah, shame that doesn't narrow it down for your debugging purposes15:44
juergbiI've verified that the docker image used in the last successful buildstream CI (on master) was the image from #8714077815:44
juergbialmost two months of BuildGrid changes that could have triggered this15:45
tlater[m]:|15:47
benschubertjuergbi, tlater[m]  I'm not sure how come we missed that? Because the docker images were not rebuilt nightly or ?15:48
juergbiyes, we have to ask the BuildGrid devs15:49
tlater[m]benschubert: I seem to recall that someone's account had been demoted due to them leaving whatever organization they were part of15:49
juergbiSotK might know more ^^15:49
tlater[m]And that GitLab didn't notify us that the job was under their ownership15:49
tlater[m]Hence we didn't have builds for a while15:50
benschubertOh I see15:50
juergbimy wild guess is https://gitlab.com/BuildGrid/buildgrid/commit/b2af88882ec4762c08ca462f6c5604d7ca621eab15:50
juergbicreates a Python dictionary with the digest hashes as keys15:51
* tlater[m] is glad it's probably not his branch introducing this though15:54
tlater[m]It had me worried that I was introducing some very strange regression :D15:54
*** lachlan has joined #buildstream15:55
* SotK reads backscroll15:56
juergbiSotK: context: buildstream remote execution test started failing. we though it was caused by a change on our side but it seems it's happening since the nightly buildgrid images are building again a few days ago15:57
juergbieven without changes on our side15:57
juergbiI'll open an issue15:59
SotKthat wild guess seems like a sensible place to start looking, the timing certainly makes it appear to be an issue in buildgrid itself16:01
juergbiI'll try whether I can quickly reproduce it with a standalone buildgrid test16:03
SotKthanks16:07
*** lachlan has quit IRC16:19
*** lachlan has joined #buildstream16:26
*** lachlan has quit IRC16:29
juergbiSotK, tlater[m]: https://gitlab.com/BuildGrid/buildgrid/issues/23016:35
tlater[m]ta juergbi16:37
gitlab-br-bottlater opened MR !1743 (tlater/fix-update-committer-security->master: update_commiters.py: Fix security vulnerabilities) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174316:42
*** santi has quit IRC16:43
*** lachlan has joined #buildstream16:48
*** traveltissues has quit IRC16:51
*** lachlan has quit IRC16:51
gitlab-br-bottlater opened MR !1744 (tlater/fix-jinja-autoescape->master: optionpool.py: Make jinja autoescape rules explicit) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174416:58
gitlab-br-botBenjaminSchubert approved MR !1744 (tlater/fix-jinja-autoescape->master: optionpool.py: Make jinja autoescape rules explicit) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174417:03
*** santi has joined #buildstream17:10
*** tme5 has quit IRC17:20
gitlab-br-botBenjaminSchubert approved MR !1743 (tlater/fix-update-committer-security->master: update_commiters.py: Fix security vulnerabilities) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174317:21
tlater[m]Such a shame marge isn't her usual self today17:22
tlater[m]benschubert: yuck, the security vulnerability scanner only allows select_autoescape, not jinja2.select_autoescape17:36
tlater[m]:|17:36
* tlater[m] wonders if he can work around that with `select_autoescape = jinja2.select_autoescape`17:36
benschubertwould have been too easy -_-'17:36
tlater[m]Eww, that actually works17:39
benschubert?17:41
tlater[m]benschubert: It doesn't care whether the function named `select_autoescape` is imported from `jinja2`17:42
tlater[m]It'll take any variable named that, and accept it as "not a security vulnerability at all anymore"17:42
tlater[m]But it will not allow any function not named that.17:42
tlater[m]So if you write your own select function, and name it something else, it will complain, even if it does select a reasonable policy.17:43
benschubertI don't know why, I'm not surprised17:47
gitlab-br-bottlater merged MR !1743 (tlater/fix-update-committer-security->master: update_commiters.py: Fix security vulnerabilities) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174317:49
*** mela9inekiddo has joined #buildstream17:52
gitlab-br-bottlater opened (was WIP) MR !1744 (tlater/fix-jinja-autoescape->master: optionpool.py: Make jinja autoescape rules explicit) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/174417:54
*** mela9inekiddo has quit IRC18:04
*** mela9inekiddo has joined #buildstream18:05
*** mela9inekiddo has quit IRC18:05
*** santi has quit IRC18:11
*** jonathanmaw has quit IRC18:26
*** lachlan has joined #buildstream18:29
*** tiagogomes has quit IRC18:38
*** lachlan has quit IRC18:44
gitlab-br-botcs-shadow opened (was WIP) MR !1706 (chandan/interactive-tests->master: Add tests for interactive BuildStream operations) on buildstream https://gitlab.com/BuildStream/buildstream/merge_requests/170618:46
*** lachlan has joined #buildstream18:47
*** lachlan has quit IRC18:51
*** rdale has quit IRC19:08
*** lachlan has joined #buildstream19:32
*** lachlan has quit IRC19:36
*** lachlan has joined #buildstream20:25
*** lachlan has quit IRC20:35
*** cs-shadow has quit IRC21:11
*** testy has joined #buildstream21:20
*** coldtom8 has joined #buildstream21:21
*** benschubert_ has joined #buildstream21:22
*** phildawson has quit IRC21:22
*** narispo has quit IRC21:22
*** slaf has quit IRC21:22
*** tristan has quit IRC21:22
*** benschubert has quit IRC21:22
*** aevri has quit IRC21:22
*** jjardon has quit IRC21:22
*** WSalmon has quit IRC21:22
*** jward has quit IRC21:22
*** valentind has quit IRC21:22
*** bethw has quit IRC21:22
*** cgmcintyre[m] has quit IRC21:22
*** gimpnet-irc[m] has quit IRC21:22
*** asingh_[m] has quit IRC21:22
*** Trevinho[m] has quit IRC21:22
*** tchaik[m] has quit IRC21:22
*** ssssam[m] has quit IRC21:22
*** jswagner has quit IRC21:22
*** reuben640[m] has quit IRC21:22
*** tlater[m] has quit IRC21:22
*** rafaelff[m] has quit IRC21:22
*** mrmcq2u[m] has quit IRC21:22
*** nielsdg has quit IRC21:22
*** dbuch has quit IRC21:22
*** kailueke[m] has quit IRC21:22
*** abderrahim[m] has quit IRC21:22
*** m_22[m] has quit IRC21:22
*** doras has quit IRC21:22
*** mattiasb has quit IRC21:22
*** albfan[m] has quit IRC21:22
*** jjardon[m] has quit IRC21:22
*** skullone[m] has quit IRC21:22
*** verdre[m] has quit IRC21:22
*** waltervargas[m] has quit IRC21:22
*** awacheux[m] has quit IRC21:22
*** pro[m] has quit IRC21:22
*** krichter[m] has quit IRC21:22
*** Kinnison has quit IRC21:22
*** coldtom has quit IRC21:22
*** benbrown has quit IRC21:22
*** persia has quit IRC21:22
*** ironfoot has quit IRC21:22
*** benschubert_ is now known as benschubert21:22
*** jjardon has joined #buildstream21:22
*** jswagner has joined #buildstream21:22
*** phildawson has joined #buildstream21:22
*** aevri has joined #buildstream21:23
*** narispo has joined #buildstream21:23
*** benbrown has joined #buildstream21:23
*** Kinnison has joined #buildstream21:23
*** persia has joined #buildstream21:23
*** ironfoot has joined #buildstream21:24
*** bethw has joined #buildstream21:24
*** slaf has joined #buildstream21:25
*** jward has joined #buildstream21:25
*** WSalmon has joined #buildstream21:25
*** valentind has joined #buildstream21:25
*** tristan has joined #buildstream21:26
*** persia has quit IRC21:34
*** persia has joined #buildstream21:34
*** asingh_[m] has joined #buildstream21:46
*** rafaelff[m] has joined #buildstream22:03
*** cgmcintyre[m] has joined #buildstream22:11
*** jjardon[m] has joined #buildstream22:30
*** tlater[m] has joined #buildstream22:39
*** albfan[m] has joined #buildstream22:54
*** mrmcq2u[m] has joined #buildstream23:01
*** awacheux[m] has joined #buildstream23:06
*** krichter[m] has joined #buildstream23:09
*** pro[m] has joined #buildstream23:26
*** gimpnet-irc[m] has joined #buildstream23:28
*** skullone[m] has joined #buildstream23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!